2026-02-23T11:43:44.988 INFO:root:teuthology version: 1.2.4.dev9+gfa17720d0 2026-02-23T11:43:45.005 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-23T11:43:45.083 INFO:teuthology.run:Config: archive_path: /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-23_09:58:48-crimson-rados-main-distro-debug-trial/66294 branch: main description: crimson-rados/perf/{0-crimson_install clusters/{crimson-fixed-1} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/segmented$/{crimson_seastore_segmented_2q} perf_ignorelist settings/optimized workloads/radosbench_4M_write} email: null first_in_suite: false flavor: debug job_id: '66294' kernel: branch: distro kdb: true sha1: distro last_in_suite: false machine_type: trial name: shraddhaag-2026-02-23_09:58:48-crimson-rados-main-distro-debug-trial no_nested_subset: false os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: global: auth client required: none auth cluster required: none auth service required: none auth supported: none debug asok: 0/0 debug auth: 0/0 debug bdev: 0/0 debug bluefs: 0/0 debug bluestore: 0/0 debug buffer: 0/0 debug civetweb: 0/0 debug client: 0/0 debug compressor: 0/0 debug context: 0/0 debug crush: 0/0 debug crypto: 0/0 debug dpdk: 0/0 debug eventtrace: 0/0 debug filer: 0/0 debug filestore: 0/0 debug finisher: 0/0 debug fuse: 0/0 debug heartbeatmap: 0/0 debug javaclient: 0/0 debug journal: 0/0 debug journaler: 0/0 debug kstore: 0/0 debug leveldb: 0/0 debug lockdep: 0/0 debug mds: 0/0 debug mds balancer: 0/0 debug mds locker: 0/0 debug mds log: 0/0 debug mds log expire: 0/0 debug mds migrator: 0/0 debug memdb: 0/0 debug mgr: 0/0 debug mgrc: 0/0 debug mon: 0/0 debug monc: 0/0 debug ms: 0/0 debug objclass: 0/0 debug objectcacher: 0/0 debug objecter: 0/0 debug optracker: 0/0 debug osd: 0/0 debug paxos: 0/0 debug perfcounter: 0/0 debug rados: 0/0 debug rbd: 0/0 debug rbd mirror: 0/0 debug rbd replay: 0/0 debug refs: 0/0 debug rgw: 0/0 debug rgw lifecycle: 0/0 debug rgw notification: 0/0 debug rgw sync: 0/0 debug rocksdb: 0/0 debug striper: 0/0 debug throttle: 0/0 debug timer: 0/0 debug tp: 0/0 enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 0/0 debug ms: 0/0 debug paxos: 0/0 osd pool default crimson: true osd: crimson cpu num: 4 crimson osd obc lru size: 10 debug filestore: 0/0 debug journal: 0/0 debug ms: 0/0 debug osd: 0/0 debug seastore: 20 debug seastore backref: 20 debug seastore cache: 20 debug seastore cleaner: 20 debug seastore device: 20 debug seastore epm: 20 debug seastore fixedkv tree: 20 debug seastore journal: 20 debug seastore lba: 20 debug seastore odata: 20 debug seastore omap: 20 debug seastore onode: 20 debug seastore t: 20 debug seastore tm: 20 osd objectstore: seastore osd shutdown pgref assert: true seastore cachepin size pershard: 64M seastore cachepin type: 2Q seastore max concurrent transactions: 8 crimson_compat: true flavor: debug log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) - \(PG_ - \(OSD_ - \(OBJECT_ - overall HEALTH sha1: e371433fcd1390a0b82cfb18effc422c1f5d8c1d ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log global: osd crush chooseleaf type: 0 osd pool default pg num: 128 osd pool default pgp num: 128 osd pool default size: 2 mon: {} install: ceph: flavor: debug sha1: e371433fcd1390a0b82cfb18effc422c1f5d8c1d workunit: branch: wip-shraddhaag-enable-cephadm-tests env: CRIMSON_COMPAT: '1' sha1: e371433fcd1390a0b82cfb18effc422c1f5d8c1d owner: scheduled_shraddhaag@soko04.front.sepia.ceph.com priority: 49 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 66 sha1: e371433fcd1390a0b82cfb18effc422c1f5d8c1d sleep_before_teardown: 0 suite: crimson-rados suite_branch: wip-shraddhaag-enable-cephadm-tests suite_path: /home/teuthworker/src/github.com_shraddhaag_ceph_e371433fcd1390a0b82cfb18effc422c1f5d8c1d/qa suite_relpath: qa suite_repo: https://github.com/shraddhaag/ceph.git suite_sha1: e371433fcd1390a0b82cfb18effc422c1f5d8c1d targets: trial186.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLDyxqOVszW7QgWdmsWYzM19QfQnsb0ZiZyzGIHN941zbMFYb7ZU9RsookcbeRuxK/WY4tvgeWIANTqANZ6NgeA= tasks: - install: extra_packages: deb: - ceph-osd-crimson - ceph-osd-crimson-dbg rpm: - ceph-osd-crimson - ceph-osd-crimson-debuginfo - pexec: all: - sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson - install: null - ceph: conf: osd: debug monc: 20 wait-for-scrub: false - ssh_keys: null - cbt: benchmarks: radosbench: concurrent_ops: 4 concurrent_procs: 2 op_size: - 4194304 pool_monitoring_list: - collectl pool_profile: replicated run_monitoring_list: - collectl time: 60 write_only: true cluster: iterations: 1 osds_per_node: 3 pool_profiles: replicated: pg_size: 256 pgp_size: 256 replication: replicated user: ubuntu monitoring_profiles: perf: args: stat -p {pid} -o {perf_dir}/perf_stat.{pid} nodes: - osds perf_cmd: perf pid_glob: crimson-osd.*.pid teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: fa17720d0088c3ac28e473468bfc79eeaff5cd38 timestamp: 2026-02-23_09:58:48 tube: trial user: shraddhaag verbose: false worker_log: /home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.3318653 2026-02-23T11:43:45.083 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_shraddhaag_ceph_e371433fcd1390a0b82cfb18effc422c1f5d8c1d/qa; will attempt to use it 2026-02-23T11:43:45.083 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_shraddhaag_ceph_e371433fcd1390a0b82cfb18effc422c1f5d8c1d/qa/tasks 2026-02-23T11:43:45.084 INFO:teuthology.run_tasks:Running task internal.check_packages... 2026-02-23T11:43:45.085 INFO:teuthology.task.internal:Checking packages... 2026-02-23T11:43:45.085 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'debug' and ceph hash 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d' 2026-02-23T11:43:45.085 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2026-02-23T11:43:45.085 INFO:teuthology.packaging:ref: None 2026-02-23T11:43:45.085 INFO:teuthology.packaging:tag: None 2026-02-23T11:43:45.085 INFO:teuthology.packaging:branch: main 2026-02-23T11:43:45.085 INFO:teuthology.packaging:sha1: e371433fcd1390a0b82cfb18effc422c1f5d8c1d 2026-02-23T11:43:45.086 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&ref=main 2026-02-23T11:43:45.213 INFO:teuthology.task.internal:Found packages for ceph version 20.3.0-5253.g5ccdd6ad 2026-02-23T11:43:45.214 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2026-02-23T11:43:45.226 INFO:teuthology.task.internal:no buildpackages task found 2026-02-23T11:43:45.226 INFO:teuthology.run_tasks:Running task internal.save_config... 2026-02-23T11:43:45.239 INFO:teuthology.task.internal:Saving configuration 2026-02-23T11:43:45.255 INFO:teuthology.run_tasks:Running task internal.check_lock... 2026-02-23T11:43:45.268 INFO:teuthology.task.internal.check_lock:Checking locks... 2026-02-23T11:43:45.345 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial186.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/shraddhaag-2026-02-23_09:58:48-crimson-rados-main-distro-debug-trial/66294', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-23 11:40:18.599009', 'locked_by': 'scheduled_shraddhaag@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLDyxqOVszW7QgWdmsWYzM19QfQnsb0ZiZyzGIHN941zbMFYb7ZU9RsookcbeRuxK/WY4tvgeWIANTqANZ6NgeA='} 2026-02-23T11:43:45.345 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2026-02-23T11:43:45.357 INFO:teuthology.task.internal:roles: ubuntu@trial186.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2026-02-23T11:43:45.357 INFO:teuthology.run_tasks:Running task console_log... 2026-02-23T11:43:45.549 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x763473b09e40>, signals=[15]) 2026-02-23T11:43:45.549 INFO:teuthology.run_tasks:Running task internal.connect... 2026-02-23T11:43:45.560 INFO:teuthology.task.internal:Opening connections... 2026-02-23T11:43:45.560 DEBUG:teuthology.task.internal:connecting to ubuntu@trial186.front.sepia.ceph.com 2026-02-23T11:43:45.561 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-23T11:43:45.621 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2026-02-23T11:43:45.634 DEBUG:teuthology.orchestra.run.trial186:> uname -m 2026-02-23T11:43:45.694 INFO:teuthology.orchestra.run.trial186.stdout:x86_64 2026-02-23T11:43:45.694 DEBUG:teuthology.orchestra.run.trial186:> cat /etc/os-release 2026-02-23T11:43:45.750 INFO:teuthology.orchestra.run.trial186.stdout:NAME="CentOS Stream" 2026-02-23T11:43:45.750 INFO:teuthology.orchestra.run.trial186.stdout:VERSION="9" 2026-02-23T11:43:45.750 INFO:teuthology.orchestra.run.trial186.stdout:ID="centos" 2026-02-23T11:43:45.750 INFO:teuthology.orchestra.run.trial186.stdout:ID_LIKE="rhel fedora" 2026-02-23T11:43:45.750 INFO:teuthology.orchestra.run.trial186.stdout:VERSION_ID="9" 2026-02-23T11:43:45.750 INFO:teuthology.orchestra.run.trial186.stdout:PLATFORM_ID="platform:el9" 2026-02-23T11:43:45.750 INFO:teuthology.orchestra.run.trial186.stdout:PRETTY_NAME="CentOS Stream 9" 2026-02-23T11:43:45.750 INFO:teuthology.orchestra.run.trial186.stdout:ANSI_COLOR="0;31" 2026-02-23T11:43:45.750 INFO:teuthology.orchestra.run.trial186.stdout:LOGO="fedora-logo-icon" 2026-02-23T11:43:45.751 INFO:teuthology.orchestra.run.trial186.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2026-02-23T11:43:45.751 INFO:teuthology.orchestra.run.trial186.stdout:HOME_URL="https://centos.org/" 2026-02-23T11:43:45.751 INFO:teuthology.orchestra.run.trial186.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2026-02-23T11:43:45.751 INFO:teuthology.orchestra.run.trial186.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2026-02-23T11:43:45.751 INFO:teuthology.orchestra.run.trial186.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2026-02-23T11:43:45.751 INFO:teuthology.lock.ops:Updating trial186.front.sepia.ceph.com on lock server 2026-02-23T11:43:45.838 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2026-02-23T11:43:45.851 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2026-02-23T11:43:45.862 INFO:teuthology.task.internal:Checking for old test directory... 2026-02-23T11:43:45.862 DEBUG:teuthology.orchestra.run.trial186:> test '!' -e /home/ubuntu/cephtest 2026-02-23T11:43:45.878 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2026-02-23T11:43:45.889 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2026-02-23T11:43:45.889 DEBUG:teuthology.orchestra.run.trial186:> test -z $(ls -A /var/lib/ceph) 2026-02-23T11:43:45.935 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2026-02-23T11:43:46.038 INFO:teuthology.run_tasks:Running task kernel... 2026-02-23T11:43:46.059 INFO:teuthology.task.kernel:normalize config orig: {'branch': 'distro', 'kdb': True, 'sha1': 'distro'} 2026-02-23T11:43:46.059 INFO:teuthology.task.kernel:config {'mon.a': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}}, timeout 300 2026-02-23T11:43:46.059 DEBUG:teuthology.orchestra.run.trial186:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-23T11:43:46.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-23T11:43:46.074 DEBUG:teuthology.orchestra.run.trial186:> uname -r 2026-02-23T11:43:46.129 INFO:teuthology.orchestra.run.trial186.stdout:5.14.0-681.el9.x86_64 2026-02-23T11:43:46.130 INFO:teuthology.task.kernel:Running kernel on trial186: 5.14.0-681.el9.x86_64 2026-02-23T11:43:46.130 DEBUG:teuthology.orchestra.run.trial186:> sudo yum install -y kernel 2026-02-23T11:43:47.291 INFO:teuthology.orchestra.run.trial186.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.9 MB 00:00 2026-02-23T11:43:50.360 INFO:teuthology.orchestra.run.trial186.stdout:CentOS Stream 9 - AppStream 11 MB/s | 27 MB 00:02 2026-02-23T11:43:54.421 INFO:teuthology.orchestra.run.trial186.stdout:CentOS Stream 9 - CRB 4.6 MB/s | 7.8 MB 00:01 2026-02-23T11:43:55.257 INFO:teuthology.orchestra.run.trial186.stdout:CentOS Stream 9 - Extras packages 185 kB/s | 20 kB 00:00 2026-02-23T11:43:56.007 INFO:teuthology.orchestra.run.trial186.stdout:Extra Packages for Enterprise Linux 29 MB/s | 20 MB 00:00 2026-02-23T11:43:59.200 INFO:teuthology.orchestra.run.trial186.stdout:lab-extras 2.5 MB/s | 50 kB 00:00 2026-02-23T11:44:00.214 INFO:teuthology.orchestra.run.trial186.stdout:Package kernel-5.14.0-645.el9.x86_64 is already installed. 2026-02-23T11:44:00.214 INFO:teuthology.orchestra.run.trial186.stdout:Package kernel-5.14.0-681.el9.x86_64 is already installed. 2026-02-23T11:44:00.233 INFO:teuthology.orchestra.run.trial186.stdout:Dependencies resolved. 2026-02-23T11:44:00.234 INFO:teuthology.orchestra.run.trial186.stdout:Nothing to do. 2026-02-23T11:44:00.234 INFO:teuthology.orchestra.run.trial186.stdout:Complete! 2026-02-23T11:44:00.258 DEBUG:teuthology.orchestra.run.trial186:> echo no | sudo yum reinstall kernel || true 2026-02-23T11:44:00.559 INFO:teuthology.orchestra.run.trial186.stdout:Last metadata expiration check: 0:00:01 ago on Mon 23 Feb 2026 11:43:59 AM UTC. 2026-02-23T11:44:00.640 INFO:teuthology.orchestra.run.trial186.stdout:Dependencies resolved. 2026-02-23T11:44:00.641 INFO:teuthology.orchestra.run.trial186.stdout:================================================================================ 2026-02-23T11:44:00.641 INFO:teuthology.orchestra.run.trial186.stdout: Package Architecture Version Repository Size 2026-02-23T11:44:00.641 INFO:teuthology.orchestra.run.trial186.stdout:================================================================================ 2026-02-23T11:44:00.642 INFO:teuthology.orchestra.run.trial186.stdout:Reinstalling: 2026-02-23T11:44:00.642 INFO:teuthology.orchestra.run.trial186.stdout: kernel x86_64 5.14.0-681.el9 baseos 803 k 2026-02-23T11:44:00.642 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:44:00.642 INFO:teuthology.orchestra.run.trial186.stdout:Transaction Summary 2026-02-23T11:44:00.642 INFO:teuthology.orchestra.run.trial186.stdout:================================================================================ 2026-02-23T11:44:00.642 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:44:00.642 INFO:teuthology.orchestra.run.trial186.stdout:Total download size: 803 k 2026-02-23T11:44:00.642 INFO:teuthology.orchestra.run.trial186.stdout:Installed size: 0 2026-02-23T11:44:00.643 INFO:teuthology.orchestra.run.trial186.stderr:Operation aborted. 2026-02-23T11:44:00.661 INFO:teuthology.orchestra.run.trial186.stdout:Is this ok [y/N]: 2026-02-23T11:44:00.661 DEBUG:teuthology.orchestra.run.trial186:> sudo yum reinstall -y kernel || true 2026-02-23T11:44:00.963 INFO:teuthology.orchestra.run.trial186.stdout:Last metadata expiration check: 0:00:01 ago on Mon 23 Feb 2026 11:43:59 AM UTC. 2026-02-23T11:44:01.044 INFO:teuthology.orchestra.run.trial186.stdout:Dependencies resolved. 2026-02-23T11:44:01.044 INFO:teuthology.orchestra.run.trial186.stdout:================================================================================ 2026-02-23T11:44:01.044 INFO:teuthology.orchestra.run.trial186.stdout: Package Architecture Version Repository Size 2026-02-23T11:44:01.044 INFO:teuthology.orchestra.run.trial186.stdout:================================================================================ 2026-02-23T11:44:01.045 INFO:teuthology.orchestra.run.trial186.stdout:Reinstalling: 2026-02-23T11:44:01.045 INFO:teuthology.orchestra.run.trial186.stdout: kernel x86_64 5.14.0-681.el9 baseos 803 k 2026-02-23T11:44:01.045 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:44:01.045 INFO:teuthology.orchestra.run.trial186.stdout:Transaction Summary 2026-02-23T11:44:01.045 INFO:teuthology.orchestra.run.trial186.stdout:================================================================================ 2026-02-23T11:44:01.045 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:44:01.045 INFO:teuthology.orchestra.run.trial186.stdout:Total download size: 803 k 2026-02-23T11:44:01.045 INFO:teuthology.orchestra.run.trial186.stdout:Installed size: 0 2026-02-23T11:44:01.045 INFO:teuthology.orchestra.run.trial186.stdout:Downloading Packages: 2026-02-23T11:44:01.775 INFO:teuthology.orchestra.run.trial186.stdout:kernel-5.14.0-681.el9.x86_64.rpm 1.2 MB/s | 803 kB 00:00 2026-02-23T11:44:01.775 INFO:teuthology.orchestra.run.trial186.stdout:-------------------------------------------------------------------------------- 2026-02-23T11:44:01.775 INFO:teuthology.orchestra.run.trial186.stdout:Total 1.1 MB/s | 803 kB 00:00 2026-02-23T11:44:01.786 INFO:teuthology.orchestra.run.trial186.stdout:Running transaction check 2026-02-23T11:44:01.819 INFO:teuthology.orchestra.run.trial186.stdout:Transaction check succeeded. 2026-02-23T11:44:01.819 INFO:teuthology.orchestra.run.trial186.stdout:Running transaction test 2026-02-23T11:44:01.823 INFO:teuthology.orchestra.run.trial186.stdout:Transaction test succeeded. 2026-02-23T11:44:01.824 INFO:teuthology.orchestra.run.trial186.stdout:Running transaction 2026-02-23T11:44:01.847 INFO:teuthology.orchestra.run.trial186.stdout: Preparing : 1/1 2026-02-23T11:44:01.852 INFO:teuthology.orchestra.run.trial186.stdout: Reinstalling : kernel-5.14.0-681.el9.x86_64 1/2 2026-02-23T11:44:01.852 INFO:teuthology.orchestra.run.trial186.stdout: Cleanup : kernel-5.14.0-681.el9.x86_64 2/2 2026-02-23T11:44:01.963 INFO:teuthology.orchestra.run.trial186.stdout: Running scriptlet: kernel-5.14.0-681.el9.x86_64 2/2 2026-02-23T11:44:01.963 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : kernel-5.14.0-681.el9.x86_64 1/2 2026-02-23T11:44:02.014 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : kernel-5.14.0-681.el9.x86_64 2/2 2026-02-23T11:44:02.015 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:44:02.015 INFO:teuthology.orchestra.run.trial186.stdout:Reinstalled: 2026-02-23T11:44:02.015 INFO:teuthology.orchestra.run.trial186.stdout: kernel-5.14.0-681.el9.x86_64 2026-02-23T11:44:02.015 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:44:02.015 INFO:teuthology.orchestra.run.trial186.stdout:Complete! 2026-02-23T11:44:02.041 DEBUG:teuthology.orchestra.run.trial186:> rpm -q kernel | sort -rV | head -n 1 2026-02-23T11:44:02.105 INFO:teuthology.orchestra.run.trial186.stdout:kernel-5.14.0-681.el9.x86_64 2026-02-23T11:44:02.106 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-681.el9.x86_64 2026-02-23T11:44:02.106 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2026-02-23T11:44:02.106 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2026-02-23T11:44:02.106 DEBUG:teuthology.orchestra.run.trial186:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-23T11:44:02.170 INFO:teuthology.orchestra.run.trial186.stdout:ttyS1 2026-02-23T11:44:02.195 DEBUG:teuthology.parallel:result is None 2026-02-23T11:44:02.195 INFO:teuthology.run_tasks:Running task internal.base... 2026-02-23T11:44:02.202 INFO:teuthology.task.internal:Creating test directory... 2026-02-23T11:44:02.203 DEBUG:teuthology.orchestra.run.trial186:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-23T11:44:02.252 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2026-02-23T11:44:02.263 INFO:teuthology.run_tasks:Running task internal.archive... 2026-02-23T11:44:02.274 INFO:teuthology.task.internal:Creating archive directory... 2026-02-23T11:44:02.275 DEBUG:teuthology.orchestra.run.trial186:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-23T11:44:02.313 INFO:teuthology.run_tasks:Running task internal.coredump... 2026-02-23T11:44:02.325 INFO:teuthology.task.internal:Enabling coredump saving... 2026-02-23T11:44:02.325 DEBUG:teuthology.orchestra.run.trial186:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-23T11:44:02.372 INFO:teuthology.orchestra.run.trial186.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-23T11:44:02.381 INFO:teuthology.orchestra.run.trial186.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-23T11:44:02.383 INFO:teuthology.run_tasks:Running task internal.sudo... 2026-02-23T11:44:02.395 INFO:teuthology.task.internal:Configuring sudo... 2026-02-23T11:44:02.395 DEBUG:teuthology.orchestra.run.trial186:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-23T11:44:02.447 INFO:teuthology.run_tasks:Running task internal.syslog... 2026-02-23T11:44:02.466 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2026-02-23T11:44:02.466 DEBUG:teuthology.orchestra.run.trial186:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-23T11:44:02.503 DEBUG:teuthology.orchestra.run.trial186:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-23T11:44:02.662 DEBUG:teuthology.orchestra.run.trial186:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-23T11:44:02.686 DEBUG:teuthology.orchestra.run.trial186:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-23T11:44:02.828 DEBUG:teuthology.orchestra.run.trial186:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-23T11:44:02.852 DEBUG:teuthology.orchestra.run.trial186:> set -ex 2026-02-23T11:44:02.852 DEBUG:teuthology.orchestra.run.trial186:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-23T11:44:02.915 DEBUG:teuthology.orchestra.run.trial186:> sudo service rsyslog restart 2026-02-23T11:44:02.983 INFO:teuthology.orchestra.run.trial186.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-23T11:44:03.345 INFO:teuthology.run_tasks:Running task internal.timer... 2026-02-23T11:44:03.364 INFO:teuthology.task.internal:Starting timer... 2026-02-23T11:44:03.364 INFO:teuthology.run_tasks:Running task pcp... 2026-02-23T11:44:03.386 INFO:teuthology.run_tasks:Running task selinux... 2026-02-23T11:44:03.409 DEBUG:teuthology.orchestra.run.trial186:> sudo service auditd rotate 2026-02-23T11:44:03.438 INFO:teuthology.orchestra.run.trial186.stdout:Rotating logs: 2026-02-23T11:44:03.440 DEBUG:teuthology.task.selinux:Getting current SELinux state 2026-02-23T11:44:03.440 DEBUG:teuthology.orchestra.run.trial186:> /usr/sbin/getenforce 2026-02-23T11:44:03.496 INFO:teuthology.orchestra.run.trial186.stdout:Permissive 2026-02-23T11:44:03.496 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@trial186.front.sepia.ceph.com': 'permissive'} 2026-02-23T11:44:03.496 DEBUG:teuthology.orchestra.run.trial186:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-23T11:44:03.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-23T11:44:03.562 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2026-02-23T11:44:03.562 DEBUG:teuthology.orchestra.run.trial186:> sudo /usr/sbin/setenforce permissive 2026-02-23T11:44:03.625 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2026-02-23T11:44:03.640 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2026-02-23T11:44:03.650 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2026-02-23T11:44:03.652 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit trial186.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2026-02-23T11:46:46.610 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@trial186.front.sepia.ceph.com')] 2026-02-23T11:46:46.612 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial186.front.sepia.ceph.com' 2026-02-23T11:46:46.614 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-23T11:46:46.721 DEBUG:teuthology.orchestra.run.trial186:> true 2026-02-23T11:46:46.795 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial186.front.sepia.ceph.com' 2026-02-23T11:46:46.796 INFO:teuthology.run_tasks:Running task clock... 2026-02-23T11:46:46.809 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2026-02-23T11:46:46.809 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-23T11:46:46.810 DEBUG:teuthology.orchestra.run.trial186:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-23T11:46:46.864 INFO:teuthology.orchestra.run.trial186.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2026-02-23T11:46:46.876 INFO:teuthology.orchestra.run.trial186.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2026-02-23T11:46:46.897 INFO:teuthology.orchestra.run.trial186.stderr:sudo: ntpd: command not found 2026-02-23T11:46:46.907 INFO:teuthology.orchestra.run.trial186.stdout:506 Cannot talk to daemon 2026-02-23T11:46:46.920 INFO:teuthology.orchestra.run.trial186.stderr:Failed to start ntp.service: Unit ntp.service not found. 2026-02-23T11:46:46.932 INFO:teuthology.orchestra.run.trial186.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2026-02-23T11:46:46.982 INFO:teuthology.orchestra.run.trial186.stderr:bash: line 1: ntpq: command not found 2026-02-23T11:46:47.004 INFO:teuthology.orchestra.run.trial186.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-23T11:46:47.004 INFO:teuthology.orchestra.run.trial186.stdout:=============================================================================== 2026-02-23T11:46:47.004 INFO:teuthology.orchestra.run.trial186.stdout:^? ntp1.lax.us.hojmark.net 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-23T11:46:47.004 INFO:teuthology.orchestra.run.trial186.stdout:^? 66.42.71.197.vultruserco> 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-23T11:46:47.004 INFO:teuthology.orchestra.run.trial186.stdout:^? 64.44.115.65 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-23T11:46:47.004 INFO:teuthology.orchestra.run.trial186.stdout:^? dns-e.ns4v.icu 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-23T11:46:47.006 INFO:teuthology.run_tasks:Running task install... 2026-02-23T11:46:47.021 DEBUG:teuthology.task.install:project ceph 2026-02-23T11:46:47.021 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'debug', 'sha1': 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d'}} 2026-02-23T11:46:47.021 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'flavor': 'debug', 'sha1': 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d'} 2026-02-23T11:46:47.021 INFO:teuthology.task.install:Using flavor: debug 2026-02-23T11:46:47.030 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-osd', 'ceph-osd-classic', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-osd', 'ceph-osd-classic', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-23T11:46:47.031 INFO:teuthology.task.install:extra packages: {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']} 2026-02-23T11:46:47.031 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'debug', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d', 'tag': None, 'wait_for_package': False} 2026-02-23T11:46:47.032 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d 2026-02-23T11:46:47.197 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-shraddhaag-enable-cephadm-tests/e371433fcd1390a0b82cfb18effc422c1f5d8c1d/centos/9/flavors/debug/ 2026-02-23T11:46:47.197 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5439.ge371433f 2026-02-23T11:46:47.587 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-shraddhaag-enable-cephadm-tests/e371433fcd1390a0b82cfb18effc422c1f5d8c1d/centos/9/flavors/debug/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-shraddhaag-enable-cephadm-tests/e371433fcd1390a0b82cfb18effc422c1f5d8c1d/centos/9/flavors/debug/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-shraddhaag-enable-cephadm-tests/e371433fcd1390a0b82cfb18effc422c1f5d8c1d/centos/9/flavors/debug/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-23T11:46:47.587 DEBUG:teuthology.orchestra.run.trial186:> set -ex 2026-02-23T11:46:47.588 DEBUG:teuthology.orchestra.run.trial186:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-23T11:46:47.613 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-osd-crimson, ceph-osd-crimson-debuginfo on remote rpm x86_64 2026-02-23T11:46:47.613 DEBUG:teuthology.orchestra.run.trial186:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/e371433fcd1390a0b82cfb18effc422c1f5d8c1d/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-23T11:46:47.678 DEBUG:teuthology.orchestra.run.trial186:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-23T11:46:47.753 DEBUG:teuthology.orchestra.run.trial186:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-23T11:46:47.818 INFO:teuthology.orchestra.run.trial186.stdout:check_obsoletes = 1 2026-02-23T11:46:47.820 DEBUG:teuthology.orchestra.run.trial186:> sudo yum clean all 2026-02-23T11:46:47.980 INFO:teuthology.orchestra.run.trial186.stdout:41 files removed 2026-02-23T11:46:47.995 DEBUG:teuthology.orchestra.run.trial186:> sudo yum -y install ceph-radosgw 2026-02-23T11:46:48.824 INFO:teuthology.orchestra.run.trial186.stdout:ceph packages for x86_64 134 kB/s | 92 kB 00:00 2026-02-23T11:46:49.465 INFO:teuthology.orchestra.run.trial186.stdout:ceph noarch packages 39 kB/s | 24 kB 00:00 2026-02-23T11:46:50.000 INFO:teuthology.orchestra.run.trial186.stdout:ceph source packages 4.0 kB/s | 2.1 kB 00:00 2026-02-23T11:46:50.710 INFO:teuthology.orchestra.run.trial186.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.9 MB 00:00 2026-02-23T11:46:53.983 INFO:teuthology.orchestra.run.trial186.stdout:CentOS Stream 9 - AppStream 9.8 MB/s | 27 MB 00:02 2026-02-23T11:46:57.458 INFO:teuthology.orchestra.run.trial186.stdout:CentOS Stream 9 - CRB 7.2 MB/s | 7.8 MB 00:01 2026-02-23T11:46:59.334 INFO:teuthology.orchestra.run.trial186.stdout:CentOS Stream 9 - Extras packages 18 kB/s | 20 kB 00:01 2026-02-23T11:47:00.281 INFO:teuthology.orchestra.run.trial186.stdout:Extra Packages for Enterprise Linux 23 MB/s | 20 MB 00:00 2026-02-23T11:47:03.449 INFO:teuthology.orchestra.run.trial186.stdout:lab-extras 2.8 MB/s | 50 kB 00:00 2026-02-23T11:47:04.655 INFO:teuthology.orchestra.run.trial186.stdout:Dependencies resolved. 2026-02-23T11:47:04.656 INFO:teuthology.orchestra.run.trial186.stdout:================================================================================ 2026-02-23T11:47:04.656 INFO:teuthology.orchestra.run.trial186.stdout: Package Arch Version Repository Size 2026-02-23T11:47:04.656 INFO:teuthology.orchestra.run.trial186.stdout:================================================================================ 2026-02-23T11:47:04.656 INFO:teuthology.orchestra.run.trial186.stdout:Installing: 2026-02-23T11:47:04.657 INFO:teuthology.orchestra.run.trial186.stdout: ceph-radosgw x86_64 2:20.3.0-5439.ge371433f.el9 ceph 11 M 2026-02-23T11:47:04.657 INFO:teuthology.orchestra.run.trial186.stdout:Upgrading: 2026-02-23T11:47:04.657 INFO:teuthology.orchestra.run.trial186.stdout: librados2 x86_64 2:20.3.0-5439.ge371433f.el9 ceph 3.3 M 2026-02-23T11:47:04.657 INFO:teuthology.orchestra.run.trial186.stdout: librbd1 x86_64 2:20.3.0-5439.ge371433f.el9 ceph 2.7 M 2026-02-23T11:47:04.657 INFO:teuthology.orchestra.run.trial186.stdout:Installing dependencies: 2026-02-23T11:47:04.657 INFO:teuthology.orchestra.run.trial186.stdout: boost-program-options x86_64 1.75.0-13.el9 appstream 104 k 2026-02-23T11:47:04.657 INFO:teuthology.orchestra.run.trial186.stdout: ceph-base x86_64 2:20.3.0-5439.ge371433f.el9 ceph 5.1 M 2026-02-23T11:47:04.657 INFO:teuthology.orchestra.run.trial186.stdout: ceph-common x86_64 2:20.3.0-5439.ge371433f.el9 ceph 21 M 2026-02-23T11:47:04.657 INFO:teuthology.orchestra.run.trial186.stdout: ceph-selinux x86_64 2:20.3.0-5439.ge371433f.el9 ceph 25 k 2026-02-23T11:47:04.657 INFO:teuthology.orchestra.run.trial186.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2026-02-23T11:47:04.658 INFO:teuthology.orchestra.run.trial186.stdout: libarrow x86_64 9.0.0-15.el9 epel 4.4 M 2026-02-23T11:47:04.658 INFO:teuthology.orchestra.run.trial186.stdout: libarrow-doc noarch 9.0.0-15.el9 epel 25 k 2026-02-23T11:47:04.658 INFO:teuthology.orchestra.run.trial186.stdout: libcephfs2 x86_64 2:20.3.0-5439.ge371433f.el9 ceph 860 k 2026-02-23T11:47:04.658 INFO:teuthology.orchestra.run.trial186.stdout: libnbd x86_64 1.20.3-4.el9 appstream 164 k 2026-02-23T11:47:04.658 INFO:teuthology.orchestra.run.trial186.stdout: liboath x86_64 2.6.12-1.el9 epel 49 k 2026-02-23T11:47:04.658 INFO:teuthology.orchestra.run.trial186.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2026-02-23T11:47:04.658 INFO:teuthology.orchestra.run.trial186.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2026-02-23T11:47:04.658 INFO:teuthology.orchestra.run.trial186.stdout: libradosstriper1 x86_64 2:20.3.0-5439.ge371433f.el9 ceph 263 k 2026-02-23T11:47:04.658 INFO:teuthology.orchestra.run.trial186.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2026-02-23T11:47:04.658 INFO:teuthology.orchestra.run.trial186.stdout: librgw2 x86_64 2:20.3.0-5439.ge371433f.el9 ceph 5.6 M 2026-02-23T11:47:04.658 INFO:teuthology.orchestra.run.trial186.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2026-02-23T11:47:04.659 INFO:teuthology.orchestra.run.trial186.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2026-02-23T11:47:04.659 INFO:teuthology.orchestra.run.trial186.stdout: mailcap noarch 2.1.49-5.el9 baseos 33 k 2026-02-23T11:47:04.659 INFO:teuthology.orchestra.run.trial186.stdout: parquet-libs x86_64 9.0.0-15.el9 epel 838 k 2026-02-23T11:47:04.659 INFO:teuthology.orchestra.run.trial186.stdout: pciutils x86_64 3.7.0-7.el9 baseos 93 k 2026-02-23T11:47:04.659 INFO:teuthology.orchestra.run.trial186.stdout: python3-ceph-argparse x86_64 2:20.3.0-5439.ge371433f.el9 ceph 45 k 2026-02-23T11:47:04.659 INFO:teuthology.orchestra.run.trial186.stdout: python3-ceph-common x86_64 2:20.3.0-5439.ge371433f.el9 ceph 186 k 2026-02-23T11:47:04.659 INFO:teuthology.orchestra.run.trial186.stdout: python3-cephfs x86_64 2:20.3.0-5439.ge371433f.el9 ceph 204 k 2026-02-23T11:47:04.659 INFO:teuthology.orchestra.run.trial186.stdout: python3-rados x86_64 2:20.3.0-5439.ge371433f.el9 ceph 324 k 2026-02-23T11:47:04.659 INFO:teuthology.orchestra.run.trial186.stdout: python3-rbd x86_64 2:20.3.0-5439.ge371433f.el9 ceph 305 k 2026-02-23T11:47:04.659 INFO:teuthology.orchestra.run.trial186.stdout: python3-rgw x86_64 2:20.3.0-5439.ge371433f.el9 ceph 100 k 2026-02-23T11:47:04.659 INFO:teuthology.orchestra.run.trial186.stdout: qatlib x86_64 25.08.0-2.el9 appstream 240 k 2026-02-23T11:47:04.660 INFO:teuthology.orchestra.run.trial186.stdout: qatzip-libs x86_64 1.3.1-1.el9 appstream 66 k 2026-02-23T11:47:04.660 INFO:teuthology.orchestra.run.trial186.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2026-02-23T11:47:04.660 INFO:teuthology.orchestra.run.trial186.stdout: thrift x86_64 0.15.0-4.el9 epel 1.6 M 2026-02-23T11:47:04.660 INFO:teuthology.orchestra.run.trial186.stdout:Installing weak dependencies: 2026-02-23T11:47:04.660 INFO:teuthology.orchestra.run.trial186.stdout: qatlib-service x86_64 25.08.0-2.el9 appstream 37 k 2026-02-23T11:47:04.660 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:47:04.660 INFO:teuthology.orchestra.run.trial186.stdout:Transaction Summary 2026-02-23T11:47:04.660 INFO:teuthology.orchestra.run.trial186.stdout:================================================================================ 2026-02-23T11:47:04.660 INFO:teuthology.orchestra.run.trial186.stdout:Install 32 Packages 2026-02-23T11:47:04.660 INFO:teuthology.orchestra.run.trial186.stdout:Upgrade 2 Packages 2026-02-23T11:47:04.660 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:47:04.660 INFO:teuthology.orchestra.run.trial186.stdout:Total download size: 60 M 2026-02-23T11:47:04.661 INFO:teuthology.orchestra.run.trial186.stdout:Downloading Packages: 2026-02-23T11:47:09.516 INFO:teuthology.orchestra.run.trial186.stdout:(1/34): ceph-base-20.3.0-5439.ge371433f.el9.x86 1.2 MB/s | 5.1 MB 00:04 2026-02-23T11:47:09.580 INFO:teuthology.orchestra.run.trial186.stdout:(2/34): ceph-selinux-20.3.0-5439.ge371433f.el9. 389 kB/s | 25 kB 00:00 2026-02-23T11:47:10.170 INFO:teuthology.orchestra.run.trial186.stdout:(3/34): libcephfs2-20.3.0-5439.ge371433f.el9.x8 1.4 MB/s | 860 kB 00:00 2026-02-23T11:47:10.373 INFO:teuthology.orchestra.run.trial186.stdout:(4/34): libradosstriper1-20.3.0-5439.ge371433f. 1.3 MB/s | 263 kB 00:00 2026-02-23T11:47:12.980 INFO:teuthology.orchestra.run.trial186.stdout:(5/34): ceph-radosgw-20.3.0-5439.ge371433f.el9. 1.5 MB/s | 11 MB 00:07 2026-02-23T11:47:13.050 INFO:teuthology.orchestra.run.trial186.stdout:(6/34): python3-ceph-argparse-20.3.0-5439.ge371 645 kB/s | 45 kB 00:00 2026-02-23T11:47:13.256 INFO:teuthology.orchestra.run.trial186.stdout:(7/34): python3-ceph-common-20.3.0-5439.ge37143 906 kB/s | 186 kB 00:00 2026-02-23T11:47:13.438 INFO:teuthology.orchestra.run.trial186.stdout:(8/34): python3-cephfs-20.3.0-5439.ge371433f.el 1.1 MB/s | 204 kB 00:00 2026-02-23T11:47:13.651 INFO:teuthology.orchestra.run.trial186.stdout:(9/34): librgw2-20.3.0-5439.ge371433f.el9.x86_6 1.7 MB/s | 5.6 MB 00:03 2026-02-23T11:47:13.688 INFO:teuthology.orchestra.run.trial186.stdout:(10/34): python3-rados-20.3.0-5439.ge371433f.el 1.3 MB/s | 324 kB 00:00 2026-02-23T11:47:13.754 INFO:teuthology.orchestra.run.trial186.stdout:(11/34): python3-rgw-20.3.0-5439.ge371433f.el9. 1.5 MB/s | 100 kB 00:00 2026-02-23T11:47:13.765 INFO:teuthology.orchestra.run.trial186.stdout:(12/34): mailcap-2.1.49-5.el9.noarch.rpm 2.9 MB/s | 33 kB 00:00 2026-02-23T11:47:13.773 INFO:teuthology.orchestra.run.trial186.stdout:(13/34): pciutils-3.7.0-7.el9.x86_64.rpm 11 MB/s | 93 kB 00:00 2026-02-23T11:47:13.841 INFO:teuthology.orchestra.run.trial186.stdout:(14/34): python3-rbd-20.3.0-5439.ge371433f.el9. 1.6 MB/s | 305 kB 00:00 2026-02-23T11:47:14.338 INFO:teuthology.orchestra.run.trial186.stdout:(15/34): boost-program-options-1.75.0-13.el9.x8 184 kB/s | 104 kB 00:00 2026-02-23T11:47:14.480 INFO:teuthology.orchestra.run.trial186.stdout:(16/34): libnbd-1.20.3-4.el9.x86_64.rpm 257 kB/s | 164 kB 00:00 2026-02-23T11:47:14.567 INFO:teuthology.orchestra.run.trial186.stdout:(17/34): librabbitmq-0.11.0-7.el9.x86_64.rpm 520 kB/s | 45 kB 00:00 2026-02-23T11:47:14.577 INFO:teuthology.orchestra.run.trial186.stdout:(18/34): libpmemobj-1.12.1-1.el9.x86_64.rpm 671 kB/s | 160 kB 00:00 2026-02-23T11:47:14.971 INFO:teuthology.orchestra.run.trial186.stdout:(19/34): lttng-ust-2.12.0-6.el9.x86_64.rpm 744 kB/s | 292 kB 00:00 2026-02-23T11:47:15.287 INFO:teuthology.orchestra.run.trial186.stdout:(20/34): qatlib-25.08.0-2.el9.x86_64.rpm 758 kB/s | 240 kB 00:00 2026-02-23T11:47:15.356 INFO:teuthology.orchestra.run.trial186.stdout:(21/34): librdkafka-1.6.1-102.el9.x86_64.rpm 839 kB/s | 662 kB 00:00 2026-02-23T11:47:15.369 INFO:teuthology.orchestra.run.trial186.stdout:(22/34): qatlib-service-25.08.0-2.el9.x86_64.rp 449 kB/s | 37 kB 00:00 2026-02-23T11:47:15.439 INFO:teuthology.orchestra.run.trial186.stdout:(23/34): qatzip-libs-1.3.1-1.el9.x86_64.rpm 800 kB/s | 66 kB 00:00 2026-02-23T11:47:15.529 INFO:teuthology.orchestra.run.trial186.stdout:(24/34): gperftools-libs-2.9.1-3.el9.x86_64.rpm 1.9 MB/s | 308 kB 00:00 2026-02-23T11:47:15.556 INFO:teuthology.orchestra.run.trial186.stdout:(25/34): libarrow-doc-9.0.0-15.el9.noarch.rpm 935 kB/s | 25 kB 00:00 2026-02-23T11:47:15.582 INFO:teuthology.orchestra.run.trial186.stdout:(26/34): liboath-2.6.12-1.el9.x86_64.rpm 1.8 MB/s | 49 kB 00:00 2026-02-23T11:47:15.611 INFO:teuthology.orchestra.run.trial186.stdout:(27/34): libunwind-1.6.2-1.el9.x86_64.rpm 2.3 MB/s | 67 kB 00:00 2026-02-23T11:47:15.663 INFO:teuthology.orchestra.run.trial186.stdout:(28/34): parquet-libs-9.0.0-15.el9.x86_64.rpm 16 MB/s | 838 kB 00:00 2026-02-23T11:47:15.685 INFO:teuthology.orchestra.run.trial186.stdout:(29/34): libarrow-9.0.0-15.el9.x86_64.rpm 18 MB/s | 4.4 MB 00:00 2026-02-23T11:47:15.691 INFO:teuthology.orchestra.run.trial186.stdout:(30/34): re2-20211101-20.el9.x86_64.rpm 6.5 MB/s | 191 kB 00:00 2026-02-23T11:47:15.717 INFO:teuthology.orchestra.run.trial186.stdout:(31/34): thrift-0.15.0-4.el9.x86_64.rpm 49 MB/s | 1.6 MB 00:00 2026-02-23T11:47:17.436 INFO:teuthology.orchestra.run.trial186.stdout:(32/34): librbd1-20.3.0-5439.ge371433f.el9.x86_ 1.6 MB/s | 2.7 MB 00:01 2026-02-23T11:47:17.695 INFO:teuthology.orchestra.run.trial186.stdout:(33/34): librados2-20.3.0-5439.ge371433f.el9.x8 1.6 MB/s | 3.3 MB 00:02 2026-02-23T11:47:22.488 INFO:teuthology.orchestra.run.trial186.stdout:(34/34): ceph-common-20.3.0-5439.ge371433f.el9. 1.2 MB/s | 21 MB 00:17 2026-02-23T11:47:22.491 INFO:teuthology.orchestra.run.trial186.stdout:-------------------------------------------------------------------------------- 2026-02-23T11:47:22.491 INFO:teuthology.orchestra.run.trial186.stdout:Total 3.4 MB/s | 60 MB 00:17 2026-02-23T11:47:22.577 INFO:teuthology.orchestra.run.trial186.stdout:Running transaction check 2026-02-23T11:47:22.594 INFO:teuthology.orchestra.run.trial186.stdout:Transaction check succeeded. 2026-02-23T11:47:22.595 INFO:teuthology.orchestra.run.trial186.stdout:Running transaction test 2026-02-23T11:47:22.838 INFO:teuthology.orchestra.run.trial186.stdout:Transaction test succeeded. 2026-02-23T11:47:22.838 INFO:teuthology.orchestra.run.trial186.stdout:Running transaction 2026-02-23T11:47:23.119 INFO:teuthology.orchestra.run.trial186.stdout: Preparing : 1/1 2026-02-23T11:47:23.263 INFO:teuthology.orchestra.run.trial186.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/36 2026-02-23T11:47:23.267 INFO:teuthology.orchestra.run.trial186.stdout: Upgrading : librados2-2:20.3.0-5439.ge371433f.el9.x86_64 2/36 2026-02-23T11:47:23.296 INFO:teuthology.orchestra.run.trial186.stdout: Running scriptlet: librados2-2:20.3.0-5439.ge371433f.el9.x86_64 2/36 2026-02-23T11:47:23.334 INFO:teuthology.orchestra.run.trial186.stdout: Installing : python3-rados-2:20.3.0-5439.ge371433f.el9.x86_64 3/36 2026-02-23T11:47:23.335 INFO:teuthology.orchestra.run.trial186.stdout: Installing : libcephfs2-2:20.3.0-5439.ge371433f.el9.x86_64 4/36 2026-02-23T11:47:23.347 INFO:teuthology.orchestra.run.trial186.stdout: Running scriptlet: libcephfs2-2:20.3.0-5439.ge371433f.el9.x86_64 4/36 2026-02-23T11:47:23.354 INFO:teuthology.orchestra.run.trial186.stdout: Installing : liboath-2.6.12-1.el9.x86_64 5/36 2026-02-23T11:47:23.357 INFO:teuthology.orchestra.run.trial186.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/36 2026-02-23T11:47:23.362 INFO:teuthology.orchestra.run.trial186.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/36 2026-02-23T11:47:23.366 INFO:teuthology.orchestra.run.trial186.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 8/36 2026-02-23T11:47:23.377 INFO:teuthology.orchestra.run.trial186.stdout: Installing : python3-ceph-argparse-2:20.3.0-5439.ge371433f.el9. 9/36 2026-02-23T11:47:23.390 INFO:teuthology.orchestra.run.trial186.stdout: Installing : python3-cephfs-2:20.3.0-5439.ge371433f.el9.x86_64 10/36 2026-02-23T11:47:23.391 INFO:teuthology.orchestra.run.trial186.stdout: Installing : libradosstriper1-2:20.3.0-5439.ge371433f.el9.x86_6 11/36 2026-02-23T11:47:23.414 INFO:teuthology.orchestra.run.trial186.stdout: Running scriptlet: libradosstriper1-2:20.3.0-5439.ge371433f.el9.x86_6 11/36 2026-02-23T11:47:23.418 INFO:teuthology.orchestra.run.trial186.stdout: Installing : thrift-0.15.0-4.el9.x86_64 12/36 2026-02-23T11:47:23.451 INFO:teuthology.orchestra.run.trial186.stdout: Installing : re2-1:20211101-20.el9.x86_64 13/36 2026-02-23T11:47:23.454 INFO:teuthology.orchestra.run.trial186.stdout: Installing : libarrow-9.0.0-15.el9.x86_64 14/36 2026-02-23T11:47:23.461 INFO:teuthology.orchestra.run.trial186.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 15/36 2026-02-23T11:47:23.463 INFO:teuthology.orchestra.run.trial186.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 16/36 2026-02-23T11:47:23.468 INFO:teuthology.orchestra.run.trial186.stdout: Installing : libarrow-doc-9.0.0-15.el9.noarch 17/36 2026-02-23T11:47:23.583 INFO:teuthology.orchestra.run.trial186.stdout: Installing : libnbd-1.20.3-4.el9.x86_64 18/36 2026-02-23T11:47:23.584 INFO:teuthology.orchestra.run.trial186.stdout: Upgrading : librbd1-2:20.3.0-5439.ge371433f.el9.x86_64 19/36 2026-02-23T11:47:23.609 INFO:teuthology.orchestra.run.trial186.stdout: Running scriptlet: librbd1-2:20.3.0-5439.ge371433f.el9.x86_64 19/36 2026-02-23T11:47:23.612 INFO:teuthology.orchestra.run.trial186.stdout: Installing : python3-rbd-2:20.3.0-5439.ge371433f.el9.x86_64 20/36 2026-02-23T11:47:23.618 INFO:teuthology.orchestra.run.trial186.stdout: Installing : boost-program-options-1.75.0-13.el9.x86_64 21/36 2026-02-23T11:47:23.865 INFO:teuthology.orchestra.run.trial186.stdout: Installing : parquet-libs-9.0.0-15.el9.x86_64 22/36 2026-02-23T11:47:23.866 INFO:teuthology.orchestra.run.trial186.stdout: Installing : librgw2-2:20.3.0-5439.ge371433f.el9.x86_64 23/36 2026-02-23T11:47:23.882 INFO:teuthology.orchestra.run.trial186.stdout: Running scriptlet: librgw2-2:20.3.0-5439.ge371433f.el9.x86_64 23/36 2026-02-23T11:47:23.886 INFO:teuthology.orchestra.run.trial186.stdout: Installing : python3-rgw-2:20.3.0-5439.ge371433f.el9.x86_64 24/36 2026-02-23T11:47:23.890 INFO:teuthology.orchestra.run.trial186.stdout: Installing : pciutils-3.7.0-7.el9.x86_64 25/36 2026-02-23T11:47:23.893 INFO:teuthology.orchestra.run.trial186.stdout: Installing : qatlib-25.08.0-2.el9.x86_64 26/36 2026-02-23T11:47:23.894 INFO:teuthology.orchestra.run.trial186.stdout: Installing : qatlib-service-25.08.0-2.el9.x86_64 27/36 2026-02-23T11:47:23.912 INFO:teuthology.orchestra.run.trial186.stdout: Running scriptlet: qatlib-service-25.08.0-2.el9.x86_64 27/36 2026-02-23T11:47:23.915 INFO:teuthology.orchestra.run.trial186.stdout: Installing : qatzip-libs-1.3.1-1.el9.x86_64 28/36 2026-02-23T11:47:23.949 INFO:teuthology.orchestra.run.trial186.stdout: Installing : mailcap-2.1.49-5.el9.noarch 29/36 2026-02-23T11:47:23.951 INFO:teuthology.orchestra.run.trial186.stdout: Installing : python3-ceph-common-2:20.3.0-5439.ge371433f.el9.x8 30/36 2026-02-23T11:47:24.893 INFO:teuthology.orchestra.run.trial186.stdout: Running scriptlet: ceph-common-2:20.3.0-5439.ge371433f.el9.x86_64 31/36 2026-02-23T11:47:24.896 INFO:teuthology.orchestra.run.trial186.stdout: Installing : ceph-common-2:20.3.0-5439.ge371433f.el9.x86_64 31/36 2026-02-23T11:47:25.160 INFO:teuthology.orchestra.run.trial186.stdout: Running scriptlet: ceph-common-2:20.3.0-5439.ge371433f.el9.x86_64 31/36 2026-02-23T11:47:25.164 INFO:teuthology.orchestra.run.trial186.stdout: Installing : ceph-base-2:20.3.0-5439.ge371433f.el9.x86_64 32/36 2026-02-23T11:47:25.198 INFO:teuthology.orchestra.run.trial186.stdout: Running scriptlet: ceph-base-2:20.3.0-5439.ge371433f.el9.x86_64 32/36 2026-02-23T11:47:25.198 INFO:teuthology.orchestra.run.trial186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2026-02-23T11:47:25.198 INFO:teuthology.orchestra.run.trial186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2026-02-23T11:47:25.199 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:47:25.201 INFO:teuthology.orchestra.run.trial186.stdout: Installing : ceph-selinux-2:20.3.0-5439.ge371433f.el9.x86_64 33/36 2026-02-23T11:47:29.673 INFO:teuthology.orchestra.run.trial186.stdout: Running scriptlet: ceph-selinux-2:20.3.0-5439.ge371433f.el9.x86_64 33/36 2026-02-23T11:47:29.673 INFO:teuthology.orchestra.run.trial186.stdout:skipping the directory /sys 2026-02-23T11:47:29.673 INFO:teuthology.orchestra.run.trial186.stdout:skipping the directory /proc 2026-02-23T11:47:29.673 INFO:teuthology.orchestra.run.trial186.stdout:skipping the directory /mnt 2026-02-23T11:47:29.674 INFO:teuthology.orchestra.run.trial186.stdout:skipping the directory /var/tmp 2026-02-23T11:47:29.674 INFO:teuthology.orchestra.run.trial186.stdout:skipping the directory /home 2026-02-23T11:47:29.674 INFO:teuthology.orchestra.run.trial186.stdout:skipping the directory /root 2026-02-23T11:47:29.674 INFO:teuthology.orchestra.run.trial186.stdout:skipping the directory /tmp 2026-02-23T11:47:29.674 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:47:30.165 INFO:teuthology.orchestra.run.trial186.stdout: Installing : ceph-radosgw-2:20.3.0-5439.ge371433f.el9.x86_64 34/36 2026-02-23T11:47:30.189 INFO:teuthology.orchestra.run.trial186.stdout: Running scriptlet: ceph-radosgw-2:20.3.0-5439.ge371433f.el9.x86_64 34/36 2026-02-23T11:47:30.189 INFO:teuthology.orchestra.run.trial186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-23T11:47:30.190 INFO:teuthology.orchestra.run.trial186.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2026-02-23T11:47:30.190 INFO:teuthology.orchestra.run.trial186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-23T11:47:30.190 INFO:teuthology.orchestra.run.trial186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-23T11:47:30.190 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:47:30.191 INFO:teuthology.orchestra.run.trial186.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 35/36 2026-02-23T11:47:30.203 INFO:teuthology.orchestra.run.trial186.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 35/36 2026-02-23T11:47:30.203 INFO:teuthology.orchestra.run.trial186.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 36/36 2026-02-23T11:47:30.258 INFO:teuthology.orchestra.run.trial186.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 36/36 2026-02-23T11:47:30.258 INFO:teuthology.orchestra.run.trial186.stdout:Creating group 'qat' with GID 992. 2026-02-23T11:47:30.258 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:47:30.555 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : ceph-base-2:20.3.0-5439.ge371433f.el9.x86_64 1/36 2026-02-23T11:47:30.556 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : ceph-common-2:20.3.0-5439.ge371433f.el9.x86_64 2/36 2026-02-23T11:47:30.556 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : ceph-radosgw-2:20.3.0-5439.ge371433f.el9.x86_64 3/36 2026-02-23T11:47:30.556 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : ceph-selinux-2:20.3.0-5439.ge371433f.el9.x86_64 4/36 2026-02-23T11:47:30.556 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : libcephfs2-2:20.3.0-5439.ge371433f.el9.x86_64 5/36 2026-02-23T11:47:30.556 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : libradosstriper1-2:20.3.0-5439.ge371433f.el9.x86_6 6/36 2026-02-23T11:47:30.556 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : librgw2-2:20.3.0-5439.ge371433f.el9.x86_64 7/36 2026-02-23T11:47:30.556 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : python3-ceph-argparse-2:20.3.0-5439.ge371433f.el9. 8/36 2026-02-23T11:47:30.556 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : python3-ceph-common-2:20.3.0-5439.ge371433f.el9.x8 9/36 2026-02-23T11:47:30.556 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : python3-cephfs-2:20.3.0-5439.ge371433f.el9.x86_64 10/36 2026-02-23T11:47:30.556 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : python3-rados-2:20.3.0-5439.ge371433f.el9.x86_64 11/36 2026-02-23T11:47:30.556 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : python3-rbd-2:20.3.0-5439.ge371433f.el9.x86_64 12/36 2026-02-23T11:47:30.557 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : python3-rgw-2:20.3.0-5439.ge371433f.el9.x86_64 13/36 2026-02-23T11:47:30.557 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : mailcap-2.1.49-5.el9.noarch 14/36 2026-02-23T11:47:30.557 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : pciutils-3.7.0-7.el9.x86_64 15/36 2026-02-23T11:47:30.557 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : boost-program-options-1.75.0-13.el9.x86_64 16/36 2026-02-23T11:47:30.557 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : libnbd-1.20.3-4.el9.x86_64 17/36 2026-02-23T11:47:30.557 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 18/36 2026-02-23T11:47:30.557 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 19/36 2026-02-23T11:47:30.557 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 20/36 2026-02-23T11:47:30.557 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 21/36 2026-02-23T11:47:30.557 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : qatlib-25.08.0-2.el9.x86_64 22/36 2026-02-23T11:47:30.557 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : qatlib-service-25.08.0-2.el9.x86_64 23/36 2026-02-23T11:47:30.557 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : qatzip-libs-1.3.1-1.el9.x86_64 24/36 2026-02-23T11:47:30.557 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 25/36 2026-02-23T11:47:30.558 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : libarrow-9.0.0-15.el9.x86_64 26/36 2026-02-23T11:47:30.558 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : libarrow-doc-9.0.0-15.el9.noarch 27/36 2026-02-23T11:47:30.558 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : liboath-2.6.12-1.el9.x86_64 28/36 2026-02-23T11:47:30.558 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 29/36 2026-02-23T11:47:30.558 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : parquet-libs-9.0.0-15.el9.x86_64 30/36 2026-02-23T11:47:30.558 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : re2-1:20211101-20.el9.x86_64 31/36 2026-02-23T11:47:30.558 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : thrift-0.15.0-4.el9.x86_64 32/36 2026-02-23T11:47:30.558 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : librados2-2:20.3.0-5439.ge371433f.el9.x86_64 33/36 2026-02-23T11:47:30.558 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 34/36 2026-02-23T11:47:30.558 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : librbd1-2:20.3.0-5439.ge371433f.el9.x86_64 35/36 2026-02-23T11:47:30.617 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 36/36 2026-02-23T11:47:30.617 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:47:30.617 INFO:teuthology.orchestra.run.trial186.stdout:Upgraded: 2026-02-23T11:47:30.617 INFO:teuthology.orchestra.run.trial186.stdout: librados2-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:47:30.617 INFO:teuthology.orchestra.run.trial186.stdout: librbd1-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:47:30.617 INFO:teuthology.orchestra.run.trial186.stdout:Installed: 2026-02-23T11:47:30.617 INFO:teuthology.orchestra.run.trial186.stdout: boost-program-options-1.75.0-13.el9.x86_64 2026-02-23T11:47:30.617 INFO:teuthology.orchestra.run.trial186.stdout: ceph-base-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:47:30.618 INFO:teuthology.orchestra.run.trial186.stdout: ceph-common-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:47:30.618 INFO:teuthology.orchestra.run.trial186.stdout: ceph-radosgw-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:47:30.618 INFO:teuthology.orchestra.run.trial186.stdout: ceph-selinux-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:47:30.618 INFO:teuthology.orchestra.run.trial186.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2026-02-23T11:47:30.618 INFO:teuthology.orchestra.run.trial186.stdout: libarrow-9.0.0-15.el9.x86_64 2026-02-23T11:47:30.618 INFO:teuthology.orchestra.run.trial186.stdout: libarrow-doc-9.0.0-15.el9.noarch 2026-02-23T11:47:30.618 INFO:teuthology.orchestra.run.trial186.stdout: libcephfs2-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:47:30.618 INFO:teuthology.orchestra.run.trial186.stdout: libnbd-1.20.3-4.el9.x86_64 2026-02-23T11:47:30.618 INFO:teuthology.orchestra.run.trial186.stdout: liboath-2.6.12-1.el9.x86_64 2026-02-23T11:47:30.618 INFO:teuthology.orchestra.run.trial186.stdout: libpmemobj-1.12.1-1.el9.x86_64 2026-02-23T11:47:30.618 INFO:teuthology.orchestra.run.trial186.stdout: librabbitmq-0.11.0-7.el9.x86_64 2026-02-23T11:47:30.618 INFO:teuthology.orchestra.run.trial186.stdout: libradosstriper1-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:47:30.619 INFO:teuthology.orchestra.run.trial186.stdout: librdkafka-1.6.1-102.el9.x86_64 2026-02-23T11:47:30.619 INFO:teuthology.orchestra.run.trial186.stdout: librgw2-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:47:30.619 INFO:teuthology.orchestra.run.trial186.stdout: libunwind-1.6.2-1.el9.x86_64 2026-02-23T11:47:30.619 INFO:teuthology.orchestra.run.trial186.stdout: lttng-ust-2.12.0-6.el9.x86_64 2026-02-23T11:47:30.619 INFO:teuthology.orchestra.run.trial186.stdout: mailcap-2.1.49-5.el9.noarch 2026-02-23T11:47:30.619 INFO:teuthology.orchestra.run.trial186.stdout: parquet-libs-9.0.0-15.el9.x86_64 2026-02-23T11:47:30.619 INFO:teuthology.orchestra.run.trial186.stdout: pciutils-3.7.0-7.el9.x86_64 2026-02-23T11:47:30.619 INFO:teuthology.orchestra.run.trial186.stdout: python3-ceph-argparse-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:47:30.619 INFO:teuthology.orchestra.run.trial186.stdout: python3-ceph-common-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:47:30.619 INFO:teuthology.orchestra.run.trial186.stdout: python3-cephfs-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:47:30.619 INFO:teuthology.orchestra.run.trial186.stdout: python3-rados-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:47:30.619 INFO:teuthology.orchestra.run.trial186.stdout: python3-rbd-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:47:30.620 INFO:teuthology.orchestra.run.trial186.stdout: python3-rgw-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:47:30.620 INFO:teuthology.orchestra.run.trial186.stdout: qatlib-25.08.0-2.el9.x86_64 2026-02-23T11:47:30.620 INFO:teuthology.orchestra.run.trial186.stdout: qatlib-service-25.08.0-2.el9.x86_64 2026-02-23T11:47:30.620 INFO:teuthology.orchestra.run.trial186.stdout: qatzip-libs-1.3.1-1.el9.x86_64 2026-02-23T11:47:30.620 INFO:teuthology.orchestra.run.trial186.stdout: re2-1:20211101-20.el9.x86_64 2026-02-23T11:47:30.620 INFO:teuthology.orchestra.run.trial186.stdout: thrift-0.15.0-4.el9.x86_64 2026-02-23T11:47:30.620 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:47:30.620 INFO:teuthology.orchestra.run.trial186.stdout:Complete! 2026-02-23T11:47:30.657 DEBUG:teuthology.orchestra.run.trial186:> sudo yum -y install ceph-test 2026-02-23T11:47:30.970 INFO:teuthology.orchestra.run.trial186.stdout:Last metadata expiration check: 0:00:27 ago on Mon 23 Feb 2026 11:47:03 AM UTC. 2026-02-23T11:47:31.054 INFO:teuthology.orchestra.run.trial186.stdout:Dependencies resolved. 2026-02-23T11:47:31.055 INFO:teuthology.orchestra.run.trial186.stdout:================================================================================ 2026-02-23T11:47:31.055 INFO:teuthology.orchestra.run.trial186.stdout: Package Arch Version Repository Size 2026-02-23T11:47:31.055 INFO:teuthology.orchestra.run.trial186.stdout:================================================================================ 2026-02-23T11:47:31.055 INFO:teuthology.orchestra.run.trial186.stdout:Installing: 2026-02-23T11:47:31.056 INFO:teuthology.orchestra.run.trial186.stdout: ceph-test x86_64 2:20.3.0-5439.ge371433f.el9 ceph 126 M 2026-02-23T11:47:31.056 INFO:teuthology.orchestra.run.trial186.stdout:Installing dependencies: 2026-02-23T11:47:31.056 INFO:teuthology.orchestra.run.trial186.stdout: hwloc-libs x86_64 2.4.1-6.el9 baseos 2.1 M 2026-02-23T11:47:31.056 INFO:teuthology.orchestra.run.trial186.stdout: libasan8 x86_64 15.2.1-7.el9 appstream 525 k 2026-02-23T11:47:31.056 INFO:teuthology.orchestra.run.trial186.stdout: libatomic x86_64 11.5.0-14.el9 baseos 23 k 2026-02-23T11:47:31.056 INFO:teuthology.orchestra.run.trial186.stdout: libcephsqlite x86_64 2:20.3.0-5439.ge371433f.el9 ceph 153 k 2026-02-23T11:47:31.056 INFO:teuthology.orchestra.run.trial186.stdout: libubsan x86_64 11.5.0-14.el9 appstream 176 k 2026-02-23T11:47:31.056 INFO:teuthology.orchestra.run.trial186.stdout: libxslt x86_64 1.1.34-12.el9 appstream 233 k 2026-02-23T11:47:31.056 INFO:teuthology.orchestra.run.trial186.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2026-02-23T11:47:31.056 INFO:teuthology.orchestra.run.trial186.stdout: protobuf x86_64 3.14.0-17.el9 appstream 1.0 M 2026-02-23T11:47:31.056 INFO:teuthology.orchestra.run.trial186.stdout: socat x86_64 1.7.4.1-8.el9 appstream 303 k 2026-02-23T11:47:31.056 INFO:teuthology.orchestra.run.trial186.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2026-02-23T11:47:31.057 INFO:teuthology.orchestra.run.trial186.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2026-02-23T11:47:31.057 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:47:31.057 INFO:teuthology.orchestra.run.trial186.stdout:Transaction Summary 2026-02-23T11:47:31.057 INFO:teuthology.orchestra.run.trial186.stdout:================================================================================ 2026-02-23T11:47:31.057 INFO:teuthology.orchestra.run.trial186.stdout:Install 12 Packages 2026-02-23T11:47:31.057 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:47:31.058 INFO:teuthology.orchestra.run.trial186.stdout:Total download size: 131 M 2026-02-23T11:47:31.058 INFO:teuthology.orchestra.run.trial186.stdout:Installed size: 977 M 2026-02-23T11:47:31.058 INFO:teuthology.orchestra.run.trial186.stdout:Downloading Packages: 2026-02-23T11:47:32.269 INFO:teuthology.orchestra.run.trial186.stdout:(1/12): libcephsqlite-20.3.0-5439.ge371433f.el9 189 kB/s | 153 kB 00:00 2026-02-23T11:47:32.678 INFO:teuthology.orchestra.run.trial186.stdout:(2/12): libatomic-11.5.0-14.el9.x86_64.rpm 56 kB/s | 23 kB 00:00 2026-02-23T11:47:32.842 INFO:teuthology.orchestra.run.trial186.stdout:(3/12): lksctp-tools-1.0.19-2.el9.x86_64.rpm 571 kB/s | 94 kB 00:00 2026-02-23T11:47:33.024 INFO:teuthology.orchestra.run.trial186.stdout:(4/12): libasan8-15.2.1-7.el9.x86_64.rpm 2.8 MB/s | 525 kB 00:00 2026-02-23T11:47:33.053 INFO:teuthology.orchestra.run.trial186.stdout:(5/12): libubsan-11.5.0-14.el9.x86_64.rpm 5.9 MB/s | 176 kB 00:00 2026-02-23T11:47:33.084 INFO:teuthology.orchestra.run.trial186.stdout:(6/12): libxslt-1.1.34-12.el9.x86_64.rpm 7.3 MB/s | 233 kB 00:00 2026-02-23T11:47:33.142 INFO:teuthology.orchestra.run.trial186.stdout:(7/12): protobuf-3.14.0-17.el9.x86_64.rpm 17 MB/s | 1.0 MB 00:00 2026-02-23T11:47:33.170 INFO:teuthology.orchestra.run.trial186.stdout:(8/12): socat-1.7.4.1-8.el9.x86_64.rpm 11 MB/s | 303 kB 00:00 2026-02-23T11:47:33.195 INFO:teuthology.orchestra.run.trial186.stdout:(9/12): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.4 MB/s | 64 kB 00:00 2026-02-23T11:47:33.522 INFO:teuthology.orchestra.run.trial186.stdout:(10/12): yaml-cpp-0.6.3-5.el9.x86_64.rpm 373 kB/s | 122 kB 00:00 2026-02-23T11:47:35.246 INFO:teuthology.orchestra.run.trial186.stdout:(11/12): hwloc-libs-2.4.1-6.el9.x86_64.rpm 567 kB/s | 2.1 MB 00:03 2026-02-23T11:49:05.799 INFO:teuthology.orchestra.run.trial186.stdout:(12/12): ceph-test-20.3.0-5439.ge371433f.el9.x8 1.3 MB/s | 126 MB 01:34 2026-02-23T11:49:05.801 INFO:teuthology.orchestra.run.trial186.stdout:-------------------------------------------------------------------------------- 2026-02-23T11:49:05.801 INFO:teuthology.orchestra.run.trial186.stdout:Total 1.4 MB/s | 131 MB 01:34 2026-02-23T11:49:05.872 INFO:teuthology.orchestra.run.trial186.stdout:Running transaction check 2026-02-23T11:49:05.883 INFO:teuthology.orchestra.run.trial186.stdout:Transaction check succeeded. 2026-02-23T11:49:05.883 INFO:teuthology.orchestra.run.trial186.stdout:Running transaction test 2026-02-23T11:49:06.312 INFO:teuthology.orchestra.run.trial186.stdout:Transaction test succeeded. 2026-02-23T11:49:06.312 INFO:teuthology.orchestra.run.trial186.stdout:Running transaction 2026-02-23T11:49:06.765 INFO:teuthology.orchestra.run.trial186.stdout: Preparing : 1/1 2026-02-23T11:49:06.773 INFO:teuthology.orchestra.run.trial186.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/12 2026-02-23T11:49:06.782 INFO:teuthology.orchestra.run.trial186.stdout: Installing : socat-1.7.4.1-8.el9.x86_64 2/12 2026-02-23T11:49:06.787 INFO:teuthology.orchestra.run.trial186.stdout: Installing : protobuf-3.14.0-17.el9.x86_64 3/12 2026-02-23T11:49:06.793 INFO:teuthology.orchestra.run.trial186.stdout: Installing : libxslt-1.1.34-12.el9.x86_64 4/12 2026-02-23T11:49:06.795 INFO:teuthology.orchestra.run.trial186.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 5/12 2026-02-23T11:49:06.799 INFO:teuthology.orchestra.run.trial186.stdout: Installing : libubsan-11.5.0-14.el9.x86_64 6/12 2026-02-23T11:49:06.800 INFO:teuthology.orchestra.run.trial186.stdout: Installing : libasan8-15.2.1-7.el9.x86_64 7/12 2026-02-23T11:49:06.817 INFO:teuthology.orchestra.run.trial186.stdout: Running scriptlet: libasan8-15.2.1-7.el9.x86_64 7/12 2026-02-23T11:49:06.819 INFO:teuthology.orchestra.run.trial186.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 8/12 2026-02-23T11:49:06.829 INFO:teuthology.orchestra.run.trial186.stdout: Installing : libatomic-11.5.0-14.el9.x86_64 9/12 2026-02-23T11:49:06.838 INFO:teuthology.orchestra.run.trial186.stdout: Installing : hwloc-libs-2.4.1-6.el9.x86_64 10/12 2026-02-23T11:49:06.838 INFO:teuthology.orchestra.run.trial186.stdout: Installing : libcephsqlite-2:20.3.0-5439.ge371433f.el9.x86_64 11/12 2026-02-23T11:49:13.724 INFO:teuthology.orchestra.run.trial186.stdout: Running scriptlet: libcephsqlite-2:20.3.0-5439.ge371433f.el9.x86_64 11/12 2026-02-23T11:49:13.743 INFO:teuthology.orchestra.run.trial186.stdout: Installing : ceph-test-2:20.3.0-5439.ge371433f.el9.x86_64 12/12 2026-02-23T11:49:14.508 INFO:teuthology.orchestra.run.trial186.stdout: Running scriptlet: ceph-test-2:20.3.0-5439.ge371433f.el9.x86_64 12/12 2026-02-23T11:49:14.508 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : ceph-test-2:20.3.0-5439.ge371433f.el9.x86_64 1/12 2026-02-23T11:49:14.508 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : libcephsqlite-2:20.3.0-5439.ge371433f.el9.x86_64 2/12 2026-02-23T11:49:14.508 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : hwloc-libs-2.4.1-6.el9.x86_64 3/12 2026-02-23T11:49:14.508 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : libatomic-11.5.0-14.el9.x86_64 4/12 2026-02-23T11:49:14.508 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 5/12 2026-02-23T11:49:14.508 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : libasan8-15.2.1-7.el9.x86_64 6/12 2026-02-23T11:49:14.508 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : libubsan-11.5.0-14.el9.x86_64 7/12 2026-02-23T11:49:14.508 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : libxslt-1.1.34-12.el9.x86_64 8/12 2026-02-23T11:49:14.508 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : protobuf-3.14.0-17.el9.x86_64 9/12 2026-02-23T11:49:14.509 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : socat-1.7.4.1-8.el9.x86_64 10/12 2026-02-23T11:49:14.509 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 11/12 2026-02-23T11:49:14.563 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 12/12 2026-02-23T11:49:14.563 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:49:14.563 INFO:teuthology.orchestra.run.trial186.stdout:Installed: 2026-02-23T11:49:14.563 INFO:teuthology.orchestra.run.trial186.stdout: ceph-test-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:49:14.563 INFO:teuthology.orchestra.run.trial186.stdout: hwloc-libs-2.4.1-6.el9.x86_64 2026-02-23T11:49:14.564 INFO:teuthology.orchestra.run.trial186.stdout: libasan8-15.2.1-7.el9.x86_64 2026-02-23T11:49:14.564 INFO:teuthology.orchestra.run.trial186.stdout: libatomic-11.5.0-14.el9.x86_64 2026-02-23T11:49:14.564 INFO:teuthology.orchestra.run.trial186.stdout: libcephsqlite-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:49:14.564 INFO:teuthology.orchestra.run.trial186.stdout: libubsan-11.5.0-14.el9.x86_64 2026-02-23T11:49:14.564 INFO:teuthology.orchestra.run.trial186.stdout: libxslt-1.1.34-12.el9.x86_64 2026-02-23T11:49:14.564 INFO:teuthology.orchestra.run.trial186.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2026-02-23T11:49:14.564 INFO:teuthology.orchestra.run.trial186.stdout: protobuf-3.14.0-17.el9.x86_64 2026-02-23T11:49:14.564 INFO:teuthology.orchestra.run.trial186.stdout: socat-1.7.4.1-8.el9.x86_64 2026-02-23T11:49:14.564 INFO:teuthology.orchestra.run.trial186.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2026-02-23T11:49:14.564 INFO:teuthology.orchestra.run.trial186.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2026-02-23T11:49:14.564 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:49:14.565 INFO:teuthology.orchestra.run.trial186.stdout:Complete! 2026-02-23T11:49:14.601 DEBUG:teuthology.orchestra.run.trial186:> sudo yum -y install ceph 2026-02-23T11:49:14.903 INFO:teuthology.orchestra.run.trial186.stdout:Last metadata expiration check: 0:02:11 ago on Mon 23 Feb 2026 11:47:03 AM UTC. 2026-02-23T11:49:14.986 INFO:teuthology.orchestra.run.trial186.stdout:Dependencies resolved. 2026-02-23T11:49:14.987 INFO:teuthology.orchestra.run.trial186.stdout:================================================================================ 2026-02-23T11:49:14.988 INFO:teuthology.orchestra.run.trial186.stdout: Package Arch Version Repository Size 2026-02-23T11:49:14.988 INFO:teuthology.orchestra.run.trial186.stdout:================================================================================ 2026-02-23T11:49:14.988 INFO:teuthology.orchestra.run.trial186.stdout:Installing: 2026-02-23T11:49:14.988 INFO:teuthology.orchestra.run.trial186.stdout: ceph x86_64 2:20.3.0-5439.ge371433f.el9 ceph 6.5 k 2026-02-23T11:49:14.988 INFO:teuthology.orchestra.run.trial186.stdout:Installing dependencies: 2026-02-23T11:49:14.988 INFO:teuthology.orchestra.run.trial186.stdout: ceph-mds x86_64 2:20.3.0-5439.ge371433f.el9 ceph 2.0 M 2026-02-23T11:49:14.988 INFO:teuthology.orchestra.run.trial186.stdout: ceph-mgr x86_64 2:20.3.0-5439.ge371433f.el9 ceph 937 k 2026-02-23T11:49:14.988 INFO:teuthology.orchestra.run.trial186.stdout: ceph-mgr-modules-core noarch 2:20.3.0-5439.ge371433f.el9 ceph-noarch 298 k 2026-02-23T11:49:14.988 INFO:teuthology.orchestra.run.trial186.stdout: ceph-mon x86_64 2:20.3.0-5439.ge371433f.el9 ceph 4.9 M 2026-02-23T11:49:14.988 INFO:teuthology.orchestra.run.trial186.stdout: ceph-osd x86_64 2:20.3.0-5439.ge371433f.el9 ceph 164 k 2026-02-23T11:49:14.988 INFO:teuthology.orchestra.run.trial186.stdout: ceph-osd-classic x86_64 2:20.3.0-5439.ge371433f.el9 ceph 15 M 2026-02-23T11:49:14.989 INFO:teuthology.orchestra.run.trial186.stdout: ceph-osd-crimson x86_64 2:20.3.0-5439.ge371433f.el9 ceph 87 M 2026-02-23T11:49:14.989 INFO:teuthology.orchestra.run.trial186.stdout: ledmon-libs x86_64 1.1.0-3.el9 baseos 40 k 2026-02-23T11:49:14.989 INFO:teuthology.orchestra.run.trial186.stdout: libconfig x86_64 1.7.2-9.el9 baseos 72 k 2026-02-23T11:49:14.989 INFO:teuthology.orchestra.run.trial186.stdout: libstoragemgmt x86_64 1.10.1-1.el9 appstream 246 k 2026-02-23T11:49:14.989 INFO:teuthology.orchestra.run.trial186.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2026-02-23T11:49:14.989 INFO:teuthology.orchestra.run.trial186.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2026-02-23T11:49:14.989 INFO:teuthology.orchestra.run.trial186.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2026-02-23T11:49:14.989 INFO:teuthology.orchestra.run.trial186.stdout: python3-autocommand noarch 2.2.2-8.el9 epel 29 k 2026-02-23T11:49:14.989 INFO:teuthology.orchestra.run.trial186.stdout: python3-backports-tarfile noarch 1.2.0-1.el9 epel 60 k 2026-02-23T11:49:14.989 INFO:teuthology.orchestra.run.trial186.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2026-02-23T11:49:14.989 INFO:teuthology.orchestra.run.trial186.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2026-02-23T11:49:14.989 INFO:teuthology.orchestra.run.trial186.stdout: python3-cheroot noarch 10.0.1-4.el9 epel 173 k 2026-02-23T11:49:14.990 INFO:teuthology.orchestra.run.trial186.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2026-02-23T11:49:14.990 INFO:teuthology.orchestra.run.trial186.stdout: python3-cryptography x86_64 36.0.1-5.el9 baseos 1.2 M 2026-02-23T11:49:14.990 INFO:teuthology.orchestra.run.trial186.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2026-02-23T11:49:14.990 INFO:teuthology.orchestra.run.trial186.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2026-02-23T11:49:14.990 INFO:teuthology.orchestra.run.trial186.stdout: python3-jaraco-collections 2026-02-23T11:49:14.990 INFO:teuthology.orchestra.run.trial186.stdout: noarch 3.0.0-8.el9 epel 23 k 2026-02-23T11:49:14.990 INFO:teuthology.orchestra.run.trial186.stdout: python3-jaraco-context noarch 6.0.1-3.el9 epel 20 k 2026-02-23T11:49:14.990 INFO:teuthology.orchestra.run.trial186.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2026-02-23T11:49:14.990 INFO:teuthology.orchestra.run.trial186.stdout: python3-jaraco-text noarch 4.0.0-2.el9 epel 26 k 2026-02-23T11:49:14.990 INFO:teuthology.orchestra.run.trial186.stdout: python3-libstoragemgmt x86_64 1.10.1-1.el9 appstream 177 k 2026-02-23T11:49:14.990 INFO:teuthology.orchestra.run.trial186.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2026-02-23T11:49:14.990 INFO:teuthology.orchestra.run.trial186.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2026-02-23T11:49:14.990 INFO:teuthology.orchestra.run.trial186.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2026-02-23T11:49:14.991 INFO:teuthology.orchestra.run.trial186.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2026-02-23T11:49:14.991 INFO:teuthology.orchestra.run.trial186.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2026-02-23T11:49:14.991 INFO:teuthology.orchestra.run.trial186.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2026-02-23T11:49:14.991 INFO:teuthology.orchestra.run.trial186.stdout: python3-pyparsing noarch 2.4.7-9.el9 baseos 150 k 2026-02-23T11:49:14.991 INFO:teuthology.orchestra.run.trial186.stdout: python3-requests noarch 2.25.1-10.el9 baseos 126 k 2026-02-23T11:49:14.991 INFO:teuthology.orchestra.run.trial186.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2026-02-23T11:49:14.991 INFO:teuthology.orchestra.run.trial186.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2026-02-23T11:49:14.991 INFO:teuthology.orchestra.run.trial186.stdout: python3-urllib3 noarch 1.26.5-7.el9 baseos 218 k 2026-02-23T11:49:14.991 INFO:teuthology.orchestra.run.trial186.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2026-02-23T11:49:14.991 INFO:teuthology.orchestra.run.trial186.stdout: unzip x86_64 6.0-59.el9 baseos 182 k 2026-02-23T11:49:14.991 INFO:teuthology.orchestra.run.trial186.stdout: zip x86_64 3.0-35.el9 baseos 266 k 2026-02-23T11:49:14.991 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:49:14.992 INFO:teuthology.orchestra.run.trial186.stdout:Transaction Summary 2026-02-23T11:49:14.992 INFO:teuthology.orchestra.run.trial186.stdout:================================================================================ 2026-02-23T11:49:14.992 INFO:teuthology.orchestra.run.trial186.stdout:Install 42 Packages 2026-02-23T11:49:14.992 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:49:14.993 INFO:teuthology.orchestra.run.trial186.stdout:Total download size: 114 M 2026-02-23T11:49:14.993 INFO:teuthology.orchestra.run.trial186.stdout:Installed size: 961 M 2026-02-23T11:49:14.993 INFO:teuthology.orchestra.run.trial186.stdout:Downloading Packages: 2026-02-23T11:49:16.052 INFO:teuthology.orchestra.run.trial186.stdout:(1/42): ceph-20.3.0-5439.ge371433f.el9.x86_64.r 28 kB/s | 6.5 kB 00:00 2026-02-23T11:49:16.473 INFO:teuthology.orchestra.run.trial186.stdout:(2/42): ceph-mgr-20.3.0-5439.ge371433f.el9.x86_ 1.4 MB/s | 937 kB 00:00 2026-02-23T11:49:16.543 INFO:teuthology.orchestra.run.trial186.stdout:(3/42): ceph-osd-20.3.0-5439.ge371433f.el9.x86_ 2.3 MB/s | 164 kB 00:00 2026-02-23T11:49:16.883 INFO:teuthology.orchestra.run.trial186.stdout:(4/42): ceph-mds-20.3.0-5439.ge371433f.el9.x86_ 1.9 MB/s | 2.0 MB 00:01 2026-02-23T11:49:17.965 INFO:teuthology.orchestra.run.trial186.stdout:(5/42): ceph-mon-20.3.0-5439.ge371433f.el9.x86_ 2.5 MB/s | 4.9 MB 00:01 2026-02-23T11:49:18.102 INFO:teuthology.orchestra.run.trial186.stdout:(6/42): ceph-mgr-modules-core-20.3.0-5439.ge371 2.1 MB/s | 298 kB 00:00 2026-02-23T11:49:18.540 INFO:teuthology.orchestra.run.trial186.stdout:(7/42): ledmon-libs-1.1.0-3.el9.x86_64.rpm 92 kB/s | 40 kB 00:00 2026-02-23T11:49:18.718 INFO:teuthology.orchestra.run.trial186.stdout:(8/42): libconfig-1.7.2-9.el9.x86_64.rpm 407 kB/s | 72 kB 00:00 2026-02-23T11:49:19.053 INFO:teuthology.orchestra.run.trial186.stdout:(9/42): python3-cffi-1.14.5-5.el9.x86_64.rpm 756 kB/s | 253 kB 00:00 2026-02-23T11:49:20.562 INFO:teuthology.orchestra.run.trial186.stdout:(10/42): python3-cryptography-36.0.1-5.el9.x86_ 845 kB/s | 1.2 MB 00:01 2026-02-23T11:49:20.740 INFO:teuthology.orchestra.run.trial186.stdout:(11/42): python3-ply-3.11-14.el9.noarch.rpm 599 kB/s | 106 kB 00:00 2026-02-23T11:49:20.918 INFO:teuthology.orchestra.run.trial186.stdout:(12/42): python3-pycparser-2.20-6.el9.noarch.rp 760 kB/s | 135 kB 00:00 2026-02-23T11:49:21.174 INFO:teuthology.orchestra.run.trial186.stdout:(13/42): python3-pyparsing-2.4.7-9.el9.noarch.r 588 kB/s | 150 kB 00:00 2026-02-23T11:49:21.352 INFO:teuthology.orchestra.run.trial186.stdout:(14/42): python3-requests-2.25.1-10.el9.noarch. 710 kB/s | 126 kB 00:00 2026-02-23T11:49:21.688 INFO:teuthology.orchestra.run.trial186.stdout:(15/42): python3-urllib3-1.26.5-7.el9.noarch.rp 650 kB/s | 218 kB 00:00 2026-02-23T11:49:21.943 INFO:teuthology.orchestra.run.trial186.stdout:(16/42): unzip-6.0-59.el9.x86_64.rpm 710 kB/s | 182 kB 00:00 2026-02-23T11:49:22.278 INFO:teuthology.orchestra.run.trial186.stdout:(17/42): zip-3.0-35.el9.x86_64.rpm 795 kB/s | 266 kB 00:00 2026-02-23T11:49:22.432 INFO:teuthology.orchestra.run.trial186.stdout:(18/42): libstoragemgmt-1.10.1-1.el9.x86_64.rpm 1.6 MB/s | 246 kB 00:00 2026-02-23T11:49:22.469 INFO:teuthology.orchestra.run.trial186.stdout:(19/42): lua-5.4.4-4.el9.x86_64.rpm 5.0 MB/s | 188 kB 00:00 2026-02-23T11:49:22.505 INFO:teuthology.orchestra.run.trial186.stdout:(20/42): python3-libstoragemgmt-1.10.1-1.el9.x8 4.8 MB/s | 177 kB 00:00 2026-02-23T11:49:22.535 INFO:teuthology.orchestra.run.trial186.stdout:(21/42): python3-packaging-20.9-5.el9.noarch.rp 2.6 MB/s | 77 kB 00:00 2026-02-23T11:49:22.562 INFO:teuthology.orchestra.run.trial186.stdout:(22/42): python3-toml-0.10.2-6.el9.noarch.rpm 1.5 MB/s | 42 kB 00:00 2026-02-23T11:49:22.639 INFO:teuthology.orchestra.run.trial186.stdout:(23/42): ceph-osd-classic-20.3.0-5439.ge371433f 2.4 MB/s | 15 MB 00:06 2026-02-23T11:49:22.661 INFO:teuthology.orchestra.run.trial186.stdout:(24/42): lua-devel-5.4.4-4.el9.x86_64.rpm 225 kB/s | 22 kB 00:00 2026-02-23T11:49:22.863 INFO:teuthology.orchestra.run.trial186.stdout:(25/42): python3-autocommand-2.2.2-8.el9.noarch 146 kB/s | 29 kB 00:00 2026-02-23T11:49:23.003 INFO:teuthology.orchestra.run.trial186.stdout:(26/42): luarocks-3.9.2-5.el9.noarch.rpm 416 kB/s | 151 kB 00:00 2026-02-23T11:49:23.005 INFO:teuthology.orchestra.run.trial186.stdout:(27/42): python3-backports-tarfile-1.2.0-1.el9. 421 kB/s | 60 kB 00:00 2026-02-23T11:49:23.077 INFO:teuthology.orchestra.run.trial186.stdout:(28/42): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 585 kB/s | 43 kB 00:00 2026-02-23T11:49:23.167 INFO:teuthology.orchestra.run.trial186.stdout:(29/42): python3-cheroot-10.0.1-4.el9.noarch.rp 1.0 MB/s | 173 kB 00:00 2026-02-23T11:49:23.231 INFO:teuthology.orchestra.run.trial186.stdout:(30/42): python3-jaraco-8.2.1-3.el9.noarch.rpm 167 kB/s | 11 kB 00:00 2026-02-23T11:49:23.242 INFO:teuthology.orchestra.run.trial186.stdout:(31/42): python3-cherrypy-18.6.1-2.el9.noarch.r 2.1 MB/s | 358 kB 00:00 2026-02-23T11:49:23.299 INFO:teuthology.orchestra.run.trial186.stdout:(32/42): python3-jaraco-classes-3.2.1-5.el9.noa 260 kB/s | 18 kB 00:00 2026-02-23T11:49:23.306 INFO:teuthology.orchestra.run.trial186.stdout:(33/42): python3-jaraco-collections-3.0.0-8.el9 363 kB/s | 23 kB 00:00 2026-02-23T11:49:23.365 INFO:teuthology.orchestra.run.trial186.stdout:(34/42): python3-jaraco-context-6.0.1-3.el9.noa 297 kB/s | 20 kB 00:00 2026-02-23T11:49:23.369 INFO:teuthology.orchestra.run.trial186.stdout:(35/42): python3-jaraco-functools-3.5.0-2.el9.n 306 kB/s | 19 kB 00:00 2026-02-23T11:49:23.433 INFO:teuthology.orchestra.run.trial186.stdout:(36/42): python3-jaraco-text-4.0.0-2.el9.noarch 390 kB/s | 26 kB 00:00 2026-02-23T11:49:23.438 INFO:teuthology.orchestra.run.trial186.stdout:(37/42): python3-more-itertools-8.12.0-2.el9.no 1.1 MB/s | 79 kB 00:00 2026-02-23T11:49:23.499 INFO:teuthology.orchestra.run.trial186.stdout:(38/42): python3-portend-3.1.0-2.el9.noarch.rpm 248 kB/s | 16 kB 00:00 2026-02-23T11:49:23.509 INFO:teuthology.orchestra.run.trial186.stdout:(39/42): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2026-02-23T11:49:23.572 INFO:teuthology.orchestra.run.trial186.stdout:(40/42): python3-zc-lockfile-2.0-10.el9.noarch. 315 kB/s | 20 kB 00:00 2026-02-23T11:49:23.575 INFO:teuthology.orchestra.run.trial186.stdout:(41/42): python3-tempora-5.0.0-2.el9.noarch.rpm 471 kB/s | 36 kB 00:00 2026-02-23T11:49:54.888 INFO:teuthology.orchestra.run.trial186.stdout:(42/42): ceph-osd-crimson-20.3.0-5439.ge371433f 2.3 MB/s | 87 MB 00:38 2026-02-23T11:49:54.890 INFO:teuthology.orchestra.run.trial186.stdout:-------------------------------------------------------------------------------- 2026-02-23T11:49:54.890 INFO:teuthology.orchestra.run.trial186.stdout:Total 2.9 MB/s | 114 MB 00:39 2026-02-23T11:49:55.020 INFO:teuthology.orchestra.run.trial186.stdout:Running transaction check 2026-02-23T11:49:55.039 INFO:teuthology.orchestra.run.trial186.stdout:Transaction check succeeded. 2026-02-23T11:49:55.039 INFO:teuthology.orchestra.run.trial186.stdout:Running transaction test 2026-02-23T11:49:55.431 INFO:teuthology.orchestra.run.trial186.stdout:Transaction test succeeded. 2026-02-23T11:49:55.432 INFO:teuthology.orchestra.run.trial186.stdout:Running transaction 2026-02-23T11:49:55.866 INFO:teuthology.orchestra.run.trial186.stdout: Preparing : 1/1 2026-02-23T11:49:55.868 INFO:teuthology.orchestra.run.trial186.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/42 2026-02-23T11:49:55.873 INFO:teuthology.orchestra.run.trial186.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/42 2026-02-23T11:49:55.877 INFO:teuthology.orchestra.run.trial186.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/42 2026-02-23T11:49:55.881 INFO:teuthology.orchestra.run.trial186.stdout: Installing : unzip-6.0-59.el9.x86_64 4/42 2026-02-23T11:49:55.887 INFO:teuthology.orchestra.run.trial186.stdout: Installing : ledmon-libs-1.1.0-3.el9.x86_64 5/42 2026-02-23T11:49:55.911 INFO:teuthology.orchestra.run.trial186.stdout: Installing : zip-3.0-35.el9.x86_64 6/42 2026-02-23T11:49:55.916 INFO:teuthology.orchestra.run.trial186.stdout: Installing : luarocks-3.9.2-5.el9.noarch 7/42 2026-02-23T11:49:55.921 INFO:teuthology.orchestra.run.trial186.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 8/42 2026-02-23T11:49:55.926 INFO:teuthology.orchestra.run.trial186.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 9/42 2026-02-23T11:49:55.932 INFO:teuthology.orchestra.run.trial186.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 10/42 2026-02-23T11:49:55.936 INFO:teuthology.orchestra.run.trial186.stdout: Installing : python3-backports-tarfile-1.2.0-1.el9.noarch 11/42 2026-02-23T11:49:55.941 INFO:teuthology.orchestra.run.trial186.stdout: Installing : python3-jaraco-context-6.0.1-3.el9.noarch 12/42 2026-02-23T11:49:55.947 INFO:teuthology.orchestra.run.trial186.stdout: Installing : python3-autocommand-2.2.2-8.el9.noarch 13/42 2026-02-23T11:49:55.950 INFO:teuthology.orchestra.run.trial186.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 14/42 2026-02-23T11:49:55.956 INFO:teuthology.orchestra.run.trial186.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 15/42 2026-02-23T11:49:55.959 INFO:teuthology.orchestra.run.trial186.stdout: Installing : python3-jaraco-text-4.0.0-2.el9.noarch 16/42 2026-02-23T11:49:55.966 INFO:teuthology.orchestra.run.trial186.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 17/42 2026-02-23T11:49:55.970 INFO:teuthology.orchestra.run.trial186.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 18/42 2026-02-23T11:49:55.993 INFO:teuthology.orchestra.run.trial186.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 19/42 2026-02-23T11:49:56.005 INFO:teuthology.orchestra.run.trial186.stdout: Installing : python3-urllib3-1.26.5-7.el9.noarch 20/42 2026-02-23T11:49:56.010 INFO:teuthology.orchestra.run.trial186.stdout: Installing : python3-requests-2.25.1-10.el9.noarch 21/42 2026-02-23T11:49:56.020 INFO:teuthology.orchestra.run.trial186.stdout: Installing : python3-pyparsing-2.4.7-9.el9.noarch 22/42 2026-02-23T11:49:56.026 INFO:teuthology.orchestra.run.trial186.stdout: Installing : python3-packaging-20.9-5.el9.noarch 23/42 2026-02-23T11:49:56.039 INFO:teuthology.orchestra.run.trial186.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/42 2026-02-23T11:49:56.055 INFO:teuthology.orchestra.run.trial186.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/42 2026-02-23T11:49:56.130 INFO:teuthology.orchestra.run.trial186.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/42 2026-02-23T11:49:56.138 INFO:teuthology.orchestra.run.trial186.stdout: Installing : python3-cryptography-36.0.1-5.el9.x86_64 27/42 2026-02-23T11:49:56.162 INFO:teuthology.orchestra.run.trial186.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/42 2026-02-23T11:49:56.191 INFO:teuthology.orchestra.run.trial186.stdout: Installing : python3-cheroot-10.0.1-4.el9.noarch 29/42 2026-02-23T11:49:56.198 INFO:teuthology.orchestra.run.trial186.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/42 2026-02-23T11:49:56.249 INFO:teuthology.orchestra.run.trial186.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/42 2026-02-23T11:49:56.291 INFO:teuthology.orchestra.run.trial186.stdout: Installing : ceph-mgr-modules-core-2:20.3.0-5439.ge371433f.el9. 32/42 2026-02-23T11:49:56.292 INFO:teuthology.orchestra.run.trial186.stdout: Installing : ceph-mgr-2:20.3.0-5439.ge371433f.el9.x86_64 33/42 2026-02-23T11:49:56.315 INFO:teuthology.orchestra.run.trial186.stdout: Running scriptlet: ceph-mgr-2:20.3.0-5439.ge371433f.el9.x86_64 33/42 2026-02-23T11:49:56.315 INFO:teuthology.orchestra.run.trial186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-23T11:49:56.315 INFO:teuthology.orchestra.run.trial186.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2026-02-23T11:49:56.315 INFO:teuthology.orchestra.run.trial186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-23T11:49:56.315 INFO:teuthology.orchestra.run.trial186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-23T11:49:56.315 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:49:56.320 INFO:teuthology.orchestra.run.trial186.stdout: Installing : libconfig-1.7.2-9.el9.x86_64 34/42 2026-02-23T11:49:56.336 INFO:teuthology.orchestra.run.trial186.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 35/42 2026-02-23T11:49:56.336 INFO:teuthology.orchestra.run.trial186.stdout:Creating group 'libstoragemgmt' with GID 991. 2026-02-23T11:49:56.336 INFO:teuthology.orchestra.run.trial186.stdout:Creating user 'libstoragemgmt' (daemon account for libstoragemgmt) with UID 991 and GID 991. 2026-02-23T11:49:56.337 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:49:56.345 INFO:teuthology.orchestra.run.trial186.stdout: Installing : libstoragemgmt-1.10.1-1.el9.x86_64 35/42 2026-02-23T11:49:56.368 INFO:teuthology.orchestra.run.trial186.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 35/42 2026-02-23T11:49:56.368 INFO:teuthology.orchestra.run.trial186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/libstoragemgmt.service → /usr/lib/systemd/system/libstoragemgmt.service. 2026-02-23T11:49:56.368 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:49:56.992 INFO:teuthology.orchestra.run.trial186.stdout: Installing : python3-libstoragemgmt-1.10.1-1.el9.x86_64 36/42 2026-02-23T11:50:02.508 INFO:teuthology.orchestra.run.trial186.stdout: Installing : ceph-osd-classic-2:20.3.0-5439.ge371433f.el9.x86_6 37/42 2026-02-23T11:50:02.517 INFO:teuthology.orchestra.run.trial186.stdout: Installing : ceph-osd-crimson-2:20.3.0-5439.ge371433f.el9.x86_6 38/42 2026-02-23T11:50:02.519 INFO:teuthology.orchestra.run.trial186.stdout: Installing : ceph-osd-2:20.3.0-5439.ge371433f.el9.x86_64 39/42 2026-02-23T11:50:02.555 INFO:teuthology.orchestra.run.trial186.stdout: Running scriptlet: ceph-osd-2:20.3.0-5439.ge371433f.el9.x86_64 39/42 2026-02-23T11:50:02.555 INFO:teuthology.orchestra.run.trial186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-23T11:50:02.556 INFO:teuthology.orchestra.run.trial186.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2026-02-23T11:50:02.556 INFO:teuthology.orchestra.run.trial186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-23T11:50:02.556 INFO:teuthology.orchestra.run.trial186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-23T11:50:02.556 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:50:02.768 INFO:teuthology.orchestra.run.trial186.stdout: Installing : ceph-mon-2:20.3.0-5439.ge371433f.el9.x86_64 40/42 2026-02-23T11:50:02.790 INFO:teuthology.orchestra.run.trial186.stdout: Running scriptlet: ceph-mon-2:20.3.0-5439.ge371433f.el9.x86_64 40/42 2026-02-23T11:50:02.790 INFO:teuthology.orchestra.run.trial186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-23T11:50:02.790 INFO:teuthology.orchestra.run.trial186.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2026-02-23T11:50:02.790 INFO:teuthology.orchestra.run.trial186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-23T11:50:02.790 INFO:teuthology.orchestra.run.trial186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-23T11:50:02.790 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:50:02.877 INFO:teuthology.orchestra.run.trial186.stdout: Installing : ceph-mds-2:20.3.0-5439.ge371433f.el9.x86_64 41/42 2026-02-23T11:50:02.898 INFO:teuthology.orchestra.run.trial186.stdout: Running scriptlet: ceph-mds-2:20.3.0-5439.ge371433f.el9.x86_64 41/42 2026-02-23T11:50:02.898 INFO:teuthology.orchestra.run.trial186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-23T11:50:02.898 INFO:teuthology.orchestra.run.trial186.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2026-02-23T11:50:02.898 INFO:teuthology.orchestra.run.trial186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-23T11:50:02.898 INFO:teuthology.orchestra.run.trial186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-23T11:50:02.898 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:50:02.899 INFO:teuthology.orchestra.run.trial186.stdout: Installing : ceph-2:20.3.0-5439.ge371433f.el9.x86_64 42/42 2026-02-23T11:50:02.905 INFO:teuthology.orchestra.run.trial186.stdout: Running scriptlet: ceph-osd-classic-2:20.3.0-5439.ge371433f.el9.x86_6 42/42 2026-02-23T11:50:02.927 INFO:teuthology.orchestra.run.trial186.stdout: Running scriptlet: ceph-osd-crimson-2:20.3.0-5439.ge371433f.el9.x86_6 42/42 2026-02-23T11:50:03.874 INFO:teuthology.orchestra.run.trial186.stdout: Running scriptlet: ceph-2:20.3.0-5439.ge371433f.el9.x86_64 42/42 2026-02-23T11:50:03.875 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : ceph-2:20.3.0-5439.ge371433f.el9.x86_64 1/42 2026-02-23T11:50:03.875 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : ceph-mds-2:20.3.0-5439.ge371433f.el9.x86_64 2/42 2026-02-23T11:50:03.875 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : ceph-mgr-2:20.3.0-5439.ge371433f.el9.x86_64 3/42 2026-02-23T11:50:03.875 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : ceph-mon-2:20.3.0-5439.ge371433f.el9.x86_64 4/42 2026-02-23T11:50:03.875 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : ceph-osd-2:20.3.0-5439.ge371433f.el9.x86_64 5/42 2026-02-23T11:50:03.875 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : ceph-osd-classic-2:20.3.0-5439.ge371433f.el9.x86_6 6/42 2026-02-23T11:50:03.875 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : ceph-osd-crimson-2:20.3.0-5439.ge371433f.el9.x86_6 7/42 2026-02-23T11:50:03.875 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : ceph-mgr-modules-core-2:20.3.0-5439.ge371433f.el9. 8/42 2026-02-23T11:50:03.875 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : ledmon-libs-1.1.0-3.el9.x86_64 9/42 2026-02-23T11:50:03.875 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : libconfig-1.7.2-9.el9.x86_64 10/42 2026-02-23T11:50:03.876 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/42 2026-02-23T11:50:03.876 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : python3-cryptography-36.0.1-5.el9.x86_64 12/42 2026-02-23T11:50:03.876 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/42 2026-02-23T11:50:03.876 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/42 2026-02-23T11:50:03.876 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : python3-pyparsing-2.4.7-9.el9.noarch 15/42 2026-02-23T11:50:03.876 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : python3-requests-2.25.1-10.el9.noarch 16/42 2026-02-23T11:50:03.876 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : python3-urllib3-1.26.5-7.el9.noarch 17/42 2026-02-23T11:50:03.876 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : unzip-6.0-59.el9.x86_64 18/42 2026-02-23T11:50:03.876 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : zip-3.0-35.el9.x86_64 19/42 2026-02-23T11:50:03.876 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : libstoragemgmt-1.10.1-1.el9.x86_64 20/42 2026-02-23T11:50:03.876 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : lua-5.4.4-4.el9.x86_64 21/42 2026-02-23T11:50:03.876 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : python3-libstoragemgmt-1.10.1-1.el9.x86_64 22/42 2026-02-23T11:50:03.876 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 23/42 2026-02-23T11:50:03.877 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 24/42 2026-02-23T11:50:03.877 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 25/42 2026-02-23T11:50:03.877 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 26/42 2026-02-23T11:50:03.877 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : python3-autocommand-2.2.2-8.el9.noarch 27/42 2026-02-23T11:50:03.877 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : python3-backports-tarfile-1.2.0-1.el9.noarch 28/42 2026-02-23T11:50:03.877 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 29/42 2026-02-23T11:50:03.877 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : python3-cheroot-10.0.1-4.el9.noarch 30/42 2026-02-23T11:50:03.877 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 31/42 2026-02-23T11:50:03.877 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 32/42 2026-02-23T11:50:03.877 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 33/42 2026-02-23T11:50:03.877 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 34/42 2026-02-23T11:50:03.877 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : python3-jaraco-context-6.0.1-3.el9.noarch 35/42 2026-02-23T11:50:03.877 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 36/42 2026-02-23T11:50:03.878 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : python3-jaraco-text-4.0.0-2.el9.noarch 37/42 2026-02-23T11:50:03.878 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 38/42 2026-02-23T11:50:03.878 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 39/42 2026-02-23T11:50:03.878 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 40/42 2026-02-23T11:50:03.878 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 41/42 2026-02-23T11:50:03.932 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 42/42 2026-02-23T11:50:03.932 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:50:03.932 INFO:teuthology.orchestra.run.trial186.stdout:Installed: 2026-02-23T11:50:03.932 INFO:teuthology.orchestra.run.trial186.stdout: ceph-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:50:03.932 INFO:teuthology.orchestra.run.trial186.stdout: ceph-mds-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:50:03.932 INFO:teuthology.orchestra.run.trial186.stdout: ceph-mgr-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:50:03.933 INFO:teuthology.orchestra.run.trial186.stdout: ceph-mgr-modules-core-2:20.3.0-5439.ge371433f.el9.noarch 2026-02-23T11:50:03.933 INFO:teuthology.orchestra.run.trial186.stdout: ceph-mon-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:50:03.933 INFO:teuthology.orchestra.run.trial186.stdout: ceph-osd-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:50:03.933 INFO:teuthology.orchestra.run.trial186.stdout: ceph-osd-classic-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:50:03.933 INFO:teuthology.orchestra.run.trial186.stdout: ceph-osd-crimson-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:50:03.933 INFO:teuthology.orchestra.run.trial186.stdout: ledmon-libs-1.1.0-3.el9.x86_64 2026-02-23T11:50:03.933 INFO:teuthology.orchestra.run.trial186.stdout: libconfig-1.7.2-9.el9.x86_64 2026-02-23T11:50:03.933 INFO:teuthology.orchestra.run.trial186.stdout: libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-23T11:50:03.933 INFO:teuthology.orchestra.run.trial186.stdout: lua-5.4.4-4.el9.x86_64 2026-02-23T11:50:03.933 INFO:teuthology.orchestra.run.trial186.stdout: lua-devel-5.4.4-4.el9.x86_64 2026-02-23T11:50:03.933 INFO:teuthology.orchestra.run.trial186.stdout: luarocks-3.9.2-5.el9.noarch 2026-02-23T11:50:03.934 INFO:teuthology.orchestra.run.trial186.stdout: python3-autocommand-2.2.2-8.el9.noarch 2026-02-23T11:50:03.934 INFO:teuthology.orchestra.run.trial186.stdout: python3-backports-tarfile-1.2.0-1.el9.noarch 2026-02-23T11:50:03.934 INFO:teuthology.orchestra.run.trial186.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2026-02-23T11:50:03.934 INFO:teuthology.orchestra.run.trial186.stdout: python3-cffi-1.14.5-5.el9.x86_64 2026-02-23T11:50:03.934 INFO:teuthology.orchestra.run.trial186.stdout: python3-cheroot-10.0.1-4.el9.noarch 2026-02-23T11:50:03.934 INFO:teuthology.orchestra.run.trial186.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2026-02-23T11:50:03.934 INFO:teuthology.orchestra.run.trial186.stdout: python3-cryptography-36.0.1-5.el9.x86_64 2026-02-23T11:50:03.934 INFO:teuthology.orchestra.run.trial186.stdout: python3-jaraco-8.2.1-3.el9.noarch 2026-02-23T11:50:03.934 INFO:teuthology.orchestra.run.trial186.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2026-02-23T11:50:03.934 INFO:teuthology.orchestra.run.trial186.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2026-02-23T11:50:03.934 INFO:teuthology.orchestra.run.trial186.stdout: python3-jaraco-context-6.0.1-3.el9.noarch 2026-02-23T11:50:03.934 INFO:teuthology.orchestra.run.trial186.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2026-02-23T11:50:03.935 INFO:teuthology.orchestra.run.trial186.stdout: python3-jaraco-text-4.0.0-2.el9.noarch 2026-02-23T11:50:03.935 INFO:teuthology.orchestra.run.trial186.stdout: python3-libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-23T11:50:03.935 INFO:teuthology.orchestra.run.trial186.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2026-02-23T11:50:03.935 INFO:teuthology.orchestra.run.trial186.stdout: python3-packaging-20.9-5.el9.noarch 2026-02-23T11:50:03.935 INFO:teuthology.orchestra.run.trial186.stdout: python3-ply-3.11-14.el9.noarch 2026-02-23T11:50:03.935 INFO:teuthology.orchestra.run.trial186.stdout: python3-portend-3.1.0-2.el9.noarch 2026-02-23T11:50:03.935 INFO:teuthology.orchestra.run.trial186.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2026-02-23T11:50:03.935 INFO:teuthology.orchestra.run.trial186.stdout: python3-pycparser-2.20-6.el9.noarch 2026-02-23T11:50:03.935 INFO:teuthology.orchestra.run.trial186.stdout: python3-pyparsing-2.4.7-9.el9.noarch 2026-02-23T11:50:03.935 INFO:teuthology.orchestra.run.trial186.stdout: python3-requests-2.25.1-10.el9.noarch 2026-02-23T11:50:03.935 INFO:teuthology.orchestra.run.trial186.stdout: python3-tempora-5.0.0-2.el9.noarch 2026-02-23T11:50:03.936 INFO:teuthology.orchestra.run.trial186.stdout: python3-toml-0.10.2-6.el9.noarch 2026-02-23T11:50:03.936 INFO:teuthology.orchestra.run.trial186.stdout: python3-urllib3-1.26.5-7.el9.noarch 2026-02-23T11:50:03.936 INFO:teuthology.orchestra.run.trial186.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2026-02-23T11:50:03.936 INFO:teuthology.orchestra.run.trial186.stdout: unzip-6.0-59.el9.x86_64 2026-02-23T11:50:03.936 INFO:teuthology.orchestra.run.trial186.stdout: zip-3.0-35.el9.x86_64 2026-02-23T11:50:03.936 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:50:03.936 INFO:teuthology.orchestra.run.trial186.stdout:Complete! 2026-02-23T11:50:03.974 DEBUG:teuthology.orchestra.run.trial186:> sudo yum -y install ceph-base 2026-02-23T11:50:04.280 INFO:teuthology.orchestra.run.trial186.stdout:Last metadata expiration check: 0:03:01 ago on Mon 23 Feb 2026 11:47:03 AM UTC. 2026-02-23T11:50:04.345 INFO:teuthology.orchestra.run.trial186.stdout:Package ceph-base-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:50:04.362 INFO:teuthology.orchestra.run.trial186.stdout:Dependencies resolved. 2026-02-23T11:50:04.363 INFO:teuthology.orchestra.run.trial186.stdout:Nothing to do. 2026-02-23T11:50:04.363 INFO:teuthology.orchestra.run.trial186.stdout:Complete! 2026-02-23T11:50:04.383 DEBUG:teuthology.orchestra.run.trial186:> sudo yum -y install cephadm 2026-02-23T11:50:04.693 INFO:teuthology.orchestra.run.trial186.stdout:Last metadata expiration check: 0:03:01 ago on Mon 23 Feb 2026 11:47:03 AM UTC. 2026-02-23T11:50:04.775 INFO:teuthology.orchestra.run.trial186.stdout:Dependencies resolved. 2026-02-23T11:50:04.775 INFO:teuthology.orchestra.run.trial186.stdout:================================================================================ 2026-02-23T11:50:04.775 INFO:teuthology.orchestra.run.trial186.stdout: Package Arch Version Repository Size 2026-02-23T11:50:04.776 INFO:teuthology.orchestra.run.trial186.stdout:================================================================================ 2026-02-23T11:50:04.776 INFO:teuthology.orchestra.run.trial186.stdout:Installing: 2026-02-23T11:50:04.776 INFO:teuthology.orchestra.run.trial186.stdout: cephadm noarch 2:20.3.0-5439.ge371433f.el9 ceph-noarch 1.0 M 2026-02-23T11:50:04.776 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:50:04.776 INFO:teuthology.orchestra.run.trial186.stdout:Transaction Summary 2026-02-23T11:50:04.776 INFO:teuthology.orchestra.run.trial186.stdout:================================================================================ 2026-02-23T11:50:04.776 INFO:teuthology.orchestra.run.trial186.stdout:Install 1 Package 2026-02-23T11:50:04.776 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:50:04.776 INFO:teuthology.orchestra.run.trial186.stdout:Total download size: 1.0 M 2026-02-23T11:50:04.776 INFO:teuthology.orchestra.run.trial186.stdout:Installed size: 1.0 M 2026-02-23T11:50:04.776 INFO:teuthology.orchestra.run.trial186.stdout:Downloading Packages: 2026-02-23T11:50:05.337 INFO:teuthology.orchestra.run.trial186.stdout:cephadm-20.3.0-5439.ge371433f.el9.noarch.rpm 1.8 MB/s | 1.0 MB 00:00 2026-02-23T11:50:05.337 INFO:teuthology.orchestra.run.trial186.stdout:-------------------------------------------------------------------------------- 2026-02-23T11:50:05.338 INFO:teuthology.orchestra.run.trial186.stdout:Total 1.8 MB/s | 1.0 MB 00:00 2026-02-23T11:50:05.338 INFO:teuthology.orchestra.run.trial186.stdout:Running transaction check 2026-02-23T11:50:05.340 INFO:teuthology.orchestra.run.trial186.stdout:Transaction check succeeded. 2026-02-23T11:50:05.340 INFO:teuthology.orchestra.run.trial186.stdout:Running transaction test 2026-02-23T11:50:05.344 INFO:teuthology.orchestra.run.trial186.stdout:Transaction test succeeded. 2026-02-23T11:50:05.344 INFO:teuthology.orchestra.run.trial186.stdout:Running transaction 2026-02-23T11:50:05.362 INFO:teuthology.orchestra.run.trial186.stdout: Preparing : 1/1 2026-02-23T11:50:05.422 INFO:teuthology.orchestra.run.trial186.stdout: Running scriptlet: cephadm-2:20.3.0-5439.ge371433f.el9.noarch 1/1 2026-02-23T11:50:05.436 INFO:teuthology.orchestra.run.trial186.stdout: Installing : cephadm-2:20.3.0-5439.ge371433f.el9.noarch 1/1 2026-02-23T11:50:05.503 INFO:teuthology.orchestra.run.trial186.stdout: Running scriptlet: cephadm-2:20.3.0-5439.ge371433f.el9.noarch 1/1 2026-02-23T11:50:05.557 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : cephadm-2:20.3.0-5439.ge371433f.el9.noarch 1/1 2026-02-23T11:50:05.557 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:50:05.558 INFO:teuthology.orchestra.run.trial186.stdout:Installed: 2026-02-23T11:50:05.558 INFO:teuthology.orchestra.run.trial186.stdout: cephadm-2:20.3.0-5439.ge371433f.el9.noarch 2026-02-23T11:50:05.558 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:50:05.558 INFO:teuthology.orchestra.run.trial186.stdout:Complete! 2026-02-23T11:50:05.584 DEBUG:teuthology.orchestra.run.trial186:> sudo yum -y install ceph-immutable-object-cache 2026-02-23T11:50:05.899 INFO:teuthology.orchestra.run.trial186.stdout:Last metadata expiration check: 0:03:02 ago on Mon 23 Feb 2026 11:47:03 AM UTC. 2026-02-23T11:50:05.981 INFO:teuthology.orchestra.run.trial186.stdout:Dependencies resolved. 2026-02-23T11:50:05.982 INFO:teuthology.orchestra.run.trial186.stdout:================================================================================ 2026-02-23T11:50:05.982 INFO:teuthology.orchestra.run.trial186.stdout: Package Arch Version Repo Size 2026-02-23T11:50:05.982 INFO:teuthology.orchestra.run.trial186.stdout:================================================================================ 2026-02-23T11:50:05.982 INFO:teuthology.orchestra.run.trial186.stdout:Installing: 2026-02-23T11:50:05.982 INFO:teuthology.orchestra.run.trial186.stdout: ceph-immutable-object-cache x86_64 2:20.3.0-5439.ge371433f.el9 ceph 149 k 2026-02-23T11:50:05.982 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:50:05.982 INFO:teuthology.orchestra.run.trial186.stdout:Transaction Summary 2026-02-23T11:50:05.982 INFO:teuthology.orchestra.run.trial186.stdout:================================================================================ 2026-02-23T11:50:05.982 INFO:teuthology.orchestra.run.trial186.stdout:Install 1 Package 2026-02-23T11:50:05.982 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:50:05.982 INFO:teuthology.orchestra.run.trial186.stdout:Total download size: 149 k 2026-02-23T11:50:05.982 INFO:teuthology.orchestra.run.trial186.stdout:Installed size: 455 k 2026-02-23T11:50:05.982 INFO:teuthology.orchestra.run.trial186.stdout:Downloading Packages: 2026-02-23T11:50:06.291 INFO:teuthology.orchestra.run.trial186.stdout:ceph-immutable-object-cache-20.3.0-5439.ge37143 482 kB/s | 149 kB 00:00 2026-02-23T11:50:06.292 INFO:teuthology.orchestra.run.trial186.stdout:-------------------------------------------------------------------------------- 2026-02-23T11:50:06.292 INFO:teuthology.orchestra.run.trial186.stdout:Total 480 kB/s | 149 kB 00:00 2026-02-23T11:50:06.292 INFO:teuthology.orchestra.run.trial186.stdout:Running transaction check 2026-02-23T11:50:06.296 INFO:teuthology.orchestra.run.trial186.stdout:Transaction check succeeded. 2026-02-23T11:50:06.296 INFO:teuthology.orchestra.run.trial186.stdout:Running transaction test 2026-02-23T11:50:06.322 INFO:teuthology.orchestra.run.trial186.stdout:Transaction test succeeded. 2026-02-23T11:50:06.323 INFO:teuthology.orchestra.run.trial186.stdout:Running transaction 2026-02-23T11:50:06.372 INFO:teuthology.orchestra.run.trial186.stdout: Preparing : 1/1 2026-02-23T11:50:06.373 INFO:teuthology.orchestra.run.trial186.stdout: Installing : ceph-immutable-object-cache-2:20.3.0-5439.ge371433f. 1/1 2026-02-23T11:50:06.395 INFO:teuthology.orchestra.run.trial186.stdout: Running scriptlet: ceph-immutable-object-cache-2:20.3.0-5439.ge371433f. 1/1 2026-02-23T11:50:06.395 INFO:teuthology.orchestra.run.trial186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-23T11:50:06.395 INFO:teuthology.orchestra.run.trial186.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2026-02-23T11:50:06.395 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:50:06.665 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : ceph-immutable-object-cache-2:20.3.0-5439.ge371433f. 1/1 2026-02-23T11:50:06.665 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:50:06.665 INFO:teuthology.orchestra.run.trial186.stdout:Installed: 2026-02-23T11:50:06.665 INFO:teuthology.orchestra.run.trial186.stdout: ceph-immutable-object-cache-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:50:06.665 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:50:06.665 INFO:teuthology.orchestra.run.trial186.stdout:Complete! 2026-02-23T11:50:06.693 DEBUG:teuthology.orchestra.run.trial186:> sudo yum -y install ceph-mgr 2026-02-23T11:50:07.001 INFO:teuthology.orchestra.run.trial186.stdout:Last metadata expiration check: 0:03:03 ago on Mon 23 Feb 2026 11:47:03 AM UTC. 2026-02-23T11:50:07.066 INFO:teuthology.orchestra.run.trial186.stdout:Package ceph-mgr-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:50:07.083 INFO:teuthology.orchestra.run.trial186.stdout:Dependencies resolved. 2026-02-23T11:50:07.084 INFO:teuthology.orchestra.run.trial186.stdout:Nothing to do. 2026-02-23T11:50:07.084 INFO:teuthology.orchestra.run.trial186.stdout:Complete! 2026-02-23T11:50:07.105 DEBUG:teuthology.orchestra.run.trial186:> sudo yum -y install ceph-mgr-dashboard 2026-02-23T11:50:07.410 INFO:teuthology.orchestra.run.trial186.stdout:Last metadata expiration check: 0:03:04 ago on Mon 23 Feb 2026 11:47:03 AM UTC. 2026-02-23T11:50:07.492 INFO:teuthology.orchestra.run.trial186.stdout:Dependencies resolved. 2026-02-23T11:50:07.493 INFO:teuthology.orchestra.run.trial186.stdout:================================================================================ 2026-02-23T11:50:07.493 INFO:teuthology.orchestra.run.trial186.stdout: Package Arch Version Repository Size 2026-02-23T11:50:07.493 INFO:teuthology.orchestra.run.trial186.stdout:================================================================================ 2026-02-23T11:50:07.494 INFO:teuthology.orchestra.run.trial186.stdout:Installing: 2026-02-23T11:50:07.494 INFO:teuthology.orchestra.run.trial186.stdout: ceph-mgr-dashboard noarch 2:20.3.0-5439.ge371433f.el9 ceph-noarch 51 M 2026-02-23T11:50:07.494 INFO:teuthology.orchestra.run.trial186.stdout:Installing dependencies: 2026-02-23T11:50:07.494 INFO:teuthology.orchestra.run.trial186.stdout: abseil-cpp x86_64 20211102.0-4.el9 epel 551 k 2026-02-23T11:50:07.494 INFO:teuthology.orchestra.run.trial186.stdout: ceph-grafana-dashboards noarch 2:20.3.0-5439.ge371433f.el9 ceph-noarch 57 k 2026-02-23T11:50:07.494 INFO:teuthology.orchestra.run.trial186.stdout: ceph-prometheus-alerts noarch 2:20.3.0-5439.ge371433f.el9 ceph-noarch 17 k 2026-02-23T11:50:07.494 INFO:teuthology.orchestra.run.trial186.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2026-02-23T11:50:07.494 INFO:teuthology.orchestra.run.trial186.stdout: protobuf-compiler x86_64 3.14.0-17.el9 crb 862 k 2026-02-23T11:50:07.494 INFO:teuthology.orchestra.run.trial186.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2026-02-23T11:50:07.494 INFO:teuthology.orchestra.run.trial186.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2026-02-23T11:50:07.494 INFO:teuthology.orchestra.run.trial186.stdout: python3-jmespath noarch 1.0.1-1.el9 appstream 48 k 2026-02-23T11:50:07.494 INFO:teuthology.orchestra.run.trial186.stdout: python3-protobuf noarch 3.14.0-17.el9 appstream 267 k 2026-02-23T11:50:07.495 INFO:teuthology.orchestra.run.trial186.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2026-02-23T11:50:07.495 INFO:teuthology.orchestra.run.trial186.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2026-02-23T11:50:07.495 INFO:teuthology.orchestra.run.trial186.stdout: python3-xmltodict noarch 0.12.0-15.el9 epel 22 k 2026-02-23T11:50:07.495 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:50:07.495 INFO:teuthology.orchestra.run.trial186.stdout:Transaction Summary 2026-02-23T11:50:07.495 INFO:teuthology.orchestra.run.trial186.stdout:================================================================================ 2026-02-23T11:50:07.495 INFO:teuthology.orchestra.run.trial186.stdout:Install 13 Packages 2026-02-23T11:50:07.495 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:50:07.495 INFO:teuthology.orchestra.run.trial186.stdout:Total download size: 55 M 2026-02-23T11:50:07.495 INFO:teuthology.orchestra.run.trial186.stdout:Installed size: 262 M 2026-02-23T11:50:07.495 INFO:teuthology.orchestra.run.trial186.stdout:Downloading Packages: 2026-02-23T11:50:08.233 INFO:teuthology.orchestra.run.trial186.stdout:(1/13): ceph-prometheus-alerts-20.3.0-5439.ge37 55 kB/s | 17 kB 00:00 2026-02-23T11:50:08.302 INFO:teuthology.orchestra.run.trial186.stdout:(2/13): ceph-grafana-dashboards-20.3.0-5439.ge3 148 kB/s | 57 kB 00:00 2026-02-23T11:50:08.867 INFO:teuthology.orchestra.run.trial186.stdout:(3/13): python3-jmespath-1.0.1-1.el9.noarch.rpm 75 kB/s | 48 kB 00:00 2026-02-23T11:50:08.997 INFO:teuthology.orchestra.run.trial186.stdout:(4/13): protobuf-compiler-3.14.0-17.el9.x86_64. 6.5 MB/s | 862 kB 00:00 2026-02-23T11:50:09.133 INFO:teuthology.orchestra.run.trial186.stdout:(5/13): abseil-cpp-20211102.0-4.el9.x86_64.rpm 4.0 MB/s | 551 kB 00:00 2026-02-23T11:50:09.150 INFO:teuthology.orchestra.run.trial186.stdout:(6/13): grpc-data-1.46.7-10.el9.noarch.rpm 1.1 MB/s | 19 kB 00:00 2026-02-23T11:50:09.169 INFO:teuthology.orchestra.run.trial186.stdout:(7/13): python3-protobuf-3.14.0-17.el9.noarch.r 309 kB/s | 267 kB 00:00 2026-02-23T11:50:09.208 INFO:teuthology.orchestra.run.trial186.stdout:(8/13): python3-grpcio-1.46.7-10.el9.x86_64.rpm 35 MB/s | 2.0 MB 00:00 2026-02-23T11:50:09.226 INFO:teuthology.orchestra.run.trial186.stdout:(9/13): python3-repoze-lru-0.7-16.el9.noarch.rp 1.7 MB/s | 31 kB 00:00 2026-02-23T11:50:09.245 INFO:teuthology.orchestra.run.trial186.stdout:(10/13): python3-routes-2.5.1-5.el9.noarch.rpm 9.7 MB/s | 188 kB 00:00 2026-02-23T11:50:09.260 INFO:teuthology.orchestra.run.trial186.stdout:(11/13): python3-grpcio-tools-1.46.7-10.el9.x86 1.5 MB/s | 144 kB 00:00 2026-02-23T11:50:09.264 INFO:teuthology.orchestra.run.trial186.stdout:(12/13): python3-xmltodict-0.12.0-15.el9.noarch 1.2 MB/s | 22 kB 00:00 2026-02-23T11:50:33.736 INFO:teuthology.orchestra.run.trial186.stdout:(13/13): ceph-mgr-dashboard-20.3.0-5439.ge37143 2.0 MB/s | 51 MB 00:25 2026-02-23T11:50:33.738 INFO:teuthology.orchestra.run.trial186.stdout:-------------------------------------------------------------------------------- 2026-02-23T11:50:33.738 INFO:teuthology.orchestra.run.trial186.stdout:Total 2.1 MB/s | 55 MB 00:26 2026-02-23T11:50:33.766 INFO:teuthology.orchestra.run.trial186.stdout:Running transaction check 2026-02-23T11:50:33.776 INFO:teuthology.orchestra.run.trial186.stdout:Transaction check succeeded. 2026-02-23T11:50:33.776 INFO:teuthology.orchestra.run.trial186.stdout:Running transaction test 2026-02-23T11:50:33.988 INFO:teuthology.orchestra.run.trial186.stdout:Transaction test succeeded. 2026-02-23T11:50:33.988 INFO:teuthology.orchestra.run.trial186.stdout:Running transaction 2026-02-23T11:50:34.216 INFO:teuthology.orchestra.run.trial186.stdout: Preparing : 1/1 2026-02-23T11:50:34.221 INFO:teuthology.orchestra.run.trial186.stdout: Installing : python3-xmltodict-0.12.0-15.el9.noarch 1/13 2026-02-23T11:50:34.233 INFO:teuthology.orchestra.run.trial186.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 2/13 2026-02-23T11:50:34.235 INFO:teuthology.orchestra.run.trial186.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 3/13 2026-02-23T11:50:34.263 INFO:teuthology.orchestra.run.trial186.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 4/13 2026-02-23T11:50:34.306 INFO:teuthology.orchestra.run.trial186.stdout: Installing : abseil-cpp-20211102.0-4.el9.x86_64 5/13 2026-02-23T11:50:34.315 INFO:teuthology.orchestra.run.trial186.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 6/13 2026-02-23T11:50:34.343 INFO:teuthology.orchestra.run.trial186.stdout: Installing : protobuf-compiler-3.14.0-17.el9.x86_64 7/13 2026-02-23T11:50:34.353 INFO:teuthology.orchestra.run.trial186.stdout: Installing : python3-protobuf-3.14.0-17.el9.noarch 8/13 2026-02-23T11:50:34.360 INFO:teuthology.orchestra.run.trial186.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/13 2026-02-23T11:50:34.363 INFO:teuthology.orchestra.run.trial186.stdout: Installing : python3-jmespath-1.0.1-1.el9.noarch 10/13 2026-02-23T11:50:34.373 INFO:teuthology.orchestra.run.trial186.stdout: Installing : ceph-prometheus-alerts-2:20.3.0-5439.ge371433f.el9 11/13 2026-02-23T11:50:37.259 INFO:teuthology.orchestra.run.trial186.stdout: Installing : ceph-grafana-dashboards-2:20.3.0-5439.ge371433f.el 12/13 2026-02-23T11:50:37.266 INFO:teuthology.orchestra.run.trial186.stdout: Installing : ceph-mgr-dashboard-2:20.3.0-5439.ge371433f.el9.noa 13/13 2026-02-23T11:50:37.525 INFO:teuthology.orchestra.run.trial186.stdout: Running scriptlet: ceph-mgr-dashboard-2:20.3.0-5439.ge371433f.el9.noa 13/13 2026-02-23T11:50:37.525 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : ceph-grafana-dashboards-2:20.3.0-5439.ge371433f.el 1/13 2026-02-23T11:50:37.525 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : ceph-mgr-dashboard-2:20.3.0-5439.ge371433f.el9.noa 2/13 2026-02-23T11:50:37.525 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : ceph-prometheus-alerts-2:20.3.0-5439.ge371433f.el9 3/13 2026-02-23T11:50:37.525 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : python3-jmespath-1.0.1-1.el9.noarch 4/13 2026-02-23T11:50:37.526 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : python3-protobuf-3.14.0-17.el9.noarch 5/13 2026-02-23T11:50:37.526 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : protobuf-compiler-3.14.0-17.el9.x86_64 6/13 2026-02-23T11:50:37.526 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : abseil-cpp-20211102.0-4.el9.x86_64 7/13 2026-02-23T11:50:37.526 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/13 2026-02-23T11:50:37.526 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/13 2026-02-23T11:50:37.526 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/13 2026-02-23T11:50:37.526 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/13 2026-02-23T11:50:37.526 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/13 2026-02-23T11:50:37.580 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : python3-xmltodict-0.12.0-15.el9.noarch 13/13 2026-02-23T11:50:37.580 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:50:37.580 INFO:teuthology.orchestra.run.trial186.stdout:Installed: 2026-02-23T11:50:37.580 INFO:teuthology.orchestra.run.trial186.stdout: abseil-cpp-20211102.0-4.el9.x86_64 2026-02-23T11:50:37.580 INFO:teuthology.orchestra.run.trial186.stdout: ceph-grafana-dashboards-2:20.3.0-5439.ge371433f.el9.noarch 2026-02-23T11:50:37.580 INFO:teuthology.orchestra.run.trial186.stdout: ceph-mgr-dashboard-2:20.3.0-5439.ge371433f.el9.noarch 2026-02-23T11:50:37.580 INFO:teuthology.orchestra.run.trial186.stdout: ceph-prometheus-alerts-2:20.3.0-5439.ge371433f.el9.noarch 2026-02-23T11:50:37.580 INFO:teuthology.orchestra.run.trial186.stdout: grpc-data-1.46.7-10.el9.noarch 2026-02-23T11:50:37.580 INFO:teuthology.orchestra.run.trial186.stdout: protobuf-compiler-3.14.0-17.el9.x86_64 2026-02-23T11:50:37.580 INFO:teuthology.orchestra.run.trial186.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2026-02-23T11:50:37.581 INFO:teuthology.orchestra.run.trial186.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2026-02-23T11:50:37.581 INFO:teuthology.orchestra.run.trial186.stdout: python3-jmespath-1.0.1-1.el9.noarch 2026-02-23T11:50:37.581 INFO:teuthology.orchestra.run.trial186.stdout: python3-protobuf-3.14.0-17.el9.noarch 2026-02-23T11:50:37.581 INFO:teuthology.orchestra.run.trial186.stdout: python3-repoze-lru-0.7-16.el9.noarch 2026-02-23T11:50:37.581 INFO:teuthology.orchestra.run.trial186.stdout: python3-routes-2.5.1-5.el9.noarch 2026-02-23T11:50:37.581 INFO:teuthology.orchestra.run.trial186.stdout: python3-xmltodict-0.12.0-15.el9.noarch 2026-02-23T11:50:37.581 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:50:37.581 INFO:teuthology.orchestra.run.trial186.stdout:Complete! 2026-02-23T11:50:37.617 DEBUG:teuthology.orchestra.run.trial186:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-23T11:50:37.922 INFO:teuthology.orchestra.run.trial186.stdout:Last metadata expiration check: 0:03:34 ago on Mon 23 Feb 2026 11:47:03 AM UTC. 2026-02-23T11:50:38.007 INFO:teuthology.orchestra.run.trial186.stdout:Dependencies resolved. 2026-02-23T11:50:38.008 INFO:teuthology.orchestra.run.trial186.stdout:======================================================================================= 2026-02-23T11:50:38.008 INFO:teuthology.orchestra.run.trial186.stdout: Package Arch Version Repository Size 2026-02-23T11:50:38.008 INFO:teuthology.orchestra.run.trial186.stdout:======================================================================================= 2026-02-23T11:50:38.008 INFO:teuthology.orchestra.run.trial186.stdout:Installing: 2026-02-23T11:50:38.008 INFO:teuthology.orchestra.run.trial186.stdout: ceph-mgr-diskprediction-local noarch 2:20.3.0-5439.ge371433f.el9 ceph-noarch 7.4 M 2026-02-23T11:50:38.008 INFO:teuthology.orchestra.run.trial186.stdout:Installing dependencies: 2026-02-23T11:50:38.008 INFO:teuthology.orchestra.run.trial186.stdout: flexiblas x86_64 3.0.4-9.el9 appstream 30 k 2026-02-23T11:50:38.008 INFO:teuthology.orchestra.run.trial186.stdout: flexiblas-netlib x86_64 3.0.4-9.el9 appstream 3.0 M 2026-02-23T11:50:38.008 INFO:teuthology.orchestra.run.trial186.stdout: flexiblas-openblas-openmp x86_64 3.0.4-9.el9 appstream 15 k 2026-02-23T11:50:38.008 INFO:teuthology.orchestra.run.trial186.stdout: libgfortran x86_64 11.5.0-14.el9 baseos 794 k 2026-02-23T11:50:38.009 INFO:teuthology.orchestra.run.trial186.stdout: libquadmath x86_64 11.5.0-14.el9 baseos 184 k 2026-02-23T11:50:38.009 INFO:teuthology.orchestra.run.trial186.stdout: openblas x86_64 0.3.29-1.el9 appstream 42 k 2026-02-23T11:50:38.009 INFO:teuthology.orchestra.run.trial186.stdout: openblas-openmp x86_64 0.3.29-1.el9 appstream 5.3 M 2026-02-23T11:50:38.009 INFO:teuthology.orchestra.run.trial186.stdout: python3-devel x86_64 3.9.25-3.el9 appstream 244 k 2026-02-23T11:50:38.009 INFO:teuthology.orchestra.run.trial186.stdout: python3-numpy x86_64 1:1.23.5-2.el9 appstream 6.1 M 2026-02-23T11:50:38.009 INFO:teuthology.orchestra.run.trial186.stdout: python3-numpy-f2py x86_64 1:1.23.5-2.el9 appstream 442 k 2026-02-23T11:50:38.009 INFO:teuthology.orchestra.run.trial186.stdout: python3-scipy x86_64 1.9.3-2.el9 appstream 19 M 2026-02-23T11:50:38.009 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:50:38.009 INFO:teuthology.orchestra.run.trial186.stdout:Transaction Summary 2026-02-23T11:50:38.009 INFO:teuthology.orchestra.run.trial186.stdout:======================================================================================= 2026-02-23T11:50:38.009 INFO:teuthology.orchestra.run.trial186.stdout:Install 12 Packages 2026-02-23T11:50:38.010 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:50:38.010 INFO:teuthology.orchestra.run.trial186.stdout:Total download size: 43 M 2026-02-23T11:50:38.010 INFO:teuthology.orchestra.run.trial186.stdout:Installed size: 234 M 2026-02-23T11:50:38.010 INFO:teuthology.orchestra.run.trial186.stdout:Downloading Packages: 2026-02-23T11:50:38.587 INFO:teuthology.orchestra.run.trial186.stdout:(1/12): libquadmath-11.5.0-14.el9.x86_64.rpm 396 kB/s | 184 kB 00:00 2026-02-23T11:50:38.614 INFO:teuthology.orchestra.run.trial186.stdout:(2/12): flexiblas-3.0.4-9.el9.x86_64.rpm 1.1 MB/s | 30 kB 00:00 2026-02-23T11:50:38.674 INFO:teuthology.orchestra.run.trial186.stdout:(3/12): flexiblas-netlib-3.0.4-9.el9.x86_64.rpm 50 MB/s | 3.0 MB 00:00 2026-02-23T11:50:38.683 INFO:teuthology.orchestra.run.trial186.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-9.el9.x 1.6 MB/s | 15 kB 00:00 2026-02-23T11:50:38.692 INFO:teuthology.orchestra.run.trial186.stdout:(5/12): openblas-0.3.29-1.el9.x86_64.rpm 4.5 MB/s | 42 kB 00:00 2026-02-23T11:50:38.720 INFO:teuthology.orchestra.run.trial186.stdout:(6/12): openblas-openmp-0.3.29-1.el9.x86_64.rpm 190 MB/s | 5.3 MB 00:00 2026-02-23T11:50:38.732 INFO:teuthology.orchestra.run.trial186.stdout:(7/12): python3-devel-3.9.25-3.el9.x86_64.rpm 20 MB/s | 244 kB 00:00 2026-02-23T11:50:38.747 INFO:teuthology.orchestra.run.trial186.stdout:(8/12): libgfortran-11.5.0-14.el9.x86_64.rpm 1.2 MB/s | 794 kB 00:00 2026-02-23T11:50:38.766 INFO:teuthology.orchestra.run.trial186.stdout:(9/12): python3-numpy-1.23.5-2.el9.x86_64.rpm 179 MB/s | 6.1 MB 00:00 2026-02-23T11:50:38.802 INFO:teuthology.orchestra.run.trial186.stdout:(10/12): python3-numpy-f2py-1.23.5-2.el9.x86_64 7.8 MB/s | 442 kB 00:00 2026-02-23T11:50:38.861 INFO:teuthology.orchestra.run.trial186.stdout:(11/12): python3-scipy-1.9.3-2.el9.x86_64.rpm 203 MB/s | 19 MB 00:00 2026-02-23T11:50:39.770 INFO:teuthology.orchestra.run.trial186.stdout:(12/12): ceph-mgr-diskprediction-local-20.3.0-5 4.5 MB/s | 7.4 MB 00:01 2026-02-23T11:50:39.771 INFO:teuthology.orchestra.run.trial186.stdout:-------------------------------------------------------------------------------- 2026-02-23T11:50:39.771 INFO:teuthology.orchestra.run.trial186.stdout:Total 24 MB/s | 43 MB 00:01 2026-02-23T11:50:39.936 INFO:teuthology.orchestra.run.trial186.stdout:Running transaction check 2026-02-23T11:50:39.950 INFO:teuthology.orchestra.run.trial186.stdout:Transaction check succeeded. 2026-02-23T11:50:39.950 INFO:teuthology.orchestra.run.trial186.stdout:Running transaction test 2026-02-23T11:50:40.152 INFO:teuthology.orchestra.run.trial186.stdout:Transaction test succeeded. 2026-02-23T11:50:40.152 INFO:teuthology.orchestra.run.trial186.stdout:Running transaction 2026-02-23T11:50:40.369 INFO:teuthology.orchestra.run.trial186.stdout: Preparing : 1/1 2026-02-23T11:50:40.371 INFO:teuthology.orchestra.run.trial186.stdout: Installing : flexiblas-3.0.4-9.el9.x86_64 1/12 2026-02-23T11:50:40.378 INFO:teuthology.orchestra.run.trial186.stdout: Installing : libquadmath-11.5.0-14.el9.x86_64 2/12 2026-02-23T11:50:40.411 INFO:teuthology.orchestra.run.trial186.stdout: Installing : libgfortran-11.5.0-14.el9.x86_64 3/12 2026-02-23T11:50:40.415 INFO:teuthology.orchestra.run.trial186.stdout: Installing : python3-devel-3.9.25-3.el9.x86_64 4/12 2026-02-23T11:50:40.474 INFO:teuthology.orchestra.run.trial186.stdout: Installing : openblas-0.3.29-1.el9.x86_64 5/12 2026-02-23T11:50:40.477 INFO:teuthology.orchestra.run.trial186.stdout: Installing : openblas-openmp-0.3.29-1.el9.x86_64 6/12 2026-02-23T11:50:40.498 INFO:teuthology.orchestra.run.trial186.stdout: Installing : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 7/12 2026-02-23T11:50:40.816 INFO:teuthology.orchestra.run.trial186.stdout: Installing : flexiblas-netlib-3.0.4-9.el9.x86_64 8/12 2026-02-23T11:50:40.887 INFO:teuthology.orchestra.run.trial186.stdout: Installing : python3-numpy-1:1.23.5-2.el9.x86_64 9/12 2026-02-23T11:50:41.543 INFO:teuthology.orchestra.run.trial186.stdout: Installing : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 10/12 2026-02-23T11:50:41.999 INFO:teuthology.orchestra.run.trial186.stdout: Installing : python3-scipy-1.9.3-2.el9.x86_64 11/12 2026-02-23T11:50:41.999 INFO:teuthology.orchestra.run.trial186.stdout: Installing : ceph-mgr-diskprediction-local-2:20.3.0-5439.ge3714 12/12 2026-02-23T11:50:42.280 INFO:teuthology.orchestra.run.trial186.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:20.3.0-5439.ge3714 12/12 2026-02-23T11:50:42.281 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : ceph-mgr-diskprediction-local-2:20.3.0-5439.ge3714 1/12 2026-02-23T11:50:42.281 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : libgfortran-11.5.0-14.el9.x86_64 2/12 2026-02-23T11:50:42.281 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : libquadmath-11.5.0-14.el9.x86_64 3/12 2026-02-23T11:50:42.281 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : flexiblas-3.0.4-9.el9.x86_64 4/12 2026-02-23T11:50:42.281 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : flexiblas-netlib-3.0.4-9.el9.x86_64 5/12 2026-02-23T11:50:42.281 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 6/12 2026-02-23T11:50:42.281 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : openblas-0.3.29-1.el9.x86_64 7/12 2026-02-23T11:50:42.281 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : openblas-openmp-0.3.29-1.el9.x86_64 8/12 2026-02-23T11:50:42.281 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : python3-devel-3.9.25-3.el9.x86_64 9/12 2026-02-23T11:50:42.282 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : python3-numpy-1:1.23.5-2.el9.x86_64 10/12 2026-02-23T11:50:42.282 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 11/12 2026-02-23T11:50:42.341 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : python3-scipy-1.9.3-2.el9.x86_64 12/12 2026-02-23T11:50:42.341 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:50:42.341 INFO:teuthology.orchestra.run.trial186.stdout:Installed: 2026-02-23T11:50:42.342 INFO:teuthology.orchestra.run.trial186.stdout: ceph-mgr-diskprediction-local-2:20.3.0-5439.ge371433f.el9.noarch 2026-02-23T11:50:42.342 INFO:teuthology.orchestra.run.trial186.stdout: flexiblas-3.0.4-9.el9.x86_64 2026-02-23T11:50:42.342 INFO:teuthology.orchestra.run.trial186.stdout: flexiblas-netlib-3.0.4-9.el9.x86_64 2026-02-23T11:50:42.342 INFO:teuthology.orchestra.run.trial186.stdout: flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 2026-02-23T11:50:42.342 INFO:teuthology.orchestra.run.trial186.stdout: libgfortran-11.5.0-14.el9.x86_64 2026-02-23T11:50:42.342 INFO:teuthology.orchestra.run.trial186.stdout: libquadmath-11.5.0-14.el9.x86_64 2026-02-23T11:50:42.342 INFO:teuthology.orchestra.run.trial186.stdout: openblas-0.3.29-1.el9.x86_64 2026-02-23T11:50:42.342 INFO:teuthology.orchestra.run.trial186.stdout: openblas-openmp-0.3.29-1.el9.x86_64 2026-02-23T11:50:42.342 INFO:teuthology.orchestra.run.trial186.stdout: python3-devel-3.9.25-3.el9.x86_64 2026-02-23T11:50:42.342 INFO:teuthology.orchestra.run.trial186.stdout: python3-numpy-1:1.23.5-2.el9.x86_64 2026-02-23T11:50:42.342 INFO:teuthology.orchestra.run.trial186.stdout: python3-numpy-f2py-1:1.23.5-2.el9.x86_64 2026-02-23T11:50:42.342 INFO:teuthology.orchestra.run.trial186.stdout: python3-scipy-1.9.3-2.el9.x86_64 2026-02-23T11:50:42.343 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:50:42.343 INFO:teuthology.orchestra.run.trial186.stdout:Complete! 2026-02-23T11:50:42.382 DEBUG:teuthology.orchestra.run.trial186:> sudo yum -y install ceph-mgr-rook 2026-02-23T11:50:42.687 INFO:teuthology.orchestra.run.trial186.stdout:Last metadata expiration check: 0:03:39 ago on Mon 23 Feb 2026 11:47:03 AM UTC. 2026-02-23T11:50:42.773 INFO:teuthology.orchestra.run.trial186.stdout:Dependencies resolved. 2026-02-23T11:50:42.774 INFO:teuthology.orchestra.run.trial186.stdout:================================================================================ 2026-02-23T11:50:42.774 INFO:teuthology.orchestra.run.trial186.stdout: Package Arch Version Repository Size 2026-02-23T11:50:42.774 INFO:teuthology.orchestra.run.trial186.stdout:================================================================================ 2026-02-23T11:50:42.774 INFO:teuthology.orchestra.run.trial186.stdout:Installing: 2026-02-23T11:50:42.775 INFO:teuthology.orchestra.run.trial186.stdout: ceph-mgr-rook noarch 2:20.3.0-5439.ge371433f.el9 ceph-noarch 50 k 2026-02-23T11:50:42.775 INFO:teuthology.orchestra.run.trial186.stdout:Installing dependencies: 2026-02-23T11:50:42.775 INFO:teuthology.orchestra.run.trial186.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2026-02-23T11:50:42.775 INFO:teuthology.orchestra.run.trial186.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2026-02-23T11:50:42.775 INFO:teuthology.orchestra.run.trial186.stdout: python3-google-auth noarch 1:2.45.0-1.el9 epel 254 k 2026-02-23T11:50:42.775 INFO:teuthology.orchestra.run.trial186.stdout: python3-kubernetes noarch 1:26.1.0-3.el9 epel 1.0 M 2026-02-23T11:50:42.775 INFO:teuthology.orchestra.run.trial186.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2026-02-23T11:50:42.775 INFO:teuthology.orchestra.run.trial186.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2026-02-23T11:50:42.775 INFO:teuthology.orchestra.run.trial186.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2026-02-23T11:50:42.775 INFO:teuthology.orchestra.run.trial186.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2026-02-23T11:50:42.775 INFO:teuthology.orchestra.run.trial186.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2026-02-23T11:50:42.776 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:50:42.776 INFO:teuthology.orchestra.run.trial186.stdout:Transaction Summary 2026-02-23T11:50:42.776 INFO:teuthology.orchestra.run.trial186.stdout:================================================================================ 2026-02-23T11:50:42.776 INFO:teuthology.orchestra.run.trial186.stdout:Install 10 Packages 2026-02-23T11:50:42.776 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:50:42.776 INFO:teuthology.orchestra.run.trial186.stdout:Total download size: 2.0 M 2026-02-23T11:50:42.776 INFO:teuthology.orchestra.run.trial186.stdout:Installed size: 25 M 2026-02-23T11:50:42.776 INFO:teuthology.orchestra.run.trial186.stdout:Downloading Packages: 2026-02-23T11:50:42.999 INFO:teuthology.orchestra.run.trial186.stdout:(1/10): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 159 kB 00:00 2026-02-23T11:50:43.007 INFO:teuthology.orchestra.run.trial186.stdout:(2/10): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 279 kB 00:00 2026-02-23T11:50:43.010 INFO:teuthology.orchestra.run.trial186.stdout:(3/10): python3-requests-oauthlib-1.3.0-12.el9. 4.7 MB/s | 54 kB 00:00 2026-02-23T11:50:43.083 INFO:teuthology.orchestra.run.trial186.stdout:(4/10): python3-certifi-2023.05.07-4.el9.noarch 194 kB/s | 14 kB 00:00 2026-02-23T11:50:43.085 INFO:teuthology.orchestra.run.trial186.stdout:(5/10): python3-cachetools-4.2.4-1.el9.noarch.r 414 kB/s | 32 kB 00:00 2026-02-23T11:50:43.175 INFO:teuthology.orchestra.run.trial186.stdout:(6/10): python3-google-auth-2.45.0-1.el9.noarch 2.7 MB/s | 254 kB 00:00 2026-02-23T11:50:43.200 INFO:teuthology.orchestra.run.trial186.stdout:(7/10): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2026-02-23T11:50:43.204 INFO:teuthology.orchestra.run.trial186.stdout:(8/10): python3-kubernetes-26.1.0-3.el9.noarch. 8.6 MB/s | 1.0 MB 00:00 2026-02-23T11:50:43.225 INFO:teuthology.orchestra.run.trial186.stdout:(9/10): python3-websocket-client-1.2.3-2.el9.no 3.5 MB/s | 90 kB 00:00 2026-02-23T11:50:43.346 INFO:teuthology.orchestra.run.trial186.stdout:(10/10): ceph-mgr-rook-20.3.0-5439.ge371433f.el 124 kB/s | 50 kB 00:00 2026-02-23T11:50:43.347 INFO:teuthology.orchestra.run.trial186.stdout:-------------------------------------------------------------------------------- 2026-02-23T11:50:43.347 INFO:teuthology.orchestra.run.trial186.stdout:Total 3.5 MB/s | 2.0 MB 00:00 2026-02-23T11:50:43.373 INFO:teuthology.orchestra.run.trial186.stdout:Running transaction check 2026-02-23T11:50:43.380 INFO:teuthology.orchestra.run.trial186.stdout:Transaction check succeeded. 2026-02-23T11:50:43.381 INFO:teuthology.orchestra.run.trial186.stdout:Running transaction test 2026-02-23T11:50:43.420 INFO:teuthology.orchestra.run.trial186.stdout:Transaction test succeeded. 2026-02-23T11:50:43.420 INFO:teuthology.orchestra.run.trial186.stdout:Running transaction 2026-02-23T11:50:43.495 INFO:teuthology.orchestra.run.trial186.stdout: Preparing : 1/1 2026-02-23T11:50:43.545 INFO:teuthology.orchestra.run.trial186.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/10 2026-02-23T11:50:43.557 INFO:teuthology.orchestra.run.trial186.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/10 2026-02-23T11:50:43.572 INFO:teuthology.orchestra.run.trial186.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/10 2026-02-23T11:50:43.576 INFO:teuthology.orchestra.run.trial186.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/10 2026-02-23T11:50:43.584 INFO:teuthology.orchestra.run.trial186.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/10 2026-02-23T11:50:43.623 INFO:teuthology.orchestra.run.trial186.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/10 2026-02-23T11:50:43.634 INFO:teuthology.orchestra.run.trial186.stdout: Installing : python3-google-auth-1:2.45.0-1.el9.noarch 7/10 2026-02-23T11:50:43.948 INFO:teuthology.orchestra.run.trial186.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 8/10 2026-02-23T11:50:43.963 INFO:teuthology.orchestra.run.trial186.stdout: Installing : python3-kubernetes-1:26.1.0-3.el9.noarch 9/10 2026-02-23T11:50:43.964 INFO:teuthology.orchestra.run.trial186.stdout: Installing : ceph-mgr-rook-2:20.3.0-5439.ge371433f.el9.noarch 10/10 2026-02-23T11:50:44.073 INFO:teuthology.orchestra.run.trial186.stdout: Running scriptlet: ceph-mgr-rook-2:20.3.0-5439.ge371433f.el9.noarch 10/10 2026-02-23T11:50:44.073 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : ceph-mgr-rook-2:20.3.0-5439.ge371433f.el9.noarch 1/10 2026-02-23T11:50:44.073 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 2/10 2026-02-23T11:50:44.073 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 3/10 2026-02-23T11:50:44.073 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 4/10 2026-02-23T11:50:44.073 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 5/10 2026-02-23T11:50:44.073 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 6/10 2026-02-23T11:50:44.073 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : python3-google-auth-1:2.45.0-1.el9.noarch 7/10 2026-02-23T11:50:44.073 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : python3-kubernetes-1:26.1.0-3.el9.noarch 8/10 2026-02-23T11:50:44.073 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 9/10 2026-02-23T11:50:44.131 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 10/10 2026-02-23T11:50:44.132 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:50:44.132 INFO:teuthology.orchestra.run.trial186.stdout:Installed: 2026-02-23T11:50:44.132 INFO:teuthology.orchestra.run.trial186.stdout: ceph-mgr-rook-2:20.3.0-5439.ge371433f.el9.noarch 2026-02-23T11:50:44.132 INFO:teuthology.orchestra.run.trial186.stdout: python3-cachetools-4.2.4-1.el9.noarch 2026-02-23T11:50:44.132 INFO:teuthology.orchestra.run.trial186.stdout: python3-certifi-2023.05.07-4.el9.noarch 2026-02-23T11:50:44.132 INFO:teuthology.orchestra.run.trial186.stdout: python3-google-auth-1:2.45.0-1.el9.noarch 2026-02-23T11:50:44.132 INFO:teuthology.orchestra.run.trial186.stdout: python3-kubernetes-1:26.1.0-3.el9.noarch 2026-02-23T11:50:44.133 INFO:teuthology.orchestra.run.trial186.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2026-02-23T11:50:44.133 INFO:teuthology.orchestra.run.trial186.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2026-02-23T11:50:44.133 INFO:teuthology.orchestra.run.trial186.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2026-02-23T11:50:44.133 INFO:teuthology.orchestra.run.trial186.stdout: python3-rsa-4.9-2.el9.noarch 2026-02-23T11:50:44.133 INFO:teuthology.orchestra.run.trial186.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2026-02-23T11:50:44.133 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:50:44.133 INFO:teuthology.orchestra.run.trial186.stdout:Complete! 2026-02-23T11:50:44.163 DEBUG:teuthology.orchestra.run.trial186:> sudo yum -y install ceph-mgr-cephadm 2026-02-23T11:50:44.475 INFO:teuthology.orchestra.run.trial186.stdout:Last metadata expiration check: 0:03:41 ago on Mon 23 Feb 2026 11:47:03 AM UTC. 2026-02-23T11:50:44.558 INFO:teuthology.orchestra.run.trial186.stdout:Dependencies resolved. 2026-02-23T11:50:44.558 INFO:teuthology.orchestra.run.trial186.stdout:================================================================================ 2026-02-23T11:50:44.558 INFO:teuthology.orchestra.run.trial186.stdout: Package Arch Version Repository Size 2026-02-23T11:50:44.558 INFO:teuthology.orchestra.run.trial186.stdout:================================================================================ 2026-02-23T11:50:44.558 INFO:teuthology.orchestra.run.trial186.stdout:Installing: 2026-02-23T11:50:44.558 INFO:teuthology.orchestra.run.trial186.stdout: ceph-mgr-cephadm noarch 2:20.3.0-5439.ge371433f.el9 ceph-noarch 184 k 2026-02-23T11:50:44.559 INFO:teuthology.orchestra.run.trial186.stdout:Installing dependencies: 2026-02-23T11:50:44.559 INFO:teuthology.orchestra.run.trial186.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2026-02-23T11:50:44.559 INFO:teuthology.orchestra.run.trial186.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2026-02-23T11:50:44.559 INFO:teuthology.orchestra.run.trial186.stdout: python3-jinja2 noarch 2.11.3-8.el9 appstream 249 k 2026-02-23T11:50:44.559 INFO:teuthology.orchestra.run.trial186.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2026-02-23T11:50:44.559 INFO:teuthology.orchestra.run.trial186.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2026-02-23T11:50:44.559 INFO:teuthology.orchestra.run.trial186.stdout: python3-typing-extensions noarch 4.15.0-1.el9 epel 86 k 2026-02-23T11:50:44.559 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:50:44.559 INFO:teuthology.orchestra.run.trial186.stdout:Transaction Summary 2026-02-23T11:50:44.559 INFO:teuthology.orchestra.run.trial186.stdout:================================================================================ 2026-02-23T11:50:44.559 INFO:teuthology.orchestra.run.trial186.stdout:Install 7 Packages 2026-02-23T11:50:44.559 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:50:44.559 INFO:teuthology.orchestra.run.trial186.stdout:Total download size: 7.1 M 2026-02-23T11:50:44.559 INFO:teuthology.orchestra.run.trial186.stdout:Installed size: 34 M 2026-02-23T11:50:44.559 INFO:teuthology.orchestra.run.trial186.stdout:Downloading Packages: 2026-02-23T11:50:45.020 INFO:teuthology.orchestra.run.trial186.stdout:(1/7): python3-jinja2-2.11.3-8.el9.noarch.rpm 4.6 MB/s | 249 kB 00:00 2026-02-23T11:50:45.030 INFO:teuthology.orchestra.run.trial186.stdout:(2/7): python3-markupsafe-1.1.1-12.el9.x86_64.r 3.4 MB/s | 35 kB 00:00 2026-02-23T11:50:45.063 INFO:teuthology.orchestra.run.trial186.stdout:(3/7): python3-babel-2.9.1-2.el9.noarch.rpm 62 MB/s | 6.0 MB 00:00 2026-02-23T11:50:45.184 INFO:teuthology.orchestra.run.trial186.stdout:(4/7): python3-natsort-7.1.1-5.el9.noarch.rpm 479 kB/s | 58 kB 00:00 2026-02-23T11:50:45.223 INFO:teuthology.orchestra.run.trial186.stdout:(5/7): python3-typing-extensions-4.15.0-1.el9.n 2.2 MB/s | 86 kB 00:00 2026-02-23T11:50:45.266 INFO:teuthology.orchestra.run.trial186.stdout:(6/7): python3-asyncssh-2.13.2-5.el9.noarch.rpm 2.3 MB/s | 548 kB 00:00 2026-02-23T11:50:45.521 INFO:teuthology.orchestra.run.trial186.stdout:(7/7): ceph-mgr-cephadm-20.3.0-5439.ge371433f.e 332 kB/s | 184 kB 00:00 2026-02-23T11:50:45.523 INFO:teuthology.orchestra.run.trial186.stdout:-------------------------------------------------------------------------------- 2026-02-23T11:50:45.524 INFO:teuthology.orchestra.run.trial186.stdout:Total 7.4 MB/s | 7.1 MB 00:00 2026-02-23T11:50:45.563 INFO:teuthology.orchestra.run.trial186.stdout:Running transaction check 2026-02-23T11:50:45.570 INFO:teuthology.orchestra.run.trial186.stdout:Transaction check succeeded. 2026-02-23T11:50:45.570 INFO:teuthology.orchestra.run.trial186.stdout:Running transaction test 2026-02-23T11:50:45.641 INFO:teuthology.orchestra.run.trial186.stdout:Transaction test succeeded. 2026-02-23T11:50:45.641 INFO:teuthology.orchestra.run.trial186.stdout:Running transaction 2026-02-23T11:50:45.730 INFO:teuthology.orchestra.run.trial186.stdout: Preparing : 1/1 2026-02-23T11:50:45.779 INFO:teuthology.orchestra.run.trial186.stdout: Installing : python3-typing-extensions-4.15.0-1.el9.noarch 1/7 2026-02-23T11:50:45.788 INFO:teuthology.orchestra.run.trial186.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/7 2026-02-23T11:50:45.794 INFO:teuthology.orchestra.run.trial186.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/7 2026-02-23T11:50:46.011 INFO:teuthology.orchestra.run.trial186.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 4/7 2026-02-23T11:50:46.034 INFO:teuthology.orchestra.run.trial186.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 5/7 2026-02-23T11:50:46.059 INFO:teuthology.orchestra.run.trial186.stdout: Installing : python3-jinja2-2.11.3-8.el9.noarch 6/7 2026-02-23T11:50:46.061 INFO:teuthology.orchestra.run.trial186.stdout: Installing : ceph-mgr-cephadm-2:20.3.0-5439.ge371433f.el9.noarch 7/7 2026-02-23T11:50:46.171 INFO:teuthology.orchestra.run.trial186.stdout: Running scriptlet: ceph-mgr-cephadm-2:20.3.0-5439.ge371433f.el9.noarch 7/7 2026-02-23T11:50:46.171 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : ceph-mgr-cephadm-2:20.3.0-5439.ge371433f.el9.noarch 1/7 2026-02-23T11:50:46.171 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/7 2026-02-23T11:50:46.171 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : python3-jinja2-2.11.3-8.el9.noarch 3/7 2026-02-23T11:50:46.171 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 4/7 2026-02-23T11:50:46.171 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 5/7 2026-02-23T11:50:46.171 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 6/7 2026-02-23T11:50:46.228 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : python3-typing-extensions-4.15.0-1.el9.noarch 7/7 2026-02-23T11:50:46.229 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:50:46.229 INFO:teuthology.orchestra.run.trial186.stdout:Installed: 2026-02-23T11:50:46.229 INFO:teuthology.orchestra.run.trial186.stdout: ceph-mgr-cephadm-2:20.3.0-5439.ge371433f.el9.noarch 2026-02-23T11:50:46.229 INFO:teuthology.orchestra.run.trial186.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2026-02-23T11:50:46.229 INFO:teuthology.orchestra.run.trial186.stdout: python3-babel-2.9.1-2.el9.noarch 2026-02-23T11:50:46.229 INFO:teuthology.orchestra.run.trial186.stdout: python3-jinja2-2.11.3-8.el9.noarch 2026-02-23T11:50:46.229 INFO:teuthology.orchestra.run.trial186.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2026-02-23T11:50:46.229 INFO:teuthology.orchestra.run.trial186.stdout: python3-natsort-7.1.1-5.el9.noarch 2026-02-23T11:50:46.229 INFO:teuthology.orchestra.run.trial186.stdout: python3-typing-extensions-4.15.0-1.el9.noarch 2026-02-23T11:50:46.229 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:50:46.230 INFO:teuthology.orchestra.run.trial186.stdout:Complete! 2026-02-23T11:50:46.260 DEBUG:teuthology.orchestra.run.trial186:> sudo yum -y install ceph-osd 2026-02-23T11:50:46.574 INFO:teuthology.orchestra.run.trial186.stdout:Last metadata expiration check: 0:03:43 ago on Mon 23 Feb 2026 11:47:03 AM UTC. 2026-02-23T11:50:46.639 INFO:teuthology.orchestra.run.trial186.stdout:Package ceph-osd-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:50:46.657 INFO:teuthology.orchestra.run.trial186.stdout:Dependencies resolved. 2026-02-23T11:50:46.658 INFO:teuthology.orchestra.run.trial186.stdout:Nothing to do. 2026-02-23T11:50:46.658 INFO:teuthology.orchestra.run.trial186.stdout:Complete! 2026-02-23T11:50:46.680 DEBUG:teuthology.orchestra.run.trial186:> sudo yum -y install ceph-osd-classic 2026-02-23T11:50:46.979 INFO:teuthology.orchestra.run.trial186.stdout:Last metadata expiration check: 0:03:43 ago on Mon 23 Feb 2026 11:47:03 AM UTC. 2026-02-23T11:50:47.043 INFO:teuthology.orchestra.run.trial186.stdout:Package ceph-osd-classic-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:50:47.061 INFO:teuthology.orchestra.run.trial186.stdout:Dependencies resolved. 2026-02-23T11:50:47.062 INFO:teuthology.orchestra.run.trial186.stdout:Nothing to do. 2026-02-23T11:50:47.062 INFO:teuthology.orchestra.run.trial186.stdout:Complete! 2026-02-23T11:50:47.083 DEBUG:teuthology.orchestra.run.trial186:> sudo yum -y install ceph-fuse 2026-02-23T11:50:47.388 INFO:teuthology.orchestra.run.trial186.stdout:Last metadata expiration check: 0:03:44 ago on Mon 23 Feb 2026 11:47:03 AM UTC. 2026-02-23T11:50:47.473 INFO:teuthology.orchestra.run.trial186.stdout:Dependencies resolved. 2026-02-23T11:50:47.473 INFO:teuthology.orchestra.run.trial186.stdout:================================================================================ 2026-02-23T11:50:47.474 INFO:teuthology.orchestra.run.trial186.stdout: Package Arch Version Repo Size 2026-02-23T11:50:47.474 INFO:teuthology.orchestra.run.trial186.stdout:================================================================================ 2026-02-23T11:50:47.474 INFO:teuthology.orchestra.run.trial186.stdout:Installing: 2026-02-23T11:50:47.474 INFO:teuthology.orchestra.run.trial186.stdout: ceph-fuse x86_64 2:20.3.0-5439.ge371433f.el9 ceph 914 k 2026-02-23T11:50:47.474 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:50:47.474 INFO:teuthology.orchestra.run.trial186.stdout:Transaction Summary 2026-02-23T11:50:47.474 INFO:teuthology.orchestra.run.trial186.stdout:================================================================================ 2026-02-23T11:50:47.474 INFO:teuthology.orchestra.run.trial186.stdout:Install 1 Package 2026-02-23T11:50:47.474 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:50:47.474 INFO:teuthology.orchestra.run.trial186.stdout:Total download size: 914 k 2026-02-23T11:50:47.475 INFO:teuthology.orchestra.run.trial186.stdout:Installed size: 2.9 M 2026-02-23T11:50:47.475 INFO:teuthology.orchestra.run.trial186.stdout:Downloading Packages: 2026-02-23T11:50:49.315 INFO:teuthology.orchestra.run.trial186.stdout:ceph-fuse-20.3.0-5439.ge371433f.el9.x86_64.rpm 497 kB/s | 914 kB 00:01 2026-02-23T11:50:49.315 INFO:teuthology.orchestra.run.trial186.stdout:-------------------------------------------------------------------------------- 2026-02-23T11:50:49.315 INFO:teuthology.orchestra.run.trial186.stdout:Total 496 kB/s | 914 kB 00:01 2026-02-23T11:50:49.315 INFO:teuthology.orchestra.run.trial186.stdout:Running transaction check 2026-02-23T11:50:49.319 INFO:teuthology.orchestra.run.trial186.stdout:Transaction check succeeded. 2026-02-23T11:50:49.320 INFO:teuthology.orchestra.run.trial186.stdout:Running transaction test 2026-02-23T11:50:49.353 INFO:teuthology.orchestra.run.trial186.stdout:Transaction test succeeded. 2026-02-23T11:50:49.353 INFO:teuthology.orchestra.run.trial186.stdout:Running transaction 2026-02-23T11:50:49.443 INFO:teuthology.orchestra.run.trial186.stdout: Preparing : 1/1 2026-02-23T11:50:49.460 INFO:teuthology.orchestra.run.trial186.stdout: Installing : ceph-fuse-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T11:50:49.673 INFO:teuthology.orchestra.run.trial186.stdout: Running scriptlet: ceph-fuse-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T11:50:49.730 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : ceph-fuse-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T11:50:49.730 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:50:49.730 INFO:teuthology.orchestra.run.trial186.stdout:Installed: 2026-02-23T11:50:49.730 INFO:teuthology.orchestra.run.trial186.stdout: ceph-fuse-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:50:49.730 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:50:49.731 INFO:teuthology.orchestra.run.trial186.stdout:Complete! 2026-02-23T11:50:49.758 DEBUG:teuthology.orchestra.run.trial186:> sudo yum -y install ceph-volume 2026-02-23T11:50:50.068 INFO:teuthology.orchestra.run.trial186.stdout:Last metadata expiration check: 0:03:47 ago on Mon 23 Feb 2026 11:47:03 AM UTC. 2026-02-23T11:50:50.152 INFO:teuthology.orchestra.run.trial186.stdout:Dependencies resolved. 2026-02-23T11:50:50.154 INFO:teuthology.orchestra.run.trial186.stdout:================================================================================ 2026-02-23T11:50:50.154 INFO:teuthology.orchestra.run.trial186.stdout: Package Arch Version Repository Size 2026-02-23T11:50:50.154 INFO:teuthology.orchestra.run.trial186.stdout:================================================================================ 2026-02-23T11:50:50.154 INFO:teuthology.orchestra.run.trial186.stdout:Installing: 2026-02-23T11:50:50.154 INFO:teuthology.orchestra.run.trial186.stdout: ceph-volume noarch 2:20.3.0-5439.ge371433f.el9 ceph-noarch 301 k 2026-02-23T11:50:50.154 INFO:teuthology.orchestra.run.trial186.stdout:Installing dependencies: 2026-02-23T11:50:50.154 INFO:teuthology.orchestra.run.trial186.stdout: cryptsetup x86_64 2.8.1-3.el9 baseos 351 k 2026-02-23T11:50:50.154 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:50:50.155 INFO:teuthology.orchestra.run.trial186.stdout:Transaction Summary 2026-02-23T11:50:50.155 INFO:teuthology.orchestra.run.trial186.stdout:================================================================================ 2026-02-23T11:50:50.155 INFO:teuthology.orchestra.run.trial186.stdout:Install 2 Packages 2026-02-23T11:50:50.155 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:50:50.155 INFO:teuthology.orchestra.run.trial186.stdout:Total download size: 652 k 2026-02-23T11:50:50.155 INFO:teuthology.orchestra.run.trial186.stdout:Installed size: 2.2 M 2026-02-23T11:50:50.155 INFO:teuthology.orchestra.run.trial186.stdout:Downloading Packages: 2026-02-23T11:50:50.350 INFO:teuthology.orchestra.run.trial186.stdout:(1/2): cryptsetup-2.8.1-3.el9.x86_64.rpm 3.2 MB/s | 351 kB 00:00 2026-02-23T11:50:50.886 INFO:teuthology.orchestra.run.trial186.stdout:(2/2): ceph-volume-20.3.0-5439.ge371433f.el9.no 469 kB/s | 301 kB 00:00 2026-02-23T11:50:50.886 INFO:teuthology.orchestra.run.trial186.stdout:-------------------------------------------------------------------------------- 2026-02-23T11:50:50.886 INFO:teuthology.orchestra.run.trial186.stdout:Total 890 kB/s | 652 kB 00:00 2026-02-23T11:50:50.897 INFO:teuthology.orchestra.run.trial186.stdout:Running transaction check 2026-02-23T11:50:50.903 INFO:teuthology.orchestra.run.trial186.stdout:Transaction check succeeded. 2026-02-23T11:50:50.903 INFO:teuthology.orchestra.run.trial186.stdout:Running transaction test 2026-02-23T11:50:50.948 INFO:teuthology.orchestra.run.trial186.stdout:Transaction test succeeded. 2026-02-23T11:50:50.948 INFO:teuthology.orchestra.run.trial186.stdout:Running transaction 2026-02-23T11:50:51.028 INFO:teuthology.orchestra.run.trial186.stdout: Preparing : 1/1 2026-02-23T11:50:51.090 INFO:teuthology.orchestra.run.trial186.stdout: Installing : cryptsetup-2.8.1-3.el9.x86_64 1/2 2026-02-23T11:50:51.092 INFO:teuthology.orchestra.run.trial186.stdout: Installing : ceph-volume-2:20.3.0-5439.ge371433f.el9.noarch 2/2 2026-02-23T11:50:51.107 INFO:teuthology.orchestra.run.trial186.stdout: Running scriptlet: ceph-volume-2:20.3.0-5439.ge371433f.el9.noarch 2/2 2026-02-23T11:50:51.107 INFO:teuthology.orchestra.run.trial186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-23T11:50:51.108 INFO:teuthology.orchestra.run.trial186.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2026-02-23T11:50:51.108 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:50:51.326 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : ceph-volume-2:20.3.0-5439.ge371433f.el9.noarch 1/2 2026-02-23T11:50:51.384 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : cryptsetup-2.8.1-3.el9.x86_64 2/2 2026-02-23T11:50:51.384 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:50:51.384 INFO:teuthology.orchestra.run.trial186.stdout:Installed: 2026-02-23T11:50:51.384 INFO:teuthology.orchestra.run.trial186.stdout: ceph-volume-2:20.3.0-5439.ge371433f.el9.noarch cryptsetup-2.8.1-3.el9.x86_64 2026-02-23T11:50:51.385 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:50:51.385 INFO:teuthology.orchestra.run.trial186.stdout:Complete! 2026-02-23T11:50:51.412 DEBUG:teuthology.orchestra.run.trial186:> sudo yum -y install librados-devel 2026-02-23T11:50:51.722 INFO:teuthology.orchestra.run.trial186.stdout:Last metadata expiration check: 0:03:48 ago on Mon 23 Feb 2026 11:47:03 AM UTC. 2026-02-23T11:50:51.806 INFO:teuthology.orchestra.run.trial186.stdout:Dependencies resolved. 2026-02-23T11:50:51.806 INFO:teuthology.orchestra.run.trial186.stdout:================================================================================ 2026-02-23T11:50:51.806 INFO:teuthology.orchestra.run.trial186.stdout: Package Arch Version Repo Size 2026-02-23T11:50:51.806 INFO:teuthology.orchestra.run.trial186.stdout:================================================================================ 2026-02-23T11:50:51.806 INFO:teuthology.orchestra.run.trial186.stdout:Installing: 2026-02-23T11:50:51.806 INFO:teuthology.orchestra.run.trial186.stdout: librados-devel x86_64 2:20.3.0-5439.ge371433f.el9 ceph 124 k 2026-02-23T11:50:51.806 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:50:51.807 INFO:teuthology.orchestra.run.trial186.stdout:Transaction Summary 2026-02-23T11:50:51.807 INFO:teuthology.orchestra.run.trial186.stdout:================================================================================ 2026-02-23T11:50:51.807 INFO:teuthology.orchestra.run.trial186.stdout:Install 1 Package 2026-02-23T11:50:51.807 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:50:51.807 INFO:teuthology.orchestra.run.trial186.stdout:Total download size: 124 k 2026-02-23T11:50:51.807 INFO:teuthology.orchestra.run.trial186.stdout:Installed size: 434 k 2026-02-23T11:50:51.807 INFO:teuthology.orchestra.run.trial186.stdout:Downloading Packages: 2026-02-23T11:50:52.303 INFO:teuthology.orchestra.run.trial186.stdout:librados-devel-20.3.0-5439.ge371433f.el9.x86_64 249 kB/s | 124 kB 00:00 2026-02-23T11:50:52.303 INFO:teuthology.orchestra.run.trial186.stdout:-------------------------------------------------------------------------------- 2026-02-23T11:50:52.303 INFO:teuthology.orchestra.run.trial186.stdout:Total 249 kB/s | 124 kB 00:00 2026-02-23T11:50:52.304 INFO:teuthology.orchestra.run.trial186.stdout:Running transaction check 2026-02-23T11:50:52.306 INFO:teuthology.orchestra.run.trial186.stdout:Transaction check succeeded. 2026-02-23T11:50:52.306 INFO:teuthology.orchestra.run.trial186.stdout:Running transaction test 2026-02-23T11:50:52.337 INFO:teuthology.orchestra.run.trial186.stdout:Transaction test succeeded. 2026-02-23T11:50:52.337 INFO:teuthology.orchestra.run.trial186.stdout:Running transaction 2026-02-23T11:50:52.390 INFO:teuthology.orchestra.run.trial186.stdout: Preparing : 1/1 2026-02-23T11:50:52.407 INFO:teuthology.orchestra.run.trial186.stdout: Installing : librados-devel-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T11:50:52.487 INFO:teuthology.orchestra.run.trial186.stdout: Running scriptlet: librados-devel-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T11:50:52.545 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : librados-devel-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T11:50:52.545 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:50:52.546 INFO:teuthology.orchestra.run.trial186.stdout:Installed: 2026-02-23T11:50:52.546 INFO:teuthology.orchestra.run.trial186.stdout: librados-devel-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:50:52.546 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:50:52.546 INFO:teuthology.orchestra.run.trial186.stdout:Complete! 2026-02-23T11:50:52.573 DEBUG:teuthology.orchestra.run.trial186:> sudo yum -y install libcephfs2 2026-02-23T11:50:52.888 INFO:teuthology.orchestra.run.trial186.stdout:Last metadata expiration check: 0:03:49 ago on Mon 23 Feb 2026 11:47:03 AM UTC. 2026-02-23T11:50:52.951 INFO:teuthology.orchestra.run.trial186.stdout:Package libcephfs2-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:50:52.969 INFO:teuthology.orchestra.run.trial186.stdout:Dependencies resolved. 2026-02-23T11:50:52.970 INFO:teuthology.orchestra.run.trial186.stdout:Nothing to do. 2026-02-23T11:50:52.970 INFO:teuthology.orchestra.run.trial186.stdout:Complete! 2026-02-23T11:50:52.991 DEBUG:teuthology.orchestra.run.trial186:> sudo yum -y install libcephfs-devel 2026-02-23T11:50:53.299 INFO:teuthology.orchestra.run.trial186.stdout:Last metadata expiration check: 0:03:50 ago on Mon 23 Feb 2026 11:47:03 AM UTC. 2026-02-23T11:50:53.385 INFO:teuthology.orchestra.run.trial186.stdout:Dependencies resolved. 2026-02-23T11:50:53.385 INFO:teuthology.orchestra.run.trial186.stdout:================================================================================ 2026-02-23T11:50:53.385 INFO:teuthology.orchestra.run.trial186.stdout: Package Arch Version Repo Size 2026-02-23T11:50:53.386 INFO:teuthology.orchestra.run.trial186.stdout:================================================================================ 2026-02-23T11:50:53.386 INFO:teuthology.orchestra.run.trial186.stdout:Installing: 2026-02-23T11:50:53.386 INFO:teuthology.orchestra.run.trial186.stdout: libcephfs-devel x86_64 2:20.3.0-5439.ge371433f.el9 ceph 37 k 2026-02-23T11:50:53.386 INFO:teuthology.orchestra.run.trial186.stdout:Installing dependencies: 2026-02-23T11:50:53.386 INFO:teuthology.orchestra.run.trial186.stdout: libcephfs-proxy2 x86_64 2:20.3.0-5439.ge371433f.el9 ceph 26 k 2026-02-23T11:50:53.386 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:50:53.386 INFO:teuthology.orchestra.run.trial186.stdout:Transaction Summary 2026-02-23T11:50:53.386 INFO:teuthology.orchestra.run.trial186.stdout:================================================================================ 2026-02-23T11:50:53.386 INFO:teuthology.orchestra.run.trial186.stdout:Install 2 Packages 2026-02-23T11:50:53.386 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:50:53.387 INFO:teuthology.orchestra.run.trial186.stdout:Total download size: 64 k 2026-02-23T11:50:53.387 INFO:teuthology.orchestra.run.trial186.stdout:Installed size: 231 k 2026-02-23T11:50:53.387 INFO:teuthology.orchestra.run.trial186.stdout:Downloading Packages: 2026-02-23T11:50:53.743 INFO:teuthology.orchestra.run.trial186.stdout:(1/2): libcephfs-devel-20.3.0-5439.ge371433f.el 104 kB/s | 37 kB 00:00 2026-02-23T11:50:53.748 INFO:teuthology.orchestra.run.trial186.stdout:(2/2): libcephfs-proxy2-20.3.0-5439.ge371433f.e 73 kB/s | 26 kB 00:00 2026-02-23T11:50:53.749 INFO:teuthology.orchestra.run.trial186.stdout:-------------------------------------------------------------------------------- 2026-02-23T11:50:53.749 INFO:teuthology.orchestra.run.trial186.stdout:Total 175 kB/s | 64 kB 00:00 2026-02-23T11:50:53.749 INFO:teuthology.orchestra.run.trial186.stdout:Running transaction check 2026-02-23T11:50:53.752 INFO:teuthology.orchestra.run.trial186.stdout:Transaction check succeeded. 2026-02-23T11:50:53.752 INFO:teuthology.orchestra.run.trial186.stdout:Running transaction test 2026-02-23T11:50:53.784 INFO:teuthology.orchestra.run.trial186.stdout:Transaction test succeeded. 2026-02-23T11:50:53.784 INFO:teuthology.orchestra.run.trial186.stdout:Running transaction 2026-02-23T11:50:53.833 INFO:teuthology.orchestra.run.trial186.stdout: Preparing : 1/1 2026-02-23T11:50:53.834 INFO:teuthology.orchestra.run.trial186.stdout: Installing : libcephfs-proxy2-2:20.3.0-5439.ge371433f.el9.x86_64 1/2 2026-02-23T11:50:53.850 INFO:teuthology.orchestra.run.trial186.stdout: Running scriptlet: libcephfs-proxy2-2:20.3.0-5439.ge371433f.el9.x86_64 1/2 2026-02-23T11:50:53.867 INFO:teuthology.orchestra.run.trial186.stdout: Installing : libcephfs-devel-2:20.3.0-5439.ge371433f.el9.x86_64 2/2 2026-02-23T11:50:53.921 INFO:teuthology.orchestra.run.trial186.stdout: Running scriptlet: libcephfs-devel-2:20.3.0-5439.ge371433f.el9.x86_64 2/2 2026-02-23T11:50:53.921 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : libcephfs-devel-2:20.3.0-5439.ge371433f.el9.x86_64 1/2 2026-02-23T11:50:53.978 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : libcephfs-proxy2-2:20.3.0-5439.ge371433f.el9.x86_64 2/2 2026-02-23T11:50:53.978 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:50:53.978 INFO:teuthology.orchestra.run.trial186.stdout:Installed: 2026-02-23T11:50:53.978 INFO:teuthology.orchestra.run.trial186.stdout: libcephfs-devel-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:50:53.978 INFO:teuthology.orchestra.run.trial186.stdout: libcephfs-proxy2-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:50:53.978 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:50:53.978 INFO:teuthology.orchestra.run.trial186.stdout:Complete! 2026-02-23T11:50:54.004 DEBUG:teuthology.orchestra.run.trial186:> sudo yum -y install librados2 2026-02-23T11:50:54.309 INFO:teuthology.orchestra.run.trial186.stdout:Last metadata expiration check: 0:03:51 ago on Mon 23 Feb 2026 11:47:03 AM UTC. 2026-02-23T11:50:54.374 INFO:teuthology.orchestra.run.trial186.stdout:Package librados2-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:50:54.392 INFO:teuthology.orchestra.run.trial186.stdout:Dependencies resolved. 2026-02-23T11:50:54.393 INFO:teuthology.orchestra.run.trial186.stdout:Nothing to do. 2026-02-23T11:50:54.393 INFO:teuthology.orchestra.run.trial186.stdout:Complete! 2026-02-23T11:50:54.414 DEBUG:teuthology.orchestra.run.trial186:> sudo yum -y install librbd1 2026-02-23T11:50:54.722 INFO:teuthology.orchestra.run.trial186.stdout:Last metadata expiration check: 0:03:51 ago on Mon 23 Feb 2026 11:47:03 AM UTC. 2026-02-23T11:50:54.789 INFO:teuthology.orchestra.run.trial186.stdout:Package librbd1-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:50:54.807 INFO:teuthology.orchestra.run.trial186.stdout:Dependencies resolved. 2026-02-23T11:50:54.808 INFO:teuthology.orchestra.run.trial186.stdout:Nothing to do. 2026-02-23T11:50:54.808 INFO:teuthology.orchestra.run.trial186.stdout:Complete! 2026-02-23T11:50:54.829 DEBUG:teuthology.orchestra.run.trial186:> sudo yum -y install python3-rados 2026-02-23T11:50:55.141 INFO:teuthology.orchestra.run.trial186.stdout:Last metadata expiration check: 0:03:52 ago on Mon 23 Feb 2026 11:47:03 AM UTC. 2026-02-23T11:50:55.205 INFO:teuthology.orchestra.run.trial186.stdout:Package python3-rados-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:50:55.223 INFO:teuthology.orchestra.run.trial186.stdout:Dependencies resolved. 2026-02-23T11:50:55.224 INFO:teuthology.orchestra.run.trial186.stdout:Nothing to do. 2026-02-23T11:50:55.224 INFO:teuthology.orchestra.run.trial186.stdout:Complete! 2026-02-23T11:50:55.245 DEBUG:teuthology.orchestra.run.trial186:> sudo yum -y install python3-rgw 2026-02-23T11:50:55.553 INFO:teuthology.orchestra.run.trial186.stdout:Last metadata expiration check: 0:03:52 ago on Mon 23 Feb 2026 11:47:03 AM UTC. 2026-02-23T11:50:55.618 INFO:teuthology.orchestra.run.trial186.stdout:Package python3-rgw-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:50:55.637 INFO:teuthology.orchestra.run.trial186.stdout:Dependencies resolved. 2026-02-23T11:50:55.637 INFO:teuthology.orchestra.run.trial186.stdout:Nothing to do. 2026-02-23T11:50:55.638 INFO:teuthology.orchestra.run.trial186.stdout:Complete! 2026-02-23T11:50:55.658 DEBUG:teuthology.orchestra.run.trial186:> sudo yum -y install python3-cephfs 2026-02-23T11:50:55.963 INFO:teuthology.orchestra.run.trial186.stdout:Last metadata expiration check: 0:03:52 ago on Mon 23 Feb 2026 11:47:03 AM UTC. 2026-02-23T11:50:56.029 INFO:teuthology.orchestra.run.trial186.stdout:Package python3-cephfs-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:50:56.047 INFO:teuthology.orchestra.run.trial186.stdout:Dependencies resolved. 2026-02-23T11:50:56.048 INFO:teuthology.orchestra.run.trial186.stdout:Nothing to do. 2026-02-23T11:50:56.048 INFO:teuthology.orchestra.run.trial186.stdout:Complete! 2026-02-23T11:50:56.069 DEBUG:teuthology.orchestra.run.trial186:> sudo yum -y install python3-rbd 2026-02-23T11:50:56.382 INFO:teuthology.orchestra.run.trial186.stdout:Last metadata expiration check: 0:03:53 ago on Mon 23 Feb 2026 11:47:03 AM UTC. 2026-02-23T11:50:56.447 INFO:teuthology.orchestra.run.trial186.stdout:Package python3-rbd-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:50:56.465 INFO:teuthology.orchestra.run.trial186.stdout:Dependencies resolved. 2026-02-23T11:50:56.466 INFO:teuthology.orchestra.run.trial186.stdout:Nothing to do. 2026-02-23T11:50:56.466 INFO:teuthology.orchestra.run.trial186.stdout:Complete! 2026-02-23T11:50:56.487 DEBUG:teuthology.orchestra.run.trial186:> sudo yum -y install rbd-fuse 2026-02-23T11:50:56.793 INFO:teuthology.orchestra.run.trial186.stdout:Last metadata expiration check: 0:03:53 ago on Mon 23 Feb 2026 11:47:03 AM UTC. 2026-02-23T11:50:56.878 INFO:teuthology.orchestra.run.trial186.stdout:Dependencies resolved. 2026-02-23T11:50:56.878 INFO:teuthology.orchestra.run.trial186.stdout:================================================================================ 2026-02-23T11:50:56.878 INFO:teuthology.orchestra.run.trial186.stdout: Package Architecture Version Repository Size 2026-02-23T11:50:56.878 INFO:teuthology.orchestra.run.trial186.stdout:================================================================================ 2026-02-23T11:50:56.878 INFO:teuthology.orchestra.run.trial186.stdout:Installing: 2026-02-23T11:50:56.879 INFO:teuthology.orchestra.run.trial186.stdout: rbd-fuse x86_64 2:20.3.0-5439.ge371433f.el9 ceph 86 k 2026-02-23T11:50:56.879 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:50:56.879 INFO:teuthology.orchestra.run.trial186.stdout:Transaction Summary 2026-02-23T11:50:56.879 INFO:teuthology.orchestra.run.trial186.stdout:================================================================================ 2026-02-23T11:50:56.879 INFO:teuthology.orchestra.run.trial186.stdout:Install 1 Package 2026-02-23T11:50:56.879 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:50:56.879 INFO:teuthology.orchestra.run.trial186.stdout:Total download size: 86 k 2026-02-23T11:50:56.879 INFO:teuthology.orchestra.run.trial186.stdout:Installed size: 236 k 2026-02-23T11:50:56.879 INFO:teuthology.orchestra.run.trial186.stdout:Downloading Packages: 2026-02-23T11:50:57.284 INFO:teuthology.orchestra.run.trial186.stdout:rbd-fuse-20.3.0-5439.ge371433f.el9.x86_64.rpm 213 kB/s | 86 kB 00:00 2026-02-23T11:50:57.285 INFO:teuthology.orchestra.run.trial186.stdout:-------------------------------------------------------------------------------- 2026-02-23T11:50:57.285 INFO:teuthology.orchestra.run.trial186.stdout:Total 213 kB/s | 86 kB 00:00 2026-02-23T11:50:57.285 INFO:teuthology.orchestra.run.trial186.stdout:Running transaction check 2026-02-23T11:50:57.289 INFO:teuthology.orchestra.run.trial186.stdout:Transaction check succeeded. 2026-02-23T11:50:57.289 INFO:teuthology.orchestra.run.trial186.stdout:Running transaction test 2026-02-23T11:50:57.320 INFO:teuthology.orchestra.run.trial186.stdout:Transaction test succeeded. 2026-02-23T11:50:57.321 INFO:teuthology.orchestra.run.trial186.stdout:Running transaction 2026-02-23T11:50:57.370 INFO:teuthology.orchestra.run.trial186.stdout: Preparing : 1/1 2026-02-23T11:50:57.387 INFO:teuthology.orchestra.run.trial186.stdout: Installing : rbd-fuse-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T11:50:57.467 INFO:teuthology.orchestra.run.trial186.stdout: Running scriptlet: rbd-fuse-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T11:50:57.525 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : rbd-fuse-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T11:50:57.525 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:50:57.525 INFO:teuthology.orchestra.run.trial186.stdout:Installed: 2026-02-23T11:50:57.525 INFO:teuthology.orchestra.run.trial186.stdout: rbd-fuse-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:50:57.525 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:50:57.526 INFO:teuthology.orchestra.run.trial186.stdout:Complete! 2026-02-23T11:50:57.554 DEBUG:teuthology.orchestra.run.trial186:> sudo yum -y install rbd-mirror 2026-02-23T11:50:57.866 INFO:teuthology.orchestra.run.trial186.stdout:Last metadata expiration check: 0:03:54 ago on Mon 23 Feb 2026 11:47:03 AM UTC. 2026-02-23T11:50:57.949 INFO:teuthology.orchestra.run.trial186.stdout:Dependencies resolved. 2026-02-23T11:50:57.950 INFO:teuthology.orchestra.run.trial186.stdout:================================================================================ 2026-02-23T11:50:57.950 INFO:teuthology.orchestra.run.trial186.stdout: Package Arch Version Repo Size 2026-02-23T11:50:57.950 INFO:teuthology.orchestra.run.trial186.stdout:================================================================================ 2026-02-23T11:50:57.950 INFO:teuthology.orchestra.run.trial186.stdout:Installing: 2026-02-23T11:50:57.950 INFO:teuthology.orchestra.run.trial186.stdout: rbd-mirror x86_64 2:20.3.0-5439.ge371433f.el9 ceph 2.6 M 2026-02-23T11:50:57.950 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:50:57.950 INFO:teuthology.orchestra.run.trial186.stdout:Transaction Summary 2026-02-23T11:50:57.950 INFO:teuthology.orchestra.run.trial186.stdout:================================================================================ 2026-02-23T11:50:57.951 INFO:teuthology.orchestra.run.trial186.stdout:Install 1 Package 2026-02-23T11:50:57.951 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:50:57.951 INFO:teuthology.orchestra.run.trial186.stdout:Total download size: 2.6 M 2026-02-23T11:50:57.951 INFO:teuthology.orchestra.run.trial186.stdout:Installed size: 11 M 2026-02-23T11:50:57.951 INFO:teuthology.orchestra.run.trial186.stdout:Downloading Packages: 2026-02-23T11:51:00.467 INFO:teuthology.orchestra.run.trial186.stdout:rbd-mirror-20.3.0-5439.ge371433f.el9.x86_64.rpm 1.0 MB/s | 2.6 MB 00:02 2026-02-23T11:51:00.467 INFO:teuthology.orchestra.run.trial186.stdout:-------------------------------------------------------------------------------- 2026-02-23T11:51:00.467 INFO:teuthology.orchestra.run.trial186.stdout:Total 1.0 MB/s | 2.6 MB 00:02 2026-02-23T11:51:00.467 INFO:teuthology.orchestra.run.trial186.stdout:Running transaction check 2026-02-23T11:51:00.472 INFO:teuthology.orchestra.run.trial186.stdout:Transaction check succeeded. 2026-02-23T11:51:00.472 INFO:teuthology.orchestra.run.trial186.stdout:Running transaction test 2026-02-23T11:51:00.510 INFO:teuthology.orchestra.run.trial186.stdout:Transaction test succeeded. 2026-02-23T11:51:00.510 INFO:teuthology.orchestra.run.trial186.stdout:Running transaction 2026-02-23T11:51:00.671 INFO:teuthology.orchestra.run.trial186.stdout: Preparing : 1/1 2026-02-23T11:51:00.673 INFO:teuthology.orchestra.run.trial186.stdout: Installing : rbd-mirror-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T11:51:00.695 INFO:teuthology.orchestra.run.trial186.stdout: Running scriptlet: rbd-mirror-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T11:51:00.696 INFO:teuthology.orchestra.run.trial186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-23T11:51:00.696 INFO:teuthology.orchestra.run.trial186.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2026-02-23T11:51:00.696 INFO:teuthology.orchestra.run.trial186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-23T11:51:00.696 INFO:teuthology.orchestra.run.trial186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-23T11:51:00.696 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:51:00.987 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : rbd-mirror-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T11:51:00.987 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:51:00.987 INFO:teuthology.orchestra.run.trial186.stdout:Installed: 2026-02-23T11:51:00.987 INFO:teuthology.orchestra.run.trial186.stdout: rbd-mirror-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:51:00.988 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:51:00.988 INFO:teuthology.orchestra.run.trial186.stdout:Complete! 2026-02-23T11:51:01.015 DEBUG:teuthology.orchestra.run.trial186:> sudo yum -y install rbd-nbd 2026-02-23T11:51:01.324 INFO:teuthology.orchestra.run.trial186.stdout:Last metadata expiration check: 0:03:58 ago on Mon 23 Feb 2026 11:47:03 AM UTC. 2026-02-23T11:51:01.408 INFO:teuthology.orchestra.run.trial186.stdout:Dependencies resolved. 2026-02-23T11:51:01.408 INFO:teuthology.orchestra.run.trial186.stdout:================================================================================ 2026-02-23T11:51:01.408 INFO:teuthology.orchestra.run.trial186.stdout: Package Architecture Version Repository Size 2026-02-23T11:51:01.408 INFO:teuthology.orchestra.run.trial186.stdout:================================================================================ 2026-02-23T11:51:01.408 INFO:teuthology.orchestra.run.trial186.stdout:Installing: 2026-02-23T11:51:01.409 INFO:teuthology.orchestra.run.trial186.stdout: rbd-nbd x86_64 2:20.3.0-5439.ge371433f.el9 ceph 166 k 2026-02-23T11:51:01.409 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:51:01.409 INFO:teuthology.orchestra.run.trial186.stdout:Transaction Summary 2026-02-23T11:51:01.409 INFO:teuthology.orchestra.run.trial186.stdout:================================================================================ 2026-02-23T11:51:01.409 INFO:teuthology.orchestra.run.trial186.stdout:Install 1 Package 2026-02-23T11:51:01.409 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:51:01.409 INFO:teuthology.orchestra.run.trial186.stdout:Total download size: 166 k 2026-02-23T11:51:01.409 INFO:teuthology.orchestra.run.trial186.stdout:Installed size: 496 k 2026-02-23T11:51:01.409 INFO:teuthology.orchestra.run.trial186.stdout:Downloading Packages: 2026-02-23T11:51:01.866 INFO:teuthology.orchestra.run.trial186.stdout:rbd-nbd-20.3.0-5439.ge371433f.el9.x86_64.rpm 365 kB/s | 166 kB 00:00 2026-02-23T11:51:01.866 INFO:teuthology.orchestra.run.trial186.stdout:-------------------------------------------------------------------------------- 2026-02-23T11:51:01.867 INFO:teuthology.orchestra.run.trial186.stdout:Total 364 kB/s | 166 kB 00:00 2026-02-23T11:51:01.867 INFO:teuthology.orchestra.run.trial186.stdout:Running transaction check 2026-02-23T11:51:01.871 INFO:teuthology.orchestra.run.trial186.stdout:Transaction check succeeded. 2026-02-23T11:51:01.872 INFO:teuthology.orchestra.run.trial186.stdout:Running transaction test 2026-02-23T11:51:01.903 INFO:teuthology.orchestra.run.trial186.stdout:Transaction test succeeded. 2026-02-23T11:51:01.903 INFO:teuthology.orchestra.run.trial186.stdout:Running transaction 2026-02-23T11:51:01.958 INFO:teuthology.orchestra.run.trial186.stdout: Preparing : 1/1 2026-02-23T11:51:01.975 INFO:teuthology.orchestra.run.trial186.stdout: Installing : rbd-nbd-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T11:51:02.058 INFO:teuthology.orchestra.run.trial186.stdout: Running scriptlet: rbd-nbd-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T11:51:02.116 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : rbd-nbd-2:20.3.0-5439.ge371433f.el9.x86_64 1/1 2026-02-23T11:51:02.116 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:51:02.116 INFO:teuthology.orchestra.run.trial186.stdout:Installed: 2026-02-23T11:51:02.116 INFO:teuthology.orchestra.run.trial186.stdout: rbd-nbd-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:51:02.116 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:51:02.116 INFO:teuthology.orchestra.run.trial186.stdout:Complete! 2026-02-23T11:51:02.145 DEBUG:teuthology.orchestra.run.trial186:> sudo yum -y install ceph-osd-crimson 2026-02-23T11:51:02.452 INFO:teuthology.orchestra.run.trial186.stdout:Last metadata expiration check: 0:03:59 ago on Mon 23 Feb 2026 11:47:03 AM UTC. 2026-02-23T11:51:02.516 INFO:teuthology.orchestra.run.trial186.stdout:Package ceph-osd-crimson-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:51:02.535 INFO:teuthology.orchestra.run.trial186.stdout:Dependencies resolved. 2026-02-23T11:51:02.535 INFO:teuthology.orchestra.run.trial186.stdout:Nothing to do. 2026-02-23T11:51:02.536 INFO:teuthology.orchestra.run.trial186.stdout:Complete! 2026-02-23T11:51:02.557 DEBUG:teuthology.orchestra.run.trial186:> sudo yum -y install ceph-osd-crimson-debuginfo 2026-02-23T11:51:02.870 INFO:teuthology.orchestra.run.trial186.stdout:Last metadata expiration check: 0:03:59 ago on Mon 23 Feb 2026 11:47:03 AM UTC. 2026-02-23T11:51:02.954 INFO:teuthology.orchestra.run.trial186.stdout:Dependencies resolved. 2026-02-23T11:51:02.955 INFO:teuthology.orchestra.run.trial186.stdout:================================================================================ 2026-02-23T11:51:02.955 INFO:teuthology.orchestra.run.trial186.stdout: Package Arch Version Repo Size 2026-02-23T11:51:02.955 INFO:teuthology.orchestra.run.trial186.stdout:================================================================================ 2026-02-23T11:51:02.955 INFO:teuthology.orchestra.run.trial186.stdout:Installing: 2026-02-23T11:51:02.955 INFO:teuthology.orchestra.run.trial186.stdout: ceph-osd-crimson-debuginfo x86_64 2:20.3.0-5439.ge371433f.el9 ceph 429 M 2026-02-23T11:51:02.955 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:51:02.955 INFO:teuthology.orchestra.run.trial186.stdout:Transaction Summary 2026-02-23T11:51:02.955 INFO:teuthology.orchestra.run.trial186.stdout:================================================================================ 2026-02-23T11:51:02.955 INFO:teuthology.orchestra.run.trial186.stdout:Install 1 Package 2026-02-23T11:51:02.956 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:51:02.956 INFO:teuthology.orchestra.run.trial186.stdout:Total download size: 429 M 2026-02-23T11:51:02.956 INFO:teuthology.orchestra.run.trial186.stdout:Installed size: 2.6 G 2026-02-23T11:51:02.956 INFO:teuthology.orchestra.run.trial186.stdout:Downloading Packages: 2026-02-23T11:53:38.744 INFO:teuthology.orchestra.run.trial186.stdout:ceph-osd-crimson-debuginfo-20.3.0-5439.ge371433 2.8 MB/s | 429 MB 02:35 2026-02-23T11:53:38.744 INFO:teuthology.orchestra.run.trial186.stdout:-------------------------------------------------------------------------------- 2026-02-23T11:53:38.744 INFO:teuthology.orchestra.run.trial186.stdout:Total 2.8 MB/s | 429 MB 02:35 2026-02-23T11:53:38.745 INFO:teuthology.orchestra.run.trial186.stdout:Running transaction check 2026-02-23T11:53:38.745 INFO:teuthology.orchestra.run.trial186.stdout:Transaction check succeeded. 2026-02-23T11:53:38.745 INFO:teuthology.orchestra.run.trial186.stdout:Running transaction test 2026-02-23T11:53:40.027 INFO:teuthology.orchestra.run.trial186.stdout:Transaction test succeeded. 2026-02-23T11:53:40.027 INFO:teuthology.orchestra.run.trial186.stdout:Running transaction 2026-02-23T11:54:03.337 INFO:teuthology.orchestra.run.trial186.stdout: Preparing : 1/1 2026-02-23T11:54:03.354 INFO:teuthology.orchestra.run.trial186.stdout: Installing : ceph-osd-crimson-debuginfo-2:20.3.0-5439.ge371433f.e 1/1 2026-02-23T11:54:05.279 INFO:teuthology.orchestra.run.trial186.stdout: Running scriptlet: ceph-osd-crimson-debuginfo-2:20.3.0-5439.ge371433f.e 1/1 2026-02-23T11:54:05.336 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : ceph-osd-crimson-debuginfo-2:20.3.0-5439.ge371433f.e 1/1 2026-02-23T11:54:05.336 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:54:05.336 INFO:teuthology.orchestra.run.trial186.stdout:Installed: 2026-02-23T11:54:05.336 INFO:teuthology.orchestra.run.trial186.stdout: ceph-osd-crimson-debuginfo-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:54:05.337 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:54:05.337 INFO:teuthology.orchestra.run.trial186.stdout:Complete! 2026-02-23T11:54:05.407 DEBUG:teuthology.parallel:result is None 2026-02-23T11:54:05.408 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d 2026-02-23T11:54:05.536 DEBUG:teuthology.orchestra.run.trial186:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-23T11:54:05.556 INFO:teuthology.orchestra.run.trial186.stdout:20.3.0-5439.ge371433f.el9 2026-02-23T11:54:05.557 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5439.ge371433f.el9 2026-02-23T11:54:05.557 INFO:teuthology.task.install:The correct ceph version 20.3.0-5439.ge371433f is installed. 2026-02-23T11:54:05.559 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2026-02-23T11:54:05.560 DEBUG:teuthology.orchestra.run.trial186:> set -ex 2026-02-23T11:54:05.560 DEBUG:teuthology.orchestra.run.trial186:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-23T11:54:05.622 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2026-02-23T11:54:05.622 DEBUG:teuthology.orchestra.run.trial186:> set -ex 2026-02-23T11:54:05.622 DEBUG:teuthology.orchestra.run.trial186:> sudo dd of=/usr/bin/daemon-helper 2026-02-23T11:54:05.688 DEBUG:teuthology.orchestra.run.trial186:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-23T11:54:05.752 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2026-02-23T11:54:05.752 DEBUG:teuthology.orchestra.run.trial186:> set -ex 2026-02-23T11:54:05.752 DEBUG:teuthology.orchestra.run.trial186:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-23T11:54:05.816 DEBUG:teuthology.orchestra.run.trial186:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-23T11:54:05.881 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2026-02-23T11:54:05.881 DEBUG:teuthology.orchestra.run.trial186:> set -ex 2026-02-23T11:54:05.882 DEBUG:teuthology.orchestra.run.trial186:> sudo dd of=/usr/bin/stdin-killer 2026-02-23T11:54:05.945 DEBUG:teuthology.orchestra.run.trial186:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-23T11:54:06.009 INFO:teuthology.run_tasks:Running task pexec... 2026-02-23T11:54:06.022 INFO:teuthology.task.pexec:Executing custom commands... 2026-02-23T11:54:06.022 DEBUG:teuthology.orchestra.run.trial186:> TESTDIR=/home/ubuntu/cephtest bash -s 2026-02-23T11:54:06.052 INFO:teuthology.task.pexec:Running commands on host ubuntu@trial186.front.sepia.ceph.com 2026-02-23T11:54:06.052 INFO:teuthology.task.pexec:sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson 2026-02-23T11:54:06.102 DEBUG:teuthology.parallel:result is None 2026-02-23T11:54:06.102 INFO:teuthology.run_tasks:Running task install... 2026-02-23T11:54:06.122 DEBUG:teuthology.task.install:project ceph 2026-02-23T11:54:06.123 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'debug', 'sha1': 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d'}} 2026-02-23T11:54:06.123 DEBUG:teuthology.task.install:config {'flavor': 'debug', 'sha1': 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d'} 2026-02-23T11:54:06.123 INFO:teuthology.task.install:Using flavor: debug 2026-02-23T11:54:06.138 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-osd', 'ceph-osd-classic', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-osd', 'ceph-osd-classic', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-23T11:54:06.138 INFO:teuthology.task.install:extra packages: [] 2026-02-23T11:54:06.139 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'debug', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d', 'tag': None, 'wait_for_package': False} 2026-02-23T11:54:06.139 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d 2026-02-23T11:54:06.276 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-shraddhaag-enable-cephadm-tests/e371433fcd1390a0b82cfb18effc422c1f5d8c1d/centos/9/flavors/debug/ 2026-02-23T11:54:06.276 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5439.ge371433f 2026-02-23T11:54:06.517 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-shraddhaag-enable-cephadm-tests/e371433fcd1390a0b82cfb18effc422c1f5d8c1d/centos/9/flavors/debug/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-shraddhaag-enable-cephadm-tests/e371433fcd1390a0b82cfb18effc422c1f5d8c1d/centos/9/flavors/debug/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-shraddhaag-enable-cephadm-tests/e371433fcd1390a0b82cfb18effc422c1f5d8c1d/centos/9/flavors/debug/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-23T11:54:06.517 DEBUG:teuthology.orchestra.run.trial186:> set -ex 2026-02-23T11:54:06.517 DEBUG:teuthology.orchestra.run.trial186:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-23T11:54:06.542 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2026-02-23T11:54:06.542 DEBUG:teuthology.orchestra.run.trial186:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/e371433fcd1390a0b82cfb18effc422c1f5d8c1d/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-23T11:54:06.609 DEBUG:teuthology.orchestra.run.trial186:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-23T11:54:06.675 DEBUG:teuthology.orchestra.run.trial186:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-23T11:54:06.732 INFO:teuthology.orchestra.run.trial186.stdout:check_obsoletes = 1 2026-02-23T11:54:06.741 DEBUG:teuthology.orchestra.run.trial186:> sudo yum clean all 2026-02-23T11:54:06.905 INFO:teuthology.orchestra.run.trial186.stdout:56 files removed 2026-02-23T11:54:06.920 DEBUG:teuthology.orchestra.run.trial186:> sudo yum -y install ceph-radosgw 2026-02-23T11:54:07.588 INFO:teuthology.orchestra.run.trial186.stdout:ceph packages for x86_64 176 kB/s | 92 kB 00:00 2026-02-23T11:54:08.092 INFO:teuthology.orchestra.run.trial186.stdout:ceph noarch packages 50 kB/s | 24 kB 00:00 2026-02-23T11:54:08.561 INFO:teuthology.orchestra.run.trial186.stdout:ceph source packages 4.6 kB/s | 2.1 kB 00:00 2026-02-23T11:54:09.383 INFO:teuthology.orchestra.run.trial186.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.9 MB 00:00 2026-02-23T11:54:12.684 INFO:teuthology.orchestra.run.trial186.stdout:CentOS Stream 9 - AppStream 9.7 MB/s | 27 MB 00:02 2026-02-23T11:54:17.464 INFO:teuthology.orchestra.run.trial186.stdout:CentOS Stream 9 - CRB 3.2 MB/s | 7.8 MB 00:02 2026-02-23T11:54:19.307 INFO:teuthology.orchestra.run.trial186.stdout:CentOS Stream 9 - Extras packages 18 kB/s | 20 kB 00:01 2026-02-23T11:54:20.781 INFO:teuthology.orchestra.run.trial186.stdout:Extra Packages for Enterprise Linux 14 MB/s | 20 MB 00:01 2026-02-23T11:54:23.927 INFO:teuthology.orchestra.run.trial186.stdout:lab-extras 1.5 MB/s | 50 kB 00:00 2026-02-23T11:54:25.109 INFO:teuthology.orchestra.run.trial186.stdout:Package ceph-radosgw-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:54:25.129 INFO:teuthology.orchestra.run.trial186.stdout:Dependencies resolved. 2026-02-23T11:54:25.129 INFO:teuthology.orchestra.run.trial186.stdout:Nothing to do. 2026-02-23T11:54:25.129 INFO:teuthology.orchestra.run.trial186.stdout:Complete! 2026-02-23T11:54:25.153 DEBUG:teuthology.orchestra.run.trial186:> sudo yum -y install ceph-test 2026-02-23T11:54:25.460 INFO:teuthology.orchestra.run.trial186.stdout:Last metadata expiration check: 0:00:02 ago on Mon 23 Feb 2026 11:54:23 AM UTC. 2026-02-23T11:54:25.526 INFO:teuthology.orchestra.run.trial186.stdout:Package ceph-test-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:54:25.545 INFO:teuthology.orchestra.run.trial186.stdout:Dependencies resolved. 2026-02-23T11:54:25.546 INFO:teuthology.orchestra.run.trial186.stdout:Nothing to do. 2026-02-23T11:54:25.546 INFO:teuthology.orchestra.run.trial186.stdout:Complete! 2026-02-23T11:54:25.566 DEBUG:teuthology.orchestra.run.trial186:> sudo yum -y install ceph 2026-02-23T11:54:25.872 INFO:teuthology.orchestra.run.trial186.stdout:Last metadata expiration check: 0:00:02 ago on Mon 23 Feb 2026 11:54:23 AM UTC. 2026-02-23T11:54:25.936 INFO:teuthology.orchestra.run.trial186.stdout:Package ceph-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:54:25.955 INFO:teuthology.orchestra.run.trial186.stdout:Dependencies resolved. 2026-02-23T11:54:25.955 INFO:teuthology.orchestra.run.trial186.stdout:Nothing to do. 2026-02-23T11:54:25.955 INFO:teuthology.orchestra.run.trial186.stdout:Complete! 2026-02-23T11:54:25.976 DEBUG:teuthology.orchestra.run.trial186:> sudo yum -y install ceph-base 2026-02-23T11:54:26.282 INFO:teuthology.orchestra.run.trial186.stdout:Last metadata expiration check: 0:00:03 ago on Mon 23 Feb 2026 11:54:23 AM UTC. 2026-02-23T11:54:26.348 INFO:teuthology.orchestra.run.trial186.stdout:Package ceph-base-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:54:26.367 INFO:teuthology.orchestra.run.trial186.stdout:Dependencies resolved. 2026-02-23T11:54:26.367 INFO:teuthology.orchestra.run.trial186.stdout:Nothing to do. 2026-02-23T11:54:26.367 INFO:teuthology.orchestra.run.trial186.stdout:Complete! 2026-02-23T11:54:26.389 DEBUG:teuthology.orchestra.run.trial186:> sudo yum -y install cephadm 2026-02-23T11:54:26.698 INFO:teuthology.orchestra.run.trial186.stdout:Last metadata expiration check: 0:00:03 ago on Mon 23 Feb 2026 11:54:23 AM UTC. 2026-02-23T11:54:26.762 INFO:teuthology.orchestra.run.trial186.stdout:Package cephadm-2:20.3.0-5439.ge371433f.el9.noarch is already installed. 2026-02-23T11:54:26.781 INFO:teuthology.orchestra.run.trial186.stdout:Dependencies resolved. 2026-02-23T11:54:26.781 INFO:teuthology.orchestra.run.trial186.stdout:Nothing to do. 2026-02-23T11:54:26.781 INFO:teuthology.orchestra.run.trial186.stdout:Complete! 2026-02-23T11:54:26.802 DEBUG:teuthology.orchestra.run.trial186:> sudo yum -y install ceph-immutable-object-cache 2026-02-23T11:54:27.119 INFO:teuthology.orchestra.run.trial186.stdout:Last metadata expiration check: 0:00:04 ago on Mon 23 Feb 2026 11:54:23 AM UTC. 2026-02-23T11:54:27.186 INFO:teuthology.orchestra.run.trial186.stdout:Package ceph-immutable-object-cache-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:54:27.204 INFO:teuthology.orchestra.run.trial186.stdout:Dependencies resolved. 2026-02-23T11:54:27.204 INFO:teuthology.orchestra.run.trial186.stdout:Nothing to do. 2026-02-23T11:54:27.204 INFO:teuthology.orchestra.run.trial186.stdout:Complete! 2026-02-23T11:54:27.226 DEBUG:teuthology.orchestra.run.trial186:> sudo yum -y install ceph-mgr 2026-02-23T11:54:27.536 INFO:teuthology.orchestra.run.trial186.stdout:Last metadata expiration check: 0:00:04 ago on Mon 23 Feb 2026 11:54:23 AM UTC. 2026-02-23T11:54:27.602 INFO:teuthology.orchestra.run.trial186.stdout:Package ceph-mgr-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:54:27.620 INFO:teuthology.orchestra.run.trial186.stdout:Dependencies resolved. 2026-02-23T11:54:27.621 INFO:teuthology.orchestra.run.trial186.stdout:Nothing to do. 2026-02-23T11:54:27.621 INFO:teuthology.orchestra.run.trial186.stdout:Complete! 2026-02-23T11:54:27.642 DEBUG:teuthology.orchestra.run.trial186:> sudo yum -y install ceph-mgr-dashboard 2026-02-23T11:54:27.954 INFO:teuthology.orchestra.run.trial186.stdout:Last metadata expiration check: 0:00:04 ago on Mon 23 Feb 2026 11:54:23 AM UTC. 2026-02-23T11:54:28.018 INFO:teuthology.orchestra.run.trial186.stdout:Package ceph-mgr-dashboard-2:20.3.0-5439.ge371433f.el9.noarch is already installed. 2026-02-23T11:54:28.037 INFO:teuthology.orchestra.run.trial186.stdout:Dependencies resolved. 2026-02-23T11:54:28.037 INFO:teuthology.orchestra.run.trial186.stdout:Nothing to do. 2026-02-23T11:54:28.038 INFO:teuthology.orchestra.run.trial186.stdout:Complete! 2026-02-23T11:54:28.058 DEBUG:teuthology.orchestra.run.trial186:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-23T11:54:28.371 INFO:teuthology.orchestra.run.trial186.stdout:Last metadata expiration check: 0:00:05 ago on Mon 23 Feb 2026 11:54:23 AM UTC. 2026-02-23T11:54:28.435 INFO:teuthology.orchestra.run.trial186.stdout:Package ceph-mgr-diskprediction-local-2:20.3.0-5439.ge371433f.el9.noarch is already installed. 2026-02-23T11:54:28.454 INFO:teuthology.orchestra.run.trial186.stdout:Dependencies resolved. 2026-02-23T11:54:28.454 INFO:teuthology.orchestra.run.trial186.stdout:Nothing to do. 2026-02-23T11:54:28.455 INFO:teuthology.orchestra.run.trial186.stdout:Complete! 2026-02-23T11:54:28.476 DEBUG:teuthology.orchestra.run.trial186:> sudo yum -y install ceph-mgr-rook 2026-02-23T11:54:28.790 INFO:teuthology.orchestra.run.trial186.stdout:Last metadata expiration check: 0:00:05 ago on Mon 23 Feb 2026 11:54:23 AM UTC. 2026-02-23T11:54:28.856 INFO:teuthology.orchestra.run.trial186.stdout:Package ceph-mgr-rook-2:20.3.0-5439.ge371433f.el9.noarch is already installed. 2026-02-23T11:54:28.874 INFO:teuthology.orchestra.run.trial186.stdout:Dependencies resolved. 2026-02-23T11:54:28.875 INFO:teuthology.orchestra.run.trial186.stdout:Nothing to do. 2026-02-23T11:54:28.875 INFO:teuthology.orchestra.run.trial186.stdout:Complete! 2026-02-23T11:54:28.897 DEBUG:teuthology.orchestra.run.trial186:> sudo yum -y install ceph-mgr-cephadm 2026-02-23T11:54:29.206 INFO:teuthology.orchestra.run.trial186.stdout:Last metadata expiration check: 0:00:06 ago on Mon 23 Feb 2026 11:54:23 AM UTC. 2026-02-23T11:54:29.271 INFO:teuthology.orchestra.run.trial186.stdout:Package ceph-mgr-cephadm-2:20.3.0-5439.ge371433f.el9.noarch is already installed. 2026-02-23T11:54:29.289 INFO:teuthology.orchestra.run.trial186.stdout:Dependencies resolved. 2026-02-23T11:54:29.290 INFO:teuthology.orchestra.run.trial186.stdout:Nothing to do. 2026-02-23T11:54:29.290 INFO:teuthology.orchestra.run.trial186.stdout:Complete! 2026-02-23T11:54:29.312 DEBUG:teuthology.orchestra.run.trial186:> sudo yum -y install ceph-osd 2026-02-23T11:54:29.621 INFO:teuthology.orchestra.run.trial186.stdout:Last metadata expiration check: 0:00:06 ago on Mon 23 Feb 2026 11:54:23 AM UTC. 2026-02-23T11:54:29.686 INFO:teuthology.orchestra.run.trial186.stdout:Package ceph-osd-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:54:29.705 INFO:teuthology.orchestra.run.trial186.stdout:Dependencies resolved. 2026-02-23T11:54:29.705 INFO:teuthology.orchestra.run.trial186.stdout:Nothing to do. 2026-02-23T11:54:29.706 INFO:teuthology.orchestra.run.trial186.stdout:Complete! 2026-02-23T11:54:29.726 DEBUG:teuthology.orchestra.run.trial186:> sudo yum -y install ceph-osd-classic 2026-02-23T11:54:30.032 INFO:teuthology.orchestra.run.trial186.stdout:Last metadata expiration check: 0:00:07 ago on Mon 23 Feb 2026 11:54:23 AM UTC. 2026-02-23T11:54:30.097 INFO:teuthology.orchestra.run.trial186.stdout:Package ceph-osd-classic-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:54:30.116 INFO:teuthology.orchestra.run.trial186.stdout:Dependencies resolved. 2026-02-23T11:54:30.116 INFO:teuthology.orchestra.run.trial186.stdout:Nothing to do. 2026-02-23T11:54:30.116 INFO:teuthology.orchestra.run.trial186.stdout:Complete! 2026-02-23T11:54:30.138 DEBUG:teuthology.orchestra.run.trial186:> sudo yum -y install ceph-fuse 2026-02-23T11:54:30.450 INFO:teuthology.orchestra.run.trial186.stdout:Last metadata expiration check: 0:00:07 ago on Mon 23 Feb 2026 11:54:23 AM UTC. 2026-02-23T11:54:30.519 INFO:teuthology.orchestra.run.trial186.stdout:Package ceph-fuse-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:54:30.537 INFO:teuthology.orchestra.run.trial186.stdout:Dependencies resolved. 2026-02-23T11:54:30.538 INFO:teuthology.orchestra.run.trial186.stdout:Nothing to do. 2026-02-23T11:54:30.538 INFO:teuthology.orchestra.run.trial186.stdout:Complete! 2026-02-23T11:54:30.559 DEBUG:teuthology.orchestra.run.trial186:> sudo yum -y install ceph-volume 2026-02-23T11:54:30.868 INFO:teuthology.orchestra.run.trial186.stdout:Last metadata expiration check: 0:00:07 ago on Mon 23 Feb 2026 11:54:23 AM UTC. 2026-02-23T11:54:30.933 INFO:teuthology.orchestra.run.trial186.stdout:Package ceph-volume-2:20.3.0-5439.ge371433f.el9.noarch is already installed. 2026-02-23T11:54:30.952 INFO:teuthology.orchestra.run.trial186.stdout:Dependencies resolved. 2026-02-23T11:54:30.953 INFO:teuthology.orchestra.run.trial186.stdout:Nothing to do. 2026-02-23T11:54:30.953 INFO:teuthology.orchestra.run.trial186.stdout:Complete! 2026-02-23T11:54:30.974 DEBUG:teuthology.orchestra.run.trial186:> sudo yum -y install librados-devel 2026-02-23T11:54:31.281 INFO:teuthology.orchestra.run.trial186.stdout:Last metadata expiration check: 0:00:08 ago on Mon 23 Feb 2026 11:54:23 AM UTC. 2026-02-23T11:54:31.347 INFO:teuthology.orchestra.run.trial186.stdout:Package librados-devel-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:54:31.366 INFO:teuthology.orchestra.run.trial186.stdout:Dependencies resolved. 2026-02-23T11:54:31.366 INFO:teuthology.orchestra.run.trial186.stdout:Nothing to do. 2026-02-23T11:54:31.366 INFO:teuthology.orchestra.run.trial186.stdout:Complete! 2026-02-23T11:54:31.388 DEBUG:teuthology.orchestra.run.trial186:> sudo yum -y install libcephfs2 2026-02-23T11:54:31.700 INFO:teuthology.orchestra.run.trial186.stdout:Last metadata expiration check: 0:00:08 ago on Mon 23 Feb 2026 11:54:23 AM UTC. 2026-02-23T11:54:31.767 INFO:teuthology.orchestra.run.trial186.stdout:Package libcephfs2-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:54:31.785 INFO:teuthology.orchestra.run.trial186.stdout:Dependencies resolved. 2026-02-23T11:54:31.786 INFO:teuthology.orchestra.run.trial186.stdout:Nothing to do. 2026-02-23T11:54:31.786 INFO:teuthology.orchestra.run.trial186.stdout:Complete! 2026-02-23T11:54:31.807 DEBUG:teuthology.orchestra.run.trial186:> sudo yum -y install libcephfs-devel 2026-02-23T11:54:32.117 INFO:teuthology.orchestra.run.trial186.stdout:Last metadata expiration check: 0:00:09 ago on Mon 23 Feb 2026 11:54:23 AM UTC. 2026-02-23T11:54:32.183 INFO:teuthology.orchestra.run.trial186.stdout:Package libcephfs-devel-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:54:32.202 INFO:teuthology.orchestra.run.trial186.stdout:Dependencies resolved. 2026-02-23T11:54:32.202 INFO:teuthology.orchestra.run.trial186.stdout:Nothing to do. 2026-02-23T11:54:32.202 INFO:teuthology.orchestra.run.trial186.stdout:Complete! 2026-02-23T11:54:32.223 DEBUG:teuthology.orchestra.run.trial186:> sudo yum -y install librados2 2026-02-23T11:54:32.536 INFO:teuthology.orchestra.run.trial186.stdout:Last metadata expiration check: 0:00:09 ago on Mon 23 Feb 2026 11:54:23 AM UTC. 2026-02-23T11:54:32.603 INFO:teuthology.orchestra.run.trial186.stdout:Package librados2-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:54:32.621 INFO:teuthology.orchestra.run.trial186.stdout:Dependencies resolved. 2026-02-23T11:54:32.622 INFO:teuthology.orchestra.run.trial186.stdout:Nothing to do. 2026-02-23T11:54:32.622 INFO:teuthology.orchestra.run.trial186.stdout:Complete! 2026-02-23T11:54:32.644 DEBUG:teuthology.orchestra.run.trial186:> sudo yum -y install librbd1 2026-02-23T11:54:32.957 INFO:teuthology.orchestra.run.trial186.stdout:Last metadata expiration check: 0:00:09 ago on Mon 23 Feb 2026 11:54:23 AM UTC. 2026-02-23T11:54:33.024 INFO:teuthology.orchestra.run.trial186.stdout:Package librbd1-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:54:33.042 INFO:teuthology.orchestra.run.trial186.stdout:Dependencies resolved. 2026-02-23T11:54:33.043 INFO:teuthology.orchestra.run.trial186.stdout:Nothing to do. 2026-02-23T11:54:33.043 INFO:teuthology.orchestra.run.trial186.stdout:Complete! 2026-02-23T11:54:33.064 DEBUG:teuthology.orchestra.run.trial186:> sudo yum -y install python3-rados 2026-02-23T11:54:33.369 INFO:teuthology.orchestra.run.trial186.stdout:Last metadata expiration check: 0:00:10 ago on Mon 23 Feb 2026 11:54:23 AM UTC. 2026-02-23T11:54:33.433 INFO:teuthology.orchestra.run.trial186.stdout:Package python3-rados-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:54:33.452 INFO:teuthology.orchestra.run.trial186.stdout:Dependencies resolved. 2026-02-23T11:54:33.452 INFO:teuthology.orchestra.run.trial186.stdout:Nothing to do. 2026-02-23T11:54:33.452 INFO:teuthology.orchestra.run.trial186.stdout:Complete! 2026-02-23T11:54:33.473 DEBUG:teuthology.orchestra.run.trial186:> sudo yum -y install python3-rgw 2026-02-23T11:54:33.780 INFO:teuthology.orchestra.run.trial186.stdout:Last metadata expiration check: 0:00:10 ago on Mon 23 Feb 2026 11:54:23 AM UTC. 2026-02-23T11:54:33.844 INFO:teuthology.orchestra.run.trial186.stdout:Package python3-rgw-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:54:33.862 INFO:teuthology.orchestra.run.trial186.stdout:Dependencies resolved. 2026-02-23T11:54:33.863 INFO:teuthology.orchestra.run.trial186.stdout:Nothing to do. 2026-02-23T11:54:33.863 INFO:teuthology.orchestra.run.trial186.stdout:Complete! 2026-02-23T11:54:33.884 DEBUG:teuthology.orchestra.run.trial186:> sudo yum -y install python3-cephfs 2026-02-23T11:54:34.188 INFO:teuthology.orchestra.run.trial186.stdout:Last metadata expiration check: 0:00:11 ago on Mon 23 Feb 2026 11:54:23 AM UTC. 2026-02-23T11:54:34.255 INFO:teuthology.orchestra.run.trial186.stdout:Package python3-cephfs-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:54:34.274 INFO:teuthology.orchestra.run.trial186.stdout:Dependencies resolved. 2026-02-23T11:54:34.275 INFO:teuthology.orchestra.run.trial186.stdout:Nothing to do. 2026-02-23T11:54:34.275 INFO:teuthology.orchestra.run.trial186.stdout:Complete! 2026-02-23T11:54:34.296 DEBUG:teuthology.orchestra.run.trial186:> sudo yum -y install python3-rbd 2026-02-23T11:54:34.611 INFO:teuthology.orchestra.run.trial186.stdout:Last metadata expiration check: 0:00:11 ago on Mon 23 Feb 2026 11:54:23 AM UTC. 2026-02-23T11:54:34.677 INFO:teuthology.orchestra.run.trial186.stdout:Package python3-rbd-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:54:34.695 INFO:teuthology.orchestra.run.trial186.stdout:Dependencies resolved. 2026-02-23T11:54:34.696 INFO:teuthology.orchestra.run.trial186.stdout:Nothing to do. 2026-02-23T11:54:34.696 INFO:teuthology.orchestra.run.trial186.stdout:Complete! 2026-02-23T11:54:34.718 DEBUG:teuthology.orchestra.run.trial186:> sudo yum -y install rbd-fuse 2026-02-23T11:54:35.033 INFO:teuthology.orchestra.run.trial186.stdout:Last metadata expiration check: 0:00:12 ago on Mon 23 Feb 2026 11:54:23 AM UTC. 2026-02-23T11:54:35.099 INFO:teuthology.orchestra.run.trial186.stdout:Package rbd-fuse-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:54:35.118 INFO:teuthology.orchestra.run.trial186.stdout:Dependencies resolved. 2026-02-23T11:54:35.118 INFO:teuthology.orchestra.run.trial186.stdout:Nothing to do. 2026-02-23T11:54:35.119 INFO:teuthology.orchestra.run.trial186.stdout:Complete! 2026-02-23T11:54:35.140 DEBUG:teuthology.orchestra.run.trial186:> sudo yum -y install rbd-mirror 2026-02-23T11:54:35.449 INFO:teuthology.orchestra.run.trial186.stdout:Last metadata expiration check: 0:00:12 ago on Mon 23 Feb 2026 11:54:23 AM UTC. 2026-02-23T11:54:35.515 INFO:teuthology.orchestra.run.trial186.stdout:Package rbd-mirror-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:54:35.534 INFO:teuthology.orchestra.run.trial186.stdout:Dependencies resolved. 2026-02-23T11:54:35.534 INFO:teuthology.orchestra.run.trial186.stdout:Nothing to do. 2026-02-23T11:54:35.534 INFO:teuthology.orchestra.run.trial186.stdout:Complete! 2026-02-23T11:54:35.555 DEBUG:teuthology.orchestra.run.trial186:> sudo yum -y install rbd-nbd 2026-02-23T11:54:35.865 INFO:teuthology.orchestra.run.trial186.stdout:Last metadata expiration check: 0:00:12 ago on Mon 23 Feb 2026 11:54:23 AM UTC. 2026-02-23T11:54:35.932 INFO:teuthology.orchestra.run.trial186.stdout:Package rbd-nbd-2:20.3.0-5439.ge371433f.el9.x86_64 is already installed. 2026-02-23T11:54:35.951 INFO:teuthology.orchestra.run.trial186.stdout:Dependencies resolved. 2026-02-23T11:54:35.952 INFO:teuthology.orchestra.run.trial186.stdout:Nothing to do. 2026-02-23T11:54:35.952 INFO:teuthology.orchestra.run.trial186.stdout:Complete! 2026-02-23T11:54:35.975 DEBUG:teuthology.parallel:result is None 2026-02-23T11:54:35.976 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=e371433fcd1390a0b82cfb18effc422c1f5d8c1d 2026-02-23T11:54:36.094 DEBUG:teuthology.orchestra.run.trial186:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-23T11:54:36.115 INFO:teuthology.orchestra.run.trial186.stdout:20.3.0-5439.ge371433f.el9 2026-02-23T11:54:36.115 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5439.ge371433f.el9 2026-02-23T11:54:36.115 INFO:teuthology.task.install:The correct ceph version 20.3.0-5439.ge371433f is installed. 2026-02-23T11:54:36.117 INFO:teuthology.task.install.util:Utilities already shipped, skip it... 2026-02-23T11:54:36.117 INFO:teuthology.run_tasks:Running task ceph... 2026-02-23T11:54:36.189 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2026-02-23T11:54:36.189 DEBUG:teuthology.orchestra.run.trial186:> sudo chmod 777 /var/log/ceph 2026-02-23T11:54:36.215 INFO:tasks.ceph:Disabling ceph logrotate... 2026-02-23T11:54:36.215 DEBUG:teuthology.orchestra.run.trial186:> sudo rm -f -- /etc/logrotate.d/ceph 2026-02-23T11:54:36.279 INFO:tasks.ceph:Creating extra log directories... 2026-02-23T11:54:36.279 DEBUG:teuthology.orchestra.run.trial186:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2026-02-23T11:54:36.352 INFO:tasks.ceph:Creating ceph cluster ceph... 2026-02-23T11:54:36.353 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore cachepin type': '2Q', 'seastore max concurrent transactions': 8}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2026-02-23T11:54:36.353 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/mnt/teuthology/shraddhaag-2026-02-23_09:58:48-crimson-rados-main-distro-debug-trial/66294', 'branch': 'main', 'description': 'crimson-rados/perf/{0-crimson_install clusters/{crimson-fixed-1} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/segmented$/{crimson_seastore_segmented_2q} perf_ignorelist settings/optimized workloads/radosbench_4M_write}', 'email': None, 'first_in_suite': False, 'flavor': 'debug', 'job_id': '66294', 'kernel': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'trial', 'name': 'shraddhaag-2026-02-23_09:58:48-crimson-rados-main-distro-debug-trial', 'no_nested_subset': False, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd': {'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore cachepin type': '2Q', 'seastore max concurrent transactions': 8}}, 'crimson_compat': True, 'flavor': 'debug', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH'], 'sha1': 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'global': {'osd crush chooseleaf type': 0, 'osd pool default pg num': 128, 'osd pool default pgp num': 128, 'osd pool default size': 2}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'debug', 'sha1': 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d'}}, 'workunit': {'branch': 'wip-shraddhaag-enable-cephadm-tests', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d'}}, 'owner': 'scheduled_shraddhaag@soko04.front.sepia.ceph.com', 'priority': 49, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 66, 'sha1': 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'wip-shraddhaag-enable-cephadm-tests', 'suite_path': '/home/teuthworker/src/github.com_shraddhaag_ceph_e371433fcd1390a0b82cfb18effc422c1f5d8c1d/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/shraddhaag/ceph.git', 'suite_sha1': 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d', 'targets': {'trial186.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLDyxqOVszW7QgWdmsWYzM19QfQnsb0ZiZyzGIHN941zbMFYb7ZU9RsookcbeRuxK/WY4tvgeWIANTqANZ6NgeA='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'flavor': 'debug', 'sha1': 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d'}}, {'pexec': {'all': ['sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson']}}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore cachepin type': '2Q', 'seastore max concurrent transactions': 8}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}}, 'wait-for-scrub': False, 'crimson_compat': True, 'flavor': 'debug', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH'], 'sha1': 'e371433fcd1390a0b82cfb18effc422c1f5d8c1d', 'cluster': 'ceph'}}, {'ssh_keys': None}, {'cbt': {'benchmarks': {'radosbench': {'concurrent_ops': 4, 'concurrent_procs': 2, 'op_size': [4194304], 'pool_monitoring_list': ['collectl'], 'pool_profile': 'replicated', 'run_monitoring_list': ['collectl'], 'time': 60, 'write_only': True}}, 'cluster': {'iterations': 1, 'osds_per_node': 3, 'pool_profiles': {'replicated': {'pg_size': 256, 'pgp_size': 256, 'replication': 'replicated'}}, 'user': 'ubuntu'}, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'fa17720d0088c3ac28e473468bfc79eeaff5cd38', 'timestamp': '2026-02-23_09:58:48', 'tube': 'trial', 'user': 'shraddhaag', 'verbose': False, 'worker_log': '/home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.3318653'} 2026-02-23T11:54:36.353 DEBUG:teuthology.orchestra.run.trial186:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2026-02-23T11:54:36.399 DEBUG:teuthology.orchestra.run.trial186:> sudo install -d -m0777 -- /var/run/ceph 2026-02-23T11:54:36.465 DEBUG:teuthology.orchestra.run.trial186:> set -ex 2026-02-23T11:54:36.465 DEBUG:teuthology.orchestra.run.trial186:> dd if=/scratch_devs of=/dev/stdout 2026-02-23T11:54:36.522 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2026-02-23T11:54:36.522 DEBUG:teuthology.orchestra.run.trial186:> stat /dev/vg_nvme/lv_1 2026-02-23T11:54:36.579 INFO:teuthology.orchestra.run.trial186.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2026-02-23T11:54:36.579 INFO:teuthology.orchestra.run.trial186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-23T11:54:36.579 INFO:teuthology.orchestra.run.trial186.stdout:Device: 6h/6d Inode: 907 Links: 1 2026-02-23T11:54:36.579 INFO:teuthology.orchestra.run.trial186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-23T11:54:36.579 INFO:teuthology.orchestra.run.trial186.stdout:Context: system_u:object_r:device_t:s0 2026-02-23T11:54:36.579 INFO:teuthology.orchestra.run.trial186.stdout:Access: 2026-02-23 11:51:00.834462874 +0000 2026-02-23T11:54:36.579 INFO:teuthology.orchestra.run.trial186.stdout:Modify: 2026-02-23 11:46:36.906332608 +0000 2026-02-23T11:54:36.579 INFO:teuthology.orchestra.run.trial186.stdout:Change: 2026-02-23 11:46:36.906332608 +0000 2026-02-23T11:54:36.580 INFO:teuthology.orchestra.run.trial186.stdout: Birth: 2026-02-23 11:46:36.906332608 +0000 2026-02-23T11:54:36.580 DEBUG:teuthology.orchestra.run.trial186:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2026-02-23T11:54:36.641 INFO:teuthology.orchestra.run.trial186.stderr:1+0 records in 2026-02-23T11:54:36.641 INFO:teuthology.orchestra.run.trial186.stderr:1+0 records out 2026-02-23T11:54:36.641 INFO:teuthology.orchestra.run.trial186.stderr:512 bytes copied, 0.000169147 s, 3.0 MB/s 2026-02-23T11:54:36.642 DEBUG:teuthology.orchestra.run.trial186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2026-02-23T11:54:36.702 DEBUG:teuthology.orchestra.run.trial186:> stat /dev/vg_nvme/lv_2 2026-02-23T11:54:36.757 INFO:teuthology.orchestra.run.trial186.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2026-02-23T11:54:36.758 INFO:teuthology.orchestra.run.trial186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-23T11:54:36.758 INFO:teuthology.orchestra.run.trial186.stdout:Device: 6h/6d Inode: 939 Links: 1 2026-02-23T11:54:36.758 INFO:teuthology.orchestra.run.trial186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-23T11:54:36.758 INFO:teuthology.orchestra.run.trial186.stdout:Context: system_u:object_r:device_t:s0 2026-02-23T11:54:36.758 INFO:teuthology.orchestra.run.trial186.stdout:Access: 2026-02-23 11:51:00.834462874 +0000 2026-02-23T11:54:36.758 INFO:teuthology.orchestra.run.trial186.stdout:Modify: 2026-02-23 11:46:37.141334972 +0000 2026-02-23T11:54:36.758 INFO:teuthology.orchestra.run.trial186.stdout:Change: 2026-02-23 11:46:37.141334972 +0000 2026-02-23T11:54:36.758 INFO:teuthology.orchestra.run.trial186.stdout: Birth: 2026-02-23 11:46:37.141334972 +0000 2026-02-23T11:54:36.758 DEBUG:teuthology.orchestra.run.trial186:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2026-02-23T11:54:36.821 INFO:teuthology.orchestra.run.trial186.stderr:1+0 records in 2026-02-23T11:54:36.821 INFO:teuthology.orchestra.run.trial186.stderr:1+0 records out 2026-02-23T11:54:36.821 INFO:teuthology.orchestra.run.trial186.stderr:512 bytes copied, 0.000155 s, 3.3 MB/s 2026-02-23T11:54:36.822 DEBUG:teuthology.orchestra.run.trial186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2026-02-23T11:54:36.879 DEBUG:teuthology.orchestra.run.trial186:> stat /dev/vg_nvme/lv_3 2026-02-23T11:54:36.936 INFO:teuthology.orchestra.run.trial186.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2026-02-23T11:54:36.936 INFO:teuthology.orchestra.run.trial186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-23T11:54:36.936 INFO:teuthology.orchestra.run.trial186.stdout:Device: 6h/6d Inode: 964 Links: 1 2026-02-23T11:54:36.936 INFO:teuthology.orchestra.run.trial186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-23T11:54:36.936 INFO:teuthology.orchestra.run.trial186.stdout:Context: system_u:object_r:device_t:s0 2026-02-23T11:54:36.936 INFO:teuthology.orchestra.run.trial186.stdout:Access: 2026-02-23 11:51:00.834462874 +0000 2026-02-23T11:54:36.936 INFO:teuthology.orchestra.run.trial186.stdout:Modify: 2026-02-23 11:46:37.404337617 +0000 2026-02-23T11:54:36.936 INFO:teuthology.orchestra.run.trial186.stdout:Change: 2026-02-23 11:46:37.404337617 +0000 2026-02-23T11:54:36.936 INFO:teuthology.orchestra.run.trial186.stdout: Birth: 2026-02-23 11:46:37.404337617 +0000 2026-02-23T11:54:36.937 DEBUG:teuthology.orchestra.run.trial186:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2026-02-23T11:54:37.000 INFO:teuthology.orchestra.run.trial186.stderr:1+0 records in 2026-02-23T11:54:37.000 INFO:teuthology.orchestra.run.trial186.stderr:1+0 records out 2026-02-23T11:54:37.001 INFO:teuthology.orchestra.run.trial186.stderr:512 bytes copied, 0.000150682 s, 3.4 MB/s 2026-02-23T11:54:37.002 DEBUG:teuthology.orchestra.run.trial186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2026-02-23T11:54:37.058 DEBUG:teuthology.orchestra.run.trial186:> stat /dev/vg_nvme/lv_4 2026-02-23T11:54:37.114 INFO:teuthology.orchestra.run.trial186.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2026-02-23T11:54:37.114 INFO:teuthology.orchestra.run.trial186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-23T11:54:37.114 INFO:teuthology.orchestra.run.trial186.stdout:Device: 6h/6d Inode: 991 Links: 1 2026-02-23T11:54:37.114 INFO:teuthology.orchestra.run.trial186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-23T11:54:37.114 INFO:teuthology.orchestra.run.trial186.stdout:Context: system_u:object_r:device_t:s0 2026-02-23T11:54:37.115 INFO:teuthology.orchestra.run.trial186.stdout:Access: 2026-02-23 11:51:00.834462874 +0000 2026-02-23T11:54:37.115 INFO:teuthology.orchestra.run.trial186.stdout:Modify: 2026-02-23 11:46:37.661340202 +0000 2026-02-23T11:54:37.115 INFO:teuthology.orchestra.run.trial186.stdout:Change: 2026-02-23 11:46:37.661340202 +0000 2026-02-23T11:54:37.115 INFO:teuthology.orchestra.run.trial186.stdout: Birth: 2026-02-23 11:46:37.661340202 +0000 2026-02-23T11:54:37.115 DEBUG:teuthology.orchestra.run.trial186:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2026-02-23T11:54:37.176 INFO:teuthology.orchestra.run.trial186.stderr:1+0 records in 2026-02-23T11:54:37.176 INFO:teuthology.orchestra.run.trial186.stderr:1+0 records out 2026-02-23T11:54:37.176 INFO:teuthology.orchestra.run.trial186.stderr:512 bytes copied, 0.000146094 s, 3.5 MB/s 2026-02-23T11:54:37.177 DEBUG:teuthology.orchestra.run.trial186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2026-02-23T11:54:37.235 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-23T11:54:37.235 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@trial186.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2026-02-23T11:54:37.235 INFO:tasks.ceph:Generating config... 2026-02-23T11:54:37.238 INFO:tasks.ceph:[osd] debug monc = 20 2026-02-23T11:54:37.238 INFO:tasks.ceph:[osd] crimson cpu num = 4 2026-02-23T11:54:37.238 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2026-02-23T11:54:37.238 INFO:tasks.ceph:[osd] debug filestore = 0/0 2026-02-23T11:54:37.238 INFO:tasks.ceph:[osd] debug journal = 0/0 2026-02-23T11:54:37.239 INFO:tasks.ceph:[osd] debug ms = 0/0 2026-02-23T11:54:37.239 INFO:tasks.ceph:[osd] debug osd = 0/0 2026-02-23T11:54:37.239 INFO:tasks.ceph:[osd] debug seastore = 20 2026-02-23T11:54:37.239 INFO:tasks.ceph:[osd] debug seastore backref = 20 2026-02-23T11:54:37.239 INFO:tasks.ceph:[osd] debug seastore cache = 20 2026-02-23T11:54:37.239 INFO:tasks.ceph:[osd] debug seastore cleaner = 20 2026-02-23T11:54:37.239 INFO:tasks.ceph:[osd] debug seastore device = 20 2026-02-23T11:54:37.239 INFO:tasks.ceph:[osd] debug seastore epm = 20 2026-02-23T11:54:37.239 INFO:tasks.ceph:[osd] debug seastore fixedkv tree = 20 2026-02-23T11:54:37.240 INFO:tasks.ceph:[osd] debug seastore journal = 20 2026-02-23T11:54:37.240 INFO:tasks.ceph:[osd] debug seastore lba = 20 2026-02-23T11:54:37.240 INFO:tasks.ceph:[osd] debug seastore odata = 20 2026-02-23T11:54:37.240 INFO:tasks.ceph:[osd] debug seastore omap = 20 2026-02-23T11:54:37.240 INFO:tasks.ceph:[osd] debug seastore onode = 20 2026-02-23T11:54:37.240 INFO:tasks.ceph:[osd] debug seastore t = 20 2026-02-23T11:54:37.240 INFO:tasks.ceph:[osd] debug seastore tm = 20 2026-02-23T11:54:37.240 INFO:tasks.ceph:[osd] osd objectstore = seastore 2026-02-23T11:54:37.240 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2026-02-23T11:54:37.240 INFO:tasks.ceph:[osd] seastore cachepin size pershard = 64M 2026-02-23T11:54:37.240 INFO:tasks.ceph:[osd] seastore cachepin type = 2Q 2026-02-23T11:54:37.240 INFO:tasks.ceph:[osd] seastore max concurrent transactions = 8 2026-02-23T11:54:37.240 INFO:tasks.ceph:[global] auth client required = none 2026-02-23T11:54:37.240 INFO:tasks.ceph:[global] auth cluster required = none 2026-02-23T11:54:37.240 INFO:tasks.ceph:[global] auth service required = none 2026-02-23T11:54:37.241 INFO:tasks.ceph:[global] auth supported = none 2026-02-23T11:54:37.241 INFO:tasks.ceph:[global] debug asok = 0/0 2026-02-23T11:54:37.241 INFO:tasks.ceph:[global] debug auth = 0/0 2026-02-23T11:54:37.241 INFO:tasks.ceph:[global] debug bdev = 0/0 2026-02-23T11:54:37.241 INFO:tasks.ceph:[global] debug bluefs = 0/0 2026-02-23T11:54:37.241 INFO:tasks.ceph:[global] debug bluestore = 0/0 2026-02-23T11:54:37.241 INFO:tasks.ceph:[global] debug buffer = 0/0 2026-02-23T11:54:37.241 INFO:tasks.ceph:[global] debug civetweb = 0/0 2026-02-23T11:54:37.241 INFO:tasks.ceph:[global] debug client = 0/0 2026-02-23T11:54:37.241 INFO:tasks.ceph:[global] debug compressor = 0/0 2026-02-23T11:54:37.241 INFO:tasks.ceph:[global] debug context = 0/0 2026-02-23T11:54:37.241 INFO:tasks.ceph:[global] debug crush = 0/0 2026-02-23T11:54:37.241 INFO:tasks.ceph:[global] debug crypto = 0/0 2026-02-23T11:54:37.241 INFO:tasks.ceph:[global] debug dpdk = 0/0 2026-02-23T11:54:37.241 INFO:tasks.ceph:[global] debug eventtrace = 0/0 2026-02-23T11:54:37.241 INFO:tasks.ceph:[global] debug filer = 0/0 2026-02-23T11:54:37.241 INFO:tasks.ceph:[global] debug filestore = 0/0 2026-02-23T11:54:37.241 INFO:tasks.ceph:[global] debug finisher = 0/0 2026-02-23T11:54:37.242 INFO:tasks.ceph:[global] debug fuse = 0/0 2026-02-23T11:54:37.242 INFO:tasks.ceph:[global] debug heartbeatmap = 0/0 2026-02-23T11:54:37.242 INFO:tasks.ceph:[global] debug javaclient = 0/0 2026-02-23T11:54:37.242 INFO:tasks.ceph:[global] debug journal = 0/0 2026-02-23T11:54:37.242 INFO:tasks.ceph:[global] debug journaler = 0/0 2026-02-23T11:54:37.242 INFO:tasks.ceph:[global] debug kstore = 0/0 2026-02-23T11:54:37.242 INFO:tasks.ceph:[global] debug leveldb = 0/0 2026-02-23T11:54:37.242 INFO:tasks.ceph:[global] debug lockdep = 0/0 2026-02-23T11:54:37.242 INFO:tasks.ceph:[global] debug mds = 0/0 2026-02-23T11:54:37.242 INFO:tasks.ceph:[global] debug mds balancer = 0/0 2026-02-23T11:54:37.242 INFO:tasks.ceph:[global] debug mds locker = 0/0 2026-02-23T11:54:37.242 INFO:tasks.ceph:[global] debug mds log = 0/0 2026-02-23T11:54:37.242 INFO:tasks.ceph:[global] debug mds log expire = 0/0 2026-02-23T11:54:37.242 INFO:tasks.ceph:[global] debug mds migrator = 0/0 2026-02-23T11:54:37.242 INFO:tasks.ceph:[global] debug memdb = 0/0 2026-02-23T11:54:37.242 INFO:tasks.ceph:[global] debug mgr = 0/0 2026-02-23T11:54:37.242 INFO:tasks.ceph:[global] debug mgrc = 0/0 2026-02-23T11:54:37.242 INFO:tasks.ceph:[global] debug mon = 0/0 2026-02-23T11:54:37.243 INFO:tasks.ceph:[global] debug monc = 0/0 2026-02-23T11:54:37.243 INFO:tasks.ceph:[global] debug ms = 0/0 2026-02-23T11:54:37.243 INFO:tasks.ceph:[global] debug objclass = 0/0 2026-02-23T11:54:37.243 INFO:tasks.ceph:[global] debug objectcacher = 0/0 2026-02-23T11:54:37.243 INFO:tasks.ceph:[global] debug objecter = 0/0 2026-02-23T11:54:37.243 INFO:tasks.ceph:[global] debug optracker = 0/0 2026-02-23T11:54:37.243 INFO:tasks.ceph:[global] debug osd = 0/0 2026-02-23T11:54:37.243 INFO:tasks.ceph:[global] debug paxos = 0/0 2026-02-23T11:54:37.243 INFO:tasks.ceph:[global] debug perfcounter = 0/0 2026-02-23T11:54:37.243 INFO:tasks.ceph:[global] debug rados = 0/0 2026-02-23T11:54:37.244 INFO:tasks.ceph:[global] debug rbd = 0/0 2026-02-23T11:54:37.244 INFO:tasks.ceph:[global] debug rbd mirror = 0/0 2026-02-23T11:54:37.244 INFO:tasks.ceph:[global] debug rbd replay = 0/0 2026-02-23T11:54:37.244 INFO:tasks.ceph:[global] debug refs = 0/0 2026-02-23T11:54:37.244 INFO:tasks.ceph:[global] debug rgw = 0/0 2026-02-23T11:54:37.244 INFO:tasks.ceph:[global] debug rgw lifecycle = 0/0 2026-02-23T11:54:37.244 INFO:tasks.ceph:[global] debug rgw notification = 0/0 2026-02-23T11:54:37.244 INFO:tasks.ceph:[global] debug rgw sync = 0/0 2026-02-23T11:54:37.244 INFO:tasks.ceph:[global] debug rocksdb = 0/0 2026-02-23T11:54:37.244 INFO:tasks.ceph:[global] debug striper = 0/0 2026-02-23T11:54:37.244 INFO:tasks.ceph:[global] debug throttle = 0/0 2026-02-23T11:54:37.244 INFO:tasks.ceph:[global] debug timer = 0/0 2026-02-23T11:54:37.244 INFO:tasks.ceph:[global] debug tp = 0/0 2026-02-23T11:54:37.244 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2026-02-23T11:54:37.244 INFO:tasks.ceph:[global] ms client mode = crc 2026-02-23T11:54:37.244 INFO:tasks.ceph:[global] ms cluster mode = crc 2026-02-23T11:54:37.245 INFO:tasks.ceph:[global] ms mon client mode = crc 2026-02-23T11:54:37.245 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2026-02-23T11:54:37.245 INFO:tasks.ceph:[global] ms mon service mode = crc 2026-02-23T11:54:37.245 INFO:tasks.ceph:[global] ms service mode = crc 2026-02-23T11:54:37.245 INFO:tasks.ceph:[mgr] debug mgr = 20 2026-02-23T11:54:37.245 INFO:tasks.ceph:[mgr] debug ms = 1 2026-02-23T11:54:37.245 INFO:tasks.ceph:[mon] debug mon = 0/0 2026-02-23T11:54:37.245 INFO:tasks.ceph:[mon] debug ms = 0/0 2026-02-23T11:54:37.245 INFO:tasks.ceph:[mon] debug paxos = 0/0 2026-02-23T11:54:37.245 INFO:tasks.ceph:[mon] osd pool default crimson = True 2026-02-23T11:54:37.245 INFO:tasks.ceph:Setting up mon.a... 2026-02-23T11:54:37.245 DEBUG:teuthology.orchestra.run.trial186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2026-02-23T11:54:37.315 INFO:teuthology.orchestra.run.trial186.stdout:creating /etc/ceph/ceph.keyring 2026-02-23T11:54:37.318 DEBUG:teuthology.orchestra.run.trial186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2026-02-23T11:54:37.403 DEBUG:teuthology.orchestra.run.trial186:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-23T11:54:37.470 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '10.20.193.186')] 2026-02-23T11:54:37.471 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'none', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '10.20.193.186', 'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': 'true', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore cachepin type': '2Q', 'seastore max concurrent transactions': 8}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': '0/0', 'debug mon': '0/0', 'debug paxos': '0/0', 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}} 2026-02-23T11:54:37.473 DEBUG:teuthology.orchestra.run.trial186:> set -ex 2026-02-23T11:54:37.473 DEBUG:teuthology.orchestra.run.trial186:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2026-02-23T11:54:37.528 DEBUG:teuthology.orchestra.run.trial186:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 10.20.193.186 --print /home/ubuntu/cephtest/ceph.monmap 2026-02-23T11:54:37.607 INFO:teuthology.orchestra.run.trial186.stderr:2026-02-23T11:54:37.603+0000 7f1f0de89f40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:54:37.607 INFO:teuthology.orchestra.run.trial186.stderr:2026-02-23T11:54:37.603+0000 7f1f0de89f40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:54:37.607 INFO:teuthology.orchestra.run.trial186.stderr:2026-02-23T11:54:37.603+0000 7f1f0de89f40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:54:37.607 INFO:teuthology.orchestra.run.trial186.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2026-02-23T11:54:37.607 INFO:teuthology.orchestra.run.trial186.stdout:monmaptool: generated fsid d91d7eac-82c0-417c-a119-6b75d467a55c 2026-02-23T11:54:37.607 INFO:teuthology.orchestra.run.trial186.stdout:setting min_mon_release = tentacle 2026-02-23T11:54:37.607 INFO:teuthology.orchestra.run.trial186.stdout:epoch 0 2026-02-23T11:54:37.607 INFO:teuthology.orchestra.run.trial186.stdout:fsid d91d7eac-82c0-417c-a119-6b75d467a55c 2026-02-23T11:54:37.608 INFO:teuthology.orchestra.run.trial186.stdout:last_changed 2026-02-23T11:54:37.605512+0000 2026-02-23T11:54:37.608 INFO:teuthology.orchestra.run.trial186.stdout:created 2026-02-23T11:54:37.605512+0000 2026-02-23T11:54:37.608 INFO:teuthology.orchestra.run.trial186.stdout:min_mon_release 20 (tentacle) 2026-02-23T11:54:37.608 INFO:teuthology.orchestra.run.trial186.stdout:election_strategy: 1 2026-02-23T11:54:37.608 INFO:teuthology.orchestra.run.trial186.stdout:0: [v2:10.20.193.186:3300/0,v1:10.20.193.186:6789/0] mon.a 2026-02-23T11:54:37.608 INFO:teuthology.orchestra.run.trial186.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2026-02-23T11:54:37.609 DEBUG:teuthology.orchestra.run.trial186:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2026-02-23T11:54:37.664 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID d91d7eac-82c0-417c-a119-6b75d467a55c... 2026-02-23T11:54:37.665 DEBUG:teuthology.orchestra.run.trial186:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2026-02-23T11:54:37.745 INFO:teuthology.orchestra.run.trial186.stdout:[global] 2026-02-23T11:54:37.746 INFO:teuthology.orchestra.run.trial186.stdout: chdir = "" 2026-02-23T11:54:37.746 INFO:teuthology.orchestra.run.trial186.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2026-02-23T11:54:37.746 INFO:teuthology.orchestra.run.trial186.stdout: auth supported = none 2026-02-23T11:54:37.746 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:54:37.746 INFO:teuthology.orchestra.run.trial186.stdout: filestore xattr use omap = true 2026-02-23T11:54:37.746 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:54:37.746 INFO:teuthology.orchestra.run.trial186.stdout: mon clock drift allowed = 1.000 2026-02-23T11:54:37.746 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:54:37.746 INFO:teuthology.orchestra.run.trial186.stdout: osd crush chooseleaf type = 0 2026-02-23T11:54:37.746 INFO:teuthology.orchestra.run.trial186.stdout: auth debug = true 2026-02-23T11:54:37.746 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:54:37.746 INFO:teuthology.orchestra.run.trial186.stdout: ms die on old message = true 2026-02-23T11:54:37.746 INFO:teuthology.orchestra.run.trial186.stdout: ms die on bug = true 2026-02-23T11:54:37.746 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:54:37.746 INFO:teuthology.orchestra.run.trial186.stdout: mon max pg per osd = 10000 # >= luminous 2026-02-23T11:54:37.746 INFO:teuthology.orchestra.run.trial186.stdout: mon pg warn max object skew = 0 2026-02-23T11:54:37.746 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:54:37.746 INFO:teuthology.orchestra.run.trial186.stdout: # disable pg_autoscaler by default for new pools 2026-02-23T11:54:37.746 INFO:teuthology.orchestra.run.trial186.stdout: osd_pool_default_pg_autoscale_mode = off 2026-02-23T11:54:37.746 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:54:37.746 INFO:teuthology.orchestra.run.trial186.stdout: osd pool default size = 2 2026-02-23T11:54:37.746 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:54:37.746 INFO:teuthology.orchestra.run.trial186.stdout: mon osd allow primary affinity = true 2026-02-23T11:54:37.746 INFO:teuthology.orchestra.run.trial186.stdout: mon osd allow pg remap = true 2026-02-23T11:54:37.746 INFO:teuthology.orchestra.run.trial186.stdout: mon warn on legacy crush tunables = false 2026-02-23T11:54:37.746 INFO:teuthology.orchestra.run.trial186.stdout: mon warn on crush straw calc version zero = false 2026-02-23T11:54:37.746 INFO:teuthology.orchestra.run.trial186.stdout: mon warn on no sortbitwise = false 2026-02-23T11:54:37.746 INFO:teuthology.orchestra.run.trial186.stdout: mon warn on osd down out interval zero = false 2026-02-23T11:54:37.746 INFO:teuthology.orchestra.run.trial186.stdout: mon warn on too few osds = false 2026-02-23T11:54:37.746 INFO:teuthology.orchestra.run.trial186.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2026-02-23T11:54:37.746 INFO:teuthology.orchestra.run.trial186.stdout: mon_warn_on_pool_no_redundancy = false 2026-02-23T11:54:37.746 INFO:teuthology.orchestra.run.trial186.stdout: mon_allow_pool_size_one = true 2026-02-23T11:54:37.746 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:54:37.746 INFO:teuthology.orchestra.run.trial186.stdout: osd pool default erasure code profile = plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2026-02-23T11:54:37.746 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:54:37.747 INFO:teuthology.orchestra.run.trial186.stdout: osd default data pool replay window = 5 2026-02-23T11:54:37.747 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:54:37.747 INFO:teuthology.orchestra.run.trial186.stdout: mon allow pool delete = true 2026-02-23T11:54:37.747 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:54:37.747 INFO:teuthology.orchestra.run.trial186.stdout: mon cluster log file level = debug 2026-02-23T11:54:37.747 INFO:teuthology.orchestra.run.trial186.stdout: debug asserts on shutdown = true 2026-02-23T11:54:37.747 INFO:teuthology.orchestra.run.trial186.stdout: mon health detail to clog = false 2026-02-23T11:54:37.747 INFO:teuthology.orchestra.run.trial186.stdout: mon host = 10.20.193.186 2026-02-23T11:54:37.747 INFO:teuthology.orchestra.run.trial186.stdout: auth client required = none 2026-02-23T11:54:37.747 INFO:teuthology.orchestra.run.trial186.stdout: auth cluster required = none 2026-02-23T11:54:37.747 INFO:teuthology.orchestra.run.trial186.stdout: auth service required = none 2026-02-23T11:54:37.747 INFO:teuthology.orchestra.run.trial186.stdout: debug asok = 0/0 2026-02-23T11:54:37.747 INFO:teuthology.orchestra.run.trial186.stdout: debug auth = 0/0 2026-02-23T11:54:37.747 INFO:teuthology.orchestra.run.trial186.stdout: debug bdev = 0/0 2026-02-23T11:54:37.747 INFO:teuthology.orchestra.run.trial186.stdout: debug bluefs = 0/0 2026-02-23T11:54:37.747 INFO:teuthology.orchestra.run.trial186.stdout: debug bluestore = 0/0 2026-02-23T11:54:37.747 INFO:teuthology.orchestra.run.trial186.stdout: debug buffer = 0/0 2026-02-23T11:54:37.747 INFO:teuthology.orchestra.run.trial186.stdout: debug civetweb = 0/0 2026-02-23T11:54:37.747 INFO:teuthology.orchestra.run.trial186.stdout: debug client = 0/0 2026-02-23T11:54:37.747 INFO:teuthology.orchestra.run.trial186.stdout: debug compressor = 0/0 2026-02-23T11:54:37.747 INFO:teuthology.orchestra.run.trial186.stdout: debug context = 0/0 2026-02-23T11:54:37.747 INFO:teuthology.orchestra.run.trial186.stdout: debug crush = 0/0 2026-02-23T11:54:37.747 INFO:teuthology.orchestra.run.trial186.stdout: debug crypto = 0/0 2026-02-23T11:54:37.747 INFO:teuthology.orchestra.run.trial186.stdout: debug dpdk = 0/0 2026-02-23T11:54:37.747 INFO:teuthology.orchestra.run.trial186.stdout: debug eventtrace = 0/0 2026-02-23T11:54:37.747 INFO:teuthology.orchestra.run.trial186.stdout: debug filer = 0/0 2026-02-23T11:54:37.747 INFO:teuthology.orchestra.run.trial186.stdout: debug filestore = 0/0 2026-02-23T11:54:37.747 INFO:teuthology.orchestra.run.trial186.stdout: debug finisher = 0/0 2026-02-23T11:54:37.747 INFO:teuthology.orchestra.run.trial186.stdout: debug fuse = 0/0 2026-02-23T11:54:37.747 INFO:teuthology.orchestra.run.trial186.stdout: debug heartbeatmap = 0/0 2026-02-23T11:54:37.747 INFO:teuthology.orchestra.run.trial186.stdout: debug javaclient = 0/0 2026-02-23T11:54:37.747 INFO:teuthology.orchestra.run.trial186.stdout: debug journal = 0/0 2026-02-23T11:54:37.747 INFO:teuthology.orchestra.run.trial186.stdout: debug journaler = 0/0 2026-02-23T11:54:37.747 INFO:teuthology.orchestra.run.trial186.stdout: debug kstore = 0/0 2026-02-23T11:54:37.747 INFO:teuthology.orchestra.run.trial186.stdout: debug leveldb = 0/0 2026-02-23T11:54:37.747 INFO:teuthology.orchestra.run.trial186.stdout: debug lockdep = 0/0 2026-02-23T11:54:37.747 INFO:teuthology.orchestra.run.trial186.stdout: debug mds = 0/0 2026-02-23T11:54:37.747 INFO:teuthology.orchestra.run.trial186.stdout: debug mds balancer = 0/0 2026-02-23T11:54:37.747 INFO:teuthology.orchestra.run.trial186.stdout: debug mds locker = 0/0 2026-02-23T11:54:37.747 INFO:teuthology.orchestra.run.trial186.stdout: debug mds log = 0/0 2026-02-23T11:54:37.747 INFO:teuthology.orchestra.run.trial186.stdout: debug mds log expire = 0/0 2026-02-23T11:54:37.747 INFO:teuthology.orchestra.run.trial186.stdout: debug mds migrator = 0/0 2026-02-23T11:54:37.748 INFO:teuthology.orchestra.run.trial186.stdout: debug memdb = 0/0 2026-02-23T11:54:37.748 INFO:teuthology.orchestra.run.trial186.stdout: debug mgr = 0/0 2026-02-23T11:54:37.748 INFO:teuthology.orchestra.run.trial186.stdout: debug mgrc = 0/0 2026-02-23T11:54:37.748 INFO:teuthology.orchestra.run.trial186.stdout: debug mon = 0/0 2026-02-23T11:54:37.748 INFO:teuthology.orchestra.run.trial186.stdout: debug monc = 0/0 2026-02-23T11:54:37.748 INFO:teuthology.orchestra.run.trial186.stdout: debug ms = 0/0 2026-02-23T11:54:37.748 INFO:teuthology.orchestra.run.trial186.stdout: debug objclass = 0/0 2026-02-23T11:54:37.748 INFO:teuthology.orchestra.run.trial186.stdout: debug objectcacher = 0/0 2026-02-23T11:54:37.748 INFO:teuthology.orchestra.run.trial186.stdout: debug objecter = 0/0 2026-02-23T11:54:37.748 INFO:teuthology.orchestra.run.trial186.stdout: debug optracker = 0/0 2026-02-23T11:54:37.748 INFO:teuthology.orchestra.run.trial186.stdout: debug osd = 0/0 2026-02-23T11:54:37.748 INFO:teuthology.orchestra.run.trial186.stdout: debug paxos = 0/0 2026-02-23T11:54:37.748 INFO:teuthology.orchestra.run.trial186.stdout: debug perfcounter = 0/0 2026-02-23T11:54:37.748 INFO:teuthology.orchestra.run.trial186.stdout: debug rados = 0/0 2026-02-23T11:54:37.748 INFO:teuthology.orchestra.run.trial186.stdout: debug rbd = 0/0 2026-02-23T11:54:37.748 INFO:teuthology.orchestra.run.trial186.stdout: debug rbd mirror = 0/0 2026-02-23T11:54:37.748 INFO:teuthology.orchestra.run.trial186.stdout: debug rbd replay = 0/0 2026-02-23T11:54:37.748 INFO:teuthology.orchestra.run.trial186.stdout: debug refs = 0/0 2026-02-23T11:54:37.748 INFO:teuthology.orchestra.run.trial186.stdout: debug rgw = 0/0 2026-02-23T11:54:37.748 INFO:teuthology.orchestra.run.trial186.stdout: debug rgw lifecycle = 0/0 2026-02-23T11:54:37.748 INFO:teuthology.orchestra.run.trial186.stdout: debug rgw notification = 0/0 2026-02-23T11:54:37.748 INFO:teuthology.orchestra.run.trial186.stdout: debug rgw sync = 0/0 2026-02-23T11:54:37.748 INFO:teuthology.orchestra.run.trial186.stdout: debug rocksdb = 0/0 2026-02-23T11:54:37.748 INFO:teuthology.orchestra.run.trial186.stdout: debug striper = 0/0 2026-02-23T11:54:37.748 INFO:teuthology.orchestra.run.trial186.stdout: debug throttle = 0/0 2026-02-23T11:54:37.748 INFO:teuthology.orchestra.run.trial186.stdout: debug timer = 0/0 2026-02-23T11:54:37.748 INFO:teuthology.orchestra.run.trial186.stdout: debug tp = 0/0 2026-02-23T11:54:37.748 INFO:teuthology.orchestra.run.trial186.stdout: enable experimental unrecoverable data corrupting features = crimson 2026-02-23T11:54:37.748 INFO:teuthology.orchestra.run.trial186.stdout: ms client mode = crc 2026-02-23T11:54:37.748 INFO:teuthology.orchestra.run.trial186.stdout: ms cluster mode = crc 2026-02-23T11:54:37.748 INFO:teuthology.orchestra.run.trial186.stdout: ms mon client mode = crc 2026-02-23T11:54:37.748 INFO:teuthology.orchestra.run.trial186.stdout: ms mon cluster mode = crc 2026-02-23T11:54:37.748 INFO:teuthology.orchestra.run.trial186.stdout: ms mon service mode = crc 2026-02-23T11:54:37.748 INFO:teuthology.orchestra.run.trial186.stdout: ms service mode = crc 2026-02-23T11:54:37.748 INFO:teuthology.orchestra.run.trial186.stdout: fsid = d91d7eac-82c0-417c-a119-6b75d467a55c 2026-02-23T11:54:37.748 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:54:37.748 INFO:teuthology.orchestra.run.trial186.stdout:[osd] 2026-02-23T11:54:37.748 INFO:teuthology.orchestra.run.trial186.stdout: osd journal size = 100 2026-02-23T11:54:37.748 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:54:37.748 INFO:teuthology.orchestra.run.trial186.stdout: osd scrub load threshold = 5.0 2026-02-23T11:54:37.748 INFO:teuthology.orchestra.run.trial186.stdout: osd scrub max interval = 600 2026-02-23T11:54:37.748 INFO:teuthology.orchestra.run.trial186.stdout: osd mclock profile = high_recovery_ops 2026-02-23T11:54:37.748 INFO:teuthology.orchestra.run.trial186.stdout: osd mclock skip benchmark = true 2026-02-23T11:54:37.749 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:54:37.749 INFO:teuthology.orchestra.run.trial186.stdout: osd recover clone overlap = true 2026-02-23T11:54:37.749 INFO:teuthology.orchestra.run.trial186.stdout: osd recovery max chunk = 1048576 2026-02-23T11:54:37.749 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:54:37.749 INFO:teuthology.orchestra.run.trial186.stdout: osd debug shutdown = true 2026-02-23T11:54:37.749 INFO:teuthology.orchestra.run.trial186.stdout: osd debug op order = true 2026-02-23T11:54:37.749 INFO:teuthology.orchestra.run.trial186.stdout: osd debug verify stray on activate = true 2026-02-23T11:54:37.749 INFO:teuthology.orchestra.run.trial186.stdout: osd debug trim objects = true 2026-02-23T11:54:37.749 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:54:37.749 INFO:teuthology.orchestra.run.trial186.stdout: osd open classes on start = true 2026-02-23T11:54:37.749 INFO:teuthology.orchestra.run.trial186.stdout: osd debug pg log writeout = true 2026-02-23T11:54:37.749 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:54:37.749 INFO:teuthology.orchestra.run.trial186.stdout: osd deep scrub update digest min age = 30 2026-02-23T11:54:37.749 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:54:37.749 INFO:teuthology.orchestra.run.trial186.stdout: osd map max advance = 10 2026-02-23T11:54:37.749 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:54:37.749 INFO:teuthology.orchestra.run.trial186.stdout: journal zero on create = true 2026-02-23T11:54:37.749 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:54:37.749 INFO:teuthology.orchestra.run.trial186.stdout: filestore ondisk finisher threads = 3 2026-02-23T11:54:37.749 INFO:teuthology.orchestra.run.trial186.stdout: filestore apply finisher threads = 3 2026-02-23T11:54:37.749 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:54:37.749 INFO:teuthology.orchestra.run.trial186.stdout: bdev debug aio = true 2026-02-23T11:54:37.749 INFO:teuthology.orchestra.run.trial186.stdout: osd debug misdirected ops = true 2026-02-23T11:54:37.749 INFO:teuthology.orchestra.run.trial186.stdout: debug monc = 20 2026-02-23T11:54:37.749 INFO:teuthology.orchestra.run.trial186.stdout: crimson cpu num = 4 2026-02-23T11:54:37.749 INFO:teuthology.orchestra.run.trial186.stdout: crimson osd obc lru size = 10 2026-02-23T11:54:37.749 INFO:teuthology.orchestra.run.trial186.stdout: debug filestore = 0/0 2026-02-23T11:54:37.749 INFO:teuthology.orchestra.run.trial186.stdout: debug journal = 0/0 2026-02-23T11:54:37.749 INFO:teuthology.orchestra.run.trial186.stdout: debug ms = 0/0 2026-02-23T11:54:37.749 INFO:teuthology.orchestra.run.trial186.stdout: debug osd = 0/0 2026-02-23T11:54:37.749 INFO:teuthology.orchestra.run.trial186.stdout: debug seastore = 20 2026-02-23T11:54:37.749 INFO:teuthology.orchestra.run.trial186.stdout: debug seastore backref = 20 2026-02-23T11:54:37.749 INFO:teuthology.orchestra.run.trial186.stdout: debug seastore cache = 20 2026-02-23T11:54:37.749 INFO:teuthology.orchestra.run.trial186.stdout: debug seastore cleaner = 20 2026-02-23T11:54:37.749 INFO:teuthology.orchestra.run.trial186.stdout: debug seastore device = 20 2026-02-23T11:54:37.749 INFO:teuthology.orchestra.run.trial186.stdout: debug seastore epm = 20 2026-02-23T11:54:37.749 INFO:teuthology.orchestra.run.trial186.stdout: debug seastore fixedkv tree = 20 2026-02-23T11:54:37.749 INFO:teuthology.orchestra.run.trial186.stdout: debug seastore journal = 20 2026-02-23T11:54:37.749 INFO:teuthology.orchestra.run.trial186.stdout: debug seastore lba = 20 2026-02-23T11:54:37.749 INFO:teuthology.orchestra.run.trial186.stdout: debug seastore odata = 20 2026-02-23T11:54:37.749 INFO:teuthology.orchestra.run.trial186.stdout: debug seastore omap = 20 2026-02-23T11:54:37.749 INFO:teuthology.orchestra.run.trial186.stdout: debug seastore onode = 20 2026-02-23T11:54:37.750 INFO:teuthology.orchestra.run.trial186.stdout: debug seastore t = 20 2026-02-23T11:54:37.750 INFO:teuthology.orchestra.run.trial186.stdout: debug seastore tm = 20 2026-02-23T11:54:37.750 INFO:teuthology.orchestra.run.trial186.stdout: osd objectstore = seastore 2026-02-23T11:54:37.750 INFO:teuthology.orchestra.run.trial186.stdout: osd shutdown pgref assert = True 2026-02-23T11:54:37.750 INFO:teuthology.orchestra.run.trial186.stdout: seastore cachepin size pershard = 64M 2026-02-23T11:54:37.750 INFO:teuthology.orchestra.run.trial186.stdout: seastore cachepin type = 2Q 2026-02-23T11:54:37.750 INFO:teuthology.orchestra.run.trial186.stdout: seastore max concurrent transactions = 8 2026-02-23T11:54:37.750 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:54:37.750 INFO:teuthology.orchestra.run.trial186.stdout:[mgr] 2026-02-23T11:54:37.750 INFO:teuthology.orchestra.run.trial186.stdout: debug ms = 1 2026-02-23T11:54:37.750 INFO:teuthology.orchestra.run.trial186.stdout: debug mgr = 20 2026-02-23T11:54:37.750 INFO:teuthology.orchestra.run.trial186.stdout: debug mon = 20 2026-02-23T11:54:37.750 INFO:teuthology.orchestra.run.trial186.stdout: debug auth = 20 2026-02-23T11:54:37.750 INFO:teuthology.orchestra.run.trial186.stdout: mon reweight min pgs per osd = 4 2026-02-23T11:54:37.750 INFO:teuthology.orchestra.run.trial186.stdout: mon reweight min bytes per osd = 10 2026-02-23T11:54:37.750 INFO:teuthology.orchestra.run.trial186.stdout: mgr/telemetry/nag = false 2026-02-23T11:54:37.750 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:54:37.750 INFO:teuthology.orchestra.run.trial186.stdout:[mon] 2026-02-23T11:54:37.750 INFO:teuthology.orchestra.run.trial186.stdout: debug ms = 0/0 2026-02-23T11:54:37.750 INFO:teuthology.orchestra.run.trial186.stdout: debug mon = 0/0 2026-02-23T11:54:37.750 INFO:teuthology.orchestra.run.trial186.stdout: debug paxos = 0/0 2026-02-23T11:54:37.750 INFO:teuthology.orchestra.run.trial186.stdout: debug auth = 20 2026-02-23T11:54:37.750 INFO:teuthology.orchestra.run.trial186.stdout: mon data avail warn = 5 2026-02-23T11:54:37.750 INFO:teuthology.orchestra.run.trial186.stdout: mon mgr mkfs grace = 240 2026-02-23T11:54:37.750 INFO:teuthology.orchestra.run.trial186.stdout: mon reweight min pgs per osd = 4 2026-02-23T11:54:37.750 INFO:teuthology.orchestra.run.trial186.stdout: mon osd reporter subtree level = osd 2026-02-23T11:54:37.750 INFO:teuthology.orchestra.run.trial186.stdout: mon osd prime pg temp = true 2026-02-23T11:54:37.750 INFO:teuthology.orchestra.run.trial186.stdout: mon reweight min bytes per osd = 10 2026-02-23T11:54:37.750 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:54:37.750 INFO:teuthology.orchestra.run.trial186.stdout: # rotate auth tickets quickly to exercise renewal paths 2026-02-23T11:54:37.750 INFO:teuthology.orchestra.run.trial186.stdout: auth mon ticket ttl = 660 # 11m 2026-02-23T11:54:37.750 INFO:teuthology.orchestra.run.trial186.stdout: auth service ticket ttl = 240 # 4m 2026-02-23T11:54:37.750 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:54:37.750 INFO:teuthology.orchestra.run.trial186.stdout: # don't complain about insecure global_id in the test suite 2026-02-23T11:54:37.750 INFO:teuthology.orchestra.run.trial186.stdout: mon_warn_on_insecure_global_id_reclaim = false 2026-02-23T11:54:37.750 INFO:teuthology.orchestra.run.trial186.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2026-02-23T11:54:37.750 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:54:37.750 INFO:teuthology.orchestra.run.trial186.stdout: # 1m isn't quite enough 2026-02-23T11:54:37.750 INFO:teuthology.orchestra.run.trial186.stdout: mon_down_mkfs_grace = 2m 2026-02-23T11:54:37.750 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:54:37.750 INFO:teuthology.orchestra.run.trial186.stdout: mon_warn_on_filestore_osds = false 2026-02-23T11:54:37.750 INFO:teuthology.orchestra.run.trial186.stdout: osd pool default crimson = True 2026-02-23T11:54:37.750 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:54:37.751 INFO:teuthology.orchestra.run.trial186.stdout:[client] 2026-02-23T11:54:37.751 INFO:teuthology.orchestra.run.trial186.stdout: rgw cache enabled = true 2026-02-23T11:54:37.751 INFO:teuthology.orchestra.run.trial186.stdout: rgw enable ops log = true 2026-02-23T11:54:37.751 INFO:teuthology.orchestra.run.trial186.stdout: rgw enable usage log = true 2026-02-23T11:54:37.751 INFO:teuthology.orchestra.run.trial186.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2026-02-23T11:54:37.751 INFO:teuthology.orchestra.run.trial186.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2026-02-23T11:54:37.751 INFO:teuthology.orchestra.run.trial186.stdout:[mon.a] 2026-02-23T11:54:37.756 INFO:tasks.ceph:Creating admin key on mon.a... 2026-02-23T11:54:37.756 DEBUG:teuthology.orchestra.run.trial186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2026-02-23T11:54:37.848 INFO:tasks.ceph:Copying monmap to all nodes... 2026-02-23T11:54:37.848 DEBUG:teuthology.orchestra.run.trial186:> set -ex 2026-02-23T11:54:37.848 DEBUG:teuthology.orchestra.run.trial186:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2026-02-23T11:54:37.863 DEBUG:teuthology.orchestra.run.trial186:> set -ex 2026-02-23T11:54:37.863 DEBUG:teuthology.orchestra.run.trial186:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2026-02-23T11:54:37.919 INFO:tasks.ceph:Sending monmap to node ubuntu@trial186.front.sepia.ceph.com 2026-02-23T11:54:37.919 DEBUG:teuthology.orchestra.run.trial186:> set -ex 2026-02-23T11:54:37.919 DEBUG:teuthology.orchestra.run.trial186:> sudo dd of=/etc/ceph/ceph.keyring 2026-02-23T11:54:37.919 DEBUG:teuthology.orchestra.run.trial186:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-23T11:54:37.992 DEBUG:teuthology.orchestra.run.trial186:> set -ex 2026-02-23T11:54:37.993 DEBUG:teuthology.orchestra.run.trial186:> dd of=/home/ubuntu/cephtest/ceph.monmap 2026-02-23T11:54:38.048 INFO:tasks.ceph:Setting up mon nodes... 2026-02-23T11:54:38.048 INFO:tasks.ceph:Setting up mgr nodes... 2026-02-23T11:54:38.048 DEBUG:teuthology.orchestra.run.trial186:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2026-02-23T11:54:38.140 INFO:teuthology.orchestra.run.trial186.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2026-02-23T11:54:38.143 INFO:tasks.ceph:Setting up mds nodes... 2026-02-23T11:54:38.143 INFO:tasks.ceph_client:Setting up client nodes... 2026-02-23T11:54:38.143 DEBUG:teuthology.orchestra.run.trial186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2026-02-23T11:54:38.183 INFO:teuthology.orchestra.run.trial186.stdout:creating /etc/ceph/ceph.client.0.keyring 2026-02-23T11:54:38.195 INFO:tasks.ceph:Running mkfs on osd nodes... 2026-02-23T11:54:38.195 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@trial186.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2026-02-23T11:54:38.195 DEBUG:teuthology.orchestra.run.trial186:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2026-02-23T11:54:38.259 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-23T11:54:38.259 INFO:tasks.ceph:role: osd.0 2026-02-23T11:54:38.260 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@trial186.front.sepia.ceph.com 2026-02-23T11:54:38.260 DEBUG:teuthology.orchestra.run.trial186:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2026-02-23T11:54:38.325 INFO:teuthology.orchestra.run.trial186.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=45875200 blks 2026-02-23T11:54:38.325 INFO:teuthology.orchestra.run.trial186.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-23T11:54:38.325 INFO:teuthology.orchestra.run.trial186.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-23T11:54:38.325 INFO:teuthology.orchestra.run.trial186.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-23T11:54:38.325 INFO:teuthology.orchestra.run.trial186.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-23T11:54:38.325 INFO:teuthology.orchestra.run.trial186.stdout: = sunit=0 swidth=0 blks 2026-02-23T11:54:38.325 INFO:teuthology.orchestra.run.trial186.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-23T11:54:38.325 INFO:teuthology.orchestra.run.trial186.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-23T11:54:38.325 INFO:teuthology.orchestra.run.trial186.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-23T11:54:38.325 INFO:teuthology.orchestra.run.trial186.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-23T11:54:38.888 INFO:teuthology.orchestra.run.trial186.stdout:Discarding blocks...Done. 2026-02-23T11:54:38.890 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@trial186.front.sepia.ceph.com -o noatime 2026-02-23T11:54:38.890 DEBUG:teuthology.orchestra.run.trial186:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2026-02-23T11:54:38.961 DEBUG:teuthology.orchestra.run.trial186:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2026-02-23T11:54:39.029 DEBUG:teuthology.orchestra.run.trial186:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2026-02-23T11:54:39.095 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-23T11:54:39.095 INFO:tasks.ceph:role: osd.1 2026-02-23T11:54:39.095 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@trial186.front.sepia.ceph.com 2026-02-23T11:54:39.095 DEBUG:teuthology.orchestra.run.trial186:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2026-02-23T11:54:39.161 INFO:teuthology.orchestra.run.trial186.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=45875200 blks 2026-02-23T11:54:39.161 INFO:teuthology.orchestra.run.trial186.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-23T11:54:39.161 INFO:teuthology.orchestra.run.trial186.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-23T11:54:39.161 INFO:teuthology.orchestra.run.trial186.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-23T11:54:39.161 INFO:teuthology.orchestra.run.trial186.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-23T11:54:39.161 INFO:teuthology.orchestra.run.trial186.stdout: = sunit=0 swidth=0 blks 2026-02-23T11:54:39.161 INFO:teuthology.orchestra.run.trial186.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-23T11:54:39.161 INFO:teuthology.orchestra.run.trial186.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-23T11:54:39.161 INFO:teuthology.orchestra.run.trial186.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-23T11:54:39.161 INFO:teuthology.orchestra.run.trial186.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-23T11:54:39.733 INFO:teuthology.orchestra.run.trial186.stdout:Discarding blocks...Done. 2026-02-23T11:54:39.735 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@trial186.front.sepia.ceph.com -o noatime 2026-02-23T11:54:39.735 DEBUG:teuthology.orchestra.run.trial186:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2026-02-23T11:54:39.806 DEBUG:teuthology.orchestra.run.trial186:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2026-02-23T11:54:39.872 DEBUG:teuthology.orchestra.run.trial186:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2026-02-23T11:54:39.943 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-23T11:54:39.943 INFO:tasks.ceph:role: osd.2 2026-02-23T11:54:39.943 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@trial186.front.sepia.ceph.com 2026-02-23T11:54:39.944 DEBUG:teuthology.orchestra.run.trial186:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2026-02-23T11:54:40.010 INFO:teuthology.orchestra.run.trial186.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=45875200 blks 2026-02-23T11:54:40.011 INFO:teuthology.orchestra.run.trial186.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-23T11:54:40.011 INFO:teuthology.orchestra.run.trial186.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-23T11:54:40.011 INFO:teuthology.orchestra.run.trial186.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-23T11:54:40.011 INFO:teuthology.orchestra.run.trial186.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-23T11:54:40.011 INFO:teuthology.orchestra.run.trial186.stdout: = sunit=0 swidth=0 blks 2026-02-23T11:54:40.011 INFO:teuthology.orchestra.run.trial186.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-23T11:54:40.011 INFO:teuthology.orchestra.run.trial186.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-23T11:54:40.011 INFO:teuthology.orchestra.run.trial186.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-23T11:54:40.011 INFO:teuthology.orchestra.run.trial186.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-23T11:54:40.574 INFO:teuthology.orchestra.run.trial186.stdout:Discarding blocks...Done. 2026-02-23T11:54:40.576 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@trial186.front.sepia.ceph.com -o noatime 2026-02-23T11:54:40.576 DEBUG:teuthology.orchestra.run.trial186:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2026-02-23T11:54:40.645 DEBUG:teuthology.orchestra.run.trial186:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2026-02-23T11:54:40.713 DEBUG:teuthology.orchestra.run.trial186:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-23T11:54:40.865 INFO:teuthology.orchestra.run.trial186.stderr:INFO  osd - OSD::main: parsing early config 2026-02-23T11:54:40.921 INFO:teuthology.orchestra.run.trial186.stderr:INFO 2026-02-23 11:54:40,919 seastar - Reactor backend: linux-aio 2026-02-23T11:54:40.931 INFO:teuthology.orchestra.run.trial186.stderr:INFO 2026-02-23 11:54:40,929 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-23T11:54:40.931 INFO:teuthology.orchestra.run.trial186.stderr:INFO 2026-02-23 11:54:40,929 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-23T11:54:40.933 INFO:teuthology.orchestra.run.trial186.stderr:==38991==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-23T11:54:40.993 INFO:teuthology.orchestra.run.trial186.stderr:DEBUG 2026-02-23 11:54:40,991 [shard 0:main] seastar - reactor::drain 2026-02-23T11:54:43.091 INFO:teuthology.orchestra.run.trial186.stderr:----------------------------------------------------- 2026-02-23T11:54:43.091 INFO:teuthology.orchestra.run.trial186.stderr:Suppressions used: 2026-02-23T11:54:43.091 INFO:teuthology.orchestra.run.trial186.stderr: count bytes template 2026-02-23T11:54:43.091 INFO:teuthology.orchestra.run.trial186.stderr: 1 8 MallocExtension::Register 2026-02-23T11:54:43.091 INFO:teuthology.orchestra.run.trial186.stderr:----------------------------------------------------- 2026-02-23T11:54:43.091 INFO:teuthology.orchestra.run.trial186.stderr: 2026-02-23T11:54:43.110 INFO:teuthology.orchestra.run.trial186.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-23T11:54:43.110 INFO:teuthology.orchestra.run.trial186.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-23T11:54:43.151 INFO:teuthology.orchestra.run.trial186.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-23T11:54:43.151 INFO:teuthology.orchestra.run.trial186.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-23T11:54:43.158 INFO:teuthology.orchestra.run.trial186.stderr:INFO 2026-02-23 11:54:43,156 seastar - Reactor backend: linux-aio 2026-02-23T11:54:43.174 INFO:teuthology.orchestra.run.trial186.stderr:INFO 2026-02-23 11:54:43,172 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-23T11:54:43.174 INFO:teuthology.orchestra.run.trial186.stderr:INFO 2026-02-23 11:54:43,172 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-23T11:54:43.178 INFO:teuthology.orchestra.run.trial186.stderr:==38989==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-23T11:54:43.178 INFO:teuthology.orchestra.run.trial186.stderr:INFO 2026-02-23 11:54:43,176 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-23T11:54:43.321 INFO:teuthology.orchestra.run.trial186.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1297e4 /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x15e37 0x9b7df96 0x9b7e348 0x9b8006f 0x9c8eee8 0x9c8f661 0x9c8fb4c 0x9c8fedf 0x9ca2b6a 0x9c052e1 0x9c042e1 0x7be8837 0x7bf23a6 0x7a51d33 0x7a8d6c5 0x711ff75 0x6f9f19a 0x6fa2f9b 0x6fa8b45 0x7482004 0x72a89fa 0x72a8afc 0x72a8bf4 0x735cb16 0x735cbcf 0x735cd82 0x735ceba 0x735cf35 0x735cfba 0x735d115 0x7425e13 0x748208d 0x728f241 0x72a6bae 0x73b5274 0x7432e6c 0x7432ee7 0x7432f62 0x74eccd2 0x75e6286 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:54:43.321 INFO:teuthology.orchestra.run.trial186.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x129859 /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x157c7 0x9b7e39c 0x9b7e5fe 0x9b7ebc0 0x9b83ce5 0x9b843c5 0x9c9ff46 0x9ca45f1 0x9c052e1 0x9c042e1 0x7be8837 0x7bf23a6 0x7a51d33 0x7a8d6c5 0x711ff75 0x6f9f19a 0x6fa2f9b 0x6fa8b45 0x7482004 0x72a89fa 0x72a8afc 0x72a8bf4 0x735cb16 0x735cbcf 0x735cd82 0x735ceba 0x735cf35 0x735cfba 0x735d115 0x7425e13 0x748208d 0x728f241 0x72a6bae 0x73b5274 0x7432e6c 0x7432ee7 0x7432f62 0x74eccd2 0x75e6286 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:54:43.321 INFO:teuthology.orchestra.run.trial186.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/ld-linux-x86-64.so.2+0x16ec6 /lib64/libasan.so.8+0xc65b1 /lib64/libasan.so.8+0x15197 0x1125e3c 0xf2e3c3 0x9b7ddd8 0x9b7e0d5 0x9b7e348 0x9b7f348 0x9b84049 0x9b843c5 0x9c9ff46 0x9ca45f1 0x9c052e1 0x9c042e1 0x7be8837 0x7bf23a6 0x7a51d33 0x7a8d6c5 0x711ff75 0x6f9f19a 0x6fa2f9b 0x6fa8b45 0x7482004 0x72a89fa 0x72a8afc 0x72a8bf4 0x735cb16 0x735cbcf 0x735cd82 0x735ceba 0x735cf35 0x735cfba 0x735d115 0x7425e13 0x748208d 0x728f241 0x72a6bae 0x73b5274 0x7432e6c 0x7432ee7 0x7432f62 0x74eccd2 0x75e6286 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:54:43.353 INFO:teuthology.orchestra.run.trial186.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc597 /lib64/libasan.so.8+0x91c1 /lib64/libasan.so.8+0xee7dc 0x133ef18 0x2a58640 0x7f7f804 0x7f7fdba 0x7f7fee6 0x7f82371 0x7f82fc9 0x7dc31d8 0x7e39297 0x7e5ae04 0x8f80d69 0x8f836d7 0x787a27f 0x78aad7c 0x6fa0365 0x75590f6 0x755924a 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9e29e18 0x9c44517 0x9c44d52 0xecf43f /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb46f54 2026-02-23T11:54:43.354 INFO:teuthology.orchestra.run.trial186.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff8a475c69 0xffffffff8a61f8c7 0xffffffff8a61fb20 0xffffffff8a49525e 0xffffffff8a7e61a9 0xffffffff8a7c9715 0xffffffff8a7d3aeb 0xffffffff8a7d3f4b 0xffffffff8a4910d2 0xffffffff8b14d451 0xffffffff8b200ba2 2026-02-23T11:54:43.420 INFO:teuthology.orchestra.run.trial186.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc65c2 /lib64/libasan.so.8+0x15197 0x2a5304b 0x7d7e215 0x7d9da0c 0x7e39ed3 0x7dbe035 0x78750f3 0x7875de8 0x79b38b2 0x7833a0a 0x78a9398 0x78f345f 0x78f36d8 0x79a02cc 0x79a042b 0x79b3a15 0x7864b7c 0x799073e 0x79907ce 0x7990872 0x73eb37a 0x7409299 0x740c6cb 0x75440fc 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9e29e18 0x9c44517 0x9c44d52 0xecf43f /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb46f54 2026-02-23T11:54:43.458 INFO:teuthology.orchestra.run.trial186.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x8cbef /lib64/libasan.so.8+0x10069c /lib64/libasan.so.8+0xc65b9 /lib64/libasan.so.8+0x15197 0x2a5304b 0x7d7e215 0x7d9da0c 0x7e39ed3 0x7dbe035 0x78750f3 0x7875de8 0x79b38b2 0x7833a0a 0x78a9398 0x78f345f 0x78f36d8 0x79a02cc 0x79a042b 0x79b3a15 0x7864b7c 0x799073e 0x79907ce 0x7990872 0x73eb37a 0x7409299 0x740c6cb 0x75440fc 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:54:43.460 INFO:teuthology.orchestra.run.trial186.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f 0x7d7e226 0x7d9da0c 0x7e39ed3 0x7dbe035 0x78750f3 0x7875de8 0x79b38b2 0x7833a0a 0x78a9398 0x78f345f 0x78f36d8 0x79a02cc 0x79a042b 0x79b3a15 0x7864b7c 0x799073e 0x79907ce 0x7990872 0x73eb37a 0x7409299 0x740c6cb 0x75440fc 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:54:43.460 INFO:teuthology.orchestra.run.trial186.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc65d9 /lib64/libasan.so.8+0x15197 0x2a5304b 0x7d7e215 0x7d9da0c 0x7e39ed3 0x7dbe035 0x78750f3 0x7875de8 0x79b38b2 0x7833a0a 0x78a9398 0x78f345f 0x78f36d8 0x79a02cc 0x79a042b 0x79b3a15 0x7864b7c 0x799073e 0x79907ce 0x7990872 0x73eb37a 0x7409299 0x740c6cb 0x75440fc 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:54:43.587 INFO:teuthology.orchestra.run.trial186.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f 0x316ca1e 0x9b7e336 0x9b7e5fe 0x9b7ebc0 0x9b83ce5 0x9b843c5 0x9c9ff46 0x9ca45f1 0x9c052e1 0x9c042e1 0x7be8837 0x7bf23a6 0x7a51d33 0x7a8d6c5 0x711ff75 0x6f3ec5e 0x6f40fc6 0x747ed38 0x72a6837 0x72a6939 0x72a6a31 0x735bce2 0x735bd9b 0x735bf4e 0x735c086 0x735c101 0x735c186 0x735c2e1 0x7425af5 0x747edc1 0x728e178 0x72a6bae 0x73b5274 0x7432e6c 0x7432ee7 0x7432f62 0x74eccd2 0x75e6286 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:54:43.587 INFO:teuthology.orchestra.run.trial186.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xb04bf /lib64/libasan.so.8+0xb1437 0xb69729 0x10912a0 0xf2e3ce 0x9b7ddd8 0x9b7e0d5 0x9b7e348 0x9b7e5fe 0x9b7f0ad 0x9b7f5ab 0x9b7fbdb 0x9b7fd45 0x9c8fd63 0x9ca2b6a 0x9c052e1 0x9c042e1 0x7be8837 0x7bf23a6 0x7a51d33 0x7a8d6c5 0x711ff75 0x6f3ec5e 0x6f40fc6 0x747ed38 0x72a6837 0x72a6939 0x72a6a31 0x735bce2 0x735bd9b 0x735bf4e 0x735c086 0x735c101 0x735c186 0x735c2e1 0x7425af5 0x747edc1 0x728e178 0x72a6bae 0x73b5274 0x7432e6c 0x7432ee7 0x7432f62 0x74eccd2 0x75e6286 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:54:43.587 INFO:teuthology.orchestra.run.trial186.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xf4e8f /lib64/libasan.so.8+0x10069c /lib64/libasan.so.8+0xc65b9 /lib64/libasan.so.8+0x157c7 0x35dbe24 0x35eaae0 0x35eabcc 0x3608f10 0x3609385 0x360943f 0x9c87d39 0x9c88176 0x9c9f909 0x9ca45f1 0x9c052e1 0x9c042e1 0x7be8837 0x7bf23a6 0x7a51d33 0x7a8d6c5 0x711ff75 0x6f3ec5e 0x6f40fc6 0x747ed38 0x72a6837 0x72a6939 0x72a6a31 0x735bce2 0x735bd9b 0x735bf4e 0x735c086 0x735c101 0x735c186 0x735c2e1 0x7425af5 0x747edc1 0x728e178 0x72a6bae 0x73b5274 0x7432e6c 0x7432ee7 0x7432f62 0x74eccd2 0x75e6286 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:54:43.669 INFO:teuthology.orchestra.run.trial186.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x15277 0x2a5304b 0x7d7e215 0x7d9da0c 0x7e39ed3 0x7dbe035 0x78750f3 0x7875de8 0x79b38b2 0x7833a0a 0x78a9398 0x78f345f 0x78f36d8 0x79a02cc 0x79a042b 0x79b3a15 0x7864b7c 0x799073e 0x79907ce 0x7990872 0x73eb37a 0x7409299 0x740c6cb 0x75440fc 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9e29e18 0x9c44517 0x9c44d52 0xecf43f /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb46f54 2026-02-23T11:54:43.683 INFO:teuthology.orchestra.run.trial186.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f 0x2a52e81 0x7d7e215 0x7d9da0c 0x7e39ed3 0x7dbe035 0x78750f3 0x7875de8 0x79b38b2 0x7833a0a 0x78a9398 0x78f345f 0x78f36d8 0x79a02cc 0x79a042b 0x79b3a15 0x7864b7c 0x799073e 0x79907ce 0x7990872 0x73eb37a 0x7409299 0x740c6cb 0x75440fc 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:54:43.683 INFO:teuthology.orchestra.run.trial186.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc65c2 /lib64/libasan.so.8+0x15197 0x2a5304b 0x7d7e215 0x7d9da0c 0x7e39ed3 0x7dbe035 0x78750f3 0x7875de8 0x79b38b2 0x7833a0a 0x78a9398 0x78f345f 0x78f36d8 0x79a02cc 0x79a042b 0x79b3a15 0x7864b7c 0x799073e 0x79907ce 0x7990872 0x73eb37a 0x7409299 0x740c6cb 0x75440fc 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:54:43.684 INFO:teuthology.orchestra.run.trial186.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc65c2 /lib64/libasan.so.8+0x15197 0x2a5304b 0x7d7e215 0x7d9da0c 0x7e39ed3 0x7dbe035 0x78750f3 0x7875de8 0x79b38b2 0x7833a0a 0x78a9398 0x78f345f 0x78f36d8 0x79a02cc 0x79a042b 0x79b3a15 0x7864b7c 0x799073e 0x79907ce 0x7990872 0x73eb37a 0x7409299 0x740c6cb 0x75440fc 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:54:45.862 INFO:teuthology.orchestra.run.trial186.stderr:----------------------------------------------------- 2026-02-23T11:54:45.862 INFO:teuthology.orchestra.run.trial186.stderr:Suppressions used: 2026-02-23T11:54:45.862 INFO:teuthology.orchestra.run.trial186.stderr: count bytes template 2026-02-23T11:54:45.862 INFO:teuthology.orchestra.run.trial186.stderr: 1 8 MallocExtension::Register 2026-02-23T11:54:45.862 INFO:teuthology.orchestra.run.trial186.stderr:----------------------------------------------------- 2026-02-23T11:54:45.862 INFO:teuthology.orchestra.run.trial186.stderr: 2026-02-23T11:54:45.897 DEBUG:teuthology.orchestra.run.trial186:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2026-02-23T11:54:45.921 DEBUG:teuthology.orchestra.run.trial186:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-23T11:54:46.075 INFO:teuthology.orchestra.run.trial186.stderr:INFO  osd - OSD::main: parsing early config 2026-02-23T11:54:46.131 INFO:teuthology.orchestra.run.trial186.stderr:INFO 2026-02-23 11:54:46,129 seastar - Reactor backend: linux-aio 2026-02-23T11:54:46.141 INFO:teuthology.orchestra.run.trial186.stderr:INFO 2026-02-23 11:54:46,139 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-23T11:54:46.141 INFO:teuthology.orchestra.run.trial186.stderr:INFO 2026-02-23 11:54:46,139 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-23T11:54:46.143 INFO:teuthology.orchestra.run.trial186.stderr:==39055==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-23T11:54:46.176 INFO:teuthology.orchestra.run.trial186.stderr:Reactor stalled for 32 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc657f /lib64/libasan.so.8+0x157c7 0x357c69b 0x357c898 0x357c96d 0x357f9ca 0x358dffd 0x358e33f 0x358e571 0x358e7f4 0x358eb73 0x358ee83 0x358ef41 0x358f4df 0x358f947 0x358fd84 0x35900aa 0x35901cc 0x359268e 0x3594527 0x35947de 0x359586f 0x359592d 0x35959cf 0x3595af8 0x3595e1d 0x359601f 0x35963a1 0x35cd7b0 0x35ce05a 0x35ce5ba 0x35ce932 0x35cedd0 0x35cf20f 0x3596cf0 0x3597d27 0x3597fde 0x359847d 0x3598599 0x359884d 0x3598bdf 0x3598f12 0x359926f 0x35994a7 0x3599ae2 0x3599e12 0x359a178 0x359a625 0x359aa64 0x359ac90 0x359b09f 0x359b356 0x359b4a7 0x359b778 0x359bb03 0x359be36 0x359c329 0x359c70b 0x359c897 0x359cccf 0x359d03b 0x359d3a0 0x359d6c3 0x359d7a5 0x359da33 0x359dc0a 0x359de71 0x35d1229 0x35d1979 0x35d1e65 0x35d21df 0x35d245e 0x35d2513 0x357b132 0x357ba95 0x357c0cf 0x357b132 0x357b541 0x357b7f8 0x357cb59 0x35c008d 0x3456bbb 0x46815ac 0x46ab703 0x46acbc9 0x46acc4e 0x46acd50 0x46ace48 0x46ad053 0x46ad239 0x46ad2b4 2026-02-23T11:54:46.202 INFO:teuthology.orchestra.run.trial186.stderr:DEBUG 2026-02-23 11:54:46,200 [shard 0:main] seastar - reactor::drain 2026-02-23T11:54:48.308 INFO:teuthology.orchestra.run.trial186.stderr:----------------------------------------------------- 2026-02-23T11:54:48.309 INFO:teuthology.orchestra.run.trial186.stderr:Suppressions used: 2026-02-23T11:54:48.309 INFO:teuthology.orchestra.run.trial186.stderr: count bytes template 2026-02-23T11:54:48.309 INFO:teuthology.orchestra.run.trial186.stderr: 1 8 MallocExtension::Register 2026-02-23T11:54:48.309 INFO:teuthology.orchestra.run.trial186.stderr:----------------------------------------------------- 2026-02-23T11:54:48.309 INFO:teuthology.orchestra.run.trial186.stderr: 2026-02-23T11:54:48.328 INFO:teuthology.orchestra.run.trial186.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-23T11:54:48.328 INFO:teuthology.orchestra.run.trial186.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-23T11:54:48.374 INFO:teuthology.orchestra.run.trial186.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-23T11:54:48.374 INFO:teuthology.orchestra.run.trial186.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-23T11:54:48.381 INFO:teuthology.orchestra.run.trial186.stderr:INFO 2026-02-23 11:54:48,379 seastar - Reactor backend: linux-aio 2026-02-23T11:54:48.389 INFO:teuthology.orchestra.run.trial186.stderr:INFO 2026-02-23 11:54:48,387 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-23T11:54:48.389 INFO:teuthology.orchestra.run.trial186.stderr:INFO 2026-02-23 11:54:48,387 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-23T11:54:48.393 INFO:teuthology.orchestra.run.trial186.stderr:==39053==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-23T11:54:48.393 INFO:teuthology.orchestra.run.trial186.stderr:INFO 2026-02-23 11:54:48,391 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-23T11:54:48.535 INFO:teuthology.orchestra.run.trial186.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f 0xcb0496 0x316fbaf 0x316ff15 0x316feda 0x317000a 0x9ca2b87 0x9c052e1 0x9c042e1 0x7be8837 0x7bf23a6 0x7a51d33 0x7a8d6c5 0x711ff75 0x6f9f19a 0x6fa2f9b 0x6fa8b45 0x7482004 0x72a89fa 0x72a8afc 0x72a8bf4 0x735cb16 0x735cbcf 0x735cd82 0x735ceba 0x735cf35 0x735cfba 0x735d115 0x7425e13 0x748208d 0x728f241 0x72a6bae 0x73b5274 0x7432e6c 0x7432ee7 0x7432f62 0x74eccd2 0x75e6286 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:54:48.536 INFO:teuthology.orchestra.run.trial186.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f 0x109116f 0xf2e3ce 0x9b7dc0b 0x9b7e0d5 0x9b7e348 0x9b7e5fe 0x9b7ebc0 0x9b83ce5 0x9b843c5 0x9c9ff46 0x9ca45f1 0x9c052e1 0x9c042e1 0x7be8837 0x7bf23a6 0x7a51d33 0x7a8d6c5 0x711ff75 0x6f9f19a 0x6fa2f9b 0x6fa8b45 0x7482004 0x72a89fa 0x72a8afc 0x72a8bf4 0x735cb16 0x735cbcf 0x735cd82 0x735ceba 0x735cf35 0x735cfba 0x735d115 0x7425e13 0x748208d 0x728f241 0x72a6bae 0x73b5274 0x7432e6c 0x7432ee7 0x7432f62 0x74eccd2 0x75e6286 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:54:48.536 INFO:teuthology.orchestra.run.trial186.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc7fd /lib64/libasan.so.8+0x8eea /lib64/libasan.so.8+0xedc45 0xcbb8b8 0x9c3919c 0x9c8811c 0x9c9f909 0x9ca45f1 0x9c052e1 0x9c042e1 0x7be8837 0x7bf23a6 0x7a51d33 0x7a8d6c5 0x711ff75 0x6f9f19a 0x6fa2f9b 0x6fa8b45 0x7482004 0x72a89fa 0x72a8afc 0x72a8bf4 0x735cb16 0x735cbcf 0x735cd82 0x735ceba 0x735cf35 0x735cfba 0x735d115 0x7425e13 0x748208d 0x728f241 0x72a6bae 0x73b5274 0x7432e6c 0x7432ee7 0x7432f62 0x74eccd2 0x75e6286 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:54:48.568 INFO:teuthology.orchestra.run.trial186.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x119155 /lib64/libasan.so.8+0xebbf /lib64/libasan.so.8+0xef905 0x316fa84 0x316fc5e 0x316ff15 0x317000a 0x6f6ef3a 0x7be96b5 0x7bf23a6 0x7a51d33 0x7a8d6c5 0x711ff75 0x6f9ff86 0x75590f6 0x755924a 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9e29e18 0x9c44517 0x9c44d52 0xecf43f /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb46f54 2026-02-23T11:54:48.608 INFO:teuthology.orchestra.run.trial186.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libstdc++.so.6+0x1138d4 /lib64/libstdc++.so.6+0x113d17 /lib64/libstdc++.so.6+0x111a9c /lib64/libstdc++.so.6+0x139f62 0xa51fd84 0x7e0331e 0x7e51f8d 0x79f7ce9 0x785a84a 0x7a903da 0x711ff75 0x6f9ff86 0x75590f6 0x755924a 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:54:48.608 INFO:teuthology.orchestra.run.trial186.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f 0x10910f5 0xf2e3ce 0x9b7dc0b 0x9b7e0d5 0x9b7e348 0x9b7f348 0x9b7f8b6 0x9b7fbdb 0x9b7fd45 0x9c8fd63 0x9ca2b6a 0x9c052e1 0x9c042e1 0x7e58655 0x7e5b450 0x8f80d69 0x8f836d7 0x787a27f 0x78aad7c 0x6fa0365 0x75590f6 0x755924a 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:54:48.638 INFO:teuthology.orchestra.run.trial186.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f 0x7d7e226 0x7d9da0c 0x7e39ed3 0x7dbe035 0x78750f3 0x7875de8 0x79b38b2 0x7833a0a 0x78a9398 0x78f345f 0x78f36d8 0x79a02cc 0x79a042b 0x79b3a15 0x7864b7c 0x799073e 0x79907ce 0x7990872 0x73eb37a 0x7409299 0x740c6cb 0x75440fc 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9e29e18 0x9c44517 0x9c44d52 0xecf43f /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb46f54 2026-02-23T11:54:48.674 INFO:teuthology.orchestra.run.trial186.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f 0x2a52ffb 0x7d7e215 0x7d9da0c 0x7e39ed3 0x7dbe035 0x78750f3 0x7875de8 0x79b38b2 0x7833a0a 0x78a9398 0x78f345f 0x78f36d8 0x79a02cc 0x79a042b 0x79b3a15 0x7864b7c 0x799073e 0x79907ce 0x7990872 0x73eb37a 0x7409299 0x740c6cb 0x75440fc 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:54:48.675 INFO:teuthology.orchestra.run.trial186.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x151e0 0x2a5304b 0x7d7e215 0x7d9da0c 0x7e39ed3 0x7dbe035 0x78750f3 0x7875de8 0x79b38b2 0x7833a0a 0x78a9398 0x78f345f 0x78f36d8 0x79a02cc 0x79a042b 0x79b3a15 0x7864b7c 0x799073e 0x79907ce 0x7990872 0x73eb37a 0x7409299 0x740c6cb 0x75440fc 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:54:48.675 INFO:teuthology.orchestra.run.trial186.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc6592 /lib64/libasan.so.8+0x15197 0x2a5304b 0x7d7e215 0x7d9da0c 0x7e39ed3 0x7dbe035 0x78750f3 0x7875de8 0x79b38b2 0x7833a0a 0x78a9398 0x78f345f 0x78f36d8 0x79a02cc 0x79a042b 0x79b3a15 0x7864b7c 0x799073e 0x79907ce 0x7990872 0x73eb37a 0x7409299 0x740c6cb 0x75440fc 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:54:48.803 INFO:teuthology.orchestra.run.trial186.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x15197 0x3606bbd 0x9b8b6b3 0x9b8b775 0x9c872e7 0x9c8747c 0x9c8780e 0x9c87a19 0x9c87d55 0x9c88176 0x9c9f909 0x9ca45f1 0x9c052e1 0x9c042e1 0x7be8837 0x7bf23a6 0x7a51d33 0x7a8d6c5 0x711ff75 0x6f3ec5e 0x6f40fc6 0x747ed38 0x72a6837 0x72a6939 0x72a6a31 0x735bce2 0x735bd9b 0x735bf4e 0x735c086 0x735c101 0x735c186 0x735c2e1 0x7425af5 0x747edc1 0x728e178 0x72a6bae 0x73b5274 0x7432e6c 0x7432ee7 0x7432f62 0x74eccd2 0x75e6286 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:54:48.803 INFO:teuthology.orchestra.run.trial186.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f 0x9b843d4 0x9c9ff46 0x9ca45f1 0x9c052e1 0x9c042e1 0x7be8837 0x7bf23a6 0x7a51d33 0x7a8d6c5 0x711ff75 0x6f3ec5e 0x6f40fc6 0x747ed38 0x72a6837 0x72a6939 0x72a6a31 0x735bce2 0x735bd9b 0x735bf4e 0x735c086 0x735c101 0x735c186 0x735c2e1 0x7425af5 0x747edc1 0x728e178 0x72a6bae 0x73b5274 0x7432e6c 0x7432ee7 0x7432f62 0x74eccd2 0x75e6286 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:54:48.803 INFO:teuthology.orchestra.run.trial186.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x119155 /lib64/libasan.so.8+0xebbf /lib64/libasan.so.8+0xec776 0xcb0505 0x9c881a3 0x9c9f909 0x9ca45f1 0x9c052e1 0x9c042e1 0x7be8837 0x7bf23a6 0x7a51d33 0x7a8d6c5 0x711ff75 0x6f3ec5e 0x6f40fc6 0x747ed38 0x72a6837 0x72a6939 0x72a6a31 0x735bce2 0x735bd9b 0x735bf4e 0x735c086 0x735c101 0x735c186 0x735c2e1 0x7425af5 0x747edc1 0x728e178 0x72a6bae 0x73b5274 0x7432e6c 0x7432ee7 0x7432f62 0x74eccd2 0x75e6286 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:54:48.886 INFO:teuthology.orchestra.run.trial186.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f 0x2a52fd4 0x7d7e215 0x7d9da0c 0x7e39ed3 0x7dbe035 0x78750f3 0x7875de8 0x79b38b2 0x7833a0a 0x78a9398 0x78f345f 0x78f36d8 0x79a02cc 0x79a042b 0x79b3a15 0x7864b7c 0x799073e 0x79907ce 0x7990872 0x73eb37a 0x7409299 0x740c6cb 0x75440fc 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9e29e18 0x9c44517 0x9c44d52 0xecf43f /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb46f54 2026-02-23T11:54:48.901 INFO:teuthology.orchestra.run.trial186.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f 0x2a52377 0x7d7e243 0x7d9da0c 0x7e39ed3 0x7dbe035 0x78750f3 0x7875de8 0x79b38b2 0x7833a0a 0x78a9398 0x78f345f 0x78f36d8 0x79a02cc 0x79a042b 0x79b3a15 0x7864b7c 0x799073e 0x79907ce 0x7990872 0x73eb37a 0x7409299 0x740c6cb 0x75440fc 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:54:48.901 INFO:teuthology.orchestra.run.trial186.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1297e4 /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x15197 0x2a5304b 0x7d7e215 0x7d9da0c 0x7e39ed3 0x7dbe035 0x78750f3 0x7875de8 0x79b38b2 0x7833a0a 0x78a9398 0x78f345f 0x78f36d8 0x79a02cc 0x79a042b 0x79b3a15 0x7864b7c 0x799073e 0x79907ce 0x7990872 0x73eb37a 0x7409299 0x740c6cb 0x75440fc 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:54:48.901 INFO:teuthology.orchestra.run.trial186.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x8cc0c /lib64/libasan.so.8+0x10069c /lib64/libasan.so.8+0xc65b9 /lib64/libasan.so.8+0x15197 0x2a5304b 0x7d7e215 0x7d9da0c 0x7e39ed3 0x7dbe035 0x78750f3 0x7875de8 0x79b38b2 0x7833a0a 0x78a9398 0x78f345f 0x78f36d8 0x79a02cc 0x79a042b 0x79b3a15 0x7864b7c 0x799073e 0x79907ce 0x7990872 0x73eb37a 0x7409299 0x740c6cb 0x75440fc 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:54:51.090 INFO:teuthology.orchestra.run.trial186.stderr:----------------------------------------------------- 2026-02-23T11:54:51.090 INFO:teuthology.orchestra.run.trial186.stderr:Suppressions used: 2026-02-23T11:54:51.090 INFO:teuthology.orchestra.run.trial186.stderr: count bytes template 2026-02-23T11:54:51.091 INFO:teuthology.orchestra.run.trial186.stderr: 1 8 MallocExtension::Register 2026-02-23T11:54:51.091 INFO:teuthology.orchestra.run.trial186.stderr:----------------------------------------------------- 2026-02-23T11:54:51.091 INFO:teuthology.orchestra.run.trial186.stderr: 2026-02-23T11:54:51.126 DEBUG:teuthology.orchestra.run.trial186:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2026-02-23T11:54:51.150 DEBUG:teuthology.orchestra.run.trial186:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-23T11:54:51.305 INFO:teuthology.orchestra.run.trial186.stderr:INFO  osd - OSD::main: parsing early config 2026-02-23T11:54:51.357 INFO:teuthology.orchestra.run.trial186.stderr:INFO 2026-02-23 11:54:51,355 seastar - Reactor backend: linux-aio 2026-02-23T11:54:51.367 INFO:teuthology.orchestra.run.trial186.stderr:INFO 2026-02-23 11:54:51,365 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-23T11:54:51.367 INFO:teuthology.orchestra.run.trial186.stderr:INFO 2026-02-23 11:54:51,365 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-23T11:54:51.369 INFO:teuthology.orchestra.run.trial186.stderr:==39118==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-23T11:54:51.430 INFO:teuthology.orchestra.run.trial186.stderr:DEBUG 2026-02-23 11:54:51,427 [shard 0:main] seastar - reactor::drain 2026-02-23T11:54:53.533 INFO:teuthology.orchestra.run.trial186.stderr:----------------------------------------------------- 2026-02-23T11:54:53.533 INFO:teuthology.orchestra.run.trial186.stderr:Suppressions used: 2026-02-23T11:54:53.533 INFO:teuthology.orchestra.run.trial186.stderr: count bytes template 2026-02-23T11:54:53.533 INFO:teuthology.orchestra.run.trial186.stderr: 1 8 MallocExtension::Register 2026-02-23T11:54:53.533 INFO:teuthology.orchestra.run.trial186.stderr:----------------------------------------------------- 2026-02-23T11:54:53.533 INFO:teuthology.orchestra.run.trial186.stderr: 2026-02-23T11:54:53.553 INFO:teuthology.orchestra.run.trial186.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-23T11:54:53.553 INFO:teuthology.orchestra.run.trial186.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-23T11:54:53.598 INFO:teuthology.orchestra.run.trial186.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-23T11:54:53.598 INFO:teuthology.orchestra.run.trial186.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-23T11:54:53.605 INFO:teuthology.orchestra.run.trial186.stderr:INFO 2026-02-23 11:54:53,603 seastar - Reactor backend: linux-aio 2026-02-23T11:54:53.616 INFO:teuthology.orchestra.run.trial186.stderr:INFO 2026-02-23 11:54:53,614 [shard 2:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-23T11:54:53.616 INFO:teuthology.orchestra.run.trial186.stderr:INFO 2026-02-23 11:54:53,614 [shard 2:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-23T11:54:53.620 INFO:teuthology.orchestra.run.trial186.stderr:==39116==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-23T11:54:53.620 INFO:teuthology.orchestra.run.trial186.stderr:INFO 2026-02-23 11:54:53,618 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-23T11:54:53.760 INFO:teuthology.orchestra.run.trial186.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1006a8 /lib64/libasan.so.8+0xebac /lib64/libasan.so.8+0xec776 0xcb0505 0x9ca656f 0x77f3d8d 0x7be876e 0x7bf23a6 0x7a51d33 0x7a8d6c5 0x711ff75 0x6f9f19a 0x6fa2f9b 0x6fa8b45 0x7482004 0x72a89fa 0x72a8afc 0x72a8bf4 0x735cb16 0x735cbcf 0x735cd82 0x735ceba 0x735cf35 0x735cfba 0x735d115 0x7425e13 0x748208d 0x728f241 0x72a6bae 0x73b5274 0x7432e6c 0x7432ee7 0x7432f62 0x74eccd2 0x75e6286 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:54:53.760 INFO:teuthology.orchestra.run.trial186.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc597 /lib64/libasan.so.8+0x91c1 /lib64/libasan.so.8+0xee7dc 0x9c864ab 0x9c865bb 0x7be87ab 0x7bf23a6 0x7a51d33 0x7a8d6c5 0x711ff75 0x6f9f19a 0x6fa2f9b 0x6fa8b45 0x7482004 0x72a89fa 0x72a8afc 0x72a8bf4 0x735cb16 0x735cbcf 0x735cd82 0x735ceba 0x735cf35 0x735cfba 0x735d115 0x7425e13 0x748208d 0x728f241 0x72a6bae 0x73b5274 0x7432e6c 0x7432ee7 0x7432f62 0x74eccd2 0x75e6286 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:54:53.760 INFO:teuthology.orchestra.run.trial186.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff8a4910cd 0xffffffff8b14d451 0xffffffff8b200ba2 2026-02-23T11:54:53.761 INFO:teuthology.orchestra.run.trial186.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x11913a /lib64/libasan.so.8+0xebbf /lib64/libasan.so.8+0xec776 0xcb0505 0x77f3def 0x7be876e 0x7bf23a6 0x7a51d33 0x7a8d6c5 0x711ff75 0x6f9f19a 0x6fa2f9b 0x6fa8b45 0x7482004 0x72a89fa 0x72a8afc 0x72a8bf4 0x735cb16 0x735cbcf 0x735cd82 0x735ceba 0x735cf35 0x735cfba 0x735d115 0x7425e13 0x748208d 0x728f241 0x72a6bae 0x73b5274 0x7432e6c 0x7432ee7 0x7432f62 0x74eccd2 0x75e6286 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:54:53.793 INFO:teuthology.orchestra.run.trial186.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x8cc0c /lib64/libasan.so.8+0x10069c /lib64/libasan.so.8+0xc65b9 /lib64/libasan.so.8+0x157c7 0x317a79f 0x317a7ec 0x31a511f 0x9ca1aab 0x9ca45f1 0x9c052e1 0x9c042e1 0x8f7f351 0x8f80566 0x8f836d7 0x787a27f 0x78aad7c 0x6fa0365 0x75590f6 0x755924a 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9e29e18 0x9c44517 0x9c44d52 0xecf43f /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb46f54 2026-02-23T11:54:53.860 INFO:teuthology.orchestra.run.trial186.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc6583 /lib64/libasan.so.8+0x15197 0x2a5304b 0x7d7e215 0x7d9da0c 0x7e39ed3 0x7dbe035 0x78750f3 0x7875de8 0x79b38b2 0x7833a0a 0x78a9398 0x78f345f 0x78f36d8 0x79a02cc 0x79a042b 0x79b3a15 0x7864b7c 0x799073e 0x79907ce 0x7990872 0x73eb37a 0x7409299 0x740c6cb 0x75440fc 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9e29e18 0x9c44517 0x9c44d52 0xecf43f /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb46f54 2026-02-23T11:54:53.899 INFO:teuthology.orchestra.run.trial186.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x151f5 0x2a5304b 0x7d7e215 0x7d9da0c 0x7e39ed3 0x7dbe035 0x78750f3 0x7875de8 0x79b38b2 0x7833a0a 0x78a9398 0x78f345f 0x78f36d8 0x79a02cc 0x79a042b 0x79b3a15 0x7864b7c 0x799073e 0x79907ce 0x7990872 0x73eb37a 0x7409299 0x740c6cb 0x75440fc 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:54:53.900 INFO:teuthology.orchestra.run.trial186.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f 0x2a53046 0x7d7e215 0x7d9da0c 0x7e39ed3 0x7dbe035 0x78750f3 0x7875de8 0x79b38b2 0x7833a0a 0x78a9398 0x78f345f 0x78f36d8 0x79a02cc 0x79a042b 0x79b3a15 0x7864b7c 0x799073e 0x79907ce 0x7990872 0x73eb37a 0x7409299 0x740c6cb 0x75440fc 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:54:53.900 INFO:teuthology.orchestra.run.trial186.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc65d8 /lib64/libasan.so.8+0x15197 0x2a5304b 0x7d7e215 0x7d9da0c 0x7e39ed3 0x7dbe035 0x78750f3 0x7875de8 0x79b38b2 0x7833a0a 0x78a9398 0x78f345f 0x78f36d8 0x79a02cc 0x79a042b 0x79b3a15 0x7864b7c 0x799073e 0x79907ce 0x7990872 0x73eb37a 0x7409299 0x740c6cb 0x75440fc 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:54:54.035 INFO:teuthology.orchestra.run.trial186.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x15882 0x1091317 0xf2e3ce 0x9b7ddd8 0x9b7e0d5 0x9b7e348 0x9b7f348 0x9b7f8b6 0x9b7fbdb 0x9b7fd45 0x9c8fd63 0x9ca2b6a 0x9c052e1 0x9c042e1 0x7be8837 0x7bf23a6 0x7a51d33 0x7a8d6c5 0x711ff75 0x6f3ec5e 0x6f40fc6 0x747ed38 0x72a6837 0x72a6939 0x72a6a31 0x735bce2 0x735bd9b 0x735bf4e 0x735c086 0x735c101 0x735c186 0x735c2e1 0x7425af5 0x747edc1 0x728e178 0x72a6bae 0x73b5274 0x7432e6c 0x7432ee7 0x7432f62 0x74eccd2 0x75e6286 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:54:54.035 INFO:teuthology.orchestra.run.trial186.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xfcaf3 /lib64/libasan.so.8+0xc6592 /lib64/libasan.so.8+0x157c7 0x1091317 0xf2e3ce 0x9b7ddd8 0x9b7e0d5 0x9b7e348 0x9b7e5fe 0x9b7f0ad 0x9b7f5ab 0x9b7fbdb 0x9b7fd45 0x9c8fd63 0x9ca2b6a 0x9c052e1 0x9c042e1 0x7be8837 0x7bf23a6 0x7a51d33 0x7a8d6c5 0x711ff75 0x6f3ec5e 0x6f40fc6 0x747ed38 0x72a6837 0x72a6939 0x72a6a31 0x735bce2 0x735bd9b 0x735bf4e 0x735c086 0x735c101 0x735c186 0x735c2e1 0x7425af5 0x747edc1 0x728e178 0x72a6bae 0x73b5274 0x7432e6c 0x7432ee7 0x7432f62 0x74eccd2 0x75e6286 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:54:54.035 INFO:teuthology.orchestra.run.trial186.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x8cbef /lib64/libasan.so.8+0x10069c /lib64/libasan.so.8+0xc65b9 /lib64/libasan.so.8+0x157c7 0x73d62f9 0x73d647f 0x73d67d2 0x77f3cf1 0x7be8169 0x7bf23a6 0x7a51d33 0x7a8d6c5 0x711ff75 0x6f3ec5e 0x6f40fc6 0x747ed38 0x72a6837 0x72a6939 0x72a6a31 0x735bce2 0x735bd9b 0x735bf4e 0x735c086 0x735c101 0x735c186 0x735c2e1 0x7425af5 0x747edc1 0x728e178 0x72a6bae 0x73b5274 0x7432e6c 0x7432ee7 0x7432f62 0x74eccd2 0x75e6286 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:54:54.100 INFO:teuthology.orchestra.run.trial186.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x12976b /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x15197 0x2a5304b 0x7d7e215 0x7d9da0c 0x7e39ed3 0x7dbe035 0x78750f3 0x7875de8 0x79b38b2 0x7833a0a 0x78a9398 0x78f345f 0x78f36d8 0x79a02cc 0x79a042b 0x79b3a15 0x7864b7c 0x799073e 0x79907ce 0x7990872 0x73eb37a 0x7409299 0x740c6cb 0x75440fc 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:54:54.117 INFO:teuthology.orchestra.run.trial186.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc6592 /lib64/libasan.so.8+0x15197 0x2a5304b 0x7d7e215 0x7d9da0c 0x7e39ed3 0x7dbe035 0x78750f3 0x7875de8 0x79b38b2 0x7833a0a 0x78a9398 0x78f345f 0x78f36d8 0x79a02cc 0x79a042b 0x79b3a15 0x7864b7c 0x799073e 0x79907ce 0x7990872 0x73eb37a 0x7409299 0x740c6cb 0x75440fc 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9e29e18 0x9c44517 0x9c44d52 0xecf43f /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb46f54 2026-02-23T11:54:54.132 INFO:teuthology.orchestra.run.trial186.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1006a8 /lib64/libasan.so.8+0xc65b9 /lib64/libasan.so.8+0x15197 0x2a5304b 0x7d7e215 0x7d9da0c 0x7e39ed3 0x7dbe035 0x78750f3 0x7875de8 0x79b38b2 0x7833a0a 0x78a9398 0x78f345f 0x78f36d8 0x79a02cc 0x79a042b 0x79b3a15 0x7864b7c 0x799073e 0x79907ce 0x7990872 0x73eb37a 0x7409299 0x740c6cb 0x75440fc 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:54:54.132 INFO:teuthology.orchestra.run.trial186.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9f10dfc 0x9d045c3 0x9d40a13 0x9d43fc6 0x9d44234 0x9d4437c 0x9d4483b /lib64/libc.so.6+0x3fc2f 0x2a5298a 0x7d7e250 0x7d9da0c 0x7e39ed3 0x7dbe035 0x78750f3 0x7875de8 0x79b38b2 0x7833a0a 0x78a9398 0x78f345f 0x78f36d8 0x79a02cc 0x79a042b 0x79b3a15 0x7864b7c 0x799073e 0x79907ce 0x7990872 0x73eb37a 0x7409299 0x740c6cb 0x75440fc 0x9e08ac1 0x9d0cb62 0x9dc6e39 0x9e285fb 0x9f988cc 0x9f98f48 0x26ed527 0xa5ce02e /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-23T11:54:56.315 INFO:teuthology.orchestra.run.trial186.stderr:----------------------------------------------------- 2026-02-23T11:54:56.315 INFO:teuthology.orchestra.run.trial186.stderr:Suppressions used: 2026-02-23T11:54:56.315 INFO:teuthology.orchestra.run.trial186.stderr: count bytes template 2026-02-23T11:54:56.315 INFO:teuthology.orchestra.run.trial186.stderr: 1 8 MallocExtension::Register 2026-02-23T11:54:56.315 INFO:teuthology.orchestra.run.trial186.stderr:----------------------------------------------------- 2026-02-23T11:54:56.315 INFO:teuthology.orchestra.run.trial186.stderr: 2026-02-23T11:54:56.349 DEBUG:teuthology.orchestra.run.trial186:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2026-02-23T11:54:56.373 INFO:tasks.ceph:Reading keys from all nodes... 2026-02-23T11:54:56.374 DEBUG:teuthology.orchestra.run.trial186:> set -ex 2026-02-23T11:54:56.374 DEBUG:teuthology.orchestra.run.trial186:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2026-02-23T11:54:56.438 DEBUG:teuthology.orchestra.run.trial186:> set -ex 2026-02-23T11:54:56.438 DEBUG:teuthology.orchestra.run.trial186:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2026-02-23T11:54:56.503 DEBUG:teuthology.orchestra.run.trial186:> set -ex 2026-02-23T11:54:56.503 DEBUG:teuthology.orchestra.run.trial186:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2026-02-23T11:54:56.568 DEBUG:teuthology.orchestra.run.trial186:> set -ex 2026-02-23T11:54:56.568 DEBUG:teuthology.orchestra.run.trial186:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2026-02-23T11:54:56.632 DEBUG:teuthology.orchestra.run.trial186:> set -ex 2026-02-23T11:54:56.632 DEBUG:teuthology.orchestra.run.trial186:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2026-02-23T11:54:56.689 INFO:tasks.ceph:Adding keys to all mons... 2026-02-23T11:54:56.689 DEBUG:teuthology.orchestra.run.trial186:> sudo tee -a /etc/ceph/ceph.keyring 2026-02-23T11:54:56.751 INFO:teuthology.orchestra.run.trial186.stdout:[mgr.x] 2026-02-23T11:54:56.751 INFO:teuthology.orchestra.run.trial186.stdout: key = AQD+P5xpLtNDCBAAUpkx15Msq9gBNuusoX6Wmw== 2026-02-23T11:54:56.751 INFO:teuthology.orchestra.run.trial186.stdout:[osd.0] 2026-02-23T11:54:56.752 INFO:teuthology.orchestra.run.trial186.stdout: key = AQADQJxpjch2DhAAKEMwK7UHkg93yGnwd9KHrQ== 2026-02-23T11:54:56.752 INFO:teuthology.orchestra.run.trial186.stdout:[osd.1] 2026-02-23T11:54:56.752 INFO:teuthology.orchestra.run.trial186.stdout: key = AQAIQJxplXw+GxAAZhKI+sKwQKmaQhpU+V4npQ== 2026-02-23T11:54:56.752 INFO:teuthology.orchestra.run.trial186.stdout:[osd.2] 2026-02-23T11:54:56.752 INFO:teuthology.orchestra.run.trial186.stdout: key = AQANQJxpScWxKBAABnYqSrS7m32fK+YNq3Nq3A== 2026-02-23T11:54:56.752 INFO:teuthology.orchestra.run.trial186.stdout:[client.0] 2026-02-23T11:54:56.752 INFO:teuthology.orchestra.run.trial186.stdout: key = AQD+P5xpgKbYChAA3/+W4HrH2WPFk0+Qe2pudQ== 2026-02-23T11:54:56.753 DEBUG:teuthology.orchestra.run.trial186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2026-02-23T11:54:56.841 DEBUG:teuthology.orchestra.run.trial186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-23T11:54:56.890 DEBUG:teuthology.orchestra.run.trial186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-23T11:54:56.940 DEBUG:teuthology.orchestra.run.trial186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-23T11:54:56.989 DEBUG:teuthology.orchestra.run.trial186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-23T11:54:57.038 INFO:tasks.ceph:Running mkfs on mon nodes... 2026-02-23T11:54:57.039 DEBUG:teuthology.orchestra.run.trial186:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2026-02-23T11:54:57.063 DEBUG:teuthology.orchestra.run.trial186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2026-02-23T11:54:57.149 INFO:teuthology.orchestra.run.trial186.stderr:2026-02-23T11:54:57.145+0000 7f1259483c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:54:57.149 INFO:teuthology.orchestra.run.trial186.stderr:2026-02-23T11:54:57.145+0000 7f1259483c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:54:57.156 DEBUG:teuthology.orchestra.run.trial186:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2026-02-23T11:54:57.181 DEBUG:teuthology.orchestra.run.trial186:> rm -- /home/ubuntu/cephtest/ceph.monmap 2026-02-23T11:54:57.237 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2026-02-23T11:54:57.237 INFO:tasks.ceph.mon.a:Restarting daemon 2026-02-23T11:54:57.237 DEBUG:teuthology.orchestra.run.trial186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2026-02-23T11:54:57.280 INFO:tasks.ceph.mon.a:Started 2026-02-23T11:54:57.280 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2026-02-23T11:54:57.280 INFO:tasks.ceph.mgr.x:Restarting daemon 2026-02-23T11:54:57.280 DEBUG:teuthology.orchestra.run.trial186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2026-02-23T11:54:57.282 INFO:tasks.ceph.mgr.x:Started 2026-02-23T11:54:57.283 DEBUG:tasks.ceph:set 0 configs 2026-02-23T11:54:57.283 DEBUG:teuthology.orchestra.run.trial186:> sudo ceph --cluster ceph config dump 2026-02-23T11:54:57.344 INFO:tasks.ceph.mon.a.trial186.stderr:2026-02-23T11:54:57.340+0000 7f5777037c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:54:57.345 INFO:tasks.ceph.mon.a.trial186.stderr:2026-02-23T11:54:57.340+0000 7f5777037c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:54:57.345 INFO:tasks.ceph.mon.a.trial186.stderr:2026-02-23T11:54:57.341+0000 7f5777037c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:54:57.347 INFO:tasks.ceph.mgr.x.trial186.stderr:2026-02-23T11:54:57.344+0000 7fb1f89c1fc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:54:57.360 INFO:teuthology.orchestra.run.trial186.stderr:2026-02-23T11:54:57.355+0000 7fc16b85a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:54:57.362 INFO:teuthology.orchestra.run.trial186.stderr:2026-02-23T11:54:57.358+0000 7fc16b85a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:54:57.393 INFO:teuthology.orchestra.run.trial186.stdout:WHO MASK LEVEL OPTION VALUE RO 2026-02-23T11:54:57.401 INFO:tasks.ceph:Setting crush tunables to default 2026-02-23T11:54:57.401 DEBUG:teuthology.orchestra.run.trial186:> sudo ceph --cluster ceph osd crush tunables default 2026-02-23T11:54:57.475 INFO:teuthology.orchestra.run.trial186.stderr:2026-02-23T11:54:57.468+0000 7f530870b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:54:57.476 INFO:teuthology.orchestra.run.trial186.stderr:2026-02-23T11:54:57.473+0000 7f530870b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:54:57.509 INFO:teuthology.orchestra.run.trial186.stderr:adjusted tunables profile to default 2026-02-23T11:54:57.517 INFO:tasks.ceph:check_enable_crimson: True 2026-02-23T11:54:57.517 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2026-02-23T11:54:57.517 DEBUG:teuthology.orchestra.run.trial186:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2026-02-23T11:54:57.549 INFO:tasks.ceph.mgr.x.trial186.stderr:2026-02-23T11:54:57.546+0000 7fb1f89c1fc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:54:57.549 INFO:tasks.ceph.mgr.x.trial186.stderr:2026-02-23T11:54:57.546+0000 7fb1f89c1fc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:54:57.592 INFO:teuthology.orchestra.run.trial186.stderr:2026-02-23T11:54:57.586+0000 7feefe5c1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:54:57.593 INFO:teuthology.orchestra.run.trial186.stderr:2026-02-23T11:54:57.590+0000 7feefe5c1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:54:57.632 INFO:tasks.ceph.mon.a.trial186.stderr:2026-02-23T11:54:57.628+0000 7f576e3f1640 -1 WARNING: experimental feature 'crimson' is enabled 2026-02-23T11:54:57.632 INFO:tasks.ceph.mon.a.trial186.stderr:Please be aware that this feature is experimental, untested, 2026-02-23T11:54:57.632 INFO:tasks.ceph.mon.a.trial186.stderr:unsupported, and may result in data corruption, data loss, 2026-02-23T11:54:57.632 INFO:tasks.ceph.mon.a.trial186.stderr:and/or irreparable damage to your cluster. Do not use 2026-02-23T11:54:57.632 INFO:tasks.ceph.mon.a.trial186.stderr:feature with important data. 2026-02-23T11:54:57.632 INFO:tasks.ceph.mon.a.trial186.stderr: 2026-02-23T11:54:58.381 INFO:tasks.ceph.mgr.x.trial186.stderr:/usr/lib64/python3.9/site-packages/scipy/__init__.py:73: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2026-02-23T11:54:58.382 INFO:tasks.ceph.mgr.x.trial186.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2026-02-23T11:54:58.382 INFO:tasks.ceph.mgr.x.trial186.stderr: from numpy import show_config as show_numpy_config 2026-02-23T11:54:58.519 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2026-02-23T11:54:58.519 DEBUG:teuthology.orchestra.run.trial186:> set -ex 2026-02-23T11:54:58.519 DEBUG:teuthology.orchestra.run.trial186:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2026-02-23T11:54:58.584 DEBUG:teuthology.orchestra.run.trial186:> set -ex 2026-02-23T11:54:58.584 DEBUG:teuthology.orchestra.run.trial186:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2026-02-23T11:54:58.648 DEBUG:teuthology.orchestra.run.trial186:> set -ex 2026-02-23T11:54:58.648 DEBUG:teuthology.orchestra.run.trial186:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2026-02-23T11:54:58.712 DEBUG:teuthology.orchestra.run.trial186:> sudo ceph --cluster ceph osd new 9b17c9dc-6333-48f6-9c07-1d5fce3a5c40 0 2026-02-23T11:54:58.825 INFO:teuthology.orchestra.run.trial186.stderr:2026-02-23T11:54:58.819+0000 7f1129fed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:54:58.826 INFO:teuthology.orchestra.run.trial186.stderr:2026-02-23T11:54:58.823+0000 7f1129fed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:54:58.863 INFO:teuthology.orchestra.run.trial186.stdout:0 2026-02-23T11:54:58.870 DEBUG:teuthology.orchestra.run.trial186:> sudo ceph --cluster ceph osd new 266e808b-9645-49e6-8c0a-49a6c2027974 1 2026-02-23T11:54:58.943 INFO:teuthology.orchestra.run.trial186.stderr:2026-02-23T11:54:58.937+0000 7fd47e2cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:54:58.944 INFO:teuthology.orchestra.run.trial186.stderr:2026-02-23T11:54:58.941+0000 7fd47e2cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:54:58.981 INFO:teuthology.orchestra.run.trial186.stdout:1 2026-02-23T11:54:58.989 DEBUG:teuthology.orchestra.run.trial186:> sudo ceph --cluster ceph osd new cb5ad62a-3278-40a0-8c19-ec2b6cdd516d 2 2026-02-23T11:54:59.064 INFO:teuthology.orchestra.run.trial186.stderr:2026-02-23T11:54:59.058+0000 7f3b1d051640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:54:59.065 INFO:teuthology.orchestra.run.trial186.stderr:2026-02-23T11:54:59.062+0000 7f3b1d051640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:54:59.102 INFO:teuthology.orchestra.run.trial186.stdout:2 2026-02-23T11:54:59.110 DEBUG:teuthology.orchestra.run.trial186:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2026-02-23T11:54:59.138 INFO:tasks.ceph.osd.0:Restarting daemon 2026-02-23T11:54:59.138 DEBUG:teuthology.orchestra.run.trial186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2026-02-23T11:54:59.180 INFO:tasks.ceph.osd.0:Started 2026-02-23T11:54:59.180 DEBUG:teuthology.orchestra.run.trial186:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2026-02-23T11:54:59.207 INFO:tasks.ceph.osd.1:Restarting daemon 2026-02-23T11:54:59.207 DEBUG:teuthology.orchestra.run.trial186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2026-02-23T11:54:59.250 INFO:tasks.ceph.osd.1:Started 2026-02-23T11:54:59.250 DEBUG:teuthology.orchestra.run.trial186:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2026-02-23T11:54:59.277 INFO:tasks.ceph.osd.2:Restarting daemon 2026-02-23T11:54:59.277 DEBUG:teuthology.orchestra.run.trial186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2026-02-23T11:54:59.320 INFO:tasks.ceph.osd.2:Started 2026-02-23T11:54:59.320 DEBUG:teuthology.orchestra.run.trial186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-23T11:54:59.398 INFO:teuthology.orchestra.run.trial186.stderr:2026-02-23T11:54:59.392+0000 7f5ea405a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:54:59.400 INFO:teuthology.orchestra.run.trial186.stderr:2026-02-23T11:54:59.396+0000 7f5ea405a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:54:59.432 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:54:59.432 INFO:teuthology.orchestra.run.trial186.stdout:{"epoch":6,"fsid":"d91d7eac-82c0-417c-a119-6b75d467a55c","created":"2026-02-23T11:54:57.351969+0000","modified":"2026-02-23T11:54:59.100287+0000","last_up_change":"0.000000","last_in_change":"2026-02-23T11:54:59.100287+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"9b17c9dc-6333-48f6-9c07-1d5fce3a5c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"266e808b-9645-49e6-8c0a-49a6c2027974","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cb5ad62a-3278-40a0-8c19-ec2b6cdd516d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T11:54:59.441 INFO:tasks.ceph.ceph_manager.ceph:[] 2026-02-23T11:54:59.441 INFO:tasks.ceph:Waiting for OSDs to come up 2026-02-23T11:54:59.743 DEBUG:teuthology.orchestra.run.trial186:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-23T11:54:59.811 INFO:teuthology.misc.health.trial186.stderr:2026-02-23T11:54:59.805+0000 7f534c30b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:54:59.812 INFO:teuthology.misc.health.trial186.stderr:2026-02-23T11:54:59.808+0000 7f534c30b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:54:59.843 INFO:teuthology.misc.health.trial186.stdout: 2026-02-23T11:54:59.843 INFO:teuthology.misc.health.trial186.stdout:{"epoch":6,"fsid":"d91d7eac-82c0-417c-a119-6b75d467a55c","created":"2026-02-23T11:54:57.351969+0000","modified":"2026-02-23T11:54:59.100287+0000","last_up_change":"0.000000","last_in_change":"2026-02-23T11:54:59.100287+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"9b17c9dc-6333-48f6-9c07-1d5fce3a5c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"266e808b-9645-49e6-8c0a-49a6c2027974","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cb5ad62a-3278-40a0-8c19-ec2b6cdd516d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T11:54:59.849 DEBUG:teuthology.misc:0 of 3 OSDs are up 2026-02-23T11:55:00.608 INFO:tasks.ceph.mgr.x.trial186.stderr:2026-02-23T11:55:00.604+0000 7fb1158c1640 -1 mgr.server handle_report got status from non-daemon mon.a 2026-02-23T11:55:06.152 DEBUG:teuthology.orchestra.run.trial186:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-23T11:55:06.248 INFO:teuthology.misc.health.trial186.stderr:2026-02-23T11:55:06.242+0000 7f2c577e4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:55:06.249 INFO:teuthology.misc.health.trial186.stderr:2026-02-23T11:55:06.246+0000 7f2c577e4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:55:06.380 INFO:teuthology.misc.health.trial186.stdout: 2026-02-23T11:55:06.380 INFO:teuthology.misc.health.trial186.stdout:{"epoch":7,"fsid":"d91d7eac-82c0-417c-a119-6b75d467a55c","created":"2026-02-23T11:54:57.351969+0000","modified":"2026-02-23T11:55:05.834744+0000","last_up_change":"0.000000","last_in_change":"2026-02-23T11:54:59.100287+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"9b17c9dc-6333-48f6-9c07-1d5fce3a5c40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"266e808b-9645-49e6-8c0a-49a6c2027974","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cb5ad62a-3278-40a0-8c19-ec2b6cdd516d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T11:55:06.394 DEBUG:teuthology.misc:0 of 3 OSDs are up 2026-02-23T11:55:12.697 DEBUG:teuthology.orchestra.run.trial186:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-23T11:55:12.767 INFO:teuthology.misc.health.trial186.stderr:2026-02-23T11:55:12.761+0000 7fc3136cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:55:12.768 INFO:teuthology.misc.health.trial186.stderr:2026-02-23T11:55:12.764+0000 7fc3136cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:55:12.880 INFO:teuthology.misc.health.trial186.stdout: 2026-02-23T11:55:12.881 INFO:teuthology.misc.health.trial186.stdout:{"epoch":13,"fsid":"d91d7eac-82c0-417c-a119-6b75d467a55c","created":"2026-02-23T11:54:57.351969+0000","modified":"2026-02-23T11:55:11.863471+0000","last_up_change":"2026-02-23T11:55:08.849356+0000","last_in_change":"2026-02-23T11:54:59.100287+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T11:55:09.617840+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9b17c9dc-6333-48f6-9c07-1d5fce3a5c40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.186:6813","nonce":3393336021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.186:6812","nonce":3393336021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.186:6802","nonce":3393336021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.186:6805","nonce":3393336021}]},"public_addr":"10.20.193.186:6813/3393336021","cluster_addr":"10.20.193.186:6812/3393336021","heartbeat_back_addr":"10.20.193.186:6802/3393336021","heartbeat_front_addr":"10.20.193.186:6805/3393336021","state":["exists","up"]},{"osd":1,"uuid":"266e808b-9645-49e6-8c0a-49a6c2027974","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.186:6808","nonce":2448563909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.186:6809","nonce":2448563909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.186:6806","nonce":2448563909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.186:6807","nonce":2448563909}]},"public_addr":"10.20.193.186:6808/2448563909","cluster_addr":"10.20.193.186:6809/2448563909","heartbeat_back_addr":"10.20.193.186:6806/2448563909","heartbeat_front_addr":"10.20.193.186:6807/2448563909","state":["exists","up"]},{"osd":2,"uuid":"cb5ad62a-3278-40a0-8c19-ec2b6cdd516d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.186:6804","nonce":770125878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.186:6803","nonce":770125878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.186:6815","nonce":770125878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.186:6814","nonce":770125878}]},"public_addr":"10.20.193.186:6804/770125878","cluster_addr":"10.20.193.186:6803/770125878","heartbeat_back_addr":"10.20.193.186:6815/770125878","heartbeat_front_addr":"10.20.193.186:6814/770125878","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T11:55:12.891 DEBUG:teuthology.misc:3 of 3 OSDs are up 2026-02-23T11:55:12.891 INFO:tasks.ceph:Creating RBD pool 2026-02-23T11:55:12.891 DEBUG:teuthology.orchestra.run.trial186:> sudo ceph --cluster ceph osd pool create rbd 8 2026-02-23T11:55:12.966 INFO:teuthology.orchestra.run.trial186.stderr:2026-02-23T11:55:12.959+0000 7f4b3485a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:55:12.967 INFO:teuthology.orchestra.run.trial186.stderr:2026-02-23T11:55:12.964+0000 7f4b3485a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:55:13.871 INFO:teuthology.orchestra.run.trial186.stderr:pool 'rbd' created 2026-02-23T11:55:13.885 DEBUG:teuthology.orchestra.run.trial186:> rbd --cluster ceph pool init rbd 2026-02-23T11:55:13.927 INFO:teuthology.orchestra.run.trial186.stderr:2026-02-23T11:55:13.923+0000 7f451e6ea240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:55:13.927 INFO:teuthology.orchestra.run.trial186.stderr:2026-02-23T11:55:13.923+0000 7f451e6ea240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:55:13.928 INFO:teuthology.orchestra.run.trial186.stderr:2026-02-23T11:55:13.925+0000 7f451e6ea240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:55:16.909 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2026-02-23T11:55:16.909 DEBUG:teuthology.orchestra.run.trial186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2026-02-23T11:55:16.909 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2026-02-23T11:55:17.027 INFO:teuthology.orchestra.run.trial186.stderr:2026-02-23T11:55:17.021+0000 7fde17ecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:55:17.029 INFO:teuthology.orchestra.run.trial186.stderr:2026-02-23T11:55:17.025+0000 7fde17ecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:55:17.140 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:55:17.149 INFO:teuthology.orchestra.run.trial186.stdout:[{"version":1,"timestamp":"0.000000","name":"","changes":[]}] 2026-02-23T11:55:17.150 INFO:tasks.ceph_manager:config epoch is 1 2026-02-23T11:55:17.150 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2026-02-23T11:55:17.150 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2026-02-23T11:55:17.151 DEBUG:teuthology.orchestra.run.trial186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2026-02-23T11:55:17.229 INFO:teuthology.orchestra.run.trial186.stderr:2026-02-23T11:55:17.223+0000 7fde9130b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:55:17.230 INFO:teuthology.orchestra.run.trial186.stderr:2026-02-23T11:55:17.226+0000 7fde9130b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:55:17.371 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:55:17.380 INFO:teuthology.orchestra.run.trial186.stdout:{"epoch":5,"flags":0,"active_gid":4104,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.186:6800","nonce":1081062777},{"type":"v1","addr":"10.20.193.186:6801","nonce":1081062777}]},"active_addr":"10.20.193.186:6801/1081062777","active_change":"2026-02-23T11:54:59.600573+0000","active_mgr_features":4544132024016699391,"available":true,"standbys":[],"modules":["iostat","nfs"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_pg_upmap_activity":{"name":"update_pg_upmap_activity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Updates pg_upmap activity stats to be used in `balancer status detail`","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often the agent on each host will gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"The first TCP port the agent will try to bind to. Up to 1000 subsequent ports will be attempted if this port cannot be bound","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this option can facilitate debugging daemons that encounter runtime problems.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"How frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"Fraction of total system memory to divide among autotuned daemons. Converged systems hosting both Ceph and compute should set a lower value; systems with more than 10 GiB of memory per OSD may benefit from a slightly larger value so that the abundance of RAM can be fully utilized","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command persistent logging","long_desc":"","tags":[],"see_also":[]},"certificate_automated_rotation_enabled":{"name":"certificate_automated_rotation_enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"This flag controls whether cephadm automatically rotates certificates upon expiration.","long_desc":"","tags":[],"see_also":[]},"certificate_check_debug_mode":{"name":"certificate_check_debug_mode","type":"bool","level":"dev","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"FOR TESTING ONLY: This flag forces the certificate check instead of waiting for certificate_check_period.","long_desc":"","tags":[],"see_also":[]},"certificate_check_period":{"name":"certificate_check_period","type":"int","level":"advanced","flags":0,"default_value":"1","min":"0","max":"30","enum_allowed":[],"desc":"Specifies how often (in days) the certificate should be checked for validity.","long_desc":"","tags":[],"see_also":[]},"certificate_duration_days":{"name":"certificate_duration_days","type":"int","level":"advanced","flags":0,"default_value":"1095","min":"90","max":"3650","enum_allowed":[],"desc":"Specifies the duration of self-signed certificates generated and signed by the cephadm root CA","long_desc":"","tags":[],"see_also":[]},"certificate_renewal_threshold_days":{"name":"certificate_renewal_threshold_days","type":"int","level":"advanced","flags":0,"default_value":"30","min":"10","max":"90","enum_allowed":[],"desc":"Specifies the lead time in days to initiate certificate renewal before expiration.","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently Podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.28.1","min":"","max":"","enum_allowed":[],"desc":"Alertmanager container image","long_desc":"","tags":[],"see_also":[]},"container_image_alloy":{"name":"container_image_alloy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/alloy:latest","min":"","max":"","enum_allowed":[],"desc":"Alloy container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"Elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:12.3.1","min":"","max":"","enum_allowed":[],"desc":"Grafana container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"Haproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.9.1","min":"","max":"","enum_allowed":[],"desc":"Node exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.6","min":"","max":"","enum_allowed":[],"desc":"Nvmeof container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"Oauth2 proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v3.6.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba metrics container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"Snmp gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"quay.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"Location of dashboards to include in Grafana","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"How frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether the hardware monitoring daemon be deployed on every host?","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at the debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"Max number of daemons per service per host: guard against mistakes and runaways","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"Max number of OSDs that will be drained simultaneously when OSDs are removed","long_desc":"","tags":[],"see_also":[]},"max_parallel_osd_upgrades":{"name":"max_parallel_osd_upgrades","type":"int","level":"advanced","flags":0,"default_value":"16","min":"","max":"","enum_allowed":[],"desc":"Maximum number of OSD daemons upgraded in parallel.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"Mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default IP address for RedFish API (OOB management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"Location of alerts to include in Prometheus Alertmanager","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry URL for login purposes. This is not the default registry.","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"Cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Customized SSH config file to distribute to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many sequential SSH connection liveness check failures before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often SSH connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"stray_daemon_check_interval":{"name":"stray_daemon_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"How frequently cephadm should check for the presence of stray daemons","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use the cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest to ensure that all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROM_ALERT_CREDENTIAL_CACHE_TTL":{"name":"PROM_ALERT_CREDENTIAL_CACHE_TTL","type":"int","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_HOSTNAME_PER_DAEMON":{"name":"RGW_HOSTNAME_PER_DAEMON","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crypto_caller":{"name":"crypto_caller","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"healthcheck_history_max_entries":{"name":"healthcheck_history_max_entries","type":"int","level":"advanced","flags":1,"default_value":"1000","min":"","max":"","enum_allowed":[],"desc":"Maximum number of health check history entries to keep","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"prometheus_tls_secret_name":{"name":"prometheus_tls_secret_name","type":"str","level":"advanced","flags":0,"default_value":"rook-ceph-prometheus-server-tls","min":"","max":"","enum_allowed":[],"desc":"name of tls secret in k8s for prometheus","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_cloning":{"name":"pause_cloning","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_purging":{"name":"pause_purging","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous subvolume purge threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"tentacle":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"force_disabled_modules":{},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"10.20.193.186:0","nonce":3661826966}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"10.20.193.186:0","nonce":375887729}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"10.20.193.186:0","nonce":2929448950}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"10.20.193.186:0","nonce":2241458087}]}]} 2026-02-23T11:55:17.381 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2026-02-23T11:55:17.381 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2026-02-23T11:55:17.381 DEBUG:teuthology.orchestra.run.trial186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-23T11:55:17.456 INFO:teuthology.orchestra.run.trial186.stderr:2026-02-23T11:55:17.451+0000 7fe41df6f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:55:17.457 INFO:teuthology.orchestra.run.trial186.stderr:2026-02-23T11:55:17.454+0000 7fe41df6f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:55:17.567 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:55:17.568 INFO:teuthology.orchestra.run.trial186.stdout:{"epoch":17,"fsid":"d91d7eac-82c0-417c-a119-6b75d467a55c","created":"2026-02-23T11:54:57.351969+0000","modified":"2026-02-23T11:55:16.881538+0000","last_up_change":"2026-02-23T11:55:08.849356+0000","last_in_change":"2026-02-23T11:54:59.100287+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T11:55:09.617840+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-23T11:55:13.080375+0000","flags":270337,"flags_names":"hashpspool,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9b17c9dc-6333-48f6-9c07-1d5fce3a5c40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.186:6813","nonce":3393336021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.186:6812","nonce":3393336021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.186:6802","nonce":3393336021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.186:6805","nonce":3393336021}]},"public_addr":"10.20.193.186:6813/3393336021","cluster_addr":"10.20.193.186:6812/3393336021","heartbeat_back_addr":"10.20.193.186:6802/3393336021","heartbeat_front_addr":"10.20.193.186:6805/3393336021","state":["exists","up"]},{"osd":1,"uuid":"266e808b-9645-49e6-8c0a-49a6c2027974","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.186:6808","nonce":2448563909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.186:6809","nonce":2448563909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.186:6806","nonce":2448563909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.186:6807","nonce":2448563909}]},"public_addr":"10.20.193.186:6808/2448563909","cluster_addr":"10.20.193.186:6809/2448563909","heartbeat_back_addr":"10.20.193.186:6806/2448563909","heartbeat_front_addr":"10.20.193.186:6807/2448563909","state":["exists","up"]},{"osd":2,"uuid":"cb5ad62a-3278-40a0-8c19-ec2b6cdd516d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.186:6804","nonce":770125878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.186:6803","nonce":770125878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.186:6815","nonce":770125878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.186:6814","nonce":770125878}]},"public_addr":"10.20.193.186:6804/770125878","cluster_addr":"10.20.193.186:6803/770125878","heartbeat_back_addr":"10.20.193.186:6815/770125878","heartbeat_front_addr":"10.20.193.186:6814/770125878","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T11:55:17.577 INFO:tasks.ceph.ceph_manager.ceph:all up! 2026-02-23T11:55:17.578 DEBUG:teuthology.orchestra.run.trial186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-23T11:55:17.652 INFO:teuthology.orchestra.run.trial186.stderr:2026-02-23T11:55:17.647+0000 7ff4698cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:55:17.654 INFO:teuthology.orchestra.run.trial186.stderr:2026-02-23T11:55:17.650+0000 7ff4698cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:55:17.766 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:55:17.766 INFO:teuthology.orchestra.run.trial186.stdout:{"epoch":17,"fsid":"d91d7eac-82c0-417c-a119-6b75d467a55c","created":"2026-02-23T11:54:57.351969+0000","modified":"2026-02-23T11:55:16.881538+0000","last_up_change":"2026-02-23T11:55:08.849356+0000","last_in_change":"2026-02-23T11:54:59.100287+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-23T11:55:09.617840+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-23T11:55:13.080375+0000","flags":270337,"flags_names":"hashpspool,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9b17c9dc-6333-48f6-9c07-1d5fce3a5c40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.186:6813","nonce":3393336021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.186:6812","nonce":3393336021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.186:6802","nonce":3393336021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.186:6805","nonce":3393336021}]},"public_addr":"10.20.193.186:6813/3393336021","cluster_addr":"10.20.193.186:6812/3393336021","heartbeat_back_addr":"10.20.193.186:6802/3393336021","heartbeat_front_addr":"10.20.193.186:6805/3393336021","state":["exists","up"]},{"osd":1,"uuid":"266e808b-9645-49e6-8c0a-49a6c2027974","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.186:6808","nonce":2448563909}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.186:6809","nonce":2448563909}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.186:6806","nonce":2448563909}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.186:6807","nonce":2448563909}]},"public_addr":"10.20.193.186:6808/2448563909","cluster_addr":"10.20.193.186:6809/2448563909","heartbeat_back_addr":"10.20.193.186:6806/2448563909","heartbeat_front_addr":"10.20.193.186:6807/2448563909","state":["exists","up"]},{"osd":2,"uuid":"cb5ad62a-3278-40a0-8c19-ec2b6cdd516d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.186:6804","nonce":770125878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.186:6803","nonce":770125878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.186:6815","nonce":770125878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.186:6814","nonce":770125878}]},"public_addr":"10.20.193.186:6804/770125878","cluster_addr":"10.20.193.186:6803/770125878","heartbeat_back_addr":"10.20.193.186:6815/770125878","heartbeat_front_addr":"10.20.193.186:6814/770125878","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-23T11:55:17.776 DEBUG:teuthology.orchestra.run.trial186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-23T11:55:17.776 DEBUG:teuthology.orchestra.run.trial186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-23T11:55:17.777 DEBUG:teuthology.orchestra.run.trial186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-23T11:55:17.858 INFO:teuthology.orchestra.run.trial186.stderr:2026-02-23T11:55:17.852+0000 7f814581a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:55:17.859 INFO:teuthology.orchestra.run.trial186.stderr:2026-02-23T11:55:17.852+0000 7f3618b6f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:55:17.859 INFO:teuthology.orchestra.run.trial186.stderr:2026-02-23T11:55:17.852+0000 7f9d5850b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:55:17.860 INFO:teuthology.orchestra.run.trial186.stderr:2026-02-23T11:55:17.856+0000 7f3618b6f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:55:17.860 INFO:teuthology.orchestra.run.trial186.stderr:2026-02-23T11:55:17.856+0000 7f814581a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:55:17.860 INFO:teuthology.orchestra.run.trial186.stderr:2026-02-23T11:55:17.856+0000 7f9d5850b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:55:17.876 INFO:teuthology.orchestra.run.trial186.stdout:42949672968 2026-02-23T11:55:17.876 DEBUG:teuthology.orchestra.run.trial186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-23T11:55:17.877 INFO:teuthology.orchestra.run.trial186.stdout:42949672968 2026-02-23T11:55:17.878 DEBUG:teuthology.orchestra.run.trial186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-23T11:55:17.879 INFO:teuthology.orchestra.run.trial186.stdout:38654705674 2026-02-23T11:55:17.879 DEBUG:teuthology.orchestra.run.trial186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-23T11:55:18.001 INFO:teuthology.orchestra.run.trial186.stderr:2026-02-23T11:55:17.994+0000 7f9243ecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:55:18.002 INFO:teuthology.orchestra.run.trial186.stderr:2026-02-23T11:55:17.994+0000 7f0d2ef9c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:55:18.002 INFO:teuthology.orchestra.run.trial186.stderr:2026-02-23T11:55:17.994+0000 7f7e2aa5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:55:18.003 INFO:teuthology.orchestra.run.trial186.stderr:2026-02-23T11:55:17.998+0000 7f9243ecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:55:18.004 INFO:teuthology.orchestra.run.trial186.stderr:2026-02-23T11:55:17.998+0000 7f0d2ef9c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:55:18.004 INFO:teuthology.orchestra.run.trial186.stderr:2026-02-23T11:55:17.998+0000 7f7e2aa5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:55:18.117 INFO:teuthology.orchestra.run.trial186.stdout:42949672964 2026-02-23T11:55:18.122 INFO:teuthology.orchestra.run.trial186.stdout:38654705673 2026-02-23T11:55:18.125 INFO:teuthology.orchestra.run.trial186.stdout:42949672964 2026-02-23T11:55:18.129 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672968 got 42949672964 for osd.2 2026-02-23T11:55:18.133 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705674 got 38654705673 for osd.1 2026-02-23T11:55:18.135 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672968 got 42949672964 for osd.0 2026-02-23T11:55:19.129 DEBUG:teuthology.orchestra.run.trial186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-23T11:55:19.134 DEBUG:teuthology.orchestra.run.trial186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-23T11:55:19.136 DEBUG:teuthology.orchestra.run.trial186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-23T11:55:19.209 INFO:teuthology.orchestra.run.trial186.stderr:2026-02-23T11:55:19.203+0000 7fd040183640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:55:19.210 INFO:teuthology.orchestra.run.trial186.stderr:2026-02-23T11:55:19.207+0000 7fd040183640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:55:19.252 INFO:teuthology.orchestra.run.trial186.stderr:2026-02-23T11:55:19.246+0000 7fac884cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:55:19.253 INFO:teuthology.orchestra.run.trial186.stderr:2026-02-23T11:55:19.247+0000 7f6a5650b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:55:19.254 INFO:teuthology.orchestra.run.trial186.stderr:2026-02-23T11:55:19.250+0000 7fac884cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:55:19.254 INFO:teuthology.orchestra.run.trial186.stderr:2026-02-23T11:55:19.251+0000 7f6a5650b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:55:19.326 INFO:teuthology.orchestra.run.trial186.stdout:42949672964 2026-02-23T11:55:19.337 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672968 got 42949672964 for osd.2 2026-02-23T11:55:19.368 INFO:teuthology.orchestra.run.trial186.stdout:42949672964 2026-02-23T11:55:19.372 INFO:teuthology.orchestra.run.trial186.stdout:38654705673 2026-02-23T11:55:19.379 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672968 got 42949672964 for osd.0 2026-02-23T11:55:19.383 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705674 got 38654705673 for osd.1 2026-02-23T11:55:20.337 DEBUG:teuthology.orchestra.run.trial186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-23T11:55:20.380 DEBUG:teuthology.orchestra.run.trial186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-23T11:55:20.383 DEBUG:teuthology.orchestra.run.trial186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-23T11:55:20.416 INFO:teuthology.orchestra.run.trial186.stderr:2026-02-23T11:55:20.410+0000 7fbc82a1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:55:20.417 INFO:teuthology.orchestra.run.trial186.stderr:2026-02-23T11:55:20.414+0000 7fbc82a1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:55:20.458 INFO:teuthology.orchestra.run.trial186.stderr:2026-02-23T11:55:20.452+0000 7f62eea8b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:55:20.459 INFO:teuthology.orchestra.run.trial186.stderr:2026-02-23T11:55:20.456+0000 7f62eea8b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:55:20.493 INFO:teuthology.orchestra.run.trial186.stderr:2026-02-23T11:55:20.487+0000 7f791d10b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:55:20.494 INFO:teuthology.orchestra.run.trial186.stderr:2026-02-23T11:55:20.491+0000 7f791d10b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:55:20.536 INFO:teuthology.orchestra.run.trial186.stdout:42949672969 2026-02-23T11:55:20.548 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672968 got 42949672969 for osd.2 2026-02-23T11:55:20.548 DEBUG:teuthology.parallel:result is None 2026-02-23T11:55:20.576 INFO:teuthology.orchestra.run.trial186.stdout:42949672969 2026-02-23T11:55:20.587 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672968 got 42949672969 for osd.0 2026-02-23T11:55:20.587 DEBUG:teuthology.parallel:result is None 2026-02-23T11:55:20.609 INFO:teuthology.orchestra.run.trial186.stdout:38654705674 2026-02-23T11:55:20.620 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705674 got 38654705674 for osd.1 2026-02-23T11:55:20.620 DEBUG:teuthology.parallel:result is None 2026-02-23T11:55:20.620 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-23T11:55:20.620 DEBUG:teuthology.orchestra.run.trial186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-23T11:55:20.738 INFO:teuthology.orchestra.run.trial186.stderr:2026-02-23T11:55:20.732+0000 7f4cd37e4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:55:20.739 INFO:teuthology.orchestra.run.trial186.stderr:2026-02-23T11:55:20.735+0000 7f4cd37e4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:55:20.848 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:55:20.848 INFO:teuthology.orchestra.run.trial186.stderr:dumped all 2026-02-23T11:55:20.859 INFO:teuthology.orchestra.run.trial186.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2026-02-23T11:55:19.604497+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156499968,"kb_used":2540,"kb_used_data":2540,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156497428,"statfs":{"total":160255967232,"available":160253366272,"internally_reserved":0,"allocated":2600960,"data_stored":2600960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.741083"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-23T11:55:16.895285+0000","last_change":"2026-02-23T11:55:16.898298+0000","last_active":"2026-02-23T11:55:16.895285+0000","last_peered":"2026-02-23T11:55:16.895285+0000","last_clean":"2026-02-23T11:55:16.895285+0000","last_became_active":"2026-02-23T11:55:14.908265+0000","last_became_peered":"2026-02-23T11:55:14.908265+0000","last_unstale":"2026-02-23T11:55:16.895285+0000","last_undegraded":"2026-02-23T11:55:16.895285+0000","last_fullsized":"2026-02-23T11:55:16.895285+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:55:13.868699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:55:13.868699+0000","last_clean_scrub_stamp":"2026-02-23T11:55:13.868699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-23T11:55:16.896339+0000","last_change":"2026-02-23T11:55:16.898171+0000","last_active":"2026-02-23T11:55:16.896339+0000","last_peered":"2026-02-23T11:55:16.896339+0000","last_clean":"2026-02-23T11:55:16.896339+0000","last_became_active":"2026-02-23T11:55:14.906589+0000","last_became_peered":"2026-02-23T11:55:14.906589+0000","last_unstale":"2026-02-23T11:55:16.896339+0000","last_undegraded":"2026-02-23T11:55:16.896339+0000","last_fullsized":"2026-02-23T11:55:16.896339+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:55:13.868699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:55:13.868699+0000","last_clean_scrub_stamp":"2026-02-23T11:55:13.868699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-23T11:55:16.896938+0000","last_change":"2026-02-23T11:55:16.899363+0000","last_active":"2026-02-23T11:55:16.896938+0000","last_peered":"2026-02-23T11:55:16.896938+0000","last_clean":"2026-02-23T11:55:16.896938+0000","last_became_active":"2026-02-23T11:55:14.905759+0000","last_became_peered":"2026-02-23T11:55:14.905759+0000","last_unstale":"2026-02-23T11:55:16.896938+0000","last_undegraded":"2026-02-23T11:55:16.896938+0000","last_fullsized":"2026-02-23T11:55:16.896938+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:55:13.868699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:55:13.868699+0000","last_clean_scrub_stamp":"2026-02-23T11:55:13.868699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-23T11:55:16.896316+0000","last_change":"2026-02-23T11:55:16.899375+0000","last_active":"2026-02-23T11:55:16.896316+0000","last_peered":"2026-02-23T11:55:16.896316+0000","last_clean":"2026-02-23T11:55:16.896316+0000","last_became_active":"2026-02-23T11:55:14.911243+0000","last_became_peered":"2026-02-23T11:55:14.911243+0000","last_unstale":"2026-02-23T11:55:16.896316+0000","last_undegraded":"2026-02-23T11:55:16.896316+0000","last_fullsized":"2026-02-23T11:55:16.896316+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:55:13.868699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:55:13.868699+0000","last_clean_scrub_stamp":"2026-02-23T11:55:13.868699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'1","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-23T11:55:16.895698+0000","last_change":"2026-02-23T11:55:16.895698+0000","last_active":"2026-02-23T11:55:16.895698+0000","last_peered":"2026-02-23T11:55:16.895698+0000","last_clean":"2026-02-23T11:55:16.895698+0000","last_became_active":"2026-02-23T11:55:14.918494+0000","last_became_peered":"2026-02-23T11:55:14.918494+0000","last_unstale":"2026-02-23T11:55:16.895698+0000","last_undegraded":"2026-02-23T11:55:16.895698+0000","last_fullsized":"2026-02-23T11:55:16.895698+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:55:13.868699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:55:13.868699+0000","last_clean_scrub_stamp":"2026-02-23T11:55:13.868699+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-23T11:55:16.894547+0000","last_change":"2026-02-23T11:55:16.896243+0000","last_active":"2026-02-23T11:55:16.894547+0000","last_peered":"2026-02-23T11:55:16.894547+0000","last_clean":"2026-02-23T11:55:16.894547+0000","last_became_active":"2026-02-23T11:55:14.923409+0000","last_became_peered":"2026-02-23T11:55:14.923409+0000","last_unstale":"2026-02-23T11:55:16.894547+0000","last_undegraded":"2026-02-23T11:55:16.894547+0000","last_fullsized":"2026-02-23T11:55:16.894547+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:55:13.868699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:55:13.868699+0000","last_clean_scrub_stamp":"2026-02-23T11:55:13.868699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-23T11:55:16.893015+0000","last_change":"2026-02-23T11:55:16.895343+0000","last_active":"2026-02-23T11:55:16.893015+0000","last_peered":"2026-02-23T11:55:16.893015+0000","last_clean":"2026-02-23T11:55:16.893015+0000","last_became_active":"2026-02-23T11:55:14.921646+0000","last_became_peered":"2026-02-23T11:55:14.921646+0000","last_unstale":"2026-02-23T11:55:16.893015+0000","last_undegraded":"2026-02-23T11:55:16.893015+0000","last_fullsized":"2026-02-23T11:55:16.893015+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:55:13.868699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:55:13.868699+0000","last_clean_scrub_stamp":"2026-02-23T11:55:13.868699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":16,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-23T11:55:16.896318+0000","last_change":"2026-02-23T11:55:16.898468+0000","last_active":"2026-02-23T11:55:16.896318+0000","last_peered":"2026-02-23T11:55:16.896318+0000","last_clean":"2026-02-23T11:55:16.896318+0000","last_became_active":"2026-02-23T11:55:14.905437+0000","last_became_peered":"2026-02-23T11:55:14.905437+0000","last_unstale":"2026-02-23T11:55:16.896318+0000","last_undegraded":"2026-02-23T11:55:16.896318+0000","last_fullsized":"2026-02-23T11:55:16.896318+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:55:13.868699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:55:13.868699+0000","last_clean_scrub_stamp":"2026-02-23T11:55:13.868699+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":90,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-23T11:55:16.895847+0000","last_change":"2026-02-23T11:55:10.893657+0000","last_active":"2026-02-23T11:55:16.895847+0000","last_peered":"2026-02-23T11:55:16.895847+0000","last_clean":"2026-02-23T11:55:16.895847+0000","last_became_active":"2026-02-23T11:55:10.886604+0000","last_became_peered":"2026-02-23T11:55:10.886604+0000","last_unstale":"2026-02-23T11:55:16.895847+0000","last_undegraded":"2026-02-23T11:55:16.895847+0000","last_fullsized":"2026-02-23T11:55:16.895847+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:55:09.853705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:55:09.853705+0000","last_clean_scrub_stamp":"2026-02-23T11:55:09.853705+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160255967232,"available":160253366272,"internally_reserved":0,"allocated":2600960,"data_stored":2600960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53418655744,"available":53417480192,"internally_reserved":0,"allocated":1175552,"data_stored":1175552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1148,"kb_used_data":1148,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165508,"statfs":{"total":53418655744,"available":53417480192,"internally_reserved":0,"allocated":1175552,"data_stored":1175552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":316,"kb_used_data":316,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52166340,"statfs":{"total":53418655744,"available":53418332160,"internally_reserved":0,"allocated":323584,"data_stored":323584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1076,"kb_used_data":1076,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165580,"statfs":{"total":53418655744,"available":53417553920,"internally_reserved":0,"allocated":1101824,"data_stored":1101824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53418655744,"available":53417480192,"internally_reserved":0,"allocated":1175552,"data_stored":1175552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53418655744,"available":53417553920,"internally_reserved":0,"allocated":1101824,"data_stored":1101824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53418655744,"available":53417480192,"internally_reserved":0,"allocated":1175552,"data_stored":1175552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53418655744,"available":53418332160,"internally_reserved":0,"allocated":323584,"data_stored":323584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-23T11:55:20.860 DEBUG:teuthology.orchestra.run.trial186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-23T11:55:20.937 INFO:teuthology.orchestra.run.trial186.stderr:2026-02-23T11:55:20.932+0000 7f3c8b4b7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:55:20.938 INFO:teuthology.orchestra.run.trial186.stderr:2026-02-23T11:55:20.935+0000 7f3c8b4b7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:55:21.047 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:55:21.048 INFO:teuthology.orchestra.run.trial186.stderr:dumped all 2026-02-23T11:55:21.057 INFO:teuthology.orchestra.run.trial186.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2026-02-23T11:55:19.604497+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156499968,"kb_used":2540,"kb_used_data":2540,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156497428,"statfs":{"total":160255967232,"available":160253366272,"internally_reserved":0,"allocated":2600960,"data_stored":2600960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.741083"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-23T11:55:16.895285+0000","last_change":"2026-02-23T11:55:16.898298+0000","last_active":"2026-02-23T11:55:16.895285+0000","last_peered":"2026-02-23T11:55:16.895285+0000","last_clean":"2026-02-23T11:55:16.895285+0000","last_became_active":"2026-02-23T11:55:14.908265+0000","last_became_peered":"2026-02-23T11:55:14.908265+0000","last_unstale":"2026-02-23T11:55:16.895285+0000","last_undegraded":"2026-02-23T11:55:16.895285+0000","last_fullsized":"2026-02-23T11:55:16.895285+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:55:13.868699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:55:13.868699+0000","last_clean_scrub_stamp":"2026-02-23T11:55:13.868699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-23T11:55:16.896339+0000","last_change":"2026-02-23T11:55:16.898171+0000","last_active":"2026-02-23T11:55:16.896339+0000","last_peered":"2026-02-23T11:55:16.896339+0000","last_clean":"2026-02-23T11:55:16.896339+0000","last_became_active":"2026-02-23T11:55:14.906589+0000","last_became_peered":"2026-02-23T11:55:14.906589+0000","last_unstale":"2026-02-23T11:55:16.896339+0000","last_undegraded":"2026-02-23T11:55:16.896339+0000","last_fullsized":"2026-02-23T11:55:16.896339+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:55:13.868699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:55:13.868699+0000","last_clean_scrub_stamp":"2026-02-23T11:55:13.868699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-23T11:55:16.896938+0000","last_change":"2026-02-23T11:55:16.899363+0000","last_active":"2026-02-23T11:55:16.896938+0000","last_peered":"2026-02-23T11:55:16.896938+0000","last_clean":"2026-02-23T11:55:16.896938+0000","last_became_active":"2026-02-23T11:55:14.905759+0000","last_became_peered":"2026-02-23T11:55:14.905759+0000","last_unstale":"2026-02-23T11:55:16.896938+0000","last_undegraded":"2026-02-23T11:55:16.896938+0000","last_fullsized":"2026-02-23T11:55:16.896938+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:55:13.868699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:55:13.868699+0000","last_clean_scrub_stamp":"2026-02-23T11:55:13.868699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-23T11:55:16.896316+0000","last_change":"2026-02-23T11:55:16.899375+0000","last_active":"2026-02-23T11:55:16.896316+0000","last_peered":"2026-02-23T11:55:16.896316+0000","last_clean":"2026-02-23T11:55:16.896316+0000","last_became_active":"2026-02-23T11:55:14.911243+0000","last_became_peered":"2026-02-23T11:55:14.911243+0000","last_unstale":"2026-02-23T11:55:16.896316+0000","last_undegraded":"2026-02-23T11:55:16.896316+0000","last_fullsized":"2026-02-23T11:55:16.896316+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:55:13.868699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:55:13.868699+0000","last_clean_scrub_stamp":"2026-02-23T11:55:13.868699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'1","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-23T11:55:16.895698+0000","last_change":"2026-02-23T11:55:16.895698+0000","last_active":"2026-02-23T11:55:16.895698+0000","last_peered":"2026-02-23T11:55:16.895698+0000","last_clean":"2026-02-23T11:55:16.895698+0000","last_became_active":"2026-02-23T11:55:14.918494+0000","last_became_peered":"2026-02-23T11:55:14.918494+0000","last_unstale":"2026-02-23T11:55:16.895698+0000","last_undegraded":"2026-02-23T11:55:16.895698+0000","last_fullsized":"2026-02-23T11:55:16.895698+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:55:13.868699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:55:13.868699+0000","last_clean_scrub_stamp":"2026-02-23T11:55:13.868699+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-23T11:55:16.894547+0000","last_change":"2026-02-23T11:55:16.896243+0000","last_active":"2026-02-23T11:55:16.894547+0000","last_peered":"2026-02-23T11:55:16.894547+0000","last_clean":"2026-02-23T11:55:16.894547+0000","last_became_active":"2026-02-23T11:55:14.923409+0000","last_became_peered":"2026-02-23T11:55:14.923409+0000","last_unstale":"2026-02-23T11:55:16.894547+0000","last_undegraded":"2026-02-23T11:55:16.894547+0000","last_fullsized":"2026-02-23T11:55:16.894547+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:55:13.868699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:55:13.868699+0000","last_clean_scrub_stamp":"2026-02-23T11:55:13.868699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-23T11:55:16.893015+0000","last_change":"2026-02-23T11:55:16.895343+0000","last_active":"2026-02-23T11:55:16.893015+0000","last_peered":"2026-02-23T11:55:16.893015+0000","last_clean":"2026-02-23T11:55:16.893015+0000","last_became_active":"2026-02-23T11:55:14.921646+0000","last_became_peered":"2026-02-23T11:55:14.921646+0000","last_unstale":"2026-02-23T11:55:16.893015+0000","last_undegraded":"2026-02-23T11:55:16.893015+0000","last_fullsized":"2026-02-23T11:55:16.893015+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:55:13.868699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:55:13.868699+0000","last_clean_scrub_stamp":"2026-02-23T11:55:13.868699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":16,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-23T11:55:16.896318+0000","last_change":"2026-02-23T11:55:16.898468+0000","last_active":"2026-02-23T11:55:16.896318+0000","last_peered":"2026-02-23T11:55:16.896318+0000","last_clean":"2026-02-23T11:55:16.896318+0000","last_became_active":"2026-02-23T11:55:14.905437+0000","last_became_peered":"2026-02-23T11:55:14.905437+0000","last_unstale":"2026-02-23T11:55:16.896318+0000","last_undegraded":"2026-02-23T11:55:16.896318+0000","last_fullsized":"2026-02-23T11:55:16.896318+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:55:13.868699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:55:13.868699+0000","last_clean_scrub_stamp":"2026-02-23T11:55:13.868699+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":90,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-23T11:55:16.895847+0000","last_change":"2026-02-23T11:55:10.893657+0000","last_active":"2026-02-23T11:55:16.895847+0000","last_peered":"2026-02-23T11:55:16.895847+0000","last_clean":"2026-02-23T11:55:16.895847+0000","last_became_active":"2026-02-23T11:55:10.886604+0000","last_became_peered":"2026-02-23T11:55:10.886604+0000","last_unstale":"2026-02-23T11:55:16.895847+0000","last_undegraded":"2026-02-23T11:55:16.895847+0000","last_fullsized":"2026-02-23T11:55:16.895847+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-23T11:55:09.853705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-23T11:55:09.853705+0000","last_clean_scrub_stamp":"2026-02-23T11:55:09.853705+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160255967232,"available":160253366272,"internally_reserved":0,"allocated":2600960,"data_stored":2600960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53418655744,"available":53417480192,"internally_reserved":0,"allocated":1175552,"data_stored":1175552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1148,"kb_used_data":1148,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165508,"statfs":{"total":53418655744,"available":53417480192,"internally_reserved":0,"allocated":1175552,"data_stored":1175552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":316,"kb_used_data":316,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52166340,"statfs":{"total":53418655744,"available":53418332160,"internally_reserved":0,"allocated":323584,"data_stored":323584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1076,"kb_used_data":1076,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165580,"statfs":{"total":53418655744,"available":53417553920,"internally_reserved":0,"allocated":1101824,"data_stored":1101824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53418655744,"available":53417480192,"internally_reserved":0,"allocated":1175552,"data_stored":1175552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53418655744,"available":53417553920,"internally_reserved":0,"allocated":1101824,"data_stored":1101824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53418655744,"available":53417480192,"internally_reserved":0,"allocated":1175552,"data_stored":1175552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53418655744,"available":53418332160,"internally_reserved":0,"allocated":323584,"data_stored":323584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-23T11:55:21.058 INFO:tasks.ceph.ceph_manager.ceph:clean! 2026-02-23T11:55:21.059 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2026-02-23T11:55:21.059 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2026-02-23T11:55:21.059 DEBUG:teuthology.orchestra.run.trial186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-23T11:55:21.136 INFO:teuthology.orchestra.run.trial186.stderr:2026-02-23T11:55:21.131+0000 7f6b7339c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:55:21.137 INFO:teuthology.orchestra.run.trial186.stderr:2026-02-23T11:55:21.133+0000 7f6b7339c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:55:21.278 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:55:21.278 INFO:teuthology.orchestra.run.trial186.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2026-02-23T11:55:21.288 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2026-02-23T11:55:21.288 INFO:teuthology.run_tasks:Running task ssh_keys... 2026-02-23T11:55:21.407 DEBUG:teuthology.orchestra.run.trial186:> echo 'StrictHostKeyChecking no 2026-02-23T11:55:21.407 DEBUG:teuthology.orchestra.run.trial186:> ' > /home/ubuntu/.ssh/config && echo 'UserKnownHostsFile ' /dev/null >> /home/ubuntu/.ssh/config && chmod 600 /home/ubuntu/.ssh/config 2026-02-23T11:55:21.425 INFO:teuthology.task.ssh_keys:generated public key AAAAB3NzaC1yc2EAAAADAQABAAABAQCp7HhS+r1LpvBa8LlAzbRu6zyyBnVDH+7hx1BbSXHIwZadHlJAYpI7ygaJC5RWh7m+s45v34gr7LdQoqZEQ5uM8lTjErxVyKg8EkZ1B6bSbHiPpb6Tb7R4/8TNql0wB8xBcPCkHGowcOqNathELtJ4P1AA/ifhfSLwUUKPyow7p/pfQ9RclBPo7AbU93dEhTcbKbVbdrbl2ifeYUYZrzkV5mX8zU9HMuOune0AiTn+Vir1dJkgyUCYHu70OQkhaqtaoVXGOycFVtPbCh5LNPJcf93EHYz+LAepqvf3uWPL8YW38+0hSuZCyLoMrnF+jNh/AM2LItBqAe+otUieo/bd 2026-02-23T11:55:21.425 INFO:teuthology.task.ssh_keys:pushing keys to trial186.front.sepia.ceph.com for ubuntu 2026-02-23T11:55:21.425 DEBUG:teuthology.orchestra.run.trial186:> rm -f -- /home/ubuntu/.ssh/id_rsa 2026-02-23T11:55:21.481 DEBUG:teuthology.orchestra.run.trial186:> set -ex 2026-02-23T11:55:21.481 DEBUG:teuthology.orchestra.run.trial186:> dd of=/home/ubuntu/.ssh/id_rsa 2026-02-23T11:55:21.481 DEBUG:teuthology.orchestra.run.trial186:> chmod 0500 /home/ubuntu/.ssh/id_rsa 2026-02-23T11:55:21.538 DEBUG:teuthology.orchestra.run.trial186:> rm -f -- /home/ubuntu/.ssh/id_rsa.pub 2026-02-23T11:55:21.593 DEBUG:teuthology.orchestra.run.trial186:> set -ex 2026-02-23T11:55:21.593 DEBUG:teuthology.orchestra.run.trial186:> dd of=/home/ubuntu/.ssh/id_rsa.pub 2026-02-23T11:55:21.650 DEBUG:teuthology.orchestra.run.trial186:> cp -v -a /home/ubuntu/.ssh/authorized_keys /home/ubuntu/.ssh/authorized_keys_2026-02-23_11:55:21:650320 2026-02-23T11:55:21.708 INFO:teuthology.orchestra.run.trial186.stdout:'/home/ubuntu/.ssh/authorized_keys' -> '/home/ubuntu/.ssh/authorized_keys_2026-02-23_11:55:21:650320' 2026-02-23T11:55:21.708 DEBUG:teuthology.orchestra.run.trial186:> set -ex 2026-02-23T11:55:21.708 DEBUG:teuthology.orchestra.run.trial186:> dd of=/home/ubuntu/.ssh/authorized_keys conv=notrunc oflag=append 2026-02-23T11:55:21.765 INFO:teuthology.run_tasks:Running task cbt... 2026-02-23T11:55:21.791 INFO:tasks.cbt:cbt configuration is {'cluster': {'user': 'ubuntu', 'head': 'ubuntu@trial186.front.sepia.ceph.com', 'osds': ['ubuntu@trial186.front.sepia.ceph.com'], 'mons': ['ubuntu@trial186.front.sepia.ceph.com'], 'clients': ['ubuntu@trial186.front.sepia.ceph.com'], 'rgws': {'ubuntu@trial186.front.sepia.ceph.com': None}, 'osds_per_node': 3, 'rebuild_every_test': False, 'use_existing': True, 'is_teuthology': True, 'iterations': 1, 'tmp_dir': '/tmp/cbt', 'pool_profiles': {'replicated': {'pg_size': 256, 'pgp_size': 256, 'replication': 'replicated'}}, 'pid_dir': '/var/run/ceph'}, 'benchmarks': {'radosbench': {'concurrent_ops': 4, 'concurrent_procs': 2, 'op_size': [4194304], 'pool_monitoring_list': ['collectl'], 'pool_profile': 'replicated', 'run_monitoring_list': ['collectl'], 'time': 60, 'write_only': True}}, 'client_endpoints': None, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}} 2026-02-23T11:55:21.791 DEBUG:teuthology.orchestra.run.trial186:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/cbt 2026-02-23T11:55:21.826 DEBUG:teuthology.orchestra.run.trial186:> set -ex 2026-02-23T11:55:21.826 DEBUG:teuthology.orchestra.run.trial186:> dd of=/home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2026-02-23T11:55:21.878 DEBUG:teuthology.orchestra.run.trial186:> git clone --depth 1 -b master https://github.com/ceph/cbt.git /home/ubuntu/cephtest/cbt 2026-02-23T11:55:21.942 INFO:teuthology.orchestra.run.trial186.stderr:Cloning into '/home/ubuntu/cephtest/cbt'... 2026-02-23T11:55:22.182 INFO:tasks.cbt:Installing collectl 2026-02-23T11:55:22.182 DEBUG:teuthology.orchestra.run.trial186:> sudo mkdir collectl && cd collectl && sudo wget https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download -O collectl.tar.gz && sudo tar -xvf collectl.tar.gz && cd collectl-4.3.1 && sudo ./INSTALL 2026-02-23T11:55:22.257 INFO:teuthology.orchestra.run.trial186.stderr:--2026-02-23 11:55:22-- https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download 2026-02-23T11:55:22.342 INFO:teuthology.orchestra.run.trial186.stderr:Resolving sourceforge.net (sourceforge.net)... 104.18.12.149, 104.18.13.149, 2606:4700::6812:d95, ... 2026-02-23T11:55:22.345 INFO:teuthology.orchestra.run.trial186.stderr:Connecting to sourceforge.net (sourceforge.net)|104.18.12.149|:443... connected. 2026-02-23T11:55:22.533 INFO:teuthology.orchestra.run.trial186.stderr:HTTP request sent, awaiting response... 302 Found 2026-02-23T11:55:22.534 INFO:teuthology.orchestra.run.trial186.stderr:Location: https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABpnEAqB81xkrn6oabxXJ1SLPn0Eechld9DYVayptil89Vc3vg3Ke-PjX2hPp7ij65ALYrSvqWHymUz8w2SYHihVSeIEw%3D%3D&use_mirror=gigenet&r= [following] 2026-02-23T11:55:22.534 INFO:teuthology.orchestra.run.trial186.stderr:--2026-02-23 11:55:22-- https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABpnEAqB81xkrn6oabxXJ1SLPn0Eechld9DYVayptil89Vc3vg3Ke-PjX2hPp7ij65ALYrSvqWHymUz8w2SYHihVSeIEw%3D%3D&use_mirror=gigenet&r= 2026-02-23T11:55:22.605 INFO:teuthology.orchestra.run.trial186.stderr:Resolving downloads.sourceforge.net (downloads.sourceforge.net)... 104.18.12.149, 104.18.13.149, 2606:4700::6812:c95, ... 2026-02-23T11:55:22.608 INFO:teuthology.orchestra.run.trial186.stderr:Connecting to downloads.sourceforge.net (downloads.sourceforge.net)|104.18.12.149|:443... connected. 2026-02-23T11:55:22.795 INFO:teuthology.orchestra.run.trial186.stderr:HTTP request sent, awaiting response... 302 Found 2026-02-23T11:55:22.795 INFO:teuthology.orchestra.run.trial186.stderr:Location: https://gigenet.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?viasf=1 [following] 2026-02-23T11:55:22.795 INFO:teuthology.orchestra.run.trial186.stderr:--2026-02-23 11:55:22-- https://gigenet.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?viasf=1 2026-02-23T11:55:22.804 INFO:teuthology.orchestra.run.trial186.stderr:Resolving gigenet.dl.sourceforge.net (gigenet.dl.sourceforge.net)... 69.65.16.141 2026-02-23T11:55:22.824 INFO:teuthology.orchestra.run.trial186.stderr:Connecting to gigenet.dl.sourceforge.net (gigenet.dl.sourceforge.net)|69.65.16.141|:443... connected. 2026-02-23T11:55:22.968 INFO:teuthology.orchestra.run.trial186.stderr:HTTP request sent, awaiting response... 200 OK 2026-02-23T11:55:22.968 INFO:teuthology.orchestra.run.trial186.stderr:Length: 620693 (606K) [application/x-gzip] 2026-02-23T11:55:22.968 INFO:teuthology.orchestra.run.trial186.stderr:Saving to: ‘collectl.tar.gz’ 2026-02-23T11:55:22.968 INFO:teuthology.orchestra.run.trial186.stderr: 2026-02-23T11:55:22.987 INFO:teuthology.orchestra.run.trial186.stderr: 0K .......... .......... .......... .......... .......... 8% 2.47M 0s 2026-02-23T11:55:23.007 INFO:teuthology.orchestra.run.trial186.stderr: 50K .......... .......... .......... .......... .......... 16% 2.54M 0s 2026-02-23T11:55:23.007 INFO:teuthology.orchestra.run.trial186.stderr: 100K .......... .......... .......... .......... .......... 24% 92.1M 0s 2026-02-23T11:55:23.026 INFO:teuthology.orchestra.run.trial186.stderr: 150K .......... .......... .......... .......... .......... 32% 2.61M 0s 2026-02-23T11:55:23.026 INFO:teuthology.orchestra.run.trial186.stderr: 200K .......... .......... .......... .......... .......... 41% 109M 0s 2026-02-23T11:55:23.027 INFO:teuthology.orchestra.run.trial186.stderr: 250K .......... .......... .......... .......... .......... 49% 116M 0s 2026-02-23T11:55:23.027 INFO:teuthology.orchestra.run.trial186.stderr: 300K .......... .......... .......... .......... .......... 57% 120M 0s 2026-02-23T11:55:23.027 INFO:teuthology.orchestra.run.trial186.stderr: 350K .......... .......... .......... .......... .......... 65% 92.4M 0s 2026-02-23T11:55:23.047 INFO:teuthology.orchestra.run.trial186.stderr: 400K .......... .......... .......... .......... .......... 74% 2.71M 0s 2026-02-23T11:55:23.048 INFO:teuthology.orchestra.run.trial186.stderr: 450K .......... .......... .......... .......... .......... 82% 131M 0s 2026-02-23T11:55:23.048 INFO:teuthology.orchestra.run.trial186.stderr: 500K .......... .......... .......... .......... .......... 90% 113M 0s 2026-02-23T11:55:23.048 INFO:teuthology.orchestra.run.trial186.stderr: 550K .......... .......... .......... .......... .......... 98% 108M 0s 2026-02-23T11:55:23.048 INFO:teuthology.orchestra.run.trial186.stderr: 600K ...... 100% 11.4T=0.08s 2026-02-23T11:55:23.048 INFO:teuthology.orchestra.run.trial186.stderr: 2026-02-23T11:55:23.048 INFO:teuthology.orchestra.run.trial186.stderr:2026-02-23 11:55:23 (7.46 MB/s) - ‘collectl.tar.gz’ saved [620693/620693] 2026-02-23T11:55:23.048 INFO:teuthology.orchestra.run.trial186.stderr: 2026-02-23T11:55:23.069 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/ 2026-02-23T11:55:23.069 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/collectl.conf 2026-02-23T11:55:23.069 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/docs/ 2026-02-23T11:55:23.069 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/docs/SlowProc.html 2026-02-23T11:55:23.069 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/docs/BuddyInfo.html 2026-02-23T11:55:23.069 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/docs/WhySummary.html 2026-02-23T11:55:23.069 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/docs/Socket.html 2026-02-23T11:55:23.069 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/docs/colmux.html 2026-02-23T11:55:23.069 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/docs/Tutorial.html 2026-02-23T11:55:23.069 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/docs/Gexpr.html 2026-02-23T11:55:23.069 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/docs/Import.html 2026-02-23T11:55:23.069 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/docs/style.css 2026-02-23T11:55:23.069 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/docs/Ganglia3.jpg 2026-02-23T11:55:23.070 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/docs/Ganglia2.jpg 2026-02-23T11:55:23.070 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/docs/FileNaming.html 2026-02-23T11:55:23.070 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/docs/TheMath.html 2026-02-23T11:55:23.070 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/docs/Environmental.html 2026-02-23T11:55:23.070 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/docs/Process.html 2026-02-23T11:55:23.070 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/docs/OutputFiles.html 2026-02-23T11:55:23.070 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/docs/HiResTime.html 2026-02-23T11:55:23.070 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/docs/OutputFormats.html 2026-02-23T11:55:23.070 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/docs/slub.jpg 2026-02-23T11:55:23.070 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/docs/Lustre.html 2026-02-23T11:55:23.070 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/docs/Documentation.html 2026-02-23T11:55:23.070 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/docs/Inodes.html 2026-02-23T11:55:23.070 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/docs/Graphite.html 2026-02-23T11:55:23.070 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/docs/InputFiles.html 2026-02-23T11:55:23.070 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/docs/Export.html 2026-02-23T11:55:23.070 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/docs/Interrupts.html 2026-02-23T11:55:23.070 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/docs/Startup.html 2026-02-23T11:55:23.070 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/docs/Performance.html 2026-02-23T11:55:23.070 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/docs/Messages.html 2026-02-23T11:55:23.070 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/docs/Logging.html 2026-02-23T11:55:23.070 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/docs/Data-detail.html 2026-02-23T11:55:23.071 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/docs/CPUs.html 2026-02-23T11:55:23.071 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/docs/Matrix.html 2026-02-23T11:55:23.071 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/docs/Data-brief.html 2026-02-23T11:55:23.071 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/docs/SlabInfo.html 2026-02-23T11:55:23.071 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/docs/Misc.html 2026-02-23T11:55:23.071 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/docs/Examples.html 2026-02-23T11:55:23.071 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/docs/OpenStack.html 2026-02-23T11:55:23.071 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/docs/Plotfiles.html 2026-02-23T11:55:23.071 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/docs/ColmuxCPU.jpg 2026-02-23T11:55:23.071 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/docs/Tutorial-Lustre.html 2026-02-23T11:55:23.071 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/docs/Data-verbose.html 2026-02-23T11:55:23.071 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/docs/Data.html 2026-02-23T11:55:23.071 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/docs/Exceptions.html 2026-02-23T11:55:23.071 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/docs/RunningAsAService.html 2026-02-23T11:55:23.071 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/docs/FAQ-collectl.html 2026-02-23T11:55:23.071 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/docs/Ganglia.jpg 2026-02-23T11:55:23.072 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/docs/OperationalModes.html 2026-02-23T11:55:23.072 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/docs/Playback.html 2026-02-23T11:55:23.072 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/docs/Hello.html 2026-02-23T11:55:23.072 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/docs/Sockets.html 2026-02-23T11:55:23.072 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/docs/Network.html 2026-02-23T11:55:23.072 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/docs/Infiniband.html 2026-02-23T11:55:23.072 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/docs/Memory.html 2026-02-23T11:55:23.072 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/docs/ColmuxLustre.jpg 2026-02-23T11:55:23.072 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/docs/Disks.html 2026-02-23T11:55:23.072 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/docs/NfsInfo.html 2026-02-23T11:55:23.072 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/service/ 2026-02-23T11:55:23.072 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/service/collectl.service 2026-02-23T11:55:23.072 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/initd/ 2026-02-23T11:55:23.072 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/initd/collectl-suse 2026-02-23T11:55:23.072 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/initd/collectl-debian 2026-02-23T11:55:23.072 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/initd/collectl-generic 2026-02-23T11:55:23.072 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/initd/collectl 2026-02-23T11:55:23.073 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/misc.ph 2026-02-23T11:55:23.073 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/gexpr.ph 2026-02-23T11:55:23.073 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/statsd.ph 2026-02-23T11:55:23.073 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/envrules.std 2026-02-23T11:55:23.073 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/hello.ph 2026-02-23T11:55:23.073 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/vmsum.ph 2026-02-23T11:55:23.073 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/proctree.ph 2026-02-23T11:55:23.073 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/README-WINDOWS 2026-02-23T11:55:23.073 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/formatit.ph 2026-02-23T11:55:23.073 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/colmux 2026-02-23T11:55:23.073 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/INSTALL 2026-02-23T11:55:23.073 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/graphite.ph 2026-02-23T11:55:23.073 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/RELEASE-collectl 2026-02-23T11:55:23.073 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/vmstat.ph 2026-02-23T11:55:23.073 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/README 2026-02-23T11:55:23.073 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/man1/ 2026-02-23T11:55:23.073 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/man1/collectl.1 2026-02-23T11:55:23.073 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/man1/colmux.1 2026-02-23T11:55:23.073 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/lexpr.ph 2026-02-23T11:55:23.073 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/client.pl 2026-02-23T11:55:23.073 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/collectl 2026-02-23T11:55:23.073 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/COPYING 2026-02-23T11:55:23.073 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/vnet.ph 2026-02-23T11:55:23.073 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/GPL 2026-02-23T11:55:23.073 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/UNINSTALL 2026-02-23T11:55:23.073 INFO:teuthology.orchestra.run.trial186.stdout:collectl-4.3.1/ARTISTIC 2026-02-23T11:55:23.111 INFO:teuthology.orchestra.run.trial186.stderr:./INSTALL: line 90: chkconfig: command not found 2026-02-23T11:55:23.113 DEBUG:teuthology.orchestra.run.trial186:> sudo yum -y install librbd-devel pdsh pdsh-rcmd-ssh perf 2026-02-23T11:55:23.471 INFO:teuthology.orchestra.run.trial186.stdout:Last metadata expiration check: 0:01:00 ago on Mon 23 Feb 2026 11:54:23 AM UTC. 2026-02-23T11:55:23.557 INFO:teuthology.orchestra.run.trial186.stdout:Dependencies resolved. 2026-02-23T11:55:23.558 INFO:teuthology.orchestra.run.trial186.stdout:================================================================================ 2026-02-23T11:55:23.558 INFO:teuthology.orchestra.run.trial186.stdout: Package Arch Version Repository Size 2026-02-23T11:55:23.558 INFO:teuthology.orchestra.run.trial186.stdout:================================================================================ 2026-02-23T11:55:23.558 INFO:teuthology.orchestra.run.trial186.stdout:Installing: 2026-02-23T11:55:23.558 INFO:teuthology.orchestra.run.trial186.stdout: librbd-devel x86_64 2:20.3.0-5439.ge371433f.el9 ceph 24 k 2026-02-23T11:55:23.558 INFO:teuthology.orchestra.run.trial186.stdout: pdsh x86_64 2.34-7.el9 epel 118 k 2026-02-23T11:55:23.558 INFO:teuthology.orchestra.run.trial186.stdout: pdsh-rcmd-ssh x86_64 2.34-7.el9 epel 13 k 2026-02-23T11:55:23.558 INFO:teuthology.orchestra.run.trial186.stdout: perf x86_64 5.14.0-681.el9 appstream 3.2 M 2026-02-23T11:55:23.558 INFO:teuthology.orchestra.run.trial186.stdout:Installing dependencies: 2026-02-23T11:55:23.558 INFO:teuthology.orchestra.run.trial186.stdout: libradospp-devel x86_64 2:20.3.0-5439.ge371433f.el9 ceph 31 k 2026-02-23T11:55:23.558 INFO:teuthology.orchestra.run.trial186.stdout: libtraceevent x86_64 1.8.4-2.el9 baseos 290 k 2026-02-23T11:55:23.558 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:55:23.558 INFO:teuthology.orchestra.run.trial186.stdout:Transaction Summary 2026-02-23T11:55:23.558 INFO:teuthology.orchestra.run.trial186.stdout:================================================================================ 2026-02-23T11:55:23.558 INFO:teuthology.orchestra.run.trial186.stdout:Install 6 Packages 2026-02-23T11:55:23.558 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:55:23.558 INFO:teuthology.orchestra.run.trial186.stdout:Total download size: 3.7 M 2026-02-23T11:55:23.558 INFO:teuthology.orchestra.run.trial186.stdout:Installed size: 14 M 2026-02-23T11:55:23.558 INFO:teuthology.orchestra.run.trial186.stdout:Downloading Packages: 2026-02-23T11:55:24.354 INFO:teuthology.orchestra.run.trial186.stdout:(1/6): librbd-devel-20.3.0-5439.ge371433f.el9.x 315 kB/s | 24 kB 00:00 2026-02-23T11:55:24.362 INFO:teuthology.orchestra.run.trial186.stdout:(2/6): libradospp-devel-20.3.0-5439.ge371433f.e 375 kB/s | 31 kB 00:00 2026-02-23T11:55:24.456 INFO:teuthology.orchestra.run.trial186.stdout:(3/6): perf-5.14.0-681.el9.x86_64.rpm 32 MB/s | 3.2 MB 00:00 2026-02-23T11:55:24.506 INFO:teuthology.orchestra.run.trial186.stdout:(4/6): pdsh-2.34-7.el9.x86_64.rpm 824 kB/s | 118 kB 00:00 2026-02-23T11:55:24.524 INFO:teuthology.orchestra.run.trial186.stdout:(5/6): pdsh-rcmd-ssh-2.34-7.el9.x86_64.rpm 194 kB/s | 13 kB 00:00 2026-02-23T11:55:25.240 INFO:teuthology.orchestra.run.trial186.stdout:(6/6): libtraceevent-1.8.4-2.el9.x86_64.rpm 302 kB/s | 290 kB 00:00 2026-02-23T11:55:25.243 INFO:teuthology.orchestra.run.trial186.stdout:-------------------------------------------------------------------------------- 2026-02-23T11:55:25.243 INFO:teuthology.orchestra.run.trial186.stdout:Total 2.2 MB/s | 3.7 MB 00:01 2026-02-23T11:55:25.267 INFO:teuthology.orchestra.run.trial186.stdout:Running transaction check 2026-02-23T11:55:25.277 INFO:teuthology.orchestra.run.trial186.stdout:Transaction check succeeded. 2026-02-23T11:55:25.277 INFO:teuthology.orchestra.run.trial186.stdout:Running transaction test 2026-02-23T11:55:25.332 INFO:teuthology.orchestra.run.trial186.stdout:Transaction test succeeded. 2026-02-23T11:55:25.333 INFO:teuthology.orchestra.run.trial186.stdout:Running transaction 2026-02-23T11:55:25.403 INFO:teuthology.orchestra.run.trial186.stdout: Preparing : 1/1 2026-02-23T11:55:25.409 INFO:teuthology.orchestra.run.trial186.stdout: Installing : pdsh-rcmd-ssh-2.34-7.el9.x86_64 1/6 2026-02-23T11:55:25.449 INFO:teuthology.orchestra.run.trial186.stdout: Installing : pdsh-2.34-7.el9.x86_64 2/6 2026-02-23T11:55:25.453 INFO:teuthology.orchestra.run.trial186.stdout: Installing : libtraceevent-1.8.4-2.el9.x86_64 3/6 2026-02-23T11:55:25.456 INFO:teuthology.orchestra.run.trial186.stdout: Installing : libradospp-devel-2:20.3.0-5439.ge371433f.el9.x86_64 4/6 2026-02-23T11:55:25.618 INFO:teuthology.orchestra.run.trial186.stdout: Installing : librbd-devel-2:20.3.0-5439.ge371433f.el9.x86_64 5/6 2026-02-23T11:55:25.638 INFO:teuthology.orchestra.run.trial186.stdout: Installing : perf-5.14.0-681.el9.x86_64 6/6 2026-02-23T11:55:25.850 INFO:teuthology.orchestra.run.trial186.stdout: Running scriptlet: perf-5.14.0-681.el9.x86_64 6/6 2026-02-23T11:55:25.850 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : libradospp-devel-2:20.3.0-5439.ge371433f.el9.x86_64 1/6 2026-02-23T11:55:25.851 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : librbd-devel-2:20.3.0-5439.ge371433f.el9.x86_64 2/6 2026-02-23T11:55:25.851 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : libtraceevent-1.8.4-2.el9.x86_64 3/6 2026-02-23T11:55:25.851 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : perf-5.14.0-681.el9.x86_64 4/6 2026-02-23T11:55:25.851 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : pdsh-2.34-7.el9.x86_64 5/6 2026-02-23T11:55:25.912 INFO:teuthology.orchestra.run.trial186.stdout: Verifying : pdsh-rcmd-ssh-2.34-7.el9.x86_64 6/6 2026-02-23T11:55:25.913 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:55:25.913 INFO:teuthology.orchestra.run.trial186.stdout:Installed: 2026-02-23T11:55:25.913 INFO:teuthology.orchestra.run.trial186.stdout: libradospp-devel-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:55:25.913 INFO:teuthology.orchestra.run.trial186.stdout: librbd-devel-2:20.3.0-5439.ge371433f.el9.x86_64 2026-02-23T11:55:25.913 INFO:teuthology.orchestra.run.trial186.stdout: libtraceevent-1.8.4-2.el9.x86_64 2026-02-23T11:55:25.913 INFO:teuthology.orchestra.run.trial186.stdout: pdsh-2.34-7.el9.x86_64 2026-02-23T11:55:25.913 INFO:teuthology.orchestra.run.trial186.stdout: pdsh-rcmd-ssh-2.34-7.el9.x86_64 2026-02-23T11:55:25.913 INFO:teuthology.orchestra.run.trial186.stdout: perf-5.14.0-681.el9.x86_64 2026-02-23T11:55:25.913 INFO:teuthology.orchestra.run.trial186.stdout: 2026-02-23T11:55:25.914 INFO:teuthology.orchestra.run.trial186.stdout:Complete! 2026-02-23T11:55:25.943 DEBUG:teuthology.orchestra.run.trial186:> python3 -m pip install -r /home/ubuntu/cephtest/cbt/requirements.txt 2026-02-23T11:55:26.080 INFO:teuthology.orchestra.run.trial186.stdout:Defaulting to user installation because normal site-packages is not writeable 2026-02-23T11:55:26.091 INFO:teuthology.orchestra.run.trial186.stdout:Requirement already satisfied: pyyaml in /usr/lib64/python3.9/site-packages (from -r /home/ubuntu/cephtest/cbt/requirements.txt (line 1)) (5.4.1) 2026-02-23T11:55:26.552 INFO:teuthology.orchestra.run.trial186.stdout:Collecting lxml 2026-02-23T11:55:26.582 INFO:teuthology.orchestra.run.trial186.stdout: Downloading lxml-6.0.2-cp39-cp39-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl (5.3 MB) 2026-02-23T11:55:26.930 INFO:teuthology.orchestra.run.trial186.stdout:Collecting matplotlib 2026-02-23T11:55:26.934 INFO:teuthology.orchestra.run.trial186.stdout: Downloading matplotlib-3.9.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (8.3 MB) 2026-02-23T11:55:26.992 INFO:teuthology.orchestra.run.trial186.stdout:Collecting cycler>=0.10 2026-02-23T11:55:26.996 INFO:teuthology.orchestra.run.trial186.stdout: Downloading cycler-0.12.1-py3-none-any.whl (8.3 kB) 2026-02-23T11:55:26.997 INFO:teuthology.orchestra.run.trial186.stdout:Requirement already satisfied: numpy>=1.23 in /usr/lib64/python3.9/site-packages (from matplotlib->-r /home/ubuntu/cephtest/cbt/requirements.txt (line 3)) (1.23.5) 2026-02-23T11:55:27.153 INFO:teuthology.orchestra.run.trial186.stdout:Collecting contourpy>=1.0.1 2026-02-23T11:55:27.156 INFO:teuthology.orchestra.run.trial186.stdout: Downloading contourpy-1.3.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (321 kB) 2026-02-23T11:55:27.159 INFO:teuthology.orchestra.run.trial186.stdout:Requirement already satisfied: packaging>=20.0 in /usr/lib/python3.9/site-packages (from matplotlib->-r /home/ubuntu/cephtest/cbt/requirements.txt (line 3)) (20.9) 2026-02-23T11:55:27.304 INFO:teuthology.orchestra.run.trial186.stdout:Collecting kiwisolver>=1.3.1 2026-02-23T11:55:27.308 INFO:teuthology.orchestra.run.trial186.stdout: Downloading kiwisolver-1.4.7-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.6 MB) 2026-02-23T11:55:27.351 INFO:teuthology.orchestra.run.trial186.stdout:Collecting importlib-resources>=3.2.0 2026-02-23T11:55:27.355 INFO:teuthology.orchestra.run.trial186.stdout: Downloading importlib_resources-6.5.2-py3-none-any.whl (37 kB) 2026-02-23T11:55:27.778 INFO:teuthology.orchestra.run.trial186.stdout:Collecting pillow>=8 2026-02-23T11:55:27.782 INFO:teuthology.orchestra.run.trial186.stdout: Downloading pillow-11.3.0-cp39-cp39-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (6.6 MB) 2026-02-23T11:55:27.815 INFO:teuthology.orchestra.run.trial186.stdout:Requirement already satisfied: python-dateutil>=2.7 in /usr/lib/python3.9/site-packages (from matplotlib->-r /home/ubuntu/cephtest/cbt/requirements.txt (line 3)) (2.9.0.post0) 2026-02-23T11:55:27.815 INFO:teuthology.orchestra.run.trial186.stdout:Requirement already satisfied: pyparsing>=2.3.1 in /usr/lib/python3.9/site-packages (from matplotlib->-r /home/ubuntu/cephtest/cbt/requirements.txt (line 3)) (2.4.7) 2026-02-23T11:55:28.150 INFO:teuthology.orchestra.run.trial186.stdout:Collecting fonttools>=4.22.0 2026-02-23T11:55:28.154 INFO:teuthology.orchestra.run.trial186.stdout: Downloading fonttools-4.60.2-cp39-cp39-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (4.8 MB) 2026-02-23T11:55:28.260 INFO:teuthology.orchestra.run.trial186.stdout:Collecting zipp>=3.1.0 2026-02-23T11:55:28.264 INFO:teuthology.orchestra.run.trial186.stdout: Downloading zipp-3.23.0-py3-none-any.whl (10 kB) 2026-02-23T11:55:28.275 INFO:teuthology.orchestra.run.trial186.stdout:Requirement already satisfied: six>=1.5 in /usr/lib/python3.9/site-packages (from python-dateutil>=2.7->matplotlib->-r /home/ubuntu/cephtest/cbt/requirements.txt (line 3)) (1.15.0) 2026-02-23T11:55:28.389 INFO:teuthology.orchestra.run.trial186.stdout:Installing collected packages: zipp, pillow, kiwisolver, importlib-resources, fonttools, cycler, contourpy, matplotlib, lxml 2026-02-23T11:55:29.523 INFO:teuthology.orchestra.run.trial186.stdout:Successfully installed contourpy-1.3.0 cycler-0.12.1 fonttools-4.60.2 importlib-resources-6.5.2 kiwisolver-1.4.7 lxml-6.0.2 matplotlib-3.9.4 pillow-11.3.0 zipp-3.23.0 2026-02-23T11:55:39.605 INFO:tasks.cbt:benchmark: radosbench 2026-02-23T11:55:39.605 DEBUG:teuthology.orchestra.run.trial186:> sudo /sbin/sysctl -q -w kernel.perf_event_paranoid=0 2026-02-23T11:55:39.632 DEBUG:teuthology.orchestra.run.trial186:> /home/ubuntu/cephtest/cbt/cbt.py -a /home/ubuntu/cephtest/archive/cbt /home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2026-02-23T11:55:39.745 INFO:teuthology.orchestra.run.trial186.stderr:11:55:39 - DEBUG - cbt - Settings.cluster: 2026-02-23T11:55:39.746 INFO:teuthology.orchestra.run.trial186.stderr: {'archive_dir': '/home/ubuntu/cephtest/archive/cbt', 2026-02-23T11:55:39.746 INFO:teuthology.orchestra.run.trial186.stderr: 'clients': ['ubuntu@trial186.front.sepia.ceph.com'], 2026-02-23T11:55:39.746 INFO:teuthology.orchestra.run.trial186.stderr: 'conf_file': '/etc/ceph/ceph.conf', 2026-02-23T11:55:39.746 INFO:teuthology.orchestra.run.trial186.stderr: 'head': 'ubuntu@trial186.front.sepia.ceph.com', 2026-02-23T11:55:39.746 INFO:teuthology.orchestra.run.trial186.stderr: 'is_teuthology': True, 2026-02-23T11:55:39.746 INFO:teuthology.orchestra.run.trial186.stderr: 'iterations': 1, 2026-02-23T11:55:39.746 INFO:teuthology.orchestra.run.trial186.stderr: 'mons': ['ubuntu@trial186.front.sepia.ceph.com'], 2026-02-23T11:55:39.746 INFO:teuthology.orchestra.run.trial186.stderr: 'osds': ['ubuntu@trial186.front.sepia.ceph.com'], 2026-02-23T11:55:39.747 INFO:teuthology.orchestra.run.trial186.stderr: 'osds_per_node': 3, 2026-02-23T11:55:39.747 INFO:teuthology.orchestra.run.trial186.stderr: 'pid_dir': '/var/run/ceph', 2026-02-23T11:55:39.747 INFO:teuthology.orchestra.run.trial186.stderr: 'pool_profiles': {'replicated': {'pg_size': 256, 2026-02-23T11:55:39.747 INFO:teuthology.orchestra.run.trial186.stderr: 'pgp_size': 256, 2026-02-23T11:55:39.747 INFO:teuthology.orchestra.run.trial186.stderr: 'replication': 'replicated'}}, 2026-02-23T11:55:39.747 INFO:teuthology.orchestra.run.trial186.stderr: 'rebuild_every_test': False, 2026-02-23T11:55:39.747 INFO:teuthology.orchestra.run.trial186.stderr: 'rgws': {'ubuntu@trial186.front.sepia.ceph.com': None}, 2026-02-23T11:55:39.747 INFO:teuthology.orchestra.run.trial186.stderr: 'tmp_dir': '/tmp/cbt', 2026-02-23T11:55:39.747 INFO:teuthology.orchestra.run.trial186.stderr: 'use_existing': True, 2026-02-23T11:55:39.747 INFO:teuthology.orchestra.run.trial186.stderr: 'user': 'ubuntu'} 2026-02-23T11:55:39.747 INFO:teuthology.orchestra.run.trial186.stderr:11:55:39 - INFO - cbt - Results dir: /home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3 2026-02-23T11:55:39.747 INFO:teuthology.orchestra.run.trial186.stderr:11:55:39 - INFO - cbt - Pausing for 60s for idle monitoring. 2026-02-23T11:55:39.748 INFO:teuthology.orchestra.run.trial186.stderr:11:55:39 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial186.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/idle_monitoring/collectl 2026-02-23T11:55:39.915 INFO:teuthology.orchestra.run.trial186.stderr:11:55:39 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial186.front.sepia.ceph.com collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/idle_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2026-02-23T11:55:39.917 INFO:teuthology.orchestra.run.trial186.stderr:11:55:39 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial186.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/idle_monitoring/perf 2026-02-23T11:55:40.080 INFO:teuthology.orchestra.run.trial186.stderr:11:55:40 - DEBUG - cbt - PerfMonitoring: remote_node 2026-02-23T11:55:40.080 INFO:teuthology.orchestra.run.trial186.stderr:11:55:40 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial186.front.sepia.ceph.com for pid in `cat /var/run/ceph/crimson-osd.*.pid`; do perf stat -p ${pid} -o /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/idle_monitoring/perf/perf_stat.${pid} & done 2026-02-23T11:56:40.142 INFO:teuthology.orchestra.run.trial186.stderr:11:56:40 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial186.front.sepia.ceph.com pkill -SIGINT -f collectl 2026-02-23T11:56:40.324 INFO:teuthology.orchestra.run.trial186.stderr:11:56:40 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial186.front.sepia.ceph.com sudo pkill -SIGINT -f perf\ 2026-02-23T11:56:40.505 INFO:teuthology.orchestra.run.trial186.stderr:11:56:40 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial186.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/idle_monitoring/perf/perf.data 2026-02-23T11:56:40.506 INFO:teuthology.orchestra.run.trial186.stderr:11:56:40 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial186.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/idle_monitoring/perf/perf_stat.* 2026-02-23T11:56:40.509 INFO:teuthology.orchestra.run.trial186.stderr:11:56:40 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial186.front.sepia.ceph.com sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/* 2026-02-23T11:56:40.684 INFO:teuthology.orchestra.run.trial186.stderr:11:56:40 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=rpdcp -f 10 -R ssh -w ubuntu@trial186.front.sepia.ceph.com -r /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/* /home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3 2026-02-23T11:56:40.849 INFO:teuthology.orchestra.run.trial186.stderr:11:56:40 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial186.front.sepia.ceph.com sudo killall -9 rados 2026-02-23T11:56:41.031 INFO:teuthology.orchestra.run.trial186.stderr:11:56:41 - INFO - cbt - Results dir: /home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3 2026-02-23T11:56:41.031 INFO:teuthology.orchestra.run.trial186.stderr:11:56:41 - INFO - cbt - Skipping existing test in /home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3. 2026-02-23T11:56:41.031 INFO:teuthology.orchestra.run.trial186.stderr:11:56:41 - INFO - cbt - Running benchmark /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004 2026-02-23T11:56:41.031 INFO:teuthology.orchestra.run.trial186.stderr:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3 2026-02-23T11:56:41.031 INFO:teuthology.orchestra.run.trial186.stderr:{'concurrent_ops': 4, 'concurrent_procs': 2, 'pool_profile': 'replicated', 'time': 60, 'write_only': True, 'op_size': 4194304, 'pool_monitoring_list': 'collectl', 'run_monitoring_list': 'collectl', 'benchmark': 'radosbench', 'iteration': 0} == iteration 0 == 2026-02-23T11:56:41.031 INFO:teuthology.orchestra.run.trial186.stderr:11:56:41 - INFO - cbt - Setting OSD Read Ahead to: 0 2026-02-23T11:56:41.031 INFO:teuthology.orchestra.run.trial186.stderr:11:56:41 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial186.front.sepia.ceph.com find /dev/disk/by-partlabel/osd-device-*data -exec readlink {} \; | cut -d"/" -f 3 | sed "s/[0-9]$//" | xargs -I{} sudo sh -c "echo 0 > /sys/block/'{}'/queue/read_ahead_kb" 2026-02-23T11:56:41.032 INFO:teuthology.orchestra.run.trial186.stderr:11:56:41 - DEBUG - cbt - Cleaning existing temporary run directory: /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004 2026-02-23T11:56:41.033 INFO:teuthology.orchestra.run.trial186.stderr:11:56:41 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial186.front.sepia.ceph.com sudo rm -rf /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004 2026-02-23T11:56:41.201 INFO:teuthology.orchestra.run.trial186.stderr:11:56:41 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial186.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/pool_monitoring/collectl 2026-02-23T11:56:41.361 INFO:teuthology.orchestra.run.trial186.stderr:11:56:41 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial186.front.sepia.ceph.com collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/pool_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2026-02-23T11:56:41.363 INFO:teuthology.orchestra.run.trial186.stderr:11:56:41 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial186.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/pool_monitoring/perf 2026-02-23T11:56:41.523 INFO:teuthology.orchestra.run.trial186.stderr:11:56:41 - DEBUG - cbt - PerfMonitoring: remote_node 2026-02-23T11:56:41.523 INFO:teuthology.orchestra.run.trial186.stderr:11:56:41 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial186.front.sepia.ceph.com for pid in `cat /var/run/ceph/crimson-osd.*.pid`; do perf stat -p ${pid} -o /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/pool_monitoring/perf/perf_stat.${pid} & done 2026-02-23T11:56:41.525 INFO:teuthology.orchestra.run.trial186.stderr:11:56:41 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial186.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool delete rados-bench-cbt rados-bench-cbt --yes-i-really-really-mean-it 2026-02-23T11:56:41.868 INFO:teuthology.orchestra.run.trial186.stderr:11:56:41 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial186.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool create rados-bench-cbt 256 256 2026-02-23T11:56:42.955 INFO:teuthology.orchestra.run.trial186.stderr:11:56:42 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial186.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool application enable rados-bench-cbt radosbench 2026-02-23T11:56:43.959 INFO:teuthology.orchestra.run.trial186.stderr:11:56:43 - INFO - cbt - Checking Health after pool creation. 2026-02-23T11:56:48.964 INFO:teuthology.orchestra.run.trial186.stderr:11:56:48 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial186.front.sepia.ceph.com /usr/bin/ceph -c /etc/ceph/ceph.conf health 2026-02-23T11:56:49.321 INFO:teuthology.orchestra.run.trial186.stderr:11:56:49 - INFO - cbt - Final Pool Health Check. 2026-02-23T11:56:54.327 INFO:teuthology.orchestra.run.trial186.stderr:11:56:54 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial186.front.sepia.ceph.com /usr/bin/ceph -c /etc/ceph/ceph.conf health 2026-02-23T11:56:54.684 INFO:teuthology.orchestra.run.trial186.stderr:11:56:54 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial186.front.sepia.ceph.com pkill -SIGINT -f collectl 2026-02-23T11:56:54.859 INFO:teuthology.orchestra.run.trial186.stderr:11:56:54 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial186.front.sepia.ceph.com sudo pkill -SIGINT -f perf\ 2026-02-23T11:56:55.044 INFO:teuthology.orchestra.run.trial186.stderr:11:56:55 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial186.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/pool_monitoring/perf/perf.data 2026-02-23T11:56:55.045 INFO:teuthology.orchestra.run.trial186.stderr:11:56:55 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial186.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/pool_monitoring/perf/perf_stat.* 2026-02-23T11:56:55.047 INFO:teuthology.orchestra.run.trial186.stderr:11:56:55 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial186.front.sepia.ceph.com sync 2026-02-23T11:56:55.267 INFO:teuthology.orchestra.run.trial186.stderr:11:56:55 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial186.front.sepia.ceph.com echo 3 | sudo tee /proc/sys/vm/drop_caches 2026-02-23T11:56:56.257 INFO:teuthology.orchestra.run.trial186.stderr:11:56:56 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial186.front.sepia.ceph.com /usr/bin/rados -c /etc/ceph/ceph.conf -v 2026-02-23T11:56:56.468 INFO:teuthology.orchestra.run.trial186.stderr:11:56:56 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial186.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/write 2026-02-23T11:56:56.630 INFO:teuthology.orchestra.run.trial186.stderr:11:56:56 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial186.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf daemon osd.0 config show > /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/write/ceph_settings.out 2026-02-23T11:56:56.918 INFO:teuthology.orchestra.run.trial186.stderr:11:56:56 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial186.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/write/collectl 2026-02-23T11:56:57.080 INFO:teuthology.orchestra.run.trial186.stderr:11:56:57 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial186.front.sepia.ceph.com collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/write/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2026-02-23T11:56:57.081 INFO:teuthology.orchestra.run.trial186.stderr:11:56:57 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial186.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/write/perf 2026-02-23T11:56:57.246 INFO:teuthology.orchestra.run.trial186.stderr:11:56:57 - DEBUG - cbt - PerfMonitoring: remote_node 2026-02-23T11:56:57.247 INFO:teuthology.orchestra.run.trial186.stderr:11:56:57 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial186.front.sepia.ceph.com for pid in `cat /var/run/ceph/crimson-osd.*.pid`; do perf stat -p ${pid} -o /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/write/perf/perf_stat.${pid} & done 2026-02-23T11:56:57.248 INFO:teuthology.orchestra.run.trial186.stderr:11:56:57 - INFO - cbt - Running radosbench write test. 2026-02-23T11:56:57.248 INFO:teuthology.orchestra.run.trial186.stderr:11:56:57 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial186.front.sepia.ceph.com /usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench -b 4194304 60 write --concurrent-ios 4 --run-name `hostname -f`-0 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/write/objecter.0.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/write/output.0 2026-02-23T11:56:57.250 INFO:teuthology.orchestra.run.trial186.stderr:11:56:57 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial186.front.sepia.ceph.com /usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench -b 4194304 60 write --concurrent-ios 4 --run-name `hostname -f`-1 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/write/objecter.1.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/write/output.1 2026-02-23T11:57:58.485 INFO:teuthology.orchestra.run.trial186.stderr:11:57:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial186.front.sepia.ceph.com pkill -SIGINT -f collectl 2026-02-23T11:57:58.666 INFO:teuthology.orchestra.run.trial186.stderr:11:57:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial186.front.sepia.ceph.com sudo pkill -SIGINT -f perf\ 2026-02-23T11:57:58.854 INFO:teuthology.orchestra.run.trial186.stderr:11:57:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial186.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/write/perf/perf.data 2026-02-23T11:57:58.856 INFO:teuthology.orchestra.run.trial186.stderr:11:57:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial186.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/write/perf/perf_stat.* 2026-02-23T11:57:58.858 INFO:teuthology.orchestra.run.trial186.stderr:11:57:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial186.front.sepia.ceph.com find /var/run/ceph/ceph-osd*.asok -maxdepth 1 -exec sudo /usr/bin/ceph --admin-daemon {} dump_historic_ops \; > /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/write/historic_ops.out 2026-02-23T11:57:59.226 INFO:teuthology.orchestra.run.trial186.stderr:11:57:59 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial186.front.sepia.ceph.com sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/write/* 2026-02-23T11:57:59.399 INFO:teuthology.orchestra.run.trial186.stderr:11:57:59 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=rpdcp -f 10 -R ssh -w ubuntu@trial186.front.sepia.ceph.com -r /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/write/* /home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3/write 2026-02-23T11:57:59.564 INFO:teuthology.orchestra.run.trial186.stderr:11:57:59 - INFO - cbt - Convert results to json format. 2026-02-23T11:57:59.576 DEBUG:teuthology.run_tasks:Unwinding manager cbt 2026-02-23T11:57:59.590 DEBUG:teuthology.orchestra.run.trial186:> rm --one-file-system -rf -- /home/ubuntu/cephtest/cbt 2026-02-23T11:57:59.618 INFO:tasks.cbt_performance:Collecting CBT performance data config 2026-02-23T11:57:59.619 INFO:tasks.cbt_performance:reading cbt preferences 2026-02-23T11:57:59.619 DEBUG:teuthology.orchestra.run.trial186:> find /home/ubuntu/cephtest -name 'json_output.*' 2026-02-23T11:57:59.677 INFO:teuthology.orchestra.run.trial186.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3/write/json_output.0.trial186.front.sepia.ceph.com 2026-02-23T11:57:59.677 INFO:teuthology.orchestra.run.trial186.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3/write/json_output.1.trial186.front.sepia.ceph.com 2026-02-23T11:57:59.678 DEBUG:teuthology.orchestra.run.trial186:> set -ex 2026-02-23T11:57:59.678 DEBUG:teuthology.orchestra.run.trial186:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3/write/json_output.0.trial186.front.sepia.ceph.com of=/dev/stdout 2026-02-23T11:57:59.736 INFO:tasks.cbt_performance:reading total cpu cycles 2026-02-23T11:57:59.736 DEBUG:teuthology.orchestra.run.trial186:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3/write -name 'perf_stat.*' 2026-02-23T11:57:59.793 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2026-02-23T11:57:59.793 INFO:tasks.cbt_performance:total cpu cycles: 0 2026-02-23T11:57:59.793 DEBUG:teuthology.orchestra.run.trial186:> set -ex 2026-02-23T11:57:59.793 DEBUG:teuthology.orchestra.run.trial186:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3/write/json_output.1.trial186.front.sepia.ceph.com of=/dev/stdout 2026-02-23T11:57:59.849 INFO:tasks.cbt_performance:reading total cpu cycles 2026-02-23T11:57:59.849 DEBUG:teuthology.orchestra.run.trial186:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3/write -name 'perf_stat.*' 2026-02-23T11:57:59.904 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2026-02-23T11:57:59.905 INFO:tasks.cbt_performance:total cpu cycles: 0 2026-02-23T11:57:59.909 WARNING:tasks.cbt_performance:Could not send performance data to http://mira118.front.sepia.ceph.com:4000/cbt_performance: HTTPConnectionPool(host='mira118.front.sepia.ceph.com', port=4000): Max retries exceeded with url: /cbt_performance (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -2] Name or service not known')) 2026-02-23T11:57:59.909 WARNING:tasks.cbt_performance:Continuing without performance data collection 2026-02-23T11:57:59.910 DEBUG:teuthology.run_tasks:Unwinding manager ssh_keys 2026-02-23T11:57:59.925 INFO:teuthology.task.ssh_keys:Cleaning up SSH keys 2026-02-23T11:57:59.925 INFO:teuthology.task.ssh_keys:cleaning up keys added for testing 2026-02-23T11:57:59.926 INFO:teuthology.task.ssh_keys: cleaning up keys for user ubuntu on trial186.front.sepia.ceph.com 2026-02-23T11:57:59.926 DEBUG:teuthology.orchestra.run.trial186:> rm -- /home/ubuntu/.ssh/id_rsa 2026-02-23T11:57:59.961 DEBUG:teuthology.orchestra.run.trial186:> rm -- /home/ubuntu/.ssh/id_rsa.pub 2026-02-23T11:58:00.018 DEBUG:teuthology.orchestra.run.trial186:> stat -c '"%a"' /home/ubuntu/.ssh/authorized_keys 2026-02-23T11:58:00.075 INFO:teuthology.orchestra.run.trial186.stdout:"600" 2026-02-23T11:58:00.075 DEBUG:teuthology.orchestra.run.trial186:> mv -- /home/ubuntu/.ssh/authorized_keys_2026-02-23_11:55:21:650320 /home/ubuntu/.ssh/authorized_keys 2026-02-23T11:58:00.133 DEBUG:teuthology.orchestra.run.trial186:> chmod 600 /home/ubuntu/.ssh/authorized_keys 2026-02-23T11:58:00.196 DEBUG:teuthology.orchestra.run.trial186:> rm /home/ubuntu/.ssh/config 2026-02-23T11:58:00.256 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2026-02-23T11:58:00.266 DEBUG:teuthology.orchestra.run.trial186:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2026-02-23T11:58:00.372 INFO:teuthology.orchestra.run.trial186.stderr:2026-02-23T11:58:00.366+0000 7f746eccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:58:00.373 INFO:teuthology.orchestra.run.trial186.stderr:2026-02-23T11:58:00.370+0000 7f746eccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-23T11:58:00.499 INFO:teuthology.misc:Shutting down mds daemons... 2026-02-23T11:58:00.500 INFO:teuthology.misc:Shutting down osd daemons... 2026-02-23T11:58:00.500 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2026-02-23T11:58:00.500 INFO:teuthology.orchestra.run:waiting for 300 2026-02-23T11:58:00.576 INFO:tasks.ceph.osd.0:Stopped 2026-02-23T11:58:00.577 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2026-02-23T11:58:00.577 INFO:teuthology.orchestra.run:waiting for 300 2026-02-23T11:58:00.654 INFO:tasks.ceph.osd.1:Stopped 2026-02-23T11:58:00.654 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2026-02-23T11:58:00.654 INFO:teuthology.orchestra.run:waiting for 300 2026-02-23T11:58:00.739 INFO:tasks.ceph.osd.2:Stopped 2026-02-23T11:58:00.739 INFO:teuthology.misc:Shutting down mgr daemons... 2026-02-23T11:58:00.739 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2026-02-23T11:58:00.739 INFO:teuthology.orchestra.run:waiting for 300 2026-02-23T11:58:00.775 INFO:tasks.ceph.mgr.x:Stopped 2026-02-23T11:58:00.775 INFO:teuthology.misc:Shutting down mon daemons... 2026-02-23T11:58:00.775 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2026-02-23T11:58:00.775 INFO:teuthology.orchestra.run:waiting for 300 2026-02-23T11:58:00.825 INFO:tasks.ceph.mon.a:Stopped 2026-02-23T11:58:00.825 INFO:tasks.ceph:Checking cluster log for badness... 2026-02-23T11:58:00.826 DEBUG:teuthology.orchestra.run.trial186:> sudo grep -E '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | grep -E -v '\(MDS_ALL_DOWN\)' | grep -E -v '\(MDS_UP_LESS_THAN_MAX\)' | grep -E -v '\(POOL_APP_NOT_ENABLED\)' | grep -E -v '\(PG_' | grep -E -v '\(OSD_' | grep -E -v '\(OBJECT_' | grep -E -v 'overall HEALTH' | head -n 1 2026-02-23T11:58:00.892 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@trial186.front.sepia.ceph.com 2026-02-23T11:58:00.892 DEBUG:teuthology.orchestra.run.trial186:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2026-02-23T11:58:00.962 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@trial186.front.sepia.ceph.com 2026-02-23T11:58:00.962 DEBUG:teuthology.orchestra.run.trial186:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2026-02-23T11:58:01.028 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@trial186.front.sepia.ceph.com 2026-02-23T11:58:01.028 DEBUG:teuthology.orchestra.run.trial186:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2026-02-23T11:58:01.097 INFO:tasks.ceph:Archiving mon data... 2026-02-23T11:58:01.119 DEBUG:teuthology.misc:Transferring archived files from trial186:/var/lib/ceph/mon/ceph-a to /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-23_09:58:48-crimson-rados-main-distro-debug-trial/66294/data/mon.a.tgz 2026-02-23T11:58:01.119 DEBUG:teuthology.orchestra.run.trial186:> mktemp 2026-02-23T11:58:01.154 INFO:teuthology.orchestra.run.trial186.stdout:/tmp/tmp.FOmnuQr0xu 2026-02-23T11:58:01.154 DEBUG:teuthology.orchestra.run.trial186:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.FOmnuQr0xu 2026-02-23T11:58:01.246 DEBUG:teuthology.orchestra.run.trial186:> sudo chmod 0666 /tmp/tmp.FOmnuQr0xu 2026-02-23T11:58:01.329 DEBUG:teuthology.orchestra.remote:trial186:/tmp/tmp.FOmnuQr0xu is 196KB 2026-02-23T11:58:01.381 DEBUG:teuthology.orchestra.run.trial186:> rm -fr /tmp/tmp.FOmnuQr0xu 2026-02-23T11:58:01.398 INFO:tasks.ceph:Cleaning ceph cluster... 2026-02-23T11:58:01.398 DEBUG:teuthology.orchestra.run.trial186:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2026-02-23T11:58:01.518 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2026-02-23T11:58:01.519 INFO:tasks.ceph:Archiving crash dumps... 2026-02-23T11:58:01.521 DEBUG:teuthology.misc:Transferring archived files from trial186:/var/lib/ceph/crash to /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-23_09:58:48-crimson-rados-main-distro-debug-trial/66294/remote/trial186/crash 2026-02-23T11:58:01.522 DEBUG:teuthology.orchestra.run.trial186:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2026-02-23T11:58:01.583 INFO:tasks.ceph:Compressing logs... 2026-02-23T11:58:01.584 DEBUG:teuthology.orchestra.run.trial186:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2026-02-23T11:58:01.653 INFO:teuthology.orchestra.run.trial186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2026-02-23T11:58:01.668 INFO:teuthology.orchestra.run.trial186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2026-02-23T11:58:01.668 INFO:teuthology.orchestra.run.trial186.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40556.log 2026-02-23T11:58:01.668 INFO:teuthology.orchestra.run.trial186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2026-02-23T11:58:01.668 INFO:teuthology.orchestra.run.trial186.stderr:/var/log/ceph/ceph-mon.a.log: /var/log/ceph/ceph-client.admin.40556.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39735.log 2026-02-23T11:58:01.668 INFO:teuthology.orchestra.run.trial186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45779.log/var/log/ceph/ceph-osd.0.log: 2026-02-23T11:58:01.668 INFO:teuthology.orchestra.run.trial186.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40556.log.gz 2026-02-23T11:58:01.668 INFO:teuthology.orchestra.run.trial186.stderr:/var/log/ceph/ceph-client.admin.39735.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41042.log 2026-02-23T11:58:01.669 INFO:teuthology.orchestra.run.trial186.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39735.log.gz 2026-02-23T11:58:01.669 INFO:teuthology.orchestra.run.trial186.stderr: 91.3% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2026-02-23T11:58:01.669 INFO:teuthology.orchestra.run.trial186.stderr:/var/log/ceph/ceph-client.admin.45779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44468.log 2026-02-23T11:58:01.669 INFO:teuthology.orchestra.run.trial186.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45779.log.gz 2026-02-23T11:58:01.669 INFO:teuthology.orchestra.run.trial186.stderr:/var/log/ceph/ceph-client.admin.41042.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40895.log 2026-02-23T11:58:01.669 INFO:teuthology.orchestra.run.trial186.stderr:/var/log/ceph/ceph-client.admin.44468.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41042.log.gz 2026-02-23T11:58:01.669 INFO:teuthology.orchestra.run.trial186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41310.log 2026-02-23T11:58:01.669 INFO:teuthology.orchestra.run.trial186.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44468.log.gz 2026-02-23T11:58:01.669 INFO:teuthology.orchestra.run.trial186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41190.log 2026-02-23T11:58:01.669 INFO:teuthology.orchestra.run.trial186.stderr:/var/log/ceph/ceph-client.admin.40895.log: /var/log/ceph/ceph-client.admin.41310.log: gzip 50.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.40895.log.gz --verbose 2026-02-23T11:58:01.669 INFO:teuthology.orchestra.run.trial186.stderr: -- /var/log/ceph/ceph-client.admin.39919.log 2026-02-23T11:58:01.669 INFO:teuthology.orchestra.run.trial186.stderr:/var/log/ceph/ceph-client.admin.41190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40894.log 2026-02-23T11:58:01.670 INFO:teuthology.orchestra.run.trial186.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41310.log.gz 2026-02-23T11:58:01.670 INFO:teuthology.orchestra.run.trial186.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41190.log.gz 2026-02-23T11:58:01.670 INFO:teuthology.orchestra.run.trial186.stderr:/var/log/ceph/ceph-client.admin.39919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41357.log 2026-02-23T11:58:01.670 INFO:teuthology.orchestra.run.trial186.stderr:gzip/var/log/ceph/ceph-client.admin.40894.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.40486.log 2026-02-23T11:58:01.670 INFO:teuthology.orchestra.run.trial186.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.39919.log.gz 2026-02-23T11:58:01.670 INFO:teuthology.orchestra.run.trial186.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40894.log.gz 2026-02-23T11:58:01.670 INFO:teuthology.orchestra.run.trial186.stderr:/var/log/ceph/ceph-client.admin.41357.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.38356.log 2026-02-23T11:58:01.670 INFO:teuthology.orchestra.run.trial186.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41357.log.gz 2026-02-23T11:58:01.670 INFO:teuthology.orchestra.run.trial186.stderr:/var/log/ceph/ceph-client.admin.40486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41136.log 2026-02-23T11:58:01.670 INFO:teuthology.orchestra.run.trial186.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40486.log.gz 2026-02-23T11:58:01.670 INFO:teuthology.orchestra.run.trial186.stderr:/var/log/ceph/ceph.tmp-client.admin.38356.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41283.log 2026-02-23T11:58:01.670 INFO:teuthology.orchestra.run.trial186.stderr:/var/log/ceph/ceph-client.admin.41136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44526.log 2026-02-23T11:58:01.671 INFO:teuthology.orchestra.run.trial186.stderr: 67.5% -- replaced with /var/log/ceph/ceph.tmp-client.admin.38356.log.gz 2026-02-23T11:58:01.671 INFO:teuthology.orchestra.run.trial186.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41136.log.gz 2026-02-23T11:58:01.671 INFO:teuthology.orchestra.run.trial186.stderr:/var/log/ceph/ceph-client.admin.41283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40695.log 2026-02-23T11:58:01.671 INFO:teuthology.orchestra.run.trial186.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41283.log.gz 2026-02-23T11:58:01.671 INFO:teuthology.orchestra.run.trial186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40013.log 2026-02-23T11:58:01.671 INFO:teuthology.orchestra.run.trial186.stderr:/var/log/ceph/ceph-client.admin.44526.log: /var/log/ceph/ceph-client.admin.40695.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41430.log 2026-02-23T11:58:01.671 INFO:teuthology.orchestra.run.trial186.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40695.log.gz 2026-02-23T11:58:01.671 INFO:teuthology.orchestra.run.trial186.stderr:/var/log/ceph/ceph-client.admin.40013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40896.log 2026-02-23T11:58:01.671 INFO:teuthology.orchestra.run.trial186.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40013.log.gz 2026-02-23T11:58:01.671 INFO:teuthology.orchestra.run.trial186.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44526.log.gz 2026-02-23T11:58:01.671 INFO:teuthology.orchestra.run.trial186.stderr:/var/log/ceph/ceph-client.admin.41430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41479.log 2026-02-23T11:58:01.671 INFO:teuthology.orchestra.run.trial186.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41430.log.gz 2026-02-23T11:58:01.671 INFO:teuthology.orchestra.run.trial186.stderr:/var/log/ceph/ceph-client.admin.40896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40793.log 2026-02-23T11:58:01.672 INFO:teuthology.orchestra.run.trial186.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40896.log.gz 2026-02-23T11:58:01.672 INFO:teuthology.orchestra.run.trial186.stderr:/var/log/ceph/ceph-client.admin.41479.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2026-02-23T11:58:01.672 INFO:teuthology.orchestra.run.trial186.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41479.log.gz 2026-02-23T11:58:01.672 INFO:teuthology.orchestra.run.trial186.stderr:/var/log/ceph/ceph-client.admin.40793.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45169.log 2026-02-23T11:58:01.672 INFO:teuthology.orchestra.run.trial186.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40793.log.gz 2026-02-23T11:58:01.672 INFO:teuthology.orchestra.run.trial186.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2026-02-23T11:58:01.672 INFO:teuthology.orchestra.run.trial186.stderr:/var/log/ceph/ceph-client.admin.45169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39966.log 2026-02-23T11:58:01.672 INFO:teuthology.orchestra.run.trial186.stderr: 66.4% -- replaced with /var/log/ceph/ceph-client.admin.45169.log.gz 2026-02-23T11:58:01.672 INFO:teuthology.orchestra.run.trial186.stderr:/var/log/ceph/ceph-osd.1.log: 87.7% -- replaced with /var/log/ceph/ceph.audit.log.gz 2026-02-23T11:58:01.672 INFO:teuthology.orchestra.run.trial186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45168.log 2026-02-23T11:58:01.672 INFO:teuthology.orchestra.run.trial186.stderr:/var/log/ceph/ceph-client.admin.39966.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40581.log 2026-02-23T11:58:01.672 INFO:teuthology.orchestra.run.trial186.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.39966.log.gz 2026-02-23T11:58:01.672 INFO:teuthology.orchestra.run.trial186.stderr:/var/log/ceph/ceph-client.admin.45168.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41528.log 2026-02-23T11:58:01.673 INFO:teuthology.orchestra.run.trial186.stderr: 66.4% -- replaced with /var/log/ceph/ceph-client.admin.45168.log.gz 2026-02-23T11:58:01.673 INFO:teuthology.orchestra.run.trial186.stderr:/var/log/ceph/ceph-client.admin.40581.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44560.log 2026-02-23T11:58:01.673 INFO:teuthology.orchestra.run.trial186.stderr: 66.9% -- replaced with /var/log/ceph/ceph-client.admin.40581.log.gz 2026-02-23T11:58:01.673 INFO:teuthology.orchestra.run.trial186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40744.log 2026-02-23T11:58:01.673 INFO:teuthology.orchestra.run.trial186.stderr:/var/log/ceph/ceph-client.admin.41528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44614.log 2026-02-23T11:58:01.673 INFO:teuthology.orchestra.run.trial186.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41528.log.gz 2026-02-23T11:58:01.673 INFO:teuthology.orchestra.run.trial186.stderr:/var/log/ceph/ceph-client.admin.44560.log: /var/log/ceph/ceph-client.admin.40744.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44560.log.gz 2026-02-23T11:58:01.673 INFO:teuthology.orchestra.run.trial186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41041.log 2026-02-23T11:58:01.673 INFO:teuthology.orchestra.run.trial186.stderr:/var/log/ceph/ceph-client.admin.44614.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2026-02-23T11:58:01.673 INFO:teuthology.orchestra.run.trial186.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.40744.log.gz 2026-02-23T11:58:01.673 INFO:teuthology.orchestra.run.trial186.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.44614.log.gz 2026-02-23T11:58:01.673 INFO:teuthology.orchestra.run.trial186.stderr:/var/log/ceph/ceph-client.admin.41041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41043.log 2026-02-23T11:58:01.673 INFO:teuthology.orchestra.run.trial186.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41041.log.gz 2026-02-23T11:58:01.674 INFO:teuthology.orchestra.run.trial186.stderr:gzip -5/var/log/ceph/ceph-mgr.x.log: --verbose -- /var/log/ceph/ceph.log 2026-02-23T11:58:01.674 INFO:teuthology.orchestra.run.trial186.stderr:/var/log/ceph/ceph-client.admin.41043.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40646.log 2026-02-23T11:58:01.674 INFO:teuthology.orchestra.run.trial186.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41043.log.gz 2026-02-23T11:58:01.674 INFO:teuthology.orchestra.run.trial186.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39639.log 2026-02-23T11:58:01.674 INFO:teuthology.orchestra.run.trial186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44412.log 2026-02-23T11:58:01.674 INFO:teuthology.orchestra.run.trial186.stderr:/var/log/ceph/ceph-client.admin.40646.log: 85.3% -- replaced with /var/log/ceph/ceph.log.gz 2026-02-23T11:58:01.674 INFO:teuthology.orchestra.run.trial186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40232.log 2026-02-23T11:58:01.674 INFO:teuthology.orchestra.run.trial186.stderr:/var/log/ceph/ceph-client.admin.39639.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.40646.log.gz 2026-02-23T11:58:01.674 INFO:teuthology.orchestra.run.trial186.stderr:/var/log/ceph/ceph-client.admin.44412.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41189.log 2026-02-23T11:58:01.674 INFO:teuthology.orchestra.run.trial186.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.39639.log.gz 2026-02-23T11:58:01.674 INFO:teuthology.orchestra.run.trial186.stderr:/var/log/ceph/ceph-client.admin.40232.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40341.log 2026-02-23T11:58:01.674 INFO:teuthology.orchestra.run.trial186.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.44412.log.gz 2026-02-23T11:58:01.674 INFO:teuthology.orchestra.run.trial186.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40232.log.gz 2026-02-23T11:58:01.675 INFO:teuthology.orchestra.run.trial186.stderr:/var/log/ceph/ceph-client.admin.41189.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40415.log 2026-02-23T11:58:01.675 INFO:teuthology.orchestra.run.trial186.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41189.log.gz 2026-02-23T11:58:01.675 INFO:teuthology.orchestra.run.trial186.stderr:/var/log/ceph/ceph-client.admin.40341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39782.log 2026-02-23T11:58:01.675 INFO:teuthology.orchestra.run.trial186.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40341.log.gz 2026-02-23T11:58:01.675 INFO:teuthology.orchestra.run.trial186.stderr:/var/log/ceph/ceph-client.admin.40415.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40415.log.gz 2026-02-23T11:58:01.675 INFO:teuthology.orchestra.run.trial186.stderr:/var/log/ceph/ceph-client.admin.39782.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39782.log.gz 2026-02-23T11:58:01.675 INFO:teuthology.orchestra.run.trial186.stderr: 92.0% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2026-02-23T11:58:03.567 INFO:teuthology.orchestra.run.trial186.stderr: 92.3% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2026-02-23T11:58:03.607 INFO:teuthology.orchestra.run.trial186.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2026-02-23T11:58:03.633 INFO:teuthology.orchestra.run.trial186.stderr: 92.3% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2026-02-23T11:58:03.634 INFO:teuthology.orchestra.run.trial186.stderr: 2026-02-23T11:58:03.634 INFO:teuthology.orchestra.run.trial186.stderr:real 0m1.992s 2026-02-23T11:58:03.635 INFO:teuthology.orchestra.run.trial186.stderr:user 0m5.524s 2026-02-23T11:58:03.635 INFO:teuthology.orchestra.run.trial186.stderr:sys 0m0.355s 2026-02-23T11:58:03.636 INFO:tasks.ceph:Archiving logs... 2026-02-23T11:58:03.636 DEBUG:teuthology.misc:Transferring archived files from trial186:/var/log/ceph to /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-23_09:58:48-crimson-rados-main-distro-debug-trial/66294/remote/trial186/log 2026-02-23T11:58:03.638 DEBUG:teuthology.orchestra.run.trial186:> sudo tar c -f - -C /var/log/ceph -- . 2026-02-23T11:58:04.317 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-23T11:58:04.386 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-23T11:58:04.397 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2026-02-23T11:58:04.397 DEBUG:teuthology.orchestra.run.trial186:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-23T11:58:04.423 DEBUG:teuthology.run_tasks:Unwinding manager clock 2026-02-23T11:58:04.437 INFO:teuthology.task.clock:Checking final clock skew... 2026-02-23T11:58:04.437 DEBUG:teuthology.orchestra.run.trial186:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-23T11:58:04.479 INFO:teuthology.orchestra.run.trial186.stderr:bash: line 1: ntpq: command not found 2026-02-23T11:58:04.494 INFO:teuthology.orchestra.run.trial186.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-23T11:58:04.494 INFO:teuthology.orchestra.run.trial186.stdout:=============================================================================== 2026-02-23T11:58:04.494 INFO:teuthology.orchestra.run.trial186.stdout:^- ntp1.lax.us.hojmark.net 2 6 377 25 -3230us[-3228us] +/- 51ms 2026-02-23T11:58:04.494 INFO:teuthology.orchestra.run.trial186.stdout:^- 66.42.71.197.vultruserco> 2 6 377 24 -441us[ -441us] +/- 65ms 2026-02-23T11:58:04.494 INFO:teuthology.orchestra.run.trial186.stdout:^* 64.44.115.65 4 6 377 24 +3203ns[+5346ns] +/- 14ms 2026-02-23T11:58:04.494 INFO:teuthology.orchestra.run.trial186.stdout:^- dns-e.ns4v.icu 2 6 377 24 +80us[ +80us] +/- 46ms 2026-02-23T11:58:04.494 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2026-02-23T11:58:04.507 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2026-02-23T11:58:04.507 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2026-02-23T11:58:04.523 DEBUG:teuthology.orchestra.run.trial186:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2026-02-23T11:58:04.593 DEBUG:teuthology.orchestra.run.trial186:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-23T11:58:04.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-23T11:58:04.620 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2026-02-23T11:58:04.637 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2026-02-23T11:58:04.653 INFO:teuthology.task.internal:Duration was 841.289158 seconds 2026-02-23T11:58:04.653 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2026-02-23T11:58:04.669 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2026-02-23T11:58:04.670 DEBUG:teuthology.orchestra.run.trial186:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-23T11:58:04.709 INFO:teuthology.orchestra.run.trial186.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-23T11:58:04.855 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2026-02-23T11:58:04.856 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial186.front.sepia.ceph.com 2026-02-23T11:58:04.856 DEBUG:teuthology.orchestra.run.trial186:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-23T11:58:04.916 INFO:teuthology.task.internal.syslog:Gathering journactl... 2026-02-23T11:58:04.917 DEBUG:teuthology.orchestra.run.trial186:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-23T11:58:05.021 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2026-02-23T11:58:05.022 DEBUG:teuthology.orchestra.run.trial186:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-23T11:58:05.044 INFO:teuthology.orchestra.run.trial186.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-23T11:58:05.045 INFO:teuthology.orchestra.run.trial186.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-23T11:58:05.045 INFO:teuthology.orchestra.run.trial186.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-23T11:58:05.045 INFO:teuthology.orchestra.run.trial186.stderr: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-02-23T11:58:05.045 INFO:teuthology.orchestra.run.trial186.stderr:/home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-23T11:58:05.056 INFO:teuthology.orchestra.run.trial186.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: 82.5% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-23T11:58:05.058 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2026-02-23T11:58:05.076 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2026-02-23T11:58:05.076 DEBUG:teuthology.orchestra.run.trial186:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-23T11:58:05.124 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2026-02-23T11:58:05.143 DEBUG:teuthology.orchestra.run.trial186:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-23T11:58:05.189 INFO:teuthology.orchestra.run.trial186.stdout:kernel.core_pattern = core 2026-02-23T11:58:05.204 DEBUG:teuthology.orchestra.run.trial186:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-23T11:58:05.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-23T11:58:05.262 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2026-02-23T11:58:05.282 INFO:teuthology.task.internal:Transferring archived files... 2026-02-23T11:58:05.283 DEBUG:teuthology.misc:Transferring archived files from trial186:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-23_09:58:48-crimson-rados-main-distro-debug-trial/66294/remote/trial186 2026-02-23T11:58:05.283 DEBUG:teuthology.orchestra.run.trial186:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-23T11:58:05.365 INFO:teuthology.task.internal:Removing archive directory... 2026-02-23T11:58:05.366 DEBUG:teuthology.orchestra.run.trial186:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-23T11:58:05.397 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2026-02-23T11:58:05.410 INFO:teuthology.task.internal:Not uploading archives. 2026-02-23T11:58:05.410 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2026-02-23T11:58:05.425 INFO:teuthology.task.internal:Tidying up after the test... 2026-02-23T11:58:05.425 DEBUG:teuthology.orchestra.run.trial186:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-23T11:58:05.454 INFO:teuthology.orchestra.run.trial186.stdout: 917576 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 23 11:58 /home/ubuntu/cephtest 2026-02-23T11:58:05.455 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2026-02-23T11:58:05.466 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2026-02-23T11:58:05.504 INFO:teuthology.run:Summary data: description: crimson-rados/perf/{0-crimson_install clusters/{crimson-fixed-1} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/segmented$/{crimson_seastore_segmented_2q} perf_ignorelist settings/optimized workloads/radosbench_4M_write} duration: 841.2891581058502 flavor: debug owner: scheduled_shraddhaag@soko04.front.sepia.ceph.com success: true 2026-02-23T11:58:05.504 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-23T11:58:05.610 INFO:teuthology.run:pass