2026-02-24T15:42:11.107 INFO:root:teuthology version: 1.2.4.dev9+gfa17720d0 2026-02-24T15:42:11.124 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-24T15:42:11.208 INFO:teuthology.run:Config: archive_path: /home/teuthworker/mnt/teuthology/liucm-2026-02-24_15:28:57-crimson-rados-wip-liucm-shard-redesign-crimson-only-distro-debug-trial/68238 branch: wip-liucm-shard-redesign-crimson-only description: crimson-rados/perf/{0-crimson_install clusters/{crimson-fixed-1} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/rbm$/{crimson_seastore_rbm} perf_ignorelist settings/optimized workloads/radosbench_4K_seq_read} email: null first_in_suite: false flavor: debug job_id: '68238' kernel: branch: distro kdb: true sha1: distro last_in_suite: false machine_type: trial name: liucm-2026-02-24_15:28:57-crimson-rados-wip-liucm-shard-redesign-crimson-only-distro-debug-trial no_nested_subset: false os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-liucm-shard-redesign-crimson-only ceph: conf: global: auth client required: none auth cluster required: none auth service required: none auth supported: none debug asok: 0/0 debug auth: 0/0 debug bdev: 0/0 debug bluefs: 0/0 debug bluestore: 0/0 debug buffer: 0/0 debug civetweb: 0/0 debug client: 0/0 debug compressor: 0/0 debug context: 0/0 debug crush: 0/0 debug crypto: 0/0 debug dpdk: 0/0 debug eventtrace: 0/0 debug filer: 0/0 debug filestore: 0/0 debug finisher: 0/0 debug fuse: 0/0 debug heartbeatmap: 0/0 debug javaclient: 0/0 debug journal: 0/0 debug journaler: 0/0 debug kstore: 0/0 debug leveldb: 0/0 debug lockdep: 0/0 debug mds: 0/0 debug mds balancer: 0/0 debug mds locker: 0/0 debug mds log: 0/0 debug mds log expire: 0/0 debug mds migrator: 0/0 debug memdb: 0/0 debug mgr: 0/0 debug mgrc: 0/0 debug mon: 0/0 debug monc: 0/0 debug ms: 0/0 debug objclass: 0/0 debug objectcacher: 0/0 debug objecter: 0/0 debug optracker: 0/0 debug osd: 0/0 debug paxos: 0/0 debug perfcounter: 0/0 debug rados: 0/0 debug rbd: 0/0 debug rbd mirror: 0/0 debug rbd replay: 0/0 debug refs: 0/0 debug rgw: 0/0 debug rgw lifecycle: 0/0 debug rgw notification: 0/0 debug rgw sync: 0/0 debug rocksdb: 0/0 debug striper: 0/0 debug throttle: 0/0 debug timer: 0/0 debug tp: 0/0 enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 0/0 debug ms: 0/0 debug paxos: 0/0 osd pool default crimson: true osd: crimson cpu num: 4 crimson osd obc lru size: 10 debug filestore: 0/0 debug journal: 0/0 debug ms: 0/0 debug osd: 0/0 debug seastore: 20 debug seastore backref: 20 debug seastore cache: 20 debug seastore cleaner: 20 debug seastore device: 20 debug seastore epm: 20 debug seastore fixedkv tree: 20 debug seastore journal: 20 debug seastore lba: 20 debug seastore odata: 20 debug seastore omap: 20 debug seastore onode: 20 debug seastore t: 20 debug seastore tm: 20 osd objectstore: seastore osd shutdown pgref assert: true seastore cachepin size pershard: 64M seastore main device type: RANDOM_BLOCK_SSD seastore max concurrent transactions: 8 crimson_compat: true flavor: debug log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) - \(PG_ - \(OSD_ - \(OBJECT_ - overall HEALTH sha1: d26e5c228e3cdea4d585bcd077f8513a649ab5d1 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log global: osd crush chooseleaf type: 0 osd pool default pg num: 128 osd pool default pgp num: 128 osd pool default size: 2 mon: {} install: ceph: flavor: debug sha1: d26e5c228e3cdea4d585bcd077f8513a649ab5d1 workunit: branch: wip-liucm-shard-redesign-crimson-only env: CRIMSON_COMPAT: '1' sha1: d26e5c228e3cdea4d585bcd077f8513a649ab5d1 owner: scheduled_liucm@soko04.front.sepia.ceph.com priority: 120 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 9834 sha1: d26e5c228e3cdea4d585bcd077f8513a649ab5d1 sleep_before_teardown: 0 suite: crimson-rados suite_branch: wip-liucm-shard-redesign-crimson-only suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_d26e5c228e3cdea4d585bcd077f8513a649ab5d1/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: d26e5c228e3cdea4d585bcd077f8513a649ab5d1 targets: trial084.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK/Xi7BILlGmIASVnslm/6eCVpGkBUUXVms9wstoJQGC+lFMV9gxCD040eOXC3EM8Tp76qDyRRALUATGCeNY+sI= tasks: - install: extra_packages: deb: - ceph-osd-crimson - ceph-osd-crimson-dbg rpm: - ceph-osd-crimson - ceph-osd-crimson-debuginfo - pexec: all: - sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson - install: null - ceph: conf: osd: debug monc: 20 wait-for-scrub: false - ssh_keys: null - cbt: benchmarks: radosbench: concurrent_ops: 4 concurrent_procs: 2 op_size: - 4096 pool_monitoring_list: - collectl pool_profile: replicated run_monitoring_list: - collectl time: 60 write_only: false cluster: iterations: 1 osds_per_node: 3 pool_profiles: replicated: pg_size: 256 pgp_size: 256 replication: replicated user: ubuntu monitoring_profiles: perf: args: stat -p {pid} -o {perf_dir}/perf_stat.{pid} nodes: - osds perf_cmd: perf pid_glob: crimson-osd.*.pid teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: fa17720d0088c3ac28e473468bfc79eeaff5cd38 timestamp: 2026-02-24_15:28:57 tube: trial user: liucm verbose: false worker_log: /home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.904699 2026-02-24T15:42:11.209 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_d26e5c228e3cdea4d585bcd077f8513a649ab5d1/qa; will attempt to use it 2026-02-24T15:42:11.209 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_d26e5c228e3cdea4d585bcd077f8513a649ab5d1/qa/tasks 2026-02-24T15:42:11.209 INFO:teuthology.run_tasks:Running task internal.check_packages... 2026-02-24T15:42:11.211 INFO:teuthology.task.internal:Checking packages... 2026-02-24T15:42:11.211 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'debug' and ceph hash 'd26e5c228e3cdea4d585bcd077f8513a649ab5d1' 2026-02-24T15:42:11.211 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2026-02-24T15:42:11.211 INFO:teuthology.packaging:ref: None 2026-02-24T15:42:11.211 INFO:teuthology.packaging:tag: None 2026-02-24T15:42:11.211 INFO:teuthology.packaging:branch: wip-liucm-shard-redesign-crimson-only 2026-02-24T15:42:11.211 INFO:teuthology.packaging:sha1: d26e5c228e3cdea4d585bcd077f8513a649ab5d1 2026-02-24T15:42:11.211 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&ref=wip-liucm-shard-redesign-crimson-only 2026-02-24T15:42:11.346 INFO:teuthology.task.internal:Found packages for ceph version 20.3.0-5343.gd26e5c22 2026-02-24T15:42:11.348 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2026-02-24T15:42:11.359 INFO:teuthology.task.internal:no buildpackages task found 2026-02-24T15:42:11.359 INFO:teuthology.run_tasks:Running task internal.save_config... 2026-02-24T15:42:11.372 INFO:teuthology.task.internal:Saving configuration 2026-02-24T15:42:11.389 INFO:teuthology.run_tasks:Running task internal.check_lock... 2026-02-24T15:42:11.401 INFO:teuthology.task.internal.check_lock:Checking locks... 2026-02-24T15:42:11.485 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial084.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/liucm-2026-02-24_15:28:57-crimson-rados-wip-liucm-shard-redesign-crimson-only-distro-debug-trial/68238', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-24 15:38:43.276956', 'locked_by': 'scheduled_liucm@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK/Xi7BILlGmIASVnslm/6eCVpGkBUUXVms9wstoJQGC+lFMV9gxCD040eOXC3EM8Tp76qDyRRALUATGCeNY+sI='} 2026-02-24T15:42:11.485 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2026-02-24T15:42:11.498 INFO:teuthology.task.internal:roles: ubuntu@trial084.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2026-02-24T15:42:11.498 INFO:teuthology.run_tasks:Running task console_log... 2026-02-24T15:42:11.577 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7e9afef32a20>, signals=[15]) 2026-02-24T15:42:11.577 INFO:teuthology.run_tasks:Running task internal.connect... 2026-02-24T15:42:11.588 INFO:teuthology.task.internal:Opening connections... 2026-02-24T15:42:11.588 DEBUG:teuthology.task.internal:connecting to ubuntu@trial084.front.sepia.ceph.com 2026-02-24T15:42:11.589 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-24T15:42:11.649 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2026-02-24T15:42:11.663 DEBUG:teuthology.orchestra.run.trial084:> uname -m 2026-02-24T15:42:11.721 INFO:teuthology.orchestra.run.trial084.stdout:x86_64 2026-02-24T15:42:11.722 DEBUG:teuthology.orchestra.run.trial084:> cat /etc/os-release 2026-02-24T15:42:11.777 INFO:teuthology.orchestra.run.trial084.stdout:NAME="CentOS Stream" 2026-02-24T15:42:11.777 INFO:teuthology.orchestra.run.trial084.stdout:VERSION="9" 2026-02-24T15:42:11.777 INFO:teuthology.orchestra.run.trial084.stdout:ID="centos" 2026-02-24T15:42:11.777 INFO:teuthology.orchestra.run.trial084.stdout:ID_LIKE="rhel fedora" 2026-02-24T15:42:11.777 INFO:teuthology.orchestra.run.trial084.stdout:VERSION_ID="9" 2026-02-24T15:42:11.777 INFO:teuthology.orchestra.run.trial084.stdout:PLATFORM_ID="platform:el9" 2026-02-24T15:42:11.777 INFO:teuthology.orchestra.run.trial084.stdout:PRETTY_NAME="CentOS Stream 9" 2026-02-24T15:42:11.778 INFO:teuthology.orchestra.run.trial084.stdout:ANSI_COLOR="0;31" 2026-02-24T15:42:11.778 INFO:teuthology.orchestra.run.trial084.stdout:LOGO="fedora-logo-icon" 2026-02-24T15:42:11.778 INFO:teuthology.orchestra.run.trial084.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2026-02-24T15:42:11.778 INFO:teuthology.orchestra.run.trial084.stdout:HOME_URL="https://centos.org/" 2026-02-24T15:42:11.778 INFO:teuthology.orchestra.run.trial084.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2026-02-24T15:42:11.778 INFO:teuthology.orchestra.run.trial084.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2026-02-24T15:42:11.778 INFO:teuthology.orchestra.run.trial084.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2026-02-24T15:42:11.779 INFO:teuthology.lock.ops:Updating trial084.front.sepia.ceph.com on lock server 2026-02-24T15:42:11.862 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2026-02-24T15:42:11.875 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2026-02-24T15:42:11.887 INFO:teuthology.task.internal:Checking for old test directory... 2026-02-24T15:42:11.887 DEBUG:teuthology.orchestra.run.trial084:> test '!' -e /home/ubuntu/cephtest 2026-02-24T15:42:11.902 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2026-02-24T15:42:11.913 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2026-02-24T15:42:11.913 DEBUG:teuthology.orchestra.run.trial084:> test -z $(ls -A /var/lib/ceph) 2026-02-24T15:42:11.959 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2026-02-24T15:42:12.065 INFO:teuthology.run_tasks:Running task kernel... 2026-02-24T15:42:12.087 INFO:teuthology.task.kernel:normalize config orig: {'branch': 'distro', 'kdb': True, 'sha1': 'distro'} 2026-02-24T15:42:12.087 INFO:teuthology.task.kernel:config {'mon.a': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}}, timeout 300 2026-02-24T15:42:12.087 DEBUG:teuthology.orchestra.run.trial084:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-24T15:42:12.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-24T15:42:12.102 DEBUG:teuthology.orchestra.run.trial084:> uname -r 2026-02-24T15:42:12.158 INFO:teuthology.orchestra.run.trial084.stdout:5.14.0-681.el9.x86_64 2026-02-24T15:42:12.158 INFO:teuthology.task.kernel:Running kernel on trial084: 5.14.0-681.el9.x86_64 2026-02-24T15:42:12.158 DEBUG:teuthology.orchestra.run.trial084:> sudo yum install -y kernel 2026-02-24T15:42:21.109 INFO:teuthology.orchestra.run.trial084.stdout:CentOS Stream 9 - BaseOS 1.0 MB/s | 8.9 MB 00:08 2026-02-24T15:42:24.343 INFO:teuthology.orchestra.run.trial084.stdout:CentOS Stream 9 - AppStream 10 MB/s | 27 MB 00:02 2026-02-24T15:42:27.829 INFO:teuthology.orchestra.run.trial084.stdout:CentOS Stream 9 - CRB 7.1 MB/s | 8.0 MB 00:01 2026-02-24T15:42:28.722 INFO:teuthology.orchestra.run.trial084.stdout:CentOS Stream 9 - Extras packages 128 kB/s | 20 kB 00:00 2026-02-24T15:42:29.337 INFO:teuthology.orchestra.run.trial084.stdout:Extra Packages for Enterprise Linux 37 MB/s | 20 MB 00:00 2026-02-24T15:42:32.500 INFO:teuthology.orchestra.run.trial084.stdout:lab-extras 2.5 MB/s | 50 kB 00:00 2026-02-24T15:42:33.509 INFO:teuthology.orchestra.run.trial084.stdout:Package kernel-5.14.0-645.el9.x86_64 is already installed. 2026-02-24T15:42:33.509 INFO:teuthology.orchestra.run.trial084.stdout:Package kernel-5.14.0-681.el9.x86_64 is already installed. 2026-02-24T15:42:33.529 INFO:teuthology.orchestra.run.trial084.stdout:Dependencies resolved. 2026-02-24T15:42:33.530 INFO:teuthology.orchestra.run.trial084.stdout:================================================================================ 2026-02-24T15:42:33.531 INFO:teuthology.orchestra.run.trial084.stdout: Package Architecture Version Repository Size 2026-02-24T15:42:33.531 INFO:teuthology.orchestra.run.trial084.stdout:================================================================================ 2026-02-24T15:42:33.531 INFO:teuthology.orchestra.run.trial084.stdout:Installing: 2026-02-24T15:42:33.531 INFO:teuthology.orchestra.run.trial084.stdout: kernel x86_64 5.14.0-686.el9 baseos 901 k 2026-02-24T15:42:33.531 INFO:teuthology.orchestra.run.trial084.stdout:Installing dependencies: 2026-02-24T15:42:33.531 INFO:teuthology.orchestra.run.trial084.stdout: kernel-core x86_64 5.14.0-686.el9 baseos 17 M 2026-02-24T15:42:33.531 INFO:teuthology.orchestra.run.trial084.stdout: kernel-modules x86_64 5.14.0-686.el9 baseos 40 M 2026-02-24T15:42:33.531 INFO:teuthology.orchestra.run.trial084.stdout: kernel-modules-core x86_64 5.14.0-686.el9 baseos 31 M 2026-02-24T15:42:33.532 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T15:42:33.532 INFO:teuthology.orchestra.run.trial084.stdout:Transaction Summary 2026-02-24T15:42:33.532 INFO:teuthology.orchestra.run.trial084.stdout:================================================================================ 2026-02-24T15:42:33.532 INFO:teuthology.orchestra.run.trial084.stdout:Install 4 Packages 2026-02-24T15:42:33.532 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T15:42:33.532 INFO:teuthology.orchestra.run.trial084.stdout:Total download size: 89 M 2026-02-24T15:42:33.532 INFO:teuthology.orchestra.run.trial084.stdout:Installed size: 135 M 2026-02-24T15:42:33.532 INFO:teuthology.orchestra.run.trial084.stdout:Downloading Packages: 2026-02-24T15:42:35.188 INFO:teuthology.orchestra.run.trial084.stdout:(1/4): kernel-5.14.0-686.el9.x86_64.rpm 575 kB/s | 901 kB 00:01 2026-02-24T15:42:45.336 INFO:teuthology.orchestra.run.trial084.stdout:(2/4): kernel-core-5.14.0-686.el9.x86_64.rpm 1.5 MB/s | 17 MB 00:11 2026-02-24T15:43:02.993 INFO:teuthology.orchestra.run.trial084.stdout:(3/4): kernel-modules-core-5.14.0-686.el9.x86_6 1.1 MB/s | 31 MB 00:27 2026-02-24T15:43:08.614 INFO:teuthology.orchestra.run.trial084.stdout:(4/4): kernel-modules-5.14.0-686.el9.x86_64.rpm 1.1 MB/s | 40 MB 00:34 2026-02-24T15:43:08.615 INFO:teuthology.orchestra.run.trial084.stdout:-------------------------------------------------------------------------------- 2026-02-24T15:43:08.616 INFO:teuthology.orchestra.run.trial084.stdout:Total 2.5 MB/s | 89 MB 00:35 2026-02-24T15:43:08.805 INFO:teuthology.orchestra.run.trial084.stdout:Running transaction check 2026-02-24T15:43:09.008 INFO:teuthology.orchestra.run.trial084.stdout:Transaction check succeeded. 2026-02-24T15:43:09.009 INFO:teuthology.orchestra.run.trial084.stdout:Running transaction test 2026-02-24T15:43:09.391 INFO:teuthology.orchestra.run.trial084.stdout:Transaction test succeeded. 2026-02-24T15:43:09.391 INFO:teuthology.orchestra.run.trial084.stdout:Running transaction 2026-02-24T15:43:10.504 INFO:teuthology.orchestra.run.trial084.stdout: Preparing : 1/1 2026-02-24T15:43:10.792 INFO:teuthology.orchestra.run.trial084.stdout: Installing : kernel-modules-core-5.14.0-686.el9.x86_64 1/4 2026-02-24T15:43:10.814 INFO:teuthology.orchestra.run.trial084.stdout: Installing : kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-24T15:43:13.363 INFO:teuthology.orchestra.run.trial084.stdout: Running scriptlet: kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-24T15:43:13.397 INFO:teuthology.orchestra.run.trial084.stdout: Installing : kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-24T15:43:16.434 INFO:teuthology.orchestra.run.trial084.stdout: Running scriptlet: kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-24T15:43:16.440 INFO:teuthology.orchestra.run.trial084.stdout: Installing : kernel-5.14.0-686.el9.x86_64 4/4 2026-02-24T15:43:19.489 INFO:teuthology.orchestra.run.trial084.stdout: Running scriptlet: kernel-modules-core-5.14.0-686.el9.x86_64 4/4 2026-02-24T15:43:37.165 INFO:teuthology.orchestra.run.trial084.stdout: Running scriptlet: kernel-core-5.14.0-686.el9.x86_64 4/4 2026-02-24T15:43:37.185 INFO:teuthology.orchestra.run.trial084.stdout: Running scriptlet: kernel-modules-5.14.0-686.el9.x86_64 4/4 2026-02-24T15:43:37.373 INFO:teuthology.orchestra.run.trial084.stdout: Running scriptlet: kernel-5.14.0-686.el9.x86_64 4/4 2026-02-24T15:43:37.373 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : kernel-5.14.0-686.el9.x86_64 1/4 2026-02-24T15:43:37.373 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-24T15:43:37.373 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-24T15:43:37.431 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : kernel-modules-core-5.14.0-686.el9.x86_64 4/4 2026-02-24T15:43:37.431 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T15:43:37.431 INFO:teuthology.orchestra.run.trial084.stdout:Installed: 2026-02-24T15:43:37.431 INFO:teuthology.orchestra.run.trial084.stdout: kernel-5.14.0-686.el9.x86_64 2026-02-24T15:43:37.432 INFO:teuthology.orchestra.run.trial084.stdout: kernel-core-5.14.0-686.el9.x86_64 2026-02-24T15:43:37.432 INFO:teuthology.orchestra.run.trial084.stdout: kernel-modules-5.14.0-686.el9.x86_64 2026-02-24T15:43:37.432 INFO:teuthology.orchestra.run.trial084.stdout: kernel-modules-core-5.14.0-686.el9.x86_64 2026-02-24T15:43:37.432 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T15:43:37.432 INFO:teuthology.orchestra.run.trial084.stdout:Complete! 2026-02-24T15:43:37.492 DEBUG:teuthology.orchestra.run.trial084:> rpm -q kernel | sort -rV | head -n 1 2026-02-24T15:43:37.521 INFO:teuthology.orchestra.run.trial084.stdout:kernel-5.14.0-686.el9.x86_64 2026-02-24T15:43:37.522 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-686.el9.x86_64 2026-02-24T15:43:37.522 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-681.el9.x86_64 Expected: 5.14.0-686.el9.x86_64 2026-02-24T15:43:37.522 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2026-02-24T15:43:37.522 DEBUG:teuthology.task.kernel:src is distro, skipping download 2026-02-24T15:43:37.522 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2026-02-24T15:43:37.522 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@trial084.front.sepia.ceph.com, path=None, version=distro) 2026-02-24T15:43:37.523 DEBUG:teuthology.orchestra.run.trial084:> rpm -q kernel | sort -rV | head -n 1 2026-02-24T15:43:37.587 INFO:teuthology.orchestra.run.trial084.stdout:kernel-5.14.0-686.el9.x86_64 2026-02-24T15:43:37.587 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-686.el9.x86_64 2026-02-24T15:43:37.587 DEBUG:teuthology.orchestra.run.trial084:> sudo rpm -qi grub2-tools 2026-02-24T15:43:37.657 INFO:teuthology.orchestra.run.trial084.stdout:Name : grub2-tools 2026-02-24T15:43:37.657 INFO:teuthology.orchestra.run.trial084.stdout:Epoch : 1 2026-02-24T15:43:37.657 INFO:teuthology.orchestra.run.trial084.stdout:Version : 2.06 2026-02-24T15:43:37.657 INFO:teuthology.orchestra.run.trial084.stdout:Release : 120.el9 2026-02-24T15:43:37.657 INFO:teuthology.orchestra.run.trial084.stdout:Architecture: x86_64 2026-02-24T15:43:37.657 INFO:teuthology.orchestra.run.trial084.stdout:Install Date: Wed 18 Feb 2026 05:18:42 PM UTC 2026-02-24T15:43:37.657 INFO:teuthology.orchestra.run.trial084.stdout:Group : Unspecified 2026-02-24T15:43:37.657 INFO:teuthology.orchestra.run.trial084.stdout:Size : 9108611 2026-02-24T15:43:37.658 INFO:teuthology.orchestra.run.trial084.stdout:License : GPLv3+ 2026-02-24T15:43:37.658 INFO:teuthology.orchestra.run.trial084.stdout:Signature : RSA/SHA256, Thu 04 Dec 2025 09:42:59 AM UTC, Key ID 05b555b38483c65d 2026-02-24T15:43:37.658 INFO:teuthology.orchestra.run.trial084.stdout:Source RPM : grub2-2.06-120.el9.src.rpm 2026-02-24T15:43:37.658 INFO:teuthology.orchestra.run.trial084.stdout:Build Date : Wed 03 Dec 2025 12:10:38 PM UTC 2026-02-24T15:43:37.658 INFO:teuthology.orchestra.run.trial084.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2026-02-24T15:43:37.658 INFO:teuthology.orchestra.run.trial084.stdout:Packager : builder@centos.org 2026-02-24T15:43:37.658 INFO:teuthology.orchestra.run.trial084.stdout:Vendor : CentOS 2026-02-24T15:43:37.658 INFO:teuthology.orchestra.run.trial084.stdout:URL : http://www.gnu.org/software/grub/ 2026-02-24T15:43:37.658 INFO:teuthology.orchestra.run.trial084.stdout:Summary : Support tools for GRUB. 2026-02-24T15:43:37.658 INFO:teuthology.orchestra.run.trial084.stdout:Description : 2026-02-24T15:43:37.658 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T15:43:37.658 INFO:teuthology.orchestra.run.trial084.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2026-02-24T15:43:37.659 INFO:teuthology.orchestra.run.trial084.stdout:customizable bootloader with modular architecture. It supports a rich 2026-02-24T15:43:37.659 INFO:teuthology.orchestra.run.trial084.stdout:variety of kernel formats, file systems, computer architectures and 2026-02-24T15:43:37.659 INFO:teuthology.orchestra.run.trial084.stdout:hardware devices. 2026-02-24T15:43:37.659 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T15:43:37.659 INFO:teuthology.orchestra.run.trial084.stdout:This subpackage provides tools for support of all platforms. 2026-02-24T15:43:37.660 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2026-02-24T15:43:37.660 INFO:teuthology.task.kernel:Updating grub on trial084 to boot 5.14.0-686.el9.x86_64 2026-02-24T15:43:37.660 DEBUG:teuthology.orchestra.run.trial084:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f -print -quit 2026-02-24T15:43:37.723 INFO:teuthology.orchestra.run.trial084.stdout:/boot/loader/entries/b0373be789df49cdafedf423703d2566-5.14.0-681.el9.x86_64.conf 2026-02-24T15:43:37.724 DEBUG:teuthology.orchestra.run.trial084:> sudo command -v grubby && echo yes || echo no 2026-02-24T15:43:37.788 INFO:teuthology.orchestra.run.trial084.stdout:/sbin/grubby 2026-02-24T15:43:37.789 INFO:teuthology.orchestra.run.trial084.stdout:yes 2026-02-24T15:43:37.789 DEBUG:teuthology.orchestra.run.trial084:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f | grep -F "5.14.0-686.el9.x86_64" | head -n 1 2026-02-24T15:43:37.855 INFO:teuthology.orchestra.run.trial084.stdout:/boot/loader/entries/b0373be789df49cdafedf423703d2566-5.14.0-686.el9.x86_64.conf 2026-02-24T15:43:37.855 DEBUG:teuthology.orchestra.run.trial084:> sudo grub2-set-default b0373be789df49cdafedf423703d2566-5.14.0-686.el9.x86_64 2026-02-24T15:43:37.927 DEBUG:teuthology.orchestra.run.trial084:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2026-02-24T15:43:38.131 INFO:teuthology.orchestra.run.trial084.stderr:Generating grub configuration file ... 2026-02-24T15:43:38.197 INFO:teuthology.orchestra.run.trial084.stderr:Found linux image: /boot/vmlinuz-5.14.0-686.el9.x86_64 2026-02-24T15:43:38.204 INFO:teuthology.orchestra.run.trial084.stderr:Found initrd image: /boot/initramfs-5.14.0-686.el9.x86_64.img 2026-02-24T15:43:38.452 INFO:teuthology.orchestra.run.trial084.stderr:Found linux image: /boot/vmlinuz-5.14.0-681.el9.x86_64 2026-02-24T15:43:38.459 INFO:teuthology.orchestra.run.trial084.stderr:Found initrd image: /boot/initramfs-5.14.0-681.el9.x86_64.img 2026-02-24T15:43:38.493 INFO:teuthology.orchestra.run.trial084.stderr:Found linux image: /boot/vmlinuz-5.14.0-645.el9.x86_64 2026-02-24T15:43:38.500 INFO:teuthology.orchestra.run.trial084.stderr:Found initrd image: /boot/initramfs-5.14.0-645.el9.x86_64.img 2026-02-24T15:43:38.531 INFO:teuthology.orchestra.run.trial084.stderr:Found linux image: /boot/vmlinuz-0-rescue-b0373be789df49cdafedf423703d2566 2026-02-24T15:43:38.537 INFO:teuthology.orchestra.run.trial084.stderr:Found initrd image: /boot/initramfs-0-rescue-b0373be789df49cdafedf423703d2566.img 2026-02-24T15:43:38.560 INFO:teuthology.orchestra.run.trial084.stderr:Found linux image: /boot/vmlinuz-0-rescue-2734456005d743dca7f3dc77c35a2d0d 2026-02-24T15:43:38.566 INFO:teuthology.orchestra.run.trial084.stderr:Found initrd image: /boot/initramfs-0-rescue-2734456005d743dca7f3dc77c35a2d0d.img 2026-02-24T15:43:38.652 INFO:teuthology.orchestra.run.trial084.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2026-02-24T15:43:38.661 INFO:teuthology.orchestra.run.trial084.stderr:done 2026-02-24T15:43:38.663 DEBUG:teuthology.orchestra.run.trial084:> test -d /sys/firmware/efi 2026-02-24T15:43:38.679 DEBUG:teuthology.orchestra.run.trial084:> sudo find /boot/efi/EFI -mindepth 1 -maxdepth 1 -type d || true 2026-02-24T15:43:38.745 INFO:teuthology.orchestra.run.trial084.stdout:/boot/efi/EFI/BOOT 2026-02-24T15:43:38.745 INFO:teuthology.orchestra.run.trial084.stdout:/boot/efi/EFI/centos 2026-02-24T15:43:38.746 DEBUG:teuthology.orchestra.run.trial084:> sudo test -f /boot/efi/EFI/BOOT/grub.cfg && echo yes || echo no 2026-02-24T15:43:38.811 INFO:teuthology.orchestra.run.trial084.stdout:no 2026-02-24T15:43:38.811 DEBUG:teuthology.orchestra.run.trial084:> sudo test -f /boot/efi/EFI/centos/grub.cfg && echo yes || echo no 2026-02-24T15:43:38.875 INFO:teuthology.orchestra.run.trial084.stdout:yes 2026-02-24T15:43:38.875 DEBUG:teuthology.orchestra.run.trial084:> sudo cp -f /boot/grub2/grub.cfg /boot/efi/EFI/centos/grub.cfg 2026-02-24T15:43:38.938 INFO:teuthology.orchestra.remote:Triggering safe hard reboot via SysRq on trial084 2026-02-24T15:43:38.939 DEBUG:teuthology.orchestra.run.trial084:> sudo bash -c ' 2026-02-24T15:43:38.939 DEBUG:teuthology.orchestra.run.trial084:> echo pci > /sys/kernel/reboot/type && echo 1 > /sys/kernel/reboot/force && sync && echo s > /proc/sysrq-trigger && echo u > /proc/sysrq-trigger && sleep 2 && echo b > /proc/sysrq-trigger 2026-02-24T15:43:38.939 DEBUG:teuthology.orchestra.run.trial084:> ' 2026-02-24T15:44:08.981 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2026-02-24T15:44:08.981 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial084.front.sepia.ceph.com' 2026-02-24T15:44:08.982 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-24T15:45:09.043 DEBUG:teuthology.orchestra.remote:timed out 2026-02-24T15:45:18.048 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial084.front.sepia.ceph.com' 2026-02-24T15:45:18.049 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-24T15:45:18.194 DEBUG:teuthology.orchestra.run.trial084:> true 2026-02-24T15:45:18.347 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial084.front.sepia.ceph.com' 2026-02-24T15:45:18.347 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-686.el9.x86_64"... 2026-02-24T15:45:18.347 DEBUG:teuthology.orchestra.run.trial084:> uname -r 2026-02-24T15:45:18.403 INFO:teuthology.orchestra.run.trial084.stdout:5.14.0-686.el9.x86_64 2026-02-24T15:45:18.403 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-686.el9.x86_64 vs 5.14.0-686.el9.x86_64 2026-02-24T15:45:18.403 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2026-02-24T15:45:18.403 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2026-02-24T15:45:19.404 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2026-02-24T15:45:19.405 DEBUG:teuthology.orchestra.run.trial084:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-24T15:45:19.435 INFO:teuthology.orchestra.run.trial084.stdout:ttyS1 2026-02-24T15:45:19.463 DEBUG:teuthology.parallel:result is None 2026-02-24T15:45:19.464 INFO:teuthology.run_tasks:Running task internal.base... 2026-02-24T15:45:19.476 INFO:teuthology.task.internal:Creating test directory... 2026-02-24T15:45:19.476 DEBUG:teuthology.orchestra.run.trial084:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-24T15:45:19.520 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2026-02-24T15:45:19.533 INFO:teuthology.run_tasks:Running task internal.archive... 2026-02-24T15:45:19.547 INFO:teuthology.task.internal:Creating archive directory... 2026-02-24T15:45:19.547 DEBUG:teuthology.orchestra.run.trial084:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-24T15:45:19.579 INFO:teuthology.run_tasks:Running task internal.coredump... 2026-02-24T15:45:19.590 INFO:teuthology.task.internal:Enabling coredump saving... 2026-02-24T15:45:19.590 DEBUG:teuthology.orchestra.run.trial084:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-24T15:45:19.641 INFO:teuthology.orchestra.run.trial084.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-24T15:45:19.650 INFO:teuthology.orchestra.run.trial084.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-24T15:45:19.652 INFO:teuthology.run_tasks:Running task internal.sudo... 2026-02-24T15:45:19.666 INFO:teuthology.task.internal:Configuring sudo... 2026-02-24T15:45:19.667 DEBUG:teuthology.orchestra.run.trial084:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-24T15:45:19.717 INFO:teuthology.run_tasks:Running task internal.syslog... 2026-02-24T15:45:19.730 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2026-02-24T15:45:19.730 DEBUG:teuthology.orchestra.run.trial084:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-24T15:45:19.773 DEBUG:teuthology.orchestra.run.trial084:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-24T15:45:19.899 DEBUG:teuthology.orchestra.run.trial084:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-24T15:45:19.922 DEBUG:teuthology.orchestra.run.trial084:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-24T15:45:20.052 DEBUG:teuthology.orchestra.run.trial084:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-24T15:45:20.077 DEBUG:teuthology.orchestra.run.trial084:> set -ex 2026-02-24T15:45:20.078 DEBUG:teuthology.orchestra.run.trial084:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-24T15:45:20.141 DEBUG:teuthology.orchestra.run.trial084:> sudo service rsyslog restart 2026-02-24T15:45:20.208 INFO:teuthology.orchestra.run.trial084.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-24T15:45:20.469 INFO:teuthology.run_tasks:Running task internal.timer... 2026-02-24T15:45:20.481 INFO:teuthology.task.internal:Starting timer... 2026-02-24T15:45:20.481 INFO:teuthology.run_tasks:Running task pcp... 2026-02-24T15:45:20.501 INFO:teuthology.run_tasks:Running task selinux... 2026-02-24T15:45:20.514 DEBUG:teuthology.orchestra.run.trial084:> sudo service auditd rotate 2026-02-24T15:45:20.542 INFO:teuthology.orchestra.run.trial084.stdout:Rotating logs: 2026-02-24T15:45:20.544 DEBUG:teuthology.task.selinux:Getting current SELinux state 2026-02-24T15:45:20.544 DEBUG:teuthology.orchestra.run.trial084:> /usr/sbin/getenforce 2026-02-24T15:45:20.601 INFO:teuthology.orchestra.run.trial084.stdout:Permissive 2026-02-24T15:45:20.601 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@trial084.front.sepia.ceph.com': 'permissive'} 2026-02-24T15:45:20.601 DEBUG:teuthology.orchestra.run.trial084:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-24T15:45:20.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-24T15:45:20.666 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2026-02-24T15:45:20.666 DEBUG:teuthology.orchestra.run.trial084:> sudo /usr/sbin/setenforce permissive 2026-02-24T15:45:20.729 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2026-02-24T15:45:20.741 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2026-02-24T15:45:20.751 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2026-02-24T15:45:20.752 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit trial084.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2026-02-24T15:48:04.269 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@trial084.front.sepia.ceph.com')] 2026-02-24T15:48:04.270 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial084.front.sepia.ceph.com' 2026-02-24T15:48:04.272 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-24T15:48:04.380 DEBUG:teuthology.orchestra.run.trial084:> true 2026-02-24T15:48:04.452 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial084.front.sepia.ceph.com' 2026-02-24T15:48:04.452 INFO:teuthology.run_tasks:Running task clock... 2026-02-24T15:48:04.469 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2026-02-24T15:48:04.469 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-24T15:48:04.470 DEBUG:teuthology.orchestra.run.trial084:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-24T15:48:04.520 INFO:teuthology.orchestra.run.trial084.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2026-02-24T15:48:04.532 INFO:teuthology.orchestra.run.trial084.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2026-02-24T15:48:04.552 INFO:teuthology.orchestra.run.trial084.stderr:sudo: ntpd: command not found 2026-02-24T15:48:04.561 INFO:teuthology.orchestra.run.trial084.stdout:506 Cannot talk to daemon 2026-02-24T15:48:04.571 INFO:teuthology.orchestra.run.trial084.stderr:Failed to start ntp.service: Unit ntp.service not found. 2026-02-24T15:48:04.583 INFO:teuthology.orchestra.run.trial084.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2026-02-24T15:48:04.625 INFO:teuthology.orchestra.run.trial084.stderr:bash: line 1: ntpq: command not found 2026-02-24T15:48:04.627 INFO:teuthology.orchestra.run.trial084.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-24T15:48:04.627 INFO:teuthology.orchestra.run.trial084.stdout:=============================================================================== 2026-02-24T15:48:04.628 INFO:teuthology.run_tasks:Running task install... 2026-02-24T15:48:04.642 DEBUG:teuthology.task.install:project ceph 2026-02-24T15:48:04.642 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'debug', 'sha1': 'd26e5c228e3cdea4d585bcd077f8513a649ab5d1'}} 2026-02-24T15:48:04.642 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'flavor': 'debug', 'sha1': 'd26e5c228e3cdea4d585bcd077f8513a649ab5d1'} 2026-02-24T15:48:04.642 INFO:teuthology.task.install:Using flavor: debug 2026-02-24T15:48:04.652 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-osd', 'ceph-osd-classic', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-osd', 'ceph-osd-classic', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-24T15:48:04.652 INFO:teuthology.task.install:extra packages: {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']} 2026-02-24T15:48:04.653 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'debug', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd26e5c228e3cdea4d585bcd077f8513a649ab5d1', 'tag': None, 'wait_for_package': False} 2026-02-24T15:48:04.654 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=d26e5c228e3cdea4d585bcd077f8513a649ab5d1 2026-02-24T15:48:04.820 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-liucm-shard-redesign-crimson-only/d26e5c228e3cdea4d585bcd077f8513a649ab5d1/centos/9/flavors/debug/ 2026-02-24T15:48:04.821 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5343.gd26e5c22 2026-02-24T15:48:05.175 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-liucm-shard-redesign-crimson-only/d26e5c228e3cdea4d585bcd077f8513a649ab5d1/centos/9/flavors/debug/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-liucm-shard-redesign-crimson-only/d26e5c228e3cdea4d585bcd077f8513a649ab5d1/centos/9/flavors/debug/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-liucm-shard-redesign-crimson-only/d26e5c228e3cdea4d585bcd077f8513a649ab5d1/centos/9/flavors/debug/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-24T15:48:05.175 DEBUG:teuthology.orchestra.run.trial084:> set -ex 2026-02-24T15:48:05.175 DEBUG:teuthology.orchestra.run.trial084:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-24T15:48:05.201 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-osd-crimson, ceph-osd-crimson-debuginfo on remote rpm x86_64 2026-02-24T15:48:05.202 DEBUG:teuthology.orchestra.run.trial084:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d26e5c228e3cdea4d585bcd077f8513a649ab5d1/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-24T15:48:05.266 DEBUG:teuthology.orchestra.run.trial084:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-24T15:48:05.340 DEBUG:teuthology.orchestra.run.trial084:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-24T15:48:05.404 INFO:teuthology.orchestra.run.trial084.stdout:check_obsoletes = 1 2026-02-24T15:48:05.406 DEBUG:teuthology.orchestra.run.trial084:> sudo yum clean all 2026-02-24T15:48:05.554 INFO:teuthology.orchestra.run.trial084.stdout:41 files removed 2026-02-24T15:48:05.567 DEBUG:teuthology.orchestra.run.trial084:> sudo yum -y install ceph-radosgw 2026-02-24T15:48:06.425 INFO:teuthology.orchestra.run.trial084.stdout:ceph packages for x86_64 129 kB/s | 92 kB 00:00 2026-02-24T15:48:07.110 INFO:teuthology.orchestra.run.trial084.stdout:ceph noarch packages 36 kB/s | 24 kB 00:00 2026-02-24T15:48:07.711 INFO:teuthology.orchestra.run.trial084.stdout:ceph source packages 3.6 kB/s | 2.1 kB 00:00 2026-02-24T15:48:11.948 INFO:teuthology.orchestra.run.trial084.stdout:CentOS Stream 9 - BaseOS 2.1 MB/s | 8.9 MB 00:04 2026-02-24T15:48:15.669 INFO:teuthology.orchestra.run.trial084.stdout:CentOS Stream 9 - AppStream 8.5 MB/s | 27 MB 00:03 2026-02-24T15:48:19.008 INFO:teuthology.orchestra.run.trial084.stdout:CentOS Stream 9 - CRB 8.8 MB/s | 8.0 MB 00:00 2026-02-24T15:48:20.074 INFO:teuthology.orchestra.run.trial084.stdout:CentOS Stream 9 - Extras packages 66 kB/s | 20 kB 00:00 2026-02-24T15:48:20.724 INFO:teuthology.orchestra.run.trial084.stdout:Extra Packages for Enterprise Linux 35 MB/s | 20 MB 00:00 2026-02-24T15:48:23.929 INFO:teuthology.orchestra.run.trial084.stdout:lab-extras 1.3 MB/s | 50 kB 00:00 2026-02-24T15:48:25.143 INFO:teuthology.orchestra.run.trial084.stdout:Dependencies resolved. 2026-02-24T15:48:25.145 INFO:teuthology.orchestra.run.trial084.stdout:================================================================================ 2026-02-24T15:48:25.146 INFO:teuthology.orchestra.run.trial084.stdout: Package Arch Version Repository Size 2026-02-24T15:48:25.146 INFO:teuthology.orchestra.run.trial084.stdout:================================================================================ 2026-02-24T15:48:25.146 INFO:teuthology.orchestra.run.trial084.stdout:Installing: 2026-02-24T15:48:25.146 INFO:teuthology.orchestra.run.trial084.stdout: ceph-radosgw x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 11 M 2026-02-24T15:48:25.146 INFO:teuthology.orchestra.run.trial084.stdout:Upgrading: 2026-02-24T15:48:25.146 INFO:teuthology.orchestra.run.trial084.stdout: librados2 x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 3.3 M 2026-02-24T15:48:25.146 INFO:teuthology.orchestra.run.trial084.stdout: librbd1 x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 2.7 M 2026-02-24T15:48:25.146 INFO:teuthology.orchestra.run.trial084.stdout:Installing dependencies: 2026-02-24T15:48:25.147 INFO:teuthology.orchestra.run.trial084.stdout: boost-program-options x86_64 1.75.0-13.el9 appstream 104 k 2026-02-24T15:48:25.147 INFO:teuthology.orchestra.run.trial084.stdout: ceph-base x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 5.1 M 2026-02-24T15:48:25.147 INFO:teuthology.orchestra.run.trial084.stdout: ceph-common x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 21 M 2026-02-24T15:48:25.147 INFO:teuthology.orchestra.run.trial084.stdout: ceph-selinux x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 25 k 2026-02-24T15:48:25.147 INFO:teuthology.orchestra.run.trial084.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2026-02-24T15:48:25.147 INFO:teuthology.orchestra.run.trial084.stdout: libarrow x86_64 9.0.0-15.el9 epel 4.4 M 2026-02-24T15:48:25.147 INFO:teuthology.orchestra.run.trial084.stdout: libarrow-doc noarch 9.0.0-15.el9 epel 25 k 2026-02-24T15:48:25.147 INFO:teuthology.orchestra.run.trial084.stdout: libcephfs2 x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 860 k 2026-02-24T15:48:25.147 INFO:teuthology.orchestra.run.trial084.stdout: libnbd x86_64 1.20.3-4.el9 appstream 164 k 2026-02-24T15:48:25.147 INFO:teuthology.orchestra.run.trial084.stdout: liboath x86_64 2.6.12-1.el9 epel 49 k 2026-02-24T15:48:25.147 INFO:teuthology.orchestra.run.trial084.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2026-02-24T15:48:25.148 INFO:teuthology.orchestra.run.trial084.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2026-02-24T15:48:25.148 INFO:teuthology.orchestra.run.trial084.stdout: libradosstriper1 x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 263 k 2026-02-24T15:48:25.148 INFO:teuthology.orchestra.run.trial084.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2026-02-24T15:48:25.148 INFO:teuthology.orchestra.run.trial084.stdout: librgw2 x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 5.6 M 2026-02-24T15:48:25.148 INFO:teuthology.orchestra.run.trial084.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2026-02-24T15:48:25.148 INFO:teuthology.orchestra.run.trial084.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2026-02-24T15:48:25.148 INFO:teuthology.orchestra.run.trial084.stdout: mailcap noarch 2.1.49-5.el9 baseos 33 k 2026-02-24T15:48:25.148 INFO:teuthology.orchestra.run.trial084.stdout: parquet-libs x86_64 9.0.0-15.el9 epel 838 k 2026-02-24T15:48:25.148 INFO:teuthology.orchestra.run.trial084.stdout: pciutils x86_64 3.7.0-7.el9 baseos 93 k 2026-02-24T15:48:25.149 INFO:teuthology.orchestra.run.trial084.stdout: python3-ceph-argparse x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 45 k 2026-02-24T15:48:25.149 INFO:teuthology.orchestra.run.trial084.stdout: python3-ceph-common x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 185 k 2026-02-24T15:48:25.149 INFO:teuthology.orchestra.run.trial084.stdout: python3-cephfs x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 203 k 2026-02-24T15:48:25.149 INFO:teuthology.orchestra.run.trial084.stdout: python3-rados x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 324 k 2026-02-24T15:48:25.149 INFO:teuthology.orchestra.run.trial084.stdout: python3-rbd x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 305 k 2026-02-24T15:48:25.149 INFO:teuthology.orchestra.run.trial084.stdout: python3-rgw x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 100 k 2026-02-24T15:48:25.149 INFO:teuthology.orchestra.run.trial084.stdout: qatlib x86_64 25.08.0-2.el9 appstream 240 k 2026-02-24T15:48:25.149 INFO:teuthology.orchestra.run.trial084.stdout: qatzip-libs x86_64 1.3.1-1.el9 appstream 66 k 2026-02-24T15:48:25.150 INFO:teuthology.orchestra.run.trial084.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2026-02-24T15:48:25.150 INFO:teuthology.orchestra.run.trial084.stdout: thrift x86_64 0.15.0-4.el9 epel 1.6 M 2026-02-24T15:48:25.150 INFO:teuthology.orchestra.run.trial084.stdout:Installing weak dependencies: 2026-02-24T15:48:25.150 INFO:teuthology.orchestra.run.trial084.stdout: qatlib-service x86_64 25.08.0-2.el9 appstream 37 k 2026-02-24T15:48:25.150 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T15:48:25.150 INFO:teuthology.orchestra.run.trial084.stdout:Transaction Summary 2026-02-24T15:48:25.150 INFO:teuthology.orchestra.run.trial084.stdout:================================================================================ 2026-02-24T15:48:25.150 INFO:teuthology.orchestra.run.trial084.stdout:Install 32 Packages 2026-02-24T15:48:25.150 INFO:teuthology.orchestra.run.trial084.stdout:Upgrade 2 Packages 2026-02-24T15:48:25.150 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T15:48:25.150 INFO:teuthology.orchestra.run.trial084.stdout:Total download size: 60 M 2026-02-24T15:48:25.151 INFO:teuthology.orchestra.run.trial084.stdout:Downloading Packages: 2026-02-24T15:48:36.560 INFO:teuthology.orchestra.run.trial084.stdout:(1/34): ceph-base-20.3.0-5343.gd26e5c22.el9.x86 486 kB/s | 5.1 MB 00:10 2026-02-24T15:48:36.630 INFO:teuthology.orchestra.run.trial084.stdout:(2/34): ceph-selinux-20.3.0-5343.gd26e5c22.el9. 358 kB/s | 25 kB 00:00 2026-02-24T15:48:37.946 INFO:teuthology.orchestra.run.trial084.stdout:(3/34): libcephfs2-20.3.0-5343.gd26e5c22.el9.x8 654 kB/s | 860 kB 00:01 2026-02-24T15:48:38.299 INFO:teuthology.orchestra.run.trial084.stdout:(4/34): libradosstriper1-20.3.0-5343.gd26e5c22. 745 kB/s | 263 kB 00:00 2026-02-24T15:48:45.955 INFO:teuthology.orchestra.run.trial084.stdout:(5/34): librgw2-20.3.0-5343.gd26e5c22.el9.x86_6 753 kB/s | 5.6 MB 00:07 2026-02-24T15:48:46.025 INFO:teuthology.orchestra.run.trial084.stdout:(6/34): python3-ceph-argparse-20.3.0-5343.gd26e 647 kB/s | 45 kB 00:00 2026-02-24T15:48:46.238 INFO:teuthology.orchestra.run.trial084.stdout:(7/34): python3-ceph-common-20.3.0-5343.gd26e5c 869 kB/s | 185 kB 00:00 2026-02-24T15:48:46.517 INFO:teuthology.orchestra.run.trial084.stdout:(8/34): python3-cephfs-20.3.0-5343.gd26e5c22.el 729 kB/s | 203 kB 00:00 2026-02-24T15:48:46.872 INFO:teuthology.orchestra.run.trial084.stdout:(9/34): python3-rados-20.3.0-5343.gd26e5c22.el9 911 kB/s | 324 kB 00:00 2026-02-24T15:48:47.227 INFO:teuthology.orchestra.run.trial084.stdout:(10/34): python3-rbd-20.3.0-5343.gd26e5c22.el9. 860 kB/s | 305 kB 00:00 2026-02-24T15:48:47.366 INFO:teuthology.orchestra.run.trial084.stdout:(11/34): python3-rgw-20.3.0-5343.gd26e5c22.el9. 716 kB/s | 100 kB 00:00 2026-02-24T15:48:47.390 INFO:teuthology.orchestra.run.trial084.stdout:(12/34): mailcap-2.1.49-5.el9.noarch.rpm 1.4 MB/s | 33 kB 00:00 2026-02-24T15:48:47.399 INFO:teuthology.orchestra.run.trial084.stdout:(13/34): pciutils-3.7.0-7.el9.x86_64.rpm 11 MB/s | 93 kB 00:00 2026-02-24T15:48:47.614 INFO:teuthology.orchestra.run.trial084.stdout:(14/34): ceph-radosgw-20.3.0-5343.gd26e5c22.el9 525 kB/s | 11 MB 00:21 2026-02-24T15:48:48.126 INFO:teuthology.orchestra.run.trial084.stdout:(15/34): boost-program-options-1.75.0-13.el9.x8 143 kB/s | 104 kB 00:00 2026-02-24T15:48:48.330 INFO:teuthology.orchestra.run.trial084.stdout:(16/34): libnbd-1.20.3-4.el9.x86_64.rpm 229 kB/s | 164 kB 00:00 2026-02-24T15:48:48.443 INFO:teuthology.orchestra.run.trial084.stdout:(17/34): libpmemobj-1.12.1-1.el9.x86_64.rpm 505 kB/s | 160 kB 00:00 2026-02-24T15:48:48.489 INFO:teuthology.orchestra.run.trial084.stdout:(18/34): librabbitmq-0.11.0-7.el9.x86_64.rpm 284 kB/s | 45 kB 00:00 2026-02-24T15:48:49.042 INFO:teuthology.orchestra.run.trial084.stdout:(19/34): lttng-ust-2.12.0-6.el9.x86_64.rpm 529 kB/s | 292 kB 00:00 2026-02-24T15:48:49.201 INFO:teuthology.orchestra.run.trial084.stdout:[MIRROR] qatlib-25.08.0-2.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-25.08.0-2.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T15:48:49.316 INFO:teuthology.orchestra.run.trial084.stdout:(20/34): librdkafka-1.6.1-102.el9.x86_64.rpm 759 kB/s | 662 kB 00:00 2026-02-24T15:48:49.476 INFO:teuthology.orchestra.run.trial084.stdout:[MIRROR] qatlib-service-25.08.0-2.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-25.08.0-2.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T15:48:49.848 INFO:teuthology.orchestra.run.trial084.stdout:[MIRROR] qatlib-25.08.0-2.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-25.08.0-2.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T15:48:49.904 INFO:teuthology.orchestra.run.trial084.stdout:(21/34): qatlib-25.08.0-2.el9.x86_64.rpm 278 kB/s | 240 kB 00:00 2026-02-24T15:48:50.119 INFO:teuthology.orchestra.run.trial084.stdout:[MIRROR] qatlib-service-25.08.0-2.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-25.08.0-2.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T15:48:50.149 INFO:teuthology.orchestra.run.trial084.stdout:(22/34): qatlib-service-25.08.0-2.el9.x86_64.rp 44 kB/s | 37 kB 00:00 2026-02-24T15:48:50.396 INFO:teuthology.orchestra.run.trial084.stdout:(23/34): gperftools-libs-2.9.1-3.el9.x86_64.rpm 1.2 MB/s | 308 kB 00:00 2026-02-24T15:48:50.461 INFO:teuthology.orchestra.run.trial084.stdout:(24/34): qatzip-libs-1.3.1-1.el9.x86_64.rpm 119 kB/s | 66 kB 00:00 2026-02-24T15:48:50.572 INFO:teuthology.orchestra.run.trial084.stdout:(25/34): libarrow-doc-9.0.0-15.el9.noarch.rpm 224 kB/s | 25 kB 00:00 2026-02-24T15:48:50.583 INFO:teuthology.orchestra.run.trial084.stdout:(26/34): libarrow-9.0.0-15.el9.x86_64.rpm 24 MB/s | 4.4 MB 00:00 2026-02-24T15:48:50.620 INFO:teuthology.orchestra.run.trial084.stdout:(27/34): libunwind-1.6.2-1.el9.x86_64.rpm 1.8 MB/s | 67 kB 00:00 2026-02-24T15:48:50.647 INFO:teuthology.orchestra.run.trial084.stdout:(28/34): liboath-2.6.12-1.el9.x86_64.rpm 655 kB/s | 49 kB 00:00 2026-02-24T15:48:50.660 INFO:teuthology.orchestra.run.trial084.stdout:(29/34): parquet-libs-9.0.0-15.el9.x86_64.rpm 20 MB/s | 838 kB 00:00 2026-02-24T15:48:50.705 INFO:teuthology.orchestra.run.trial084.stdout:(30/34): thrift-0.15.0-4.el9.x86_64.rpm 35 MB/s | 1.6 MB 00:00 2026-02-24T15:48:50.747 INFO:teuthology.orchestra.run.trial084.stdout:(31/34): re2-20211101-20.el9.x86_64.rpm 1.9 MB/s | 191 kB 00:00 2026-02-24T15:48:55.487 INFO:teuthology.orchestra.run.trial084.stdout:(32/34): librados2-20.3.0-5343.gd26e5c22.el9.x8 703 kB/s | 3.3 MB 00:04 2026-02-24T15:48:58.216 INFO:teuthology.orchestra.run.trial084.stdout:(33/34): librbd1-20.3.0-5343.gd26e5c22.el9.x86_ 369 kB/s | 2.7 MB 00:07 2026-02-24T15:49:26.172 INFO:teuthology.orchestra.run.trial084.stdout:(34/34): ceph-common-20.3.0-5343.gd26e5c22.el9. 353 kB/s | 21 MB 01:00 2026-02-24T15:49:26.176 INFO:teuthology.orchestra.run.trial084.stdout:-------------------------------------------------------------------------------- 2026-02-24T15:49:26.177 INFO:teuthology.orchestra.run.trial084.stdout:Total 1.0 MB/s | 60 MB 01:01 2026-02-24T15:49:26.255 INFO:teuthology.orchestra.run.trial084.stdout:Running transaction check 2026-02-24T15:49:26.274 INFO:teuthology.orchestra.run.trial084.stdout:Transaction check succeeded. 2026-02-24T15:49:26.274 INFO:teuthology.orchestra.run.trial084.stdout:Running transaction test 2026-02-24T15:49:26.516 INFO:teuthology.orchestra.run.trial084.stdout:Transaction test succeeded. 2026-02-24T15:49:26.516 INFO:teuthology.orchestra.run.trial084.stdout:Running transaction 2026-02-24T15:49:26.798 INFO:teuthology.orchestra.run.trial084.stdout: Preparing : 1/1 2026-02-24T15:49:26.939 INFO:teuthology.orchestra.run.trial084.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/36 2026-02-24T15:49:26.943 INFO:teuthology.orchestra.run.trial084.stdout: Upgrading : librados2-2:20.3.0-5343.gd26e5c22.el9.x86_64 2/36 2026-02-24T15:49:26.973 INFO:teuthology.orchestra.run.trial084.stdout: Running scriptlet: librados2-2:20.3.0-5343.gd26e5c22.el9.x86_64 2/36 2026-02-24T15:49:27.009 INFO:teuthology.orchestra.run.trial084.stdout: Installing : python3-rados-2:20.3.0-5343.gd26e5c22.el9.x86_64 3/36 2026-02-24T15:49:27.009 INFO:teuthology.orchestra.run.trial084.stdout: Installing : libcephfs2-2:20.3.0-5343.gd26e5c22.el9.x86_64 4/36 2026-02-24T15:49:27.021 INFO:teuthology.orchestra.run.trial084.stdout: Running scriptlet: libcephfs2-2:20.3.0-5343.gd26e5c22.el9.x86_64 4/36 2026-02-24T15:49:27.029 INFO:teuthology.orchestra.run.trial084.stdout: Installing : liboath-2.6.12-1.el9.x86_64 5/36 2026-02-24T15:49:27.032 INFO:teuthology.orchestra.run.trial084.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/36 2026-02-24T15:49:27.035 INFO:teuthology.orchestra.run.trial084.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/36 2026-02-24T15:49:27.040 INFO:teuthology.orchestra.run.trial084.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 8/36 2026-02-24T15:49:27.051 INFO:teuthology.orchestra.run.trial084.stdout: Installing : python3-ceph-argparse-2:20.3.0-5343.gd26e5c22.el9. 9/36 2026-02-24T15:49:27.066 INFO:teuthology.orchestra.run.trial084.stdout: Installing : python3-cephfs-2:20.3.0-5343.gd26e5c22.el9.x86_64 10/36 2026-02-24T15:49:27.067 INFO:teuthology.orchestra.run.trial084.stdout: Installing : libradosstriper1-2:20.3.0-5343.gd26e5c22.el9.x86_6 11/36 2026-02-24T15:49:27.090 INFO:teuthology.orchestra.run.trial084.stdout: Running scriptlet: libradosstriper1-2:20.3.0-5343.gd26e5c22.el9.x86_6 11/36 2026-02-24T15:49:27.094 INFO:teuthology.orchestra.run.trial084.stdout: Installing : thrift-0.15.0-4.el9.x86_64 12/36 2026-02-24T15:49:27.127 INFO:teuthology.orchestra.run.trial084.stdout: Installing : re2-1:20211101-20.el9.x86_64 13/36 2026-02-24T15:49:27.131 INFO:teuthology.orchestra.run.trial084.stdout: Installing : libarrow-9.0.0-15.el9.x86_64 14/36 2026-02-24T15:49:27.137 INFO:teuthology.orchestra.run.trial084.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 15/36 2026-02-24T15:49:27.139 INFO:teuthology.orchestra.run.trial084.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 16/36 2026-02-24T15:49:27.144 INFO:teuthology.orchestra.run.trial084.stdout: Installing : libarrow-doc-9.0.0-15.el9.noarch 17/36 2026-02-24T15:49:27.257 INFO:teuthology.orchestra.run.trial084.stdout: Installing : libnbd-1.20.3-4.el9.x86_64 18/36 2026-02-24T15:49:27.258 INFO:teuthology.orchestra.run.trial084.stdout: Upgrading : librbd1-2:20.3.0-5343.gd26e5c22.el9.x86_64 19/36 2026-02-24T15:49:27.283 INFO:teuthology.orchestra.run.trial084.stdout: Running scriptlet: librbd1-2:20.3.0-5343.gd26e5c22.el9.x86_64 19/36 2026-02-24T15:49:27.285 INFO:teuthology.orchestra.run.trial084.stdout: Installing : python3-rbd-2:20.3.0-5343.gd26e5c22.el9.x86_64 20/36 2026-02-24T15:49:27.291 INFO:teuthology.orchestra.run.trial084.stdout: Installing : boost-program-options-1.75.0-13.el9.x86_64 21/36 2026-02-24T15:49:27.539 INFO:teuthology.orchestra.run.trial084.stdout: Installing : parquet-libs-9.0.0-15.el9.x86_64 22/36 2026-02-24T15:49:27.540 INFO:teuthology.orchestra.run.trial084.stdout: Installing : librgw2-2:20.3.0-5343.gd26e5c22.el9.x86_64 23/36 2026-02-24T15:49:27.556 INFO:teuthology.orchestra.run.trial084.stdout: Running scriptlet: librgw2-2:20.3.0-5343.gd26e5c22.el9.x86_64 23/36 2026-02-24T15:49:27.561 INFO:teuthology.orchestra.run.trial084.stdout: Installing : python3-rgw-2:20.3.0-5343.gd26e5c22.el9.x86_64 24/36 2026-02-24T15:49:27.564 INFO:teuthology.orchestra.run.trial084.stdout: Installing : pciutils-3.7.0-7.el9.x86_64 25/36 2026-02-24T15:49:27.567 INFO:teuthology.orchestra.run.trial084.stdout: Installing : qatlib-25.08.0-2.el9.x86_64 26/36 2026-02-24T15:49:27.567 INFO:teuthology.orchestra.run.trial084.stdout: Installing : qatlib-service-25.08.0-2.el9.x86_64 27/36 2026-02-24T15:49:27.585 INFO:teuthology.orchestra.run.trial084.stdout: Running scriptlet: qatlib-service-25.08.0-2.el9.x86_64 27/36 2026-02-24T15:49:27.587 INFO:teuthology.orchestra.run.trial084.stdout: Installing : qatzip-libs-1.3.1-1.el9.x86_64 28/36 2026-02-24T15:49:27.622 INFO:teuthology.orchestra.run.trial084.stdout: Installing : mailcap-2.1.49-5.el9.noarch 29/36 2026-02-24T15:49:27.624 INFO:teuthology.orchestra.run.trial084.stdout: Installing : python3-ceph-common-2:20.3.0-5343.gd26e5c22.el9.x8 30/36 2026-02-24T15:49:28.557 INFO:teuthology.orchestra.run.trial084.stdout: Running scriptlet: ceph-common-2:20.3.0-5343.gd26e5c22.el9.x86_64 31/36 2026-02-24T15:49:28.560 INFO:teuthology.orchestra.run.trial084.stdout: Installing : ceph-common-2:20.3.0-5343.gd26e5c22.el9.x86_64 31/36 2026-02-24T15:49:28.820 INFO:teuthology.orchestra.run.trial084.stdout: Running scriptlet: ceph-common-2:20.3.0-5343.gd26e5c22.el9.x86_64 31/36 2026-02-24T15:49:28.823 INFO:teuthology.orchestra.run.trial084.stdout: Installing : ceph-base-2:20.3.0-5343.gd26e5c22.el9.x86_64 32/36 2026-02-24T15:49:28.861 INFO:teuthology.orchestra.run.trial084.stdout: Running scriptlet: ceph-base-2:20.3.0-5343.gd26e5c22.el9.x86_64 32/36 2026-02-24T15:49:28.861 INFO:teuthology.orchestra.run.trial084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2026-02-24T15:49:28.862 INFO:teuthology.orchestra.run.trial084.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2026-02-24T15:49:28.862 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T15:49:28.864 INFO:teuthology.orchestra.run.trial084.stdout: Installing : ceph-selinux-2:20.3.0-5343.gd26e5c22.el9.x86_64 33/36 2026-02-24T15:49:33.308 INFO:teuthology.orchestra.run.trial084.stdout: Running scriptlet: ceph-selinux-2:20.3.0-5343.gd26e5c22.el9.x86_64 33/36 2026-02-24T15:49:33.309 INFO:teuthology.orchestra.run.trial084.stdout:skipping the directory /sys 2026-02-24T15:49:33.309 INFO:teuthology.orchestra.run.trial084.stdout:skipping the directory /proc 2026-02-24T15:49:33.309 INFO:teuthology.orchestra.run.trial084.stdout:skipping the directory /mnt 2026-02-24T15:49:33.309 INFO:teuthology.orchestra.run.trial084.stdout:skipping the directory /var/tmp 2026-02-24T15:49:33.309 INFO:teuthology.orchestra.run.trial084.stdout:skipping the directory /home 2026-02-24T15:49:33.309 INFO:teuthology.orchestra.run.trial084.stdout:skipping the directory /root 2026-02-24T15:49:33.309 INFO:teuthology.orchestra.run.trial084.stdout:skipping the directory /tmp 2026-02-24T15:49:33.309 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T15:49:33.798 INFO:teuthology.orchestra.run.trial084.stdout: Installing : ceph-radosgw-2:20.3.0-5343.gd26e5c22.el9.x86_64 34/36 2026-02-24T15:49:33.821 INFO:teuthology.orchestra.run.trial084.stdout: Running scriptlet: ceph-radosgw-2:20.3.0-5343.gd26e5c22.el9.x86_64 34/36 2026-02-24T15:49:33.821 INFO:teuthology.orchestra.run.trial084.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T15:49:33.821 INFO:teuthology.orchestra.run.trial084.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2026-02-24T15:49:33.821 INFO:teuthology.orchestra.run.trial084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-24T15:49:33.821 INFO:teuthology.orchestra.run.trial084.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-24T15:49:33.821 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T15:49:33.823 INFO:teuthology.orchestra.run.trial084.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 35/36 2026-02-24T15:49:33.835 INFO:teuthology.orchestra.run.trial084.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 35/36 2026-02-24T15:49:33.835 INFO:teuthology.orchestra.run.trial084.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 36/36 2026-02-24T15:49:33.887 INFO:teuthology.orchestra.run.trial084.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 36/36 2026-02-24T15:49:33.887 INFO:teuthology.orchestra.run.trial084.stdout:Creating group 'qat' with GID 992. 2026-02-24T15:49:33.887 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T15:49:34.181 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : ceph-base-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/36 2026-02-24T15:49:34.182 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : ceph-common-2:20.3.0-5343.gd26e5c22.el9.x86_64 2/36 2026-02-24T15:49:34.182 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : ceph-radosgw-2:20.3.0-5343.gd26e5c22.el9.x86_64 3/36 2026-02-24T15:49:34.182 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : ceph-selinux-2:20.3.0-5343.gd26e5c22.el9.x86_64 4/36 2026-02-24T15:49:34.182 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : libcephfs2-2:20.3.0-5343.gd26e5c22.el9.x86_64 5/36 2026-02-24T15:49:34.182 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : libradosstriper1-2:20.3.0-5343.gd26e5c22.el9.x86_6 6/36 2026-02-24T15:49:34.182 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : librgw2-2:20.3.0-5343.gd26e5c22.el9.x86_64 7/36 2026-02-24T15:49:34.182 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : python3-ceph-argparse-2:20.3.0-5343.gd26e5c22.el9. 8/36 2026-02-24T15:49:34.182 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : python3-ceph-common-2:20.3.0-5343.gd26e5c22.el9.x8 9/36 2026-02-24T15:49:34.182 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : python3-cephfs-2:20.3.0-5343.gd26e5c22.el9.x86_64 10/36 2026-02-24T15:49:34.182 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : python3-rados-2:20.3.0-5343.gd26e5c22.el9.x86_64 11/36 2026-02-24T15:49:34.182 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : python3-rbd-2:20.3.0-5343.gd26e5c22.el9.x86_64 12/36 2026-02-24T15:49:34.182 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : python3-rgw-2:20.3.0-5343.gd26e5c22.el9.x86_64 13/36 2026-02-24T15:49:34.182 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : mailcap-2.1.49-5.el9.noarch 14/36 2026-02-24T15:49:34.183 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : pciutils-3.7.0-7.el9.x86_64 15/36 2026-02-24T15:49:34.183 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : boost-program-options-1.75.0-13.el9.x86_64 16/36 2026-02-24T15:49:34.183 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : libnbd-1.20.3-4.el9.x86_64 17/36 2026-02-24T15:49:34.183 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 18/36 2026-02-24T15:49:34.183 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 19/36 2026-02-24T15:49:34.183 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 20/36 2026-02-24T15:49:34.183 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 21/36 2026-02-24T15:49:34.183 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : qatlib-25.08.0-2.el9.x86_64 22/36 2026-02-24T15:49:34.183 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : qatlib-service-25.08.0-2.el9.x86_64 23/36 2026-02-24T15:49:34.183 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : qatzip-libs-1.3.1-1.el9.x86_64 24/36 2026-02-24T15:49:34.183 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 25/36 2026-02-24T15:49:34.183 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : libarrow-9.0.0-15.el9.x86_64 26/36 2026-02-24T15:49:34.183 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : libarrow-doc-9.0.0-15.el9.noarch 27/36 2026-02-24T15:49:34.183 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : liboath-2.6.12-1.el9.x86_64 28/36 2026-02-24T15:49:34.183 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 29/36 2026-02-24T15:49:34.184 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : parquet-libs-9.0.0-15.el9.x86_64 30/36 2026-02-24T15:49:34.184 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : re2-1:20211101-20.el9.x86_64 31/36 2026-02-24T15:49:34.184 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : thrift-0.15.0-4.el9.x86_64 32/36 2026-02-24T15:49:34.184 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : librados2-2:20.3.0-5343.gd26e5c22.el9.x86_64 33/36 2026-02-24T15:49:34.184 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 34/36 2026-02-24T15:49:34.184 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : librbd1-2:20.3.0-5343.gd26e5c22.el9.x86_64 35/36 2026-02-24T15:49:34.249 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 36/36 2026-02-24T15:49:34.249 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T15:49:34.249 INFO:teuthology.orchestra.run.trial084.stdout:Upgraded: 2026-02-24T15:49:34.249 INFO:teuthology.orchestra.run.trial084.stdout: librados2-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:49:34.250 INFO:teuthology.orchestra.run.trial084.stdout: librbd1-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:49:34.250 INFO:teuthology.orchestra.run.trial084.stdout:Installed: 2026-02-24T15:49:34.250 INFO:teuthology.orchestra.run.trial084.stdout: boost-program-options-1.75.0-13.el9.x86_64 2026-02-24T15:49:34.250 INFO:teuthology.orchestra.run.trial084.stdout: ceph-base-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:49:34.250 INFO:teuthology.orchestra.run.trial084.stdout: ceph-common-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:49:34.250 INFO:teuthology.orchestra.run.trial084.stdout: ceph-radosgw-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:49:34.250 INFO:teuthology.orchestra.run.trial084.stdout: ceph-selinux-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:49:34.250 INFO:teuthology.orchestra.run.trial084.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2026-02-24T15:49:34.250 INFO:teuthology.orchestra.run.trial084.stdout: libarrow-9.0.0-15.el9.x86_64 2026-02-24T15:49:34.250 INFO:teuthology.orchestra.run.trial084.stdout: libarrow-doc-9.0.0-15.el9.noarch 2026-02-24T15:49:34.250 INFO:teuthology.orchestra.run.trial084.stdout: libcephfs2-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:49:34.251 INFO:teuthology.orchestra.run.trial084.stdout: libnbd-1.20.3-4.el9.x86_64 2026-02-24T15:49:34.251 INFO:teuthology.orchestra.run.trial084.stdout: liboath-2.6.12-1.el9.x86_64 2026-02-24T15:49:34.251 INFO:teuthology.orchestra.run.trial084.stdout: libpmemobj-1.12.1-1.el9.x86_64 2026-02-24T15:49:34.251 INFO:teuthology.orchestra.run.trial084.stdout: librabbitmq-0.11.0-7.el9.x86_64 2026-02-24T15:49:34.251 INFO:teuthology.orchestra.run.trial084.stdout: libradosstriper1-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:49:34.251 INFO:teuthology.orchestra.run.trial084.stdout: librdkafka-1.6.1-102.el9.x86_64 2026-02-24T15:49:34.251 INFO:teuthology.orchestra.run.trial084.stdout: librgw2-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:49:34.251 INFO:teuthology.orchestra.run.trial084.stdout: libunwind-1.6.2-1.el9.x86_64 2026-02-24T15:49:34.251 INFO:teuthology.orchestra.run.trial084.stdout: lttng-ust-2.12.0-6.el9.x86_64 2026-02-24T15:49:34.251 INFO:teuthology.orchestra.run.trial084.stdout: mailcap-2.1.49-5.el9.noarch 2026-02-24T15:49:34.252 INFO:teuthology.orchestra.run.trial084.stdout: parquet-libs-9.0.0-15.el9.x86_64 2026-02-24T15:49:34.252 INFO:teuthology.orchestra.run.trial084.stdout: pciutils-3.7.0-7.el9.x86_64 2026-02-24T15:49:34.252 INFO:teuthology.orchestra.run.trial084.stdout: python3-ceph-argparse-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:49:34.252 INFO:teuthology.orchestra.run.trial084.stdout: python3-ceph-common-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:49:34.252 INFO:teuthology.orchestra.run.trial084.stdout: python3-cephfs-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:49:34.252 INFO:teuthology.orchestra.run.trial084.stdout: python3-rados-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:49:34.252 INFO:teuthology.orchestra.run.trial084.stdout: python3-rbd-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:49:34.252 INFO:teuthology.orchestra.run.trial084.stdout: python3-rgw-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:49:34.252 INFO:teuthology.orchestra.run.trial084.stdout: qatlib-25.08.0-2.el9.x86_64 2026-02-24T15:49:34.252 INFO:teuthology.orchestra.run.trial084.stdout: qatlib-service-25.08.0-2.el9.x86_64 2026-02-24T15:49:34.252 INFO:teuthology.orchestra.run.trial084.stdout: qatzip-libs-1.3.1-1.el9.x86_64 2026-02-24T15:49:34.253 INFO:teuthology.orchestra.run.trial084.stdout: re2-1:20211101-20.el9.x86_64 2026-02-24T15:49:34.253 INFO:teuthology.orchestra.run.trial084.stdout: thrift-0.15.0-4.el9.x86_64 2026-02-24T15:49:34.253 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T15:49:34.253 INFO:teuthology.orchestra.run.trial084.stdout:Complete! 2026-02-24T15:49:34.288 DEBUG:teuthology.orchestra.run.trial084:> sudo yum -y install ceph-test 2026-02-24T15:49:34.605 INFO:teuthology.orchestra.run.trial084.stdout:Last metadata expiration check: 0:01:11 ago on Tue 24 Feb 2026 03:48:23 PM UTC. 2026-02-24T15:49:34.688 INFO:teuthology.orchestra.run.trial084.stdout:Dependencies resolved. 2026-02-24T15:49:34.689 INFO:teuthology.orchestra.run.trial084.stdout:================================================================================ 2026-02-24T15:49:34.689 INFO:teuthology.orchestra.run.trial084.stdout: Package Arch Version Repository Size 2026-02-24T15:49:34.689 INFO:teuthology.orchestra.run.trial084.stdout:================================================================================ 2026-02-24T15:49:34.690 INFO:teuthology.orchestra.run.trial084.stdout:Installing: 2026-02-24T15:49:34.690 INFO:teuthology.orchestra.run.trial084.stdout: ceph-test x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 126 M 2026-02-24T15:49:34.690 INFO:teuthology.orchestra.run.trial084.stdout:Installing dependencies: 2026-02-24T15:49:34.690 INFO:teuthology.orchestra.run.trial084.stdout: hwloc-libs x86_64 2.4.1-6.el9 baseos 2.1 M 2026-02-24T15:49:34.690 INFO:teuthology.orchestra.run.trial084.stdout: libasan8 x86_64 15.2.1-7.el9 appstream 525 k 2026-02-24T15:49:34.690 INFO:teuthology.orchestra.run.trial084.stdout: libatomic x86_64 11.5.0-14.el9 baseos 23 k 2026-02-24T15:49:34.690 INFO:teuthology.orchestra.run.trial084.stdout: libcephsqlite x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 153 k 2026-02-24T15:49:34.690 INFO:teuthology.orchestra.run.trial084.stdout: libubsan x86_64 11.5.0-14.el9 appstream 176 k 2026-02-24T15:49:34.690 INFO:teuthology.orchestra.run.trial084.stdout: libxslt x86_64 1.1.34-12.el9 appstream 233 k 2026-02-24T15:49:34.690 INFO:teuthology.orchestra.run.trial084.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2026-02-24T15:49:34.690 INFO:teuthology.orchestra.run.trial084.stdout: protobuf x86_64 3.14.0-17.el9 appstream 1.0 M 2026-02-24T15:49:34.691 INFO:teuthology.orchestra.run.trial084.stdout: socat x86_64 1.7.4.1-8.el9 appstream 303 k 2026-02-24T15:49:34.691 INFO:teuthology.orchestra.run.trial084.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2026-02-24T15:49:34.691 INFO:teuthology.orchestra.run.trial084.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2026-02-24T15:49:34.691 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T15:49:34.691 INFO:teuthology.orchestra.run.trial084.stdout:Transaction Summary 2026-02-24T15:49:34.691 INFO:teuthology.orchestra.run.trial084.stdout:================================================================================ 2026-02-24T15:49:34.691 INFO:teuthology.orchestra.run.trial084.stdout:Install 12 Packages 2026-02-24T15:49:34.691 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T15:49:34.691 INFO:teuthology.orchestra.run.trial084.stdout:Total download size: 131 M 2026-02-24T15:49:34.691 INFO:teuthology.orchestra.run.trial084.stdout:Installed size: 981 M 2026-02-24T15:49:34.691 INFO:teuthology.orchestra.run.trial084.stdout:Downloading Packages: 2026-02-24T15:49:35.302 INFO:teuthology.orchestra.run.trial084.stdout:(1/12): hwloc-libs-2.4.1-6.el9.x86_64.rpm 20 MB/s | 2.1 MB 00:00 2026-02-24T15:49:35.316 INFO:teuthology.orchestra.run.trial084.stdout:(2/12): libatomic-11.5.0-14.el9.x86_64.rpm 1.5 MB/s | 23 kB 00:00 2026-02-24T15:49:35.329 INFO:teuthology.orchestra.run.trial084.stdout:(3/12): lksctp-tools-1.0.19-2.el9.x86_64.rpm 7.3 MB/s | 94 kB 00:00 2026-02-24T15:49:36.116 INFO:teuthology.orchestra.run.trial084.stdout:(4/12): libcephsqlite-20.3.0-5343.gd26e5c22.el9 166 kB/s | 153 kB 00:00 2026-02-24T15:49:36.430 INFO:teuthology.orchestra.run.trial084.stdout:(5/12): libasan8-15.2.1-7.el9.x86_64.rpm 476 kB/s | 525 kB 00:01 2026-02-24T15:49:36.841 INFO:teuthology.orchestra.run.trial084.stdout:(6/12): libubsan-11.5.0-14.el9.x86_64.rpm 243 kB/s | 176 kB 00:00 2026-02-24T15:49:37.194 INFO:teuthology.orchestra.run.trial084.stdout:(7/12): libxslt-1.1.34-12.el9.x86_64.rpm 305 kB/s | 233 kB 00:00 2026-02-24T15:49:37.773 INFO:teuthology.orchestra.run.trial084.stdout:(8/12): protobuf-3.14.0-17.el9.x86_64.rpm 1.1 MB/s | 1.0 MB 00:00 2026-02-24T15:49:37.982 INFO:teuthology.orchestra.run.trial084.stdout:(9/12): socat-1.7.4.1-8.el9.x86_64.rpm 385 kB/s | 303 kB 00:00 2026-02-24T15:49:38.015 INFO:teuthology.orchestra.run.trial084.stdout:(10/12): yaml-cpp-0.6.3-5.el9.x86_64.rpm 3.6 MB/s | 122 kB 00:00 2026-02-24T15:49:38.400 INFO:teuthology.orchestra.run.trial084.stdout:(11/12): xmlstarlet-1.6.1-20.el9.x86_64.rpm 102 kB/s | 64 kB 00:00 2026-02-24T15:56:41.683 INFO:teuthology.orchestra.run.trial084.stdout:(12/12): ceph-test-20.3.0-5343.gd26e5c22.el9.x8 303 kB/s | 126 MB 07:06 2026-02-24T15:56:41.684 INFO:teuthology.orchestra.run.trial084.stdout:-------------------------------------------------------------------------------- 2026-02-24T15:56:41.684 INFO:teuthology.orchestra.run.trial084.stdout:Total 314 kB/s | 131 MB 07:06 2026-02-24T15:56:41.750 INFO:teuthology.orchestra.run.trial084.stdout:Running transaction check 2026-02-24T15:56:41.761 INFO:teuthology.orchestra.run.trial084.stdout:Transaction check succeeded. 2026-02-24T15:56:41.761 INFO:teuthology.orchestra.run.trial084.stdout:Running transaction test 2026-02-24T15:56:42.192 INFO:teuthology.orchestra.run.trial084.stdout:Transaction test succeeded. 2026-02-24T15:56:42.192 INFO:teuthology.orchestra.run.trial084.stdout:Running transaction 2026-02-24T15:56:42.649 INFO:teuthology.orchestra.run.trial084.stdout: Preparing : 1/1 2026-02-24T15:56:42.658 INFO:teuthology.orchestra.run.trial084.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/12 2026-02-24T15:56:42.666 INFO:teuthology.orchestra.run.trial084.stdout: Installing : socat-1.7.4.1-8.el9.x86_64 2/12 2026-02-24T15:56:42.671 INFO:teuthology.orchestra.run.trial084.stdout: Installing : protobuf-3.14.0-17.el9.x86_64 3/12 2026-02-24T15:56:42.676 INFO:teuthology.orchestra.run.trial084.stdout: Installing : libxslt-1.1.34-12.el9.x86_64 4/12 2026-02-24T15:56:42.678 INFO:teuthology.orchestra.run.trial084.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 5/12 2026-02-24T15:56:42.683 INFO:teuthology.orchestra.run.trial084.stdout: Installing : libubsan-11.5.0-14.el9.x86_64 6/12 2026-02-24T15:56:42.683 INFO:teuthology.orchestra.run.trial084.stdout: Installing : libasan8-15.2.1-7.el9.x86_64 7/12 2026-02-24T15:56:42.699 INFO:teuthology.orchestra.run.trial084.stdout: Running scriptlet: libasan8-15.2.1-7.el9.x86_64 7/12 2026-02-24T15:56:42.701 INFO:teuthology.orchestra.run.trial084.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 8/12 2026-02-24T15:56:42.711 INFO:teuthology.orchestra.run.trial084.stdout: Installing : libatomic-11.5.0-14.el9.x86_64 9/12 2026-02-24T15:56:42.719 INFO:teuthology.orchestra.run.trial084.stdout: Installing : hwloc-libs-2.4.1-6.el9.x86_64 10/12 2026-02-24T15:56:42.720 INFO:teuthology.orchestra.run.trial084.stdout: Installing : libcephsqlite-2:20.3.0-5343.gd26e5c22.el9.x86_64 11/12 2026-02-24T15:56:49.619 INFO:teuthology.orchestra.run.trial084.stdout: Running scriptlet: libcephsqlite-2:20.3.0-5343.gd26e5c22.el9.x86_64 11/12 2026-02-24T15:56:49.638 INFO:teuthology.orchestra.run.trial084.stdout: Installing : ceph-test-2:20.3.0-5343.gd26e5c22.el9.x86_64 12/12 2026-02-24T15:56:50.384 INFO:teuthology.orchestra.run.trial084.stdout: Running scriptlet: ceph-test-2:20.3.0-5343.gd26e5c22.el9.x86_64 12/12 2026-02-24T15:56:50.384 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : ceph-test-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/12 2026-02-24T15:56:50.385 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : libcephsqlite-2:20.3.0-5343.gd26e5c22.el9.x86_64 2/12 2026-02-24T15:56:50.385 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : hwloc-libs-2.4.1-6.el9.x86_64 3/12 2026-02-24T15:56:50.385 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : libatomic-11.5.0-14.el9.x86_64 4/12 2026-02-24T15:56:50.385 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 5/12 2026-02-24T15:56:50.385 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : libasan8-15.2.1-7.el9.x86_64 6/12 2026-02-24T15:56:50.385 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : libubsan-11.5.0-14.el9.x86_64 7/12 2026-02-24T15:56:50.385 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : libxslt-1.1.34-12.el9.x86_64 8/12 2026-02-24T15:56:50.385 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : protobuf-3.14.0-17.el9.x86_64 9/12 2026-02-24T15:56:50.385 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : socat-1.7.4.1-8.el9.x86_64 10/12 2026-02-24T15:56:50.385 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 11/12 2026-02-24T15:56:50.447 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 12/12 2026-02-24T15:56:50.447 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T15:56:50.447 INFO:teuthology.orchestra.run.trial084.stdout:Installed: 2026-02-24T15:56:50.447 INFO:teuthology.orchestra.run.trial084.stdout: ceph-test-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:56:50.447 INFO:teuthology.orchestra.run.trial084.stdout: hwloc-libs-2.4.1-6.el9.x86_64 2026-02-24T15:56:50.447 INFO:teuthology.orchestra.run.trial084.stdout: libasan8-15.2.1-7.el9.x86_64 2026-02-24T15:56:50.447 INFO:teuthology.orchestra.run.trial084.stdout: libatomic-11.5.0-14.el9.x86_64 2026-02-24T15:56:50.448 INFO:teuthology.orchestra.run.trial084.stdout: libcephsqlite-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:56:50.448 INFO:teuthology.orchestra.run.trial084.stdout: libubsan-11.5.0-14.el9.x86_64 2026-02-24T15:56:50.448 INFO:teuthology.orchestra.run.trial084.stdout: libxslt-1.1.34-12.el9.x86_64 2026-02-24T15:56:50.448 INFO:teuthology.orchestra.run.trial084.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2026-02-24T15:56:50.448 INFO:teuthology.orchestra.run.trial084.stdout: protobuf-3.14.0-17.el9.x86_64 2026-02-24T15:56:50.448 INFO:teuthology.orchestra.run.trial084.stdout: socat-1.7.4.1-8.el9.x86_64 2026-02-24T15:56:50.448 INFO:teuthology.orchestra.run.trial084.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2026-02-24T15:56:50.448 INFO:teuthology.orchestra.run.trial084.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2026-02-24T15:56:50.448 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T15:56:50.448 INFO:teuthology.orchestra.run.trial084.stdout:Complete! 2026-02-24T15:56:50.489 DEBUG:teuthology.orchestra.run.trial084:> sudo yum -y install ceph 2026-02-24T15:56:50.806 INFO:teuthology.orchestra.run.trial084.stdout:Last metadata expiration check: 0:08:27 ago on Tue 24 Feb 2026 03:48:23 PM UTC. 2026-02-24T15:56:50.890 INFO:teuthology.orchestra.run.trial084.stdout:Dependencies resolved. 2026-02-24T15:56:50.891 INFO:teuthology.orchestra.run.trial084.stdout:================================================================================ 2026-02-24T15:56:50.891 INFO:teuthology.orchestra.run.trial084.stdout: Package Arch Version Repository Size 2026-02-24T15:56:50.891 INFO:teuthology.orchestra.run.trial084.stdout:================================================================================ 2026-02-24T15:56:50.892 INFO:teuthology.orchestra.run.trial084.stdout:Installing: 2026-02-24T15:56:50.892 INFO:teuthology.orchestra.run.trial084.stdout: ceph x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 6.5 k 2026-02-24T15:56:50.892 INFO:teuthology.orchestra.run.trial084.stdout:Installing dependencies: 2026-02-24T15:56:50.892 INFO:teuthology.orchestra.run.trial084.stdout: ceph-mds x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 2.0 M 2026-02-24T15:56:50.892 INFO:teuthology.orchestra.run.trial084.stdout: ceph-mgr x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 937 k 2026-02-24T15:56:50.892 INFO:teuthology.orchestra.run.trial084.stdout: ceph-mgr-modules-core noarch 2:20.3.0-5343.gd26e5c22.el9 ceph-noarch 297 k 2026-02-24T15:56:50.892 INFO:teuthology.orchestra.run.trial084.stdout: ceph-mon x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 4.9 M 2026-02-24T15:56:50.892 INFO:teuthology.orchestra.run.trial084.stdout: ceph-osd x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 163 k 2026-02-24T15:56:50.892 INFO:teuthology.orchestra.run.trial084.stdout: ceph-osd-classic x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 15 M 2026-02-24T15:56:50.892 INFO:teuthology.orchestra.run.trial084.stdout: ceph-osd-crimson x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 88 M 2026-02-24T15:56:50.892 INFO:teuthology.orchestra.run.trial084.stdout: ledmon-libs x86_64 1.1.0-3.el9 baseos 40 k 2026-02-24T15:56:50.893 INFO:teuthology.orchestra.run.trial084.stdout: libconfig x86_64 1.7.2-9.el9 baseos 72 k 2026-02-24T15:56:50.893 INFO:teuthology.orchestra.run.trial084.stdout: libstoragemgmt x86_64 1.10.1-1.el9 appstream 246 k 2026-02-24T15:56:50.893 INFO:teuthology.orchestra.run.trial084.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2026-02-24T15:56:50.893 INFO:teuthology.orchestra.run.trial084.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2026-02-24T15:56:50.893 INFO:teuthology.orchestra.run.trial084.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2026-02-24T15:56:50.893 INFO:teuthology.orchestra.run.trial084.stdout: python3-autocommand noarch 2.2.2-8.el9 epel 29 k 2026-02-24T15:56:50.893 INFO:teuthology.orchestra.run.trial084.stdout: python3-backports-tarfile noarch 1.2.0-1.el9 epel 60 k 2026-02-24T15:56:50.893 INFO:teuthology.orchestra.run.trial084.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2026-02-24T15:56:50.893 INFO:teuthology.orchestra.run.trial084.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2026-02-24T15:56:50.893 INFO:teuthology.orchestra.run.trial084.stdout: python3-cheroot noarch 10.0.1-4.el9 epel 173 k 2026-02-24T15:56:50.893 INFO:teuthology.orchestra.run.trial084.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2026-02-24T15:56:50.893 INFO:teuthology.orchestra.run.trial084.stdout: python3-cryptography x86_64 36.0.1-5.el9 baseos 1.2 M 2026-02-24T15:56:50.894 INFO:teuthology.orchestra.run.trial084.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2026-02-24T15:56:50.894 INFO:teuthology.orchestra.run.trial084.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2026-02-24T15:56:50.894 INFO:teuthology.orchestra.run.trial084.stdout: python3-jaraco-collections 2026-02-24T15:56:50.894 INFO:teuthology.orchestra.run.trial084.stdout: noarch 3.0.0-8.el9 epel 23 k 2026-02-24T15:56:50.894 INFO:teuthology.orchestra.run.trial084.stdout: python3-jaraco-context noarch 6.0.1-3.el9 epel 20 k 2026-02-24T15:56:50.894 INFO:teuthology.orchestra.run.trial084.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2026-02-24T15:56:50.894 INFO:teuthology.orchestra.run.trial084.stdout: python3-jaraco-text noarch 4.0.0-2.el9 epel 26 k 2026-02-24T15:56:50.894 INFO:teuthology.orchestra.run.trial084.stdout: python3-libstoragemgmt x86_64 1.10.1-1.el9 appstream 177 k 2026-02-24T15:56:50.894 INFO:teuthology.orchestra.run.trial084.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2026-02-24T15:56:50.894 INFO:teuthology.orchestra.run.trial084.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2026-02-24T15:56:50.894 INFO:teuthology.orchestra.run.trial084.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2026-02-24T15:56:50.894 INFO:teuthology.orchestra.run.trial084.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2026-02-24T15:56:50.895 INFO:teuthology.orchestra.run.trial084.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2026-02-24T15:56:50.895 INFO:teuthology.orchestra.run.trial084.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2026-02-24T15:56:50.895 INFO:teuthology.orchestra.run.trial084.stdout: python3-pyparsing noarch 2.4.7-9.el9 baseos 150 k 2026-02-24T15:56:50.895 INFO:teuthology.orchestra.run.trial084.stdout: python3-requests noarch 2.25.1-10.el9 baseos 126 k 2026-02-24T15:56:50.895 INFO:teuthology.orchestra.run.trial084.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2026-02-24T15:56:50.895 INFO:teuthology.orchestra.run.trial084.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2026-02-24T15:56:50.895 INFO:teuthology.orchestra.run.trial084.stdout: python3-urllib3 noarch 1.26.5-7.el9 baseos 218 k 2026-02-24T15:56:50.895 INFO:teuthology.orchestra.run.trial084.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2026-02-24T15:56:50.895 INFO:teuthology.orchestra.run.trial084.stdout: unzip x86_64 6.0-59.el9 baseos 182 k 2026-02-24T15:56:50.895 INFO:teuthology.orchestra.run.trial084.stdout: zip x86_64 3.0-35.el9 baseos 266 k 2026-02-24T15:56:50.895 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T15:56:50.895 INFO:teuthology.orchestra.run.trial084.stdout:Transaction Summary 2026-02-24T15:56:50.896 INFO:teuthology.orchestra.run.trial084.stdout:================================================================================ 2026-02-24T15:56:50.896 INFO:teuthology.orchestra.run.trial084.stdout:Install 42 Packages 2026-02-24T15:56:50.896 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T15:56:50.896 INFO:teuthology.orchestra.run.trial084.stdout:Total download size: 115 M 2026-02-24T15:56:50.897 INFO:teuthology.orchestra.run.trial084.stdout:Installed size: 974 M 2026-02-24T15:56:50.897 INFO:teuthology.orchestra.run.trial084.stdout:Downloading Packages: 2026-02-24T15:56:52.265 INFO:teuthology.orchestra.run.trial084.stdout:(1/42): ceph-20.3.0-5343.gd26e5c22.el9.x86_64.r 7.5 kB/s | 6.5 kB 00:00 2026-02-24T15:56:54.976 INFO:teuthology.orchestra.run.trial084.stdout:(2/42): ceph-mgr-20.3.0-5343.gd26e5c22.el9.x86_ 262 kB/s | 937 kB 00:03 2026-02-24T15:56:55.577 INFO:teuthology.orchestra.run.trial084.stdout:(3/42): ceph-osd-20.3.0-5343.gd26e5c22.el9.x86_ 272 kB/s | 163 kB 00:00 2026-02-24T15:56:57.593 INFO:teuthology.orchestra.run.trial084.stdout:(4/42): ceph-mds-20.3.0-5343.gd26e5c22.el9.x86_ 334 kB/s | 2.0 MB 00:06 2026-02-24T15:57:12.872 INFO:teuthology.orchestra.run.trial084.stdout:(5/42): ceph-mon-20.3.0-5343.gd26e5c22.el9.x86_ 242 kB/s | 4.9 MB 00:20 2026-02-24T15:57:14.057 INFO:teuthology.orchestra.run.trial084.stdout:(6/42): ceph-mgr-modules-core-20.3.0-5343.gd26e 250 kB/s | 297 kB 00:01 2026-02-24T15:57:14.466 INFO:teuthology.orchestra.run.trial084.stdout:(7/42): ledmon-libs-1.1.0-3.el9.x86_64.rpm 99 kB/s | 40 kB 00:00 2026-02-24T15:57:14.627 INFO:teuthology.orchestra.run.trial084.stdout:(8/42): libconfig-1.7.2-9.el9.x86_64.rpm 445 kB/s | 72 kB 00:00 2026-02-24T15:57:14.948 INFO:teuthology.orchestra.run.trial084.stdout:(9/42): python3-cffi-1.14.5-5.el9.x86_64.rpm 791 kB/s | 253 kB 00:00 2026-02-24T15:57:16.444 INFO:teuthology.orchestra.run.trial084.stdout:(10/42): python3-cryptography-36.0.1-5.el9.x86_ 853 kB/s | 1.2 MB 00:01 2026-02-24T15:57:16.606 INFO:teuthology.orchestra.run.trial084.stdout:(11/42): python3-ply-3.11-14.el9.noarch.rpm 657 kB/s | 106 kB 00:00 2026-02-24T15:57:16.768 INFO:teuthology.orchestra.run.trial084.stdout:(12/42): python3-pycparser-2.20-6.el9.noarch.rp 834 kB/s | 135 kB 00:00 2026-02-24T15:57:17.008 INFO:teuthology.orchestra.run.trial084.stdout:(13/42): python3-pyparsing-2.4.7-9.el9.noarch.r 627 kB/s | 150 kB 00:00 2026-02-24T15:57:17.170 INFO:teuthology.orchestra.run.trial084.stdout:(14/42): python3-requests-2.25.1-10.el9.noarch. 780 kB/s | 126 kB 00:00 2026-02-24T15:57:17.490 INFO:teuthology.orchestra.run.trial084.stdout:(15/42): python3-urllib3-1.26.5-7.el9.noarch.rp 681 kB/s | 218 kB 00:00 2026-02-24T15:57:17.730 INFO:teuthology.orchestra.run.trial084.stdout:(16/42): unzip-6.0-59.el9.x86_64.rpm 757 kB/s | 182 kB 00:00 2026-02-24T15:57:18.048 INFO:teuthology.orchestra.run.trial084.stdout:(17/42): zip-3.0-35.el9.x86_64.rpm 834 kB/s | 266 kB 00:00 2026-02-24T15:57:18.367 INFO:teuthology.orchestra.run.trial084.stdout:(18/42): libstoragemgmt-1.10.1-1.el9.x86_64.rpm 774 kB/s | 246 kB 00:00 2026-02-24T15:57:18.608 INFO:teuthology.orchestra.run.trial084.stdout:(19/42): lua-5.4.4-4.el9.x86_64.rpm 783 kB/s | 188 kB 00:00 2026-02-24T15:57:18.849 INFO:teuthology.orchestra.run.trial084.stdout:(20/42): python3-libstoragemgmt-1.10.1-1.el9.x8 732 kB/s | 177 kB 00:00 2026-02-24T15:57:19.011 INFO:teuthology.orchestra.run.trial084.stdout:(21/42): python3-packaging-20.9-5.el9.noarch.rp 477 kB/s | 77 kB 00:00 2026-02-24T15:57:19.095 INFO:teuthology.orchestra.run.trial084.stdout:(22/42): python3-toml-0.10.2-6.el9.noarch.rpm 498 kB/s | 42 kB 00:00 2026-02-24T15:57:19.121 INFO:teuthology.orchestra.run.trial084.stdout:(23/42): lua-devel-5.4.4-4.el9.x86_64.rpm 854 kB/s | 22 kB 00:00 2026-02-24T15:57:19.157 INFO:teuthology.orchestra.run.trial084.stdout:(24/42): luarocks-3.9.2-5.el9.noarch.rpm 4.1 MB/s | 151 kB 00:00 2026-02-24T15:57:19.161 INFO:teuthology.orchestra.run.trial084.stdout:(25/42): python3-autocommand-2.2.2-8.el9.noarch 9.1 MB/s | 29 kB 00:00 2026-02-24T15:57:19.164 INFO:teuthology.orchestra.run.trial084.stdout:(26/42): python3-backports-tarfile-1.2.0-1.el9. 18 MB/s | 60 kB 00:00 2026-02-24T15:57:19.167 INFO:teuthology.orchestra.run.trial084.stdout:(27/42): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 13 MB/s | 43 kB 00:00 2026-02-24T15:57:19.173 INFO:teuthology.orchestra.run.trial084.stdout:(28/42): python3-cheroot-10.0.1-4.el9.noarch.rp 33 MB/s | 173 kB 00:00 2026-02-24T15:57:19.180 INFO:teuthology.orchestra.run.trial084.stdout:(29/42): python3-cherrypy-18.6.1-2.el9.noarch.r 48 MB/s | 358 kB 00:00 2026-02-24T15:57:19.185 INFO:teuthology.orchestra.run.trial084.stdout:(30/42): python3-jaraco-8.2.1-3.el9.noarch.rpm 2.0 MB/s | 11 kB 00:00 2026-02-24T15:57:19.190 INFO:teuthology.orchestra.run.trial084.stdout:(31/42): python3-jaraco-classes-3.2.1-5.el9.noa 3.7 MB/s | 18 kB 00:00 2026-02-24T15:57:19.193 INFO:teuthology.orchestra.run.trial084.stdout:(32/42): python3-jaraco-collections-3.0.0-8.el9 7.3 MB/s | 23 kB 00:00 2026-02-24T15:57:19.196 INFO:teuthology.orchestra.run.trial084.stdout:(33/42): python3-jaraco-context-6.0.1-3.el9.noa 6.1 MB/s | 20 kB 00:00 2026-02-24T15:57:19.200 INFO:teuthology.orchestra.run.trial084.stdout:(34/42): python3-jaraco-functools-3.5.0-2.el9.n 5.0 MB/s | 19 kB 00:00 2026-02-24T15:57:19.204 INFO:teuthology.orchestra.run.trial084.stdout:(35/42): python3-jaraco-text-4.0.0-2.el9.noarch 8.4 MB/s | 26 kB 00:00 2026-02-24T15:57:19.207 INFO:teuthology.orchestra.run.trial084.stdout:(36/42): python3-more-itertools-8.12.0-2.el9.no 23 MB/s | 79 kB 00:00 2026-02-24T15:57:19.210 INFO:teuthology.orchestra.run.trial084.stdout:(37/42): python3-portend-3.1.0-2.el9.noarch.rpm 4.9 MB/s | 16 kB 00:00 2026-02-24T15:57:19.214 INFO:teuthology.orchestra.run.trial084.stdout:(38/42): python3-pyOpenSSL-21.0.0-1.el9.noarch. 25 MB/s | 90 kB 00:00 2026-02-24T15:57:19.218 INFO:teuthology.orchestra.run.trial084.stdout:(39/42): python3-tempora-5.0.0-2.el9.noarch.rpm 8.9 MB/s | 36 kB 00:00 2026-02-24T15:57:19.222 INFO:teuthology.orchestra.run.trial084.stdout:(40/42): python3-zc-lockfile-2.0-10.el9.noarch. 5.4 MB/s | 20 kB 00:00 2026-02-24T15:57:39.989 INFO:teuthology.orchestra.run.trial084.stdout:(41/42): ceph-osd-classic-20.3.0-5343.gd26e5c22 340 kB/s | 15 MB 00:44 2026-02-24T16:00:06.458 INFO:teuthology.orchestra.run.trial084.stdout:(42/42): ceph-osd-crimson-20.3.0-5343.gd26e5c22 476 kB/s | 88 MB 03:08 2026-02-24T16:00:06.461 INFO:teuthology.orchestra.run.trial084.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:00:06.461 INFO:teuthology.orchestra.run.trial084.stdout:Total 604 kB/s | 115 MB 03:15 2026-02-24T16:00:06.583 INFO:teuthology.orchestra.run.trial084.stdout:Running transaction check 2026-02-24T16:00:06.602 INFO:teuthology.orchestra.run.trial084.stdout:Transaction check succeeded. 2026-02-24T16:00:06.603 INFO:teuthology.orchestra.run.trial084.stdout:Running transaction test 2026-02-24T16:00:07.000 INFO:teuthology.orchestra.run.trial084.stdout:Transaction test succeeded. 2026-02-24T16:00:07.001 INFO:teuthology.orchestra.run.trial084.stdout:Running transaction 2026-02-24T16:00:07.430 INFO:teuthology.orchestra.run.trial084.stdout: Preparing : 1/1 2026-02-24T16:00:07.432 INFO:teuthology.orchestra.run.trial084.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/42 2026-02-24T16:00:07.437 INFO:teuthology.orchestra.run.trial084.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/42 2026-02-24T16:00:07.442 INFO:teuthology.orchestra.run.trial084.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/42 2026-02-24T16:00:07.446 INFO:teuthology.orchestra.run.trial084.stdout: Installing : unzip-6.0-59.el9.x86_64 4/42 2026-02-24T16:00:07.452 INFO:teuthology.orchestra.run.trial084.stdout: Installing : ledmon-libs-1.1.0-3.el9.x86_64 5/42 2026-02-24T16:00:07.475 INFO:teuthology.orchestra.run.trial084.stdout: Installing : zip-3.0-35.el9.x86_64 6/42 2026-02-24T16:00:07.480 INFO:teuthology.orchestra.run.trial084.stdout: Installing : luarocks-3.9.2-5.el9.noarch 7/42 2026-02-24T16:00:07.485 INFO:teuthology.orchestra.run.trial084.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 8/42 2026-02-24T16:00:07.489 INFO:teuthology.orchestra.run.trial084.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 9/42 2026-02-24T16:00:07.495 INFO:teuthology.orchestra.run.trial084.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 10/42 2026-02-24T16:00:07.499 INFO:teuthology.orchestra.run.trial084.stdout: Installing : python3-backports-tarfile-1.2.0-1.el9.noarch 11/42 2026-02-24T16:00:07.505 INFO:teuthology.orchestra.run.trial084.stdout: Installing : python3-jaraco-context-6.0.1-3.el9.noarch 12/42 2026-02-24T16:00:07.510 INFO:teuthology.orchestra.run.trial084.stdout: Installing : python3-autocommand-2.2.2-8.el9.noarch 13/42 2026-02-24T16:00:07.514 INFO:teuthology.orchestra.run.trial084.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 14/42 2026-02-24T16:00:07.519 INFO:teuthology.orchestra.run.trial084.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 15/42 2026-02-24T16:00:07.523 INFO:teuthology.orchestra.run.trial084.stdout: Installing : python3-jaraco-text-4.0.0-2.el9.noarch 16/42 2026-02-24T16:00:07.530 INFO:teuthology.orchestra.run.trial084.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 17/42 2026-02-24T16:00:07.533 INFO:teuthology.orchestra.run.trial084.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 18/42 2026-02-24T16:00:07.555 INFO:teuthology.orchestra.run.trial084.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 19/42 2026-02-24T16:00:07.568 INFO:teuthology.orchestra.run.trial084.stdout: Installing : python3-urllib3-1.26.5-7.el9.noarch 20/42 2026-02-24T16:00:07.573 INFO:teuthology.orchestra.run.trial084.stdout: Installing : python3-requests-2.25.1-10.el9.noarch 21/42 2026-02-24T16:00:07.583 INFO:teuthology.orchestra.run.trial084.stdout: Installing : python3-pyparsing-2.4.7-9.el9.noarch 22/42 2026-02-24T16:00:07.589 INFO:teuthology.orchestra.run.trial084.stdout: Installing : python3-packaging-20.9-5.el9.noarch 23/42 2026-02-24T16:00:07.602 INFO:teuthology.orchestra.run.trial084.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/42 2026-02-24T16:00:07.618 INFO:teuthology.orchestra.run.trial084.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/42 2026-02-24T16:00:07.692 INFO:teuthology.orchestra.run.trial084.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/42 2026-02-24T16:00:07.701 INFO:teuthology.orchestra.run.trial084.stdout: Installing : python3-cryptography-36.0.1-5.el9.x86_64 27/42 2026-02-24T16:00:07.724 INFO:teuthology.orchestra.run.trial084.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/42 2026-02-24T16:00:07.753 INFO:teuthology.orchestra.run.trial084.stdout: Installing : python3-cheroot-10.0.1-4.el9.noarch 29/42 2026-02-24T16:00:07.758 INFO:teuthology.orchestra.run.trial084.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/42 2026-02-24T16:00:07.810 INFO:teuthology.orchestra.run.trial084.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/42 2026-02-24T16:00:07.850 INFO:teuthology.orchestra.run.trial084.stdout: Installing : ceph-mgr-modules-core-2:20.3.0-5343.gd26e5c22.el9. 32/42 2026-02-24T16:00:07.851 INFO:teuthology.orchestra.run.trial084.stdout: Installing : ceph-mgr-2:20.3.0-5343.gd26e5c22.el9.x86_64 33/42 2026-02-24T16:00:07.875 INFO:teuthology.orchestra.run.trial084.stdout: Running scriptlet: ceph-mgr-2:20.3.0-5343.gd26e5c22.el9.x86_64 33/42 2026-02-24T16:00:07.875 INFO:teuthology.orchestra.run.trial084.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T16:00:07.875 INFO:teuthology.orchestra.run.trial084.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2026-02-24T16:00:07.876 INFO:teuthology.orchestra.run.trial084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-24T16:00:07.876 INFO:teuthology.orchestra.run.trial084.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-24T16:00:07.876 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:00:07.880 INFO:teuthology.orchestra.run.trial084.stdout: Installing : libconfig-1.7.2-9.el9.x86_64 34/42 2026-02-24T16:00:07.896 INFO:teuthology.orchestra.run.trial084.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 35/42 2026-02-24T16:00:07.896 INFO:teuthology.orchestra.run.trial084.stdout:Creating group 'libstoragemgmt' with GID 991. 2026-02-24T16:00:07.896 INFO:teuthology.orchestra.run.trial084.stdout:Creating user 'libstoragemgmt' (daemon account for libstoragemgmt) with UID 991 and GID 991. 2026-02-24T16:00:07.897 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:00:07.906 INFO:teuthology.orchestra.run.trial084.stdout: Installing : libstoragemgmt-1.10.1-1.el9.x86_64 35/42 2026-02-24T16:00:07.929 INFO:teuthology.orchestra.run.trial084.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 35/42 2026-02-24T16:00:07.929 INFO:teuthology.orchestra.run.trial084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/libstoragemgmt.service → /usr/lib/systemd/system/libstoragemgmt.service. 2026-02-24T16:00:07.929 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:00:08.553 INFO:teuthology.orchestra.run.trial084.stdout: Installing : python3-libstoragemgmt-1.10.1-1.el9.x86_64 36/42 2026-02-24T16:00:14.082 INFO:teuthology.orchestra.run.trial084.stdout: Installing : ceph-osd-classic-2:20.3.0-5343.gd26e5c22.el9.x86_6 37/42 2026-02-24T16:00:14.092 INFO:teuthology.orchestra.run.trial084.stdout: Installing : ceph-osd-crimson-2:20.3.0-5343.gd26e5c22.el9.x86_6 38/42 2026-02-24T16:00:14.093 INFO:teuthology.orchestra.run.trial084.stdout: Installing : ceph-osd-2:20.3.0-5343.gd26e5c22.el9.x86_64 39/42 2026-02-24T16:00:14.117 INFO:teuthology.orchestra.run.trial084.stdout: Running scriptlet: ceph-osd-2:20.3.0-5343.gd26e5c22.el9.x86_64 39/42 2026-02-24T16:00:14.118 INFO:teuthology.orchestra.run.trial084.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T16:00:14.118 INFO:teuthology.orchestra.run.trial084.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2026-02-24T16:00:14.118 INFO:teuthology.orchestra.run.trial084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-24T16:00:14.118 INFO:teuthology.orchestra.run.trial084.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-24T16:00:14.118 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:00:14.337 INFO:teuthology.orchestra.run.trial084.stdout: Installing : ceph-mon-2:20.3.0-5343.gd26e5c22.el9.x86_64 40/42 2026-02-24T16:00:14.357 INFO:teuthology.orchestra.run.trial084.stdout: Running scriptlet: ceph-mon-2:20.3.0-5343.gd26e5c22.el9.x86_64 40/42 2026-02-24T16:00:14.358 INFO:teuthology.orchestra.run.trial084.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T16:00:14.358 INFO:teuthology.orchestra.run.trial084.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2026-02-24T16:00:14.358 INFO:teuthology.orchestra.run.trial084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-24T16:00:14.358 INFO:teuthology.orchestra.run.trial084.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-24T16:00:14.358 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:00:14.443 INFO:teuthology.orchestra.run.trial084.stdout: Installing : ceph-mds-2:20.3.0-5343.gd26e5c22.el9.x86_64 41/42 2026-02-24T16:00:14.464 INFO:teuthology.orchestra.run.trial084.stdout: Running scriptlet: ceph-mds-2:20.3.0-5343.gd26e5c22.el9.x86_64 41/42 2026-02-24T16:00:14.464 INFO:teuthology.orchestra.run.trial084.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T16:00:14.464 INFO:teuthology.orchestra.run.trial084.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2026-02-24T16:00:14.464 INFO:teuthology.orchestra.run.trial084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-24T16:00:14.464 INFO:teuthology.orchestra.run.trial084.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-24T16:00:14.464 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:00:14.465 INFO:teuthology.orchestra.run.trial084.stdout: Installing : ceph-2:20.3.0-5343.gd26e5c22.el9.x86_64 42/42 2026-02-24T16:00:14.471 INFO:teuthology.orchestra.run.trial084.stdout: Running scriptlet: ceph-osd-classic-2:20.3.0-5343.gd26e5c22.el9.x86_6 42/42 2026-02-24T16:00:14.494 INFO:teuthology.orchestra.run.trial084.stdout: Running scriptlet: ceph-osd-crimson-2:20.3.0-5343.gd26e5c22.el9.x86_6 42/42 2026-02-24T16:00:15.461 INFO:teuthology.orchestra.run.trial084.stdout: Running scriptlet: ceph-2:20.3.0-5343.gd26e5c22.el9.x86_64 42/42 2026-02-24T16:00:15.461 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : ceph-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/42 2026-02-24T16:00:15.461 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : ceph-mds-2:20.3.0-5343.gd26e5c22.el9.x86_64 2/42 2026-02-24T16:00:15.461 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : ceph-mgr-2:20.3.0-5343.gd26e5c22.el9.x86_64 3/42 2026-02-24T16:00:15.461 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : ceph-mon-2:20.3.0-5343.gd26e5c22.el9.x86_64 4/42 2026-02-24T16:00:15.462 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : ceph-osd-2:20.3.0-5343.gd26e5c22.el9.x86_64 5/42 2026-02-24T16:00:15.462 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : ceph-osd-classic-2:20.3.0-5343.gd26e5c22.el9.x86_6 6/42 2026-02-24T16:00:15.462 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : ceph-osd-crimson-2:20.3.0-5343.gd26e5c22.el9.x86_6 7/42 2026-02-24T16:00:15.462 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : ceph-mgr-modules-core-2:20.3.0-5343.gd26e5c22.el9. 8/42 2026-02-24T16:00:15.462 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : ledmon-libs-1.1.0-3.el9.x86_64 9/42 2026-02-24T16:00:15.462 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : libconfig-1.7.2-9.el9.x86_64 10/42 2026-02-24T16:00:15.462 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/42 2026-02-24T16:00:15.462 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : python3-cryptography-36.0.1-5.el9.x86_64 12/42 2026-02-24T16:00:15.462 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/42 2026-02-24T16:00:15.462 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/42 2026-02-24T16:00:15.462 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : python3-pyparsing-2.4.7-9.el9.noarch 15/42 2026-02-24T16:00:15.462 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : python3-requests-2.25.1-10.el9.noarch 16/42 2026-02-24T16:00:15.463 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : python3-urllib3-1.26.5-7.el9.noarch 17/42 2026-02-24T16:00:15.463 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : unzip-6.0-59.el9.x86_64 18/42 2026-02-24T16:00:15.463 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : zip-3.0-35.el9.x86_64 19/42 2026-02-24T16:00:15.463 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : libstoragemgmt-1.10.1-1.el9.x86_64 20/42 2026-02-24T16:00:15.463 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : lua-5.4.4-4.el9.x86_64 21/42 2026-02-24T16:00:15.463 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : python3-libstoragemgmt-1.10.1-1.el9.x86_64 22/42 2026-02-24T16:00:15.463 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 23/42 2026-02-24T16:00:15.463 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 24/42 2026-02-24T16:00:15.463 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 25/42 2026-02-24T16:00:15.463 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 26/42 2026-02-24T16:00:15.463 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : python3-autocommand-2.2.2-8.el9.noarch 27/42 2026-02-24T16:00:15.463 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : python3-backports-tarfile-1.2.0-1.el9.noarch 28/42 2026-02-24T16:00:15.464 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 29/42 2026-02-24T16:00:15.464 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : python3-cheroot-10.0.1-4.el9.noarch 30/42 2026-02-24T16:00:15.464 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 31/42 2026-02-24T16:00:15.464 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 32/42 2026-02-24T16:00:15.464 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 33/42 2026-02-24T16:00:15.464 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 34/42 2026-02-24T16:00:15.464 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : python3-jaraco-context-6.0.1-3.el9.noarch 35/42 2026-02-24T16:00:15.464 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 36/42 2026-02-24T16:00:15.464 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : python3-jaraco-text-4.0.0-2.el9.noarch 37/42 2026-02-24T16:00:15.464 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 38/42 2026-02-24T16:00:15.464 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 39/42 2026-02-24T16:00:15.464 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 40/42 2026-02-24T16:00:15.464 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 41/42 2026-02-24T16:00:15.520 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 42/42 2026-02-24T16:00:15.521 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:00:15.521 INFO:teuthology.orchestra.run.trial084.stdout:Installed: 2026-02-24T16:00:15.521 INFO:teuthology.orchestra.run.trial084.stdout: ceph-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T16:00:15.521 INFO:teuthology.orchestra.run.trial084.stdout: ceph-mds-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T16:00:15.521 INFO:teuthology.orchestra.run.trial084.stdout: ceph-mgr-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T16:00:15.521 INFO:teuthology.orchestra.run.trial084.stdout: ceph-mgr-modules-core-2:20.3.0-5343.gd26e5c22.el9.noarch 2026-02-24T16:00:15.521 INFO:teuthology.orchestra.run.trial084.stdout: ceph-mon-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T16:00:15.521 INFO:teuthology.orchestra.run.trial084.stdout: ceph-osd-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T16:00:15.521 INFO:teuthology.orchestra.run.trial084.stdout: ceph-osd-classic-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T16:00:15.521 INFO:teuthology.orchestra.run.trial084.stdout: ceph-osd-crimson-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T16:00:15.522 INFO:teuthology.orchestra.run.trial084.stdout: ledmon-libs-1.1.0-3.el9.x86_64 2026-02-24T16:00:15.522 INFO:teuthology.orchestra.run.trial084.stdout: libconfig-1.7.2-9.el9.x86_64 2026-02-24T16:00:15.522 INFO:teuthology.orchestra.run.trial084.stdout: libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-24T16:00:15.522 INFO:teuthology.orchestra.run.trial084.stdout: lua-5.4.4-4.el9.x86_64 2026-02-24T16:00:15.522 INFO:teuthology.orchestra.run.trial084.stdout: lua-devel-5.4.4-4.el9.x86_64 2026-02-24T16:00:15.522 INFO:teuthology.orchestra.run.trial084.stdout: luarocks-3.9.2-5.el9.noarch 2026-02-24T16:00:15.522 INFO:teuthology.orchestra.run.trial084.stdout: python3-autocommand-2.2.2-8.el9.noarch 2026-02-24T16:00:15.522 INFO:teuthology.orchestra.run.trial084.stdout: python3-backports-tarfile-1.2.0-1.el9.noarch 2026-02-24T16:00:15.522 INFO:teuthology.orchestra.run.trial084.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2026-02-24T16:00:15.522 INFO:teuthology.orchestra.run.trial084.stdout: python3-cffi-1.14.5-5.el9.x86_64 2026-02-24T16:00:15.522 INFO:teuthology.orchestra.run.trial084.stdout: python3-cheroot-10.0.1-4.el9.noarch 2026-02-24T16:00:15.522 INFO:teuthology.orchestra.run.trial084.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2026-02-24T16:00:15.523 INFO:teuthology.orchestra.run.trial084.stdout: python3-cryptography-36.0.1-5.el9.x86_64 2026-02-24T16:00:15.523 INFO:teuthology.orchestra.run.trial084.stdout: python3-jaraco-8.2.1-3.el9.noarch 2026-02-24T16:00:15.523 INFO:teuthology.orchestra.run.trial084.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2026-02-24T16:00:15.523 INFO:teuthology.orchestra.run.trial084.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2026-02-24T16:00:15.523 INFO:teuthology.orchestra.run.trial084.stdout: python3-jaraco-context-6.0.1-3.el9.noarch 2026-02-24T16:00:15.523 INFO:teuthology.orchestra.run.trial084.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2026-02-24T16:00:15.523 INFO:teuthology.orchestra.run.trial084.stdout: python3-jaraco-text-4.0.0-2.el9.noarch 2026-02-24T16:00:15.523 INFO:teuthology.orchestra.run.trial084.stdout: python3-libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-24T16:00:15.523 INFO:teuthology.orchestra.run.trial084.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2026-02-24T16:00:15.523 INFO:teuthology.orchestra.run.trial084.stdout: python3-packaging-20.9-5.el9.noarch 2026-02-24T16:00:15.523 INFO:teuthology.orchestra.run.trial084.stdout: python3-ply-3.11-14.el9.noarch 2026-02-24T16:00:15.523 INFO:teuthology.orchestra.run.trial084.stdout: python3-portend-3.1.0-2.el9.noarch 2026-02-24T16:00:15.523 INFO:teuthology.orchestra.run.trial084.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2026-02-24T16:00:15.524 INFO:teuthology.orchestra.run.trial084.stdout: python3-pycparser-2.20-6.el9.noarch 2026-02-24T16:00:15.524 INFO:teuthology.orchestra.run.trial084.stdout: python3-pyparsing-2.4.7-9.el9.noarch 2026-02-24T16:00:15.524 INFO:teuthology.orchestra.run.trial084.stdout: python3-requests-2.25.1-10.el9.noarch 2026-02-24T16:00:15.524 INFO:teuthology.orchestra.run.trial084.stdout: python3-tempora-5.0.0-2.el9.noarch 2026-02-24T16:00:15.524 INFO:teuthology.orchestra.run.trial084.stdout: python3-toml-0.10.2-6.el9.noarch 2026-02-24T16:00:15.524 INFO:teuthology.orchestra.run.trial084.stdout: python3-urllib3-1.26.5-7.el9.noarch 2026-02-24T16:00:15.524 INFO:teuthology.orchestra.run.trial084.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2026-02-24T16:00:15.524 INFO:teuthology.orchestra.run.trial084.stdout: unzip-6.0-59.el9.x86_64 2026-02-24T16:00:15.524 INFO:teuthology.orchestra.run.trial084.stdout: zip-3.0-35.el9.x86_64 2026-02-24T16:00:15.524 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:00:15.524 INFO:teuthology.orchestra.run.trial084.stdout:Complete! 2026-02-24T16:00:15.564 DEBUG:teuthology.orchestra.run.trial084:> sudo yum -y install ceph-base 2026-02-24T16:00:15.885 INFO:teuthology.orchestra.run.trial084.stdout:Last metadata expiration check: 0:11:52 ago on Tue 24 Feb 2026 03:48:23 PM UTC. 2026-02-24T16:00:15.950 INFO:teuthology.orchestra.run.trial084.stdout:Package ceph-base-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:00:15.968 INFO:teuthology.orchestra.run.trial084.stdout:Dependencies resolved. 2026-02-24T16:00:15.968 INFO:teuthology.orchestra.run.trial084.stdout:Nothing to do. 2026-02-24T16:00:15.969 INFO:teuthology.orchestra.run.trial084.stdout:Complete! 2026-02-24T16:00:15.990 DEBUG:teuthology.orchestra.run.trial084:> sudo yum -y install cephadm 2026-02-24T16:00:16.305 INFO:teuthology.orchestra.run.trial084.stdout:Last metadata expiration check: 0:11:53 ago on Tue 24 Feb 2026 03:48:23 PM UTC. 2026-02-24T16:00:16.387 INFO:teuthology.orchestra.run.trial084.stdout:Dependencies resolved. 2026-02-24T16:00:16.388 INFO:teuthology.orchestra.run.trial084.stdout:================================================================================ 2026-02-24T16:00:16.388 INFO:teuthology.orchestra.run.trial084.stdout: Package Arch Version Repository Size 2026-02-24T16:00:16.388 INFO:teuthology.orchestra.run.trial084.stdout:================================================================================ 2026-02-24T16:00:16.389 INFO:teuthology.orchestra.run.trial084.stdout:Installing: 2026-02-24T16:00:16.389 INFO:teuthology.orchestra.run.trial084.stdout: cephadm noarch 2:20.3.0-5343.gd26e5c22.el9 ceph-noarch 1.0 M 2026-02-24T16:00:16.389 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:00:16.389 INFO:teuthology.orchestra.run.trial084.stdout:Transaction Summary 2026-02-24T16:00:16.389 INFO:teuthology.orchestra.run.trial084.stdout:================================================================================ 2026-02-24T16:00:16.389 INFO:teuthology.orchestra.run.trial084.stdout:Install 1 Package 2026-02-24T16:00:16.389 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:00:16.389 INFO:teuthology.orchestra.run.trial084.stdout:Total download size: 1.0 M 2026-02-24T16:00:16.389 INFO:teuthology.orchestra.run.trial084.stdout:Installed size: 1.0 M 2026-02-24T16:00:16.389 INFO:teuthology.orchestra.run.trial084.stdout:Downloading Packages: 2026-02-24T16:00:19.498 INFO:teuthology.orchestra.run.trial084.stdout:cephadm-20.3.0-5343.gd26e5c22.el9.noarch.rpm 335 kB/s | 1.0 MB 00:03 2026-02-24T16:00:19.499 INFO:teuthology.orchestra.run.trial084.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:00:19.500 INFO:teuthology.orchestra.run.trial084.stdout:Total 334 kB/s | 1.0 MB 00:03 2026-02-24T16:00:19.500 INFO:teuthology.orchestra.run.trial084.stdout:Running transaction check 2026-02-24T16:00:19.502 INFO:teuthology.orchestra.run.trial084.stdout:Transaction check succeeded. 2026-02-24T16:00:19.502 INFO:teuthology.orchestra.run.trial084.stdout:Running transaction test 2026-02-24T16:00:19.506 INFO:teuthology.orchestra.run.trial084.stdout:Transaction test succeeded. 2026-02-24T16:00:19.506 INFO:teuthology.orchestra.run.trial084.stdout:Running transaction 2026-02-24T16:00:19.525 INFO:teuthology.orchestra.run.trial084.stdout: Preparing : 1/1 2026-02-24T16:00:19.586 INFO:teuthology.orchestra.run.trial084.stdout: Running scriptlet: cephadm-2:20.3.0-5343.gd26e5c22.el9.noarch 1/1 2026-02-24T16:00:19.600 INFO:teuthology.orchestra.run.trial084.stdout: Installing : cephadm-2:20.3.0-5343.gd26e5c22.el9.noarch 1/1 2026-02-24T16:00:19.669 INFO:teuthology.orchestra.run.trial084.stdout: Running scriptlet: cephadm-2:20.3.0-5343.gd26e5c22.el9.noarch 1/1 2026-02-24T16:00:19.728 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : cephadm-2:20.3.0-5343.gd26e5c22.el9.noarch 1/1 2026-02-24T16:00:19.728 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:00:19.729 INFO:teuthology.orchestra.run.trial084.stdout:Installed: 2026-02-24T16:00:19.729 INFO:teuthology.orchestra.run.trial084.stdout: cephadm-2:20.3.0-5343.gd26e5c22.el9.noarch 2026-02-24T16:00:19.729 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:00:19.729 INFO:teuthology.orchestra.run.trial084.stdout:Complete! 2026-02-24T16:00:19.755 DEBUG:teuthology.orchestra.run.trial084:> sudo yum -y install ceph-immutable-object-cache 2026-02-24T16:00:20.076 INFO:teuthology.orchestra.run.trial084.stdout:Last metadata expiration check: 0:11:57 ago on Tue 24 Feb 2026 03:48:23 PM UTC. 2026-02-24T16:00:20.161 INFO:teuthology.orchestra.run.trial084.stdout:Dependencies resolved. 2026-02-24T16:00:20.162 INFO:teuthology.orchestra.run.trial084.stdout:================================================================================ 2026-02-24T16:00:20.162 INFO:teuthology.orchestra.run.trial084.stdout: Package Arch Version Repo Size 2026-02-24T16:00:20.162 INFO:teuthology.orchestra.run.trial084.stdout:================================================================================ 2026-02-24T16:00:20.162 INFO:teuthology.orchestra.run.trial084.stdout:Installing: 2026-02-24T16:00:20.162 INFO:teuthology.orchestra.run.trial084.stdout: ceph-immutable-object-cache x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 149 k 2026-02-24T16:00:20.162 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:00:20.163 INFO:teuthology.orchestra.run.trial084.stdout:Transaction Summary 2026-02-24T16:00:20.163 INFO:teuthology.orchestra.run.trial084.stdout:================================================================================ 2026-02-24T16:00:20.163 INFO:teuthology.orchestra.run.trial084.stdout:Install 1 Package 2026-02-24T16:00:20.163 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:00:20.163 INFO:teuthology.orchestra.run.trial084.stdout:Total download size: 149 k 2026-02-24T16:00:20.163 INFO:teuthology.orchestra.run.trial084.stdout:Installed size: 455 k 2026-02-24T16:00:20.163 INFO:teuthology.orchestra.run.trial084.stdout:Downloading Packages: 2026-02-24T16:00:21.166 INFO:teuthology.orchestra.run.trial084.stdout:ceph-immutable-object-cache-20.3.0-5343.gd26e5c 149 kB/s | 149 kB 00:01 2026-02-24T16:00:21.166 INFO:teuthology.orchestra.run.trial084.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:00:21.166 INFO:teuthology.orchestra.run.trial084.stdout:Total 149 kB/s | 149 kB 00:01 2026-02-24T16:00:21.167 INFO:teuthology.orchestra.run.trial084.stdout:Running transaction check 2026-02-24T16:00:21.171 INFO:teuthology.orchestra.run.trial084.stdout:Transaction check succeeded. 2026-02-24T16:00:21.171 INFO:teuthology.orchestra.run.trial084.stdout:Running transaction test 2026-02-24T16:00:21.197 INFO:teuthology.orchestra.run.trial084.stdout:Transaction test succeeded. 2026-02-24T16:00:21.197 INFO:teuthology.orchestra.run.trial084.stdout:Running transaction 2026-02-24T16:00:21.247 INFO:teuthology.orchestra.run.trial084.stdout: Preparing : 1/1 2026-02-24T16:00:21.248 INFO:teuthology.orchestra.run.trial084.stdout: Installing : ceph-immutable-object-cache-2:20.3.0-5343.gd26e5c22. 1/1 2026-02-24T16:00:21.271 INFO:teuthology.orchestra.run.trial084.stdout: Running scriptlet: ceph-immutable-object-cache-2:20.3.0-5343.gd26e5c22. 1/1 2026-02-24T16:00:21.271 INFO:teuthology.orchestra.run.trial084.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T16:00:21.271 INFO:teuthology.orchestra.run.trial084.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2026-02-24T16:00:21.271 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:00:21.564 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : ceph-immutable-object-cache-2:20.3.0-5343.gd26e5c22. 1/1 2026-02-24T16:00:21.564 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:00:21.565 INFO:teuthology.orchestra.run.trial084.stdout:Installed: 2026-02-24T16:00:21.565 INFO:teuthology.orchestra.run.trial084.stdout: ceph-immutable-object-cache-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T16:00:21.565 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:00:21.565 INFO:teuthology.orchestra.run.trial084.stdout:Complete! 2026-02-24T16:00:21.593 DEBUG:teuthology.orchestra.run.trial084:> sudo yum -y install ceph-mgr 2026-02-24T16:00:21.917 INFO:teuthology.orchestra.run.trial084.stdout:Last metadata expiration check: 0:11:58 ago on Tue 24 Feb 2026 03:48:23 PM UTC. 2026-02-24T16:00:21.981 INFO:teuthology.orchestra.run.trial084.stdout:Package ceph-mgr-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:00:21.999 INFO:teuthology.orchestra.run.trial084.stdout:Dependencies resolved. 2026-02-24T16:00:21.999 INFO:teuthology.orchestra.run.trial084.stdout:Nothing to do. 2026-02-24T16:00:21.999 INFO:teuthology.orchestra.run.trial084.stdout:Complete! 2026-02-24T16:00:22.021 DEBUG:teuthology.orchestra.run.trial084:> sudo yum -y install ceph-mgr-dashboard 2026-02-24T16:00:22.333 INFO:teuthology.orchestra.run.trial084.stdout:Last metadata expiration check: 0:11:59 ago on Tue 24 Feb 2026 03:48:23 PM UTC. 2026-02-24T16:00:22.416 INFO:teuthology.orchestra.run.trial084.stdout:Dependencies resolved. 2026-02-24T16:00:22.417 INFO:teuthology.orchestra.run.trial084.stdout:================================================================================ 2026-02-24T16:00:22.417 INFO:teuthology.orchestra.run.trial084.stdout: Package Arch Version Repository Size 2026-02-24T16:00:22.417 INFO:teuthology.orchestra.run.trial084.stdout:================================================================================ 2026-02-24T16:00:22.417 INFO:teuthology.orchestra.run.trial084.stdout:Installing: 2026-02-24T16:00:22.417 INFO:teuthology.orchestra.run.trial084.stdout: ceph-mgr-dashboard noarch 2:20.3.0-5343.gd26e5c22.el9 ceph-noarch 51 M 2026-02-24T16:00:22.417 INFO:teuthology.orchestra.run.trial084.stdout:Installing dependencies: 2026-02-24T16:00:22.417 INFO:teuthology.orchestra.run.trial084.stdout: abseil-cpp x86_64 20211102.0-4.el9 epel 551 k 2026-02-24T16:00:22.418 INFO:teuthology.orchestra.run.trial084.stdout: ceph-grafana-dashboards noarch 2:20.3.0-5343.gd26e5c22.el9 ceph-noarch 57 k 2026-02-24T16:00:22.418 INFO:teuthology.orchestra.run.trial084.stdout: ceph-prometheus-alerts noarch 2:20.3.0-5343.gd26e5c22.el9 ceph-noarch 17 k 2026-02-24T16:00:22.418 INFO:teuthology.orchestra.run.trial084.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2026-02-24T16:00:22.418 INFO:teuthology.orchestra.run.trial084.stdout: protobuf-compiler x86_64 3.14.0-17.el9 crb 862 k 2026-02-24T16:00:22.418 INFO:teuthology.orchestra.run.trial084.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2026-02-24T16:00:22.418 INFO:teuthology.orchestra.run.trial084.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2026-02-24T16:00:22.418 INFO:teuthology.orchestra.run.trial084.stdout: python3-jmespath noarch 1.0.1-1.el9 appstream 48 k 2026-02-24T16:00:22.418 INFO:teuthology.orchestra.run.trial084.stdout: python3-protobuf noarch 3.14.0-17.el9 appstream 267 k 2026-02-24T16:00:22.418 INFO:teuthology.orchestra.run.trial084.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2026-02-24T16:00:22.418 INFO:teuthology.orchestra.run.trial084.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2026-02-24T16:00:22.418 INFO:teuthology.orchestra.run.trial084.stdout: python3-xmltodict noarch 0.12.0-15.el9 epel 22 k 2026-02-24T16:00:22.418 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:00:22.419 INFO:teuthology.orchestra.run.trial084.stdout:Transaction Summary 2026-02-24T16:00:22.419 INFO:teuthology.orchestra.run.trial084.stdout:================================================================================ 2026-02-24T16:00:22.419 INFO:teuthology.orchestra.run.trial084.stdout:Install 13 Packages 2026-02-24T16:00:22.419 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:00:22.419 INFO:teuthology.orchestra.run.trial084.stdout:Total download size: 55 M 2026-02-24T16:00:22.420 INFO:teuthology.orchestra.run.trial084.stdout:Installed size: 262 M 2026-02-24T16:00:22.420 INFO:teuthology.orchestra.run.trial084.stdout:Downloading Packages: 2026-02-24T16:00:23.709 INFO:teuthology.orchestra.run.trial084.stdout:(1/13): ceph-grafana-dashboards-20.3.0-5343.gd2 105 kB/s | 57 kB 00:00 2026-02-24T16:00:23.813 INFO:teuthology.orchestra.run.trial084.stdout:(2/13): python3-jmespath-1.0.1-1.el9.noarch.rpm 459 kB/s | 48 kB 00:00 2026-02-24T16:00:23.901 INFO:teuthology.orchestra.run.trial084.stdout:(3/13): python3-protobuf-3.14.0-17.el9.noarch.r 3.0 MB/s | 267 kB 00:00 2026-02-24T16:00:25.630 INFO:teuthology.orchestra.run.trial084.stdout:(4/13): ceph-prometheus-alerts-20.3.0-5343.gd26 7.0 kB/s | 17 kB 00:02 2026-02-24T16:00:25.979 INFO:teuthology.orchestra.run.trial084.stdout:(5/13): abseil-cpp-20211102.0-4.el9.x86_64.rpm 1.5 MB/s | 551 kB 00:00 2026-02-24T16:00:26.017 INFO:teuthology.orchestra.run.trial084.stdout:(6/13): grpc-data-1.46.7-10.el9.noarch.rpm 512 kB/s | 19 kB 00:00 2026-02-24T16:00:26.145 INFO:teuthology.orchestra.run.trial084.stdout:(7/13): python3-grpcio-1.46.7-10.el9.x86_64.rpm 16 MB/s | 2.0 MB 00:00 2026-02-24T16:00:26.184 INFO:teuthology.orchestra.run.trial084.stdout:(8/13): python3-grpcio-tools-1.46.7-10.el9.x86_ 3.6 MB/s | 144 kB 00:00 2026-02-24T16:00:26.222 INFO:teuthology.orchestra.run.trial084.stdout:(9/13): python3-repoze-lru-0.7-16.el9.noarch.rp 815 kB/s | 31 kB 00:00 2026-02-24T16:00:26.262 INFO:teuthology.orchestra.run.trial084.stdout:(10/13): python3-routes-2.5.1-5.el9.noarch.rpm 4.7 MB/s | 188 kB 00:00 2026-02-24T16:00:26.299 INFO:teuthology.orchestra.run.trial084.stdout:(11/13): python3-xmltodict-0.12.0-15.el9.noarch 588 kB/s | 22 kB 00:00 2026-02-24T16:00:26.351 INFO:teuthology.orchestra.run.trial084.stdout:(12/13): protobuf-compiler-3.14.0-17.el9.x86_64 352 kB/s | 862 kB 00:02 2026-02-24T16:01:52.943 INFO:teuthology.orchestra.run.trial084.stdout:(13/13): ceph-mgr-dashboard-20.3.0-5343.gd26e5c 578 kB/s | 51 MB 01:29 2026-02-24T16:01:52.946 INFO:teuthology.orchestra.run.trial084.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:01:52.946 INFO:teuthology.orchestra.run.trial084.stdout:Total 621 kB/s | 55 MB 01:30 2026-02-24T16:01:52.969 INFO:teuthology.orchestra.run.trial084.stdout:Running transaction check 2026-02-24T16:01:52.980 INFO:teuthology.orchestra.run.trial084.stdout:Transaction check succeeded. 2026-02-24T16:01:52.980 INFO:teuthology.orchestra.run.trial084.stdout:Running transaction test 2026-02-24T16:01:53.193 INFO:teuthology.orchestra.run.trial084.stdout:Transaction test succeeded. 2026-02-24T16:01:53.193 INFO:teuthology.orchestra.run.trial084.stdout:Running transaction 2026-02-24T16:01:53.423 INFO:teuthology.orchestra.run.trial084.stdout: Preparing : 1/1 2026-02-24T16:01:53.428 INFO:teuthology.orchestra.run.trial084.stdout: Installing : python3-xmltodict-0.12.0-15.el9.noarch 1/13 2026-02-24T16:01:53.439 INFO:teuthology.orchestra.run.trial084.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 2/13 2026-02-24T16:01:53.442 INFO:teuthology.orchestra.run.trial084.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 3/13 2026-02-24T16:01:53.469 INFO:teuthology.orchestra.run.trial084.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 4/13 2026-02-24T16:01:53.513 INFO:teuthology.orchestra.run.trial084.stdout: Installing : abseil-cpp-20211102.0-4.el9.x86_64 5/13 2026-02-24T16:01:53.522 INFO:teuthology.orchestra.run.trial084.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 6/13 2026-02-24T16:01:53.550 INFO:teuthology.orchestra.run.trial084.stdout: Installing : protobuf-compiler-3.14.0-17.el9.x86_64 7/13 2026-02-24T16:01:53.559 INFO:teuthology.orchestra.run.trial084.stdout: Installing : python3-protobuf-3.14.0-17.el9.noarch 8/13 2026-02-24T16:01:53.566 INFO:teuthology.orchestra.run.trial084.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/13 2026-02-24T16:01:53.569 INFO:teuthology.orchestra.run.trial084.stdout: Installing : python3-jmespath-1.0.1-1.el9.noarch 10/13 2026-02-24T16:01:53.578 INFO:teuthology.orchestra.run.trial084.stdout: Installing : ceph-prometheus-alerts-2:20.3.0-5343.gd26e5c22.el9 11/13 2026-02-24T16:01:56.516 INFO:teuthology.orchestra.run.trial084.stdout: Installing : ceph-grafana-dashboards-2:20.3.0-5343.gd26e5c22.el 12/13 2026-02-24T16:01:56.523 INFO:teuthology.orchestra.run.trial084.stdout: Installing : ceph-mgr-dashboard-2:20.3.0-5343.gd26e5c22.el9.noa 13/13 2026-02-24T16:01:56.786 INFO:teuthology.orchestra.run.trial084.stdout: Running scriptlet: ceph-mgr-dashboard-2:20.3.0-5343.gd26e5c22.el9.noa 13/13 2026-02-24T16:01:56.787 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : ceph-grafana-dashboards-2:20.3.0-5343.gd26e5c22.el 1/13 2026-02-24T16:01:56.787 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : ceph-mgr-dashboard-2:20.3.0-5343.gd26e5c22.el9.noa 2/13 2026-02-24T16:01:56.787 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : ceph-prometheus-alerts-2:20.3.0-5343.gd26e5c22.el9 3/13 2026-02-24T16:01:56.787 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : python3-jmespath-1.0.1-1.el9.noarch 4/13 2026-02-24T16:01:56.787 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : python3-protobuf-3.14.0-17.el9.noarch 5/13 2026-02-24T16:01:56.787 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : protobuf-compiler-3.14.0-17.el9.x86_64 6/13 2026-02-24T16:01:56.787 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : abseil-cpp-20211102.0-4.el9.x86_64 7/13 2026-02-24T16:01:56.787 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/13 2026-02-24T16:01:56.787 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/13 2026-02-24T16:01:56.787 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/13 2026-02-24T16:01:56.788 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/13 2026-02-24T16:01:56.788 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/13 2026-02-24T16:01:56.847 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : python3-xmltodict-0.12.0-15.el9.noarch 13/13 2026-02-24T16:01:56.848 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:01:56.848 INFO:teuthology.orchestra.run.trial084.stdout:Installed: 2026-02-24T16:01:56.848 INFO:teuthology.orchestra.run.trial084.stdout: abseil-cpp-20211102.0-4.el9.x86_64 2026-02-24T16:01:56.848 INFO:teuthology.orchestra.run.trial084.stdout: ceph-grafana-dashboards-2:20.3.0-5343.gd26e5c22.el9.noarch 2026-02-24T16:01:56.848 INFO:teuthology.orchestra.run.trial084.stdout: ceph-mgr-dashboard-2:20.3.0-5343.gd26e5c22.el9.noarch 2026-02-24T16:01:56.848 INFO:teuthology.orchestra.run.trial084.stdout: ceph-prometheus-alerts-2:20.3.0-5343.gd26e5c22.el9.noarch 2026-02-24T16:01:56.848 INFO:teuthology.orchestra.run.trial084.stdout: grpc-data-1.46.7-10.el9.noarch 2026-02-24T16:01:56.848 INFO:teuthology.orchestra.run.trial084.stdout: protobuf-compiler-3.14.0-17.el9.x86_64 2026-02-24T16:01:56.848 INFO:teuthology.orchestra.run.trial084.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2026-02-24T16:01:56.848 INFO:teuthology.orchestra.run.trial084.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2026-02-24T16:01:56.848 INFO:teuthology.orchestra.run.trial084.stdout: python3-jmespath-1.0.1-1.el9.noarch 2026-02-24T16:01:56.849 INFO:teuthology.orchestra.run.trial084.stdout: python3-protobuf-3.14.0-17.el9.noarch 2026-02-24T16:01:56.849 INFO:teuthology.orchestra.run.trial084.stdout: python3-repoze-lru-0.7-16.el9.noarch 2026-02-24T16:01:56.849 INFO:teuthology.orchestra.run.trial084.stdout: python3-routes-2.5.1-5.el9.noarch 2026-02-24T16:01:56.849 INFO:teuthology.orchestra.run.trial084.stdout: python3-xmltodict-0.12.0-15.el9.noarch 2026-02-24T16:01:56.849 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:01:56.849 INFO:teuthology.orchestra.run.trial084.stdout:Complete! 2026-02-24T16:01:56.886 DEBUG:teuthology.orchestra.run.trial084:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-24T16:01:57.207 INFO:teuthology.orchestra.run.trial084.stdout:Last metadata expiration check: 0:13:34 ago on Tue 24 Feb 2026 03:48:23 PM UTC. 2026-02-24T16:01:57.291 INFO:teuthology.orchestra.run.trial084.stdout:Dependencies resolved. 2026-02-24T16:01:57.291 INFO:teuthology.orchestra.run.trial084.stdout:======================================================================================= 2026-02-24T16:01:57.292 INFO:teuthology.orchestra.run.trial084.stdout: Package Arch Version Repository Size 2026-02-24T16:01:57.292 INFO:teuthology.orchestra.run.trial084.stdout:======================================================================================= 2026-02-24T16:01:57.292 INFO:teuthology.orchestra.run.trial084.stdout:Installing: 2026-02-24T16:01:57.292 INFO:teuthology.orchestra.run.trial084.stdout: ceph-mgr-diskprediction-local noarch 2:20.3.0-5343.gd26e5c22.el9 ceph-noarch 7.4 M 2026-02-24T16:01:57.292 INFO:teuthology.orchestra.run.trial084.stdout:Installing dependencies: 2026-02-24T16:01:57.292 INFO:teuthology.orchestra.run.trial084.stdout: flexiblas x86_64 3.0.4-9.el9 appstream 30 k 2026-02-24T16:01:57.292 INFO:teuthology.orchestra.run.trial084.stdout: flexiblas-netlib x86_64 3.0.4-9.el9 appstream 3.0 M 2026-02-24T16:01:57.292 INFO:teuthology.orchestra.run.trial084.stdout: flexiblas-openblas-openmp x86_64 3.0.4-9.el9 appstream 15 k 2026-02-24T16:01:57.292 INFO:teuthology.orchestra.run.trial084.stdout: libgfortran x86_64 11.5.0-14.el9 baseos 794 k 2026-02-24T16:01:57.293 INFO:teuthology.orchestra.run.trial084.stdout: libquadmath x86_64 11.5.0-14.el9 baseos 184 k 2026-02-24T16:01:57.293 INFO:teuthology.orchestra.run.trial084.stdout: openblas x86_64 0.3.29-1.el9 appstream 42 k 2026-02-24T16:01:57.293 INFO:teuthology.orchestra.run.trial084.stdout: openblas-openmp x86_64 0.3.29-1.el9 appstream 5.3 M 2026-02-24T16:01:57.293 INFO:teuthology.orchestra.run.trial084.stdout: python3-devel x86_64 3.9.25-3.el9 appstream 244 k 2026-02-24T16:01:57.293 INFO:teuthology.orchestra.run.trial084.stdout: python3-numpy x86_64 1:1.23.5-2.el9 appstream 6.1 M 2026-02-24T16:01:57.293 INFO:teuthology.orchestra.run.trial084.stdout: python3-numpy-f2py x86_64 1:1.23.5-2.el9 appstream 442 k 2026-02-24T16:01:57.293 INFO:teuthology.orchestra.run.trial084.stdout: python3-scipy x86_64 1.9.3-2.el9 appstream 19 M 2026-02-24T16:01:57.293 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:01:57.293 INFO:teuthology.orchestra.run.trial084.stdout:Transaction Summary 2026-02-24T16:01:57.293 INFO:teuthology.orchestra.run.trial084.stdout:======================================================================================= 2026-02-24T16:01:57.293 INFO:teuthology.orchestra.run.trial084.stdout:Install 12 Packages 2026-02-24T16:01:57.294 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:01:57.294 INFO:teuthology.orchestra.run.trial084.stdout:Total download size: 43 M 2026-02-24T16:01:57.294 INFO:teuthology.orchestra.run.trial084.stdout:Installed size: 234 M 2026-02-24T16:01:57.295 INFO:teuthology.orchestra.run.trial084.stdout:Downloading Packages: 2026-02-24T16:01:58.166 INFO:teuthology.orchestra.run.trial084.stdout:(1/12): libquadmath-11.5.0-14.el9.x86_64.rpm 267 kB/s | 184 kB 00:00 2026-02-24T16:01:58.182 INFO:teuthology.orchestra.run.trial084.stdout:(2/12): flexiblas-3.0.4-9.el9.x86_64.rpm 1.8 MB/s | 30 kB 00:00 2026-02-24T16:01:58.210 INFO:teuthology.orchestra.run.trial084.stdout:(3/12): flexiblas-netlib-3.0.4-9.el9.x86_64.rpm 107 MB/s | 3.0 MB 00:00 2026-02-24T16:01:58.214 INFO:teuthology.orchestra.run.trial084.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-9.el9.x 3.6 MB/s | 15 kB 00:00 2026-02-24T16:01:58.219 INFO:teuthology.orchestra.run.trial084.stdout:(5/12): openblas-0.3.29-1.el9.x86_64.rpm 9.5 MB/s | 42 kB 00:00 2026-02-24T16:01:58.242 INFO:teuthology.orchestra.run.trial084.stdout:(6/12): openblas-openmp-0.3.29-1.el9.x86_64.rpm 230 MB/s | 5.3 MB 00:00 2026-02-24T16:01:58.249 INFO:teuthology.orchestra.run.trial084.stdout:(7/12): python3-devel-3.9.25-3.el9.x86_64.rpm 34 MB/s | 244 kB 00:00 2026-02-24T16:01:58.277 INFO:teuthology.orchestra.run.trial084.stdout:(8/12): python3-numpy-1.23.5-2.el9.x86_64.rpm 227 MB/s | 6.1 MB 00:00 2026-02-24T16:01:58.285 INFO:teuthology.orchestra.run.trial084.stdout:(9/12): python3-numpy-f2py-1.23.5-2.el9.x86_64. 55 MB/s | 442 kB 00:00 2026-02-24T16:01:58.355 INFO:teuthology.orchestra.run.trial084.stdout:(10/12): python3-scipy-1.9.3-2.el9.x86_64.rpm 276 MB/s | 19 MB 00:00 2026-02-24T16:01:58.865 INFO:teuthology.orchestra.run.trial084.stdout:(11/12): libgfortran-11.5.0-14.el9.x86_64.rpm 572 kB/s | 794 kB 00:01 2026-02-24T16:02:13.435 INFO:teuthology.orchestra.run.trial084.stdout:(12/12): ceph-mgr-diskprediction-local-20.3.0-5 474 kB/s | 7.4 MB 00:15 2026-02-24T16:02:13.437 INFO:teuthology.orchestra.run.trial084.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:02:13.437 INFO:teuthology.orchestra.run.trial084.stdout:Total 2.6 MB/s | 43 MB 00:16 2026-02-24T16:02:13.595 INFO:teuthology.orchestra.run.trial084.stdout:Running transaction check 2026-02-24T16:02:13.609 INFO:teuthology.orchestra.run.trial084.stdout:Transaction check succeeded. 2026-02-24T16:02:13.609 INFO:teuthology.orchestra.run.trial084.stdout:Running transaction test 2026-02-24T16:02:13.811 INFO:teuthology.orchestra.run.trial084.stdout:Transaction test succeeded. 2026-02-24T16:02:13.812 INFO:teuthology.orchestra.run.trial084.stdout:Running transaction 2026-02-24T16:02:14.031 INFO:teuthology.orchestra.run.trial084.stdout: Preparing : 1/1 2026-02-24T16:02:14.034 INFO:teuthology.orchestra.run.trial084.stdout: Installing : flexiblas-3.0.4-9.el9.x86_64 1/12 2026-02-24T16:02:14.040 INFO:teuthology.orchestra.run.trial084.stdout: Installing : libquadmath-11.5.0-14.el9.x86_64 2/12 2026-02-24T16:02:14.073 INFO:teuthology.orchestra.run.trial084.stdout: Installing : libgfortran-11.5.0-14.el9.x86_64 3/12 2026-02-24T16:02:14.078 INFO:teuthology.orchestra.run.trial084.stdout: Installing : python3-devel-3.9.25-3.el9.x86_64 4/12 2026-02-24T16:02:14.137 INFO:teuthology.orchestra.run.trial084.stdout: Installing : openblas-0.3.29-1.el9.x86_64 5/12 2026-02-24T16:02:14.139 INFO:teuthology.orchestra.run.trial084.stdout: Installing : openblas-openmp-0.3.29-1.el9.x86_64 6/12 2026-02-24T16:02:14.161 INFO:teuthology.orchestra.run.trial084.stdout: Installing : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 7/12 2026-02-24T16:02:14.479 INFO:teuthology.orchestra.run.trial084.stdout: Installing : flexiblas-netlib-3.0.4-9.el9.x86_64 8/12 2026-02-24T16:02:14.551 INFO:teuthology.orchestra.run.trial084.stdout: Installing : python3-numpy-1:1.23.5-2.el9.x86_64 9/12 2026-02-24T16:02:15.210 INFO:teuthology.orchestra.run.trial084.stdout: Installing : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 10/12 2026-02-24T16:02:15.668 INFO:teuthology.orchestra.run.trial084.stdout: Installing : python3-scipy-1.9.3-2.el9.x86_64 11/12 2026-02-24T16:02:15.669 INFO:teuthology.orchestra.run.trial084.stdout: Installing : ceph-mgr-diskprediction-local-2:20.3.0-5343.gd26e5 12/12 2026-02-24T16:02:15.959 INFO:teuthology.orchestra.run.trial084.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:20.3.0-5343.gd26e5 12/12 2026-02-24T16:02:15.959 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : ceph-mgr-diskprediction-local-2:20.3.0-5343.gd26e5 1/12 2026-02-24T16:02:15.959 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : libgfortran-11.5.0-14.el9.x86_64 2/12 2026-02-24T16:02:15.959 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : libquadmath-11.5.0-14.el9.x86_64 3/12 2026-02-24T16:02:15.959 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : flexiblas-3.0.4-9.el9.x86_64 4/12 2026-02-24T16:02:15.959 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : flexiblas-netlib-3.0.4-9.el9.x86_64 5/12 2026-02-24T16:02:15.959 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 6/12 2026-02-24T16:02:15.959 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : openblas-0.3.29-1.el9.x86_64 7/12 2026-02-24T16:02:15.960 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : openblas-openmp-0.3.29-1.el9.x86_64 8/12 2026-02-24T16:02:15.960 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : python3-devel-3.9.25-3.el9.x86_64 9/12 2026-02-24T16:02:15.960 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : python3-numpy-1:1.23.5-2.el9.x86_64 10/12 2026-02-24T16:02:15.960 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 11/12 2026-02-24T16:02:16.022 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : python3-scipy-1.9.3-2.el9.x86_64 12/12 2026-02-24T16:02:16.022 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:02:16.022 INFO:teuthology.orchestra.run.trial084.stdout:Installed: 2026-02-24T16:02:16.022 INFO:teuthology.orchestra.run.trial084.stdout: ceph-mgr-diskprediction-local-2:20.3.0-5343.gd26e5c22.el9.noarch 2026-02-24T16:02:16.022 INFO:teuthology.orchestra.run.trial084.stdout: flexiblas-3.0.4-9.el9.x86_64 2026-02-24T16:02:16.023 INFO:teuthology.orchestra.run.trial084.stdout: flexiblas-netlib-3.0.4-9.el9.x86_64 2026-02-24T16:02:16.023 INFO:teuthology.orchestra.run.trial084.stdout: flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 2026-02-24T16:02:16.023 INFO:teuthology.orchestra.run.trial084.stdout: libgfortran-11.5.0-14.el9.x86_64 2026-02-24T16:02:16.023 INFO:teuthology.orchestra.run.trial084.stdout: libquadmath-11.5.0-14.el9.x86_64 2026-02-24T16:02:16.023 INFO:teuthology.orchestra.run.trial084.stdout: openblas-0.3.29-1.el9.x86_64 2026-02-24T16:02:16.023 INFO:teuthology.orchestra.run.trial084.stdout: openblas-openmp-0.3.29-1.el9.x86_64 2026-02-24T16:02:16.023 INFO:teuthology.orchestra.run.trial084.stdout: python3-devel-3.9.25-3.el9.x86_64 2026-02-24T16:02:16.023 INFO:teuthology.orchestra.run.trial084.stdout: python3-numpy-1:1.23.5-2.el9.x86_64 2026-02-24T16:02:16.023 INFO:teuthology.orchestra.run.trial084.stdout: python3-numpy-f2py-1:1.23.5-2.el9.x86_64 2026-02-24T16:02:16.023 INFO:teuthology.orchestra.run.trial084.stdout: python3-scipy-1.9.3-2.el9.x86_64 2026-02-24T16:02:16.023 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:02:16.024 INFO:teuthology.orchestra.run.trial084.stdout:Complete! 2026-02-24T16:02:16.058 DEBUG:teuthology.orchestra.run.trial084:> sudo yum -y install ceph-mgr-rook 2026-02-24T16:02:16.376 INFO:teuthology.orchestra.run.trial084.stdout:Last metadata expiration check: 0:13:53 ago on Tue 24 Feb 2026 03:48:23 PM UTC. 2026-02-24T16:02:16.460 INFO:teuthology.orchestra.run.trial084.stdout:Dependencies resolved. 2026-02-24T16:02:16.460 INFO:teuthology.orchestra.run.trial084.stdout:================================================================================ 2026-02-24T16:02:16.461 INFO:teuthology.orchestra.run.trial084.stdout: Package Arch Version Repository Size 2026-02-24T16:02:16.461 INFO:teuthology.orchestra.run.trial084.stdout:================================================================================ 2026-02-24T16:02:16.461 INFO:teuthology.orchestra.run.trial084.stdout:Installing: 2026-02-24T16:02:16.461 INFO:teuthology.orchestra.run.trial084.stdout: ceph-mgr-rook noarch 2:20.3.0-5343.gd26e5c22.el9 ceph-noarch 50 k 2026-02-24T16:02:16.461 INFO:teuthology.orchestra.run.trial084.stdout:Installing dependencies: 2026-02-24T16:02:16.461 INFO:teuthology.orchestra.run.trial084.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2026-02-24T16:02:16.461 INFO:teuthology.orchestra.run.trial084.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2026-02-24T16:02:16.461 INFO:teuthology.orchestra.run.trial084.stdout: python3-google-auth noarch 1:2.45.0-1.el9 epel 254 k 2026-02-24T16:02:16.461 INFO:teuthology.orchestra.run.trial084.stdout: python3-kubernetes noarch 1:26.1.0-3.el9 epel 1.0 M 2026-02-24T16:02:16.461 INFO:teuthology.orchestra.run.trial084.stdout: python3-pyasn1 noarch 0.4.8-7.el9 appstream 157 k 2026-02-24T16:02:16.462 INFO:teuthology.orchestra.run.trial084.stdout: python3-pyasn1-modules noarch 0.4.8-7.el9 appstream 277 k 2026-02-24T16:02:16.462 INFO:teuthology.orchestra.run.trial084.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2026-02-24T16:02:16.462 INFO:teuthology.orchestra.run.trial084.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2026-02-24T16:02:16.462 INFO:teuthology.orchestra.run.trial084.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2026-02-24T16:02:16.462 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:02:16.462 INFO:teuthology.orchestra.run.trial084.stdout:Transaction Summary 2026-02-24T16:02:16.462 INFO:teuthology.orchestra.run.trial084.stdout:================================================================================ 2026-02-24T16:02:16.462 INFO:teuthology.orchestra.run.trial084.stdout:Install 10 Packages 2026-02-24T16:02:16.462 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:02:16.462 INFO:teuthology.orchestra.run.trial084.stdout:Total download size: 2.0 M 2026-02-24T16:02:16.462 INFO:teuthology.orchestra.run.trial084.stdout:Installed size: 25 M 2026-02-24T16:02:16.462 INFO:teuthology.orchestra.run.trial084.stdout:Downloading Packages: 2026-02-24T16:02:16.691 INFO:teuthology.orchestra.run.trial084.stdout:(1/10): python3-pyasn1-0.4.8-7.el9.noarch.rpm 1.7 MB/s | 157 kB 00:00 2026-02-24T16:02:16.693 INFO:teuthology.orchestra.run.trial084.stdout:(2/10): python3-pyasn1-modules-0.4.8-7.el9.noar 3.0 MB/s | 277 kB 00:00 2026-02-24T16:02:16.714 INFO:teuthology.orchestra.run.trial084.stdout:(3/10): python3-requests-oauthlib-1.3.0-12.el9. 2.3 MB/s | 54 kB 00:00 2026-02-24T16:02:16.779 INFO:teuthology.orchestra.run.trial084.stdout:(4/10): python3-cachetools-4.2.4-1.el9.noarch.r 375 kB/s | 32 kB 00:00 2026-02-24T16:02:16.782 INFO:teuthology.orchestra.run.trial084.stdout:(5/10): python3-certifi-2023.05.07-4.el9.noarch 207 kB/s | 14 kB 00:00 2026-02-24T16:02:16.874 INFO:teuthology.orchestra.run.trial084.stdout:(6/10): python3-google-auth-2.45.0-1.el9.noarch 2.6 MB/s | 254 kB 00:00 2026-02-24T16:02:16.900 INFO:teuthology.orchestra.run.trial084.stdout:(7/10): python3-rsa-4.9-2.el9.noarch.rpm 2.2 MB/s | 59 kB 00:00 2026-02-24T16:02:16.927 INFO:teuthology.orchestra.run.trial084.stdout:(8/10): python3-websocket-client-1.2.3-2.el9.no 3.2 MB/s | 90 kB 00:00 2026-02-24T16:02:16.937 INFO:teuthology.orchestra.run.trial084.stdout:(9/10): python3-kubernetes-26.1.0-3.el9.noarch. 6.6 MB/s | 1.0 MB 00:00 2026-02-24T16:02:17.021 INFO:teuthology.orchestra.run.trial084.stdout:(10/10): ceph-mgr-rook-20.3.0-5343.gd26e5c22.el 120 kB/s | 50 kB 00:00 2026-02-24T16:02:17.021 INFO:teuthology.orchestra.run.trial084.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:02:17.022 INFO:teuthology.orchestra.run.trial084.stdout:Total 3.5 MB/s | 2.0 MB 00:00 2026-02-24T16:02:17.046 INFO:teuthology.orchestra.run.trial084.stdout:Running transaction check 2026-02-24T16:02:17.053 INFO:teuthology.orchestra.run.trial084.stdout:Transaction check succeeded. 2026-02-24T16:02:17.053 INFO:teuthology.orchestra.run.trial084.stdout:Running transaction test 2026-02-24T16:02:17.092 INFO:teuthology.orchestra.run.trial084.stdout:Transaction test succeeded. 2026-02-24T16:02:17.093 INFO:teuthology.orchestra.run.trial084.stdout:Running transaction 2026-02-24T16:02:17.168 INFO:teuthology.orchestra.run.trial084.stdout: Preparing : 1/1 2026-02-24T16:02:17.219 INFO:teuthology.orchestra.run.trial084.stdout: Installing : python3-pyasn1-0.4.8-7.el9.noarch 1/10 2026-02-24T16:02:17.232 INFO:teuthology.orchestra.run.trial084.stdout: Installing : python3-pyasn1-modules-0.4.8-7.el9.noarch 2/10 2026-02-24T16:02:17.246 INFO:teuthology.orchestra.run.trial084.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/10 2026-02-24T16:02:17.251 INFO:teuthology.orchestra.run.trial084.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/10 2026-02-24T16:02:17.259 INFO:teuthology.orchestra.run.trial084.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/10 2026-02-24T16:02:17.298 INFO:teuthology.orchestra.run.trial084.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/10 2026-02-24T16:02:17.310 INFO:teuthology.orchestra.run.trial084.stdout: Installing : python3-google-auth-1:2.45.0-1.el9.noarch 7/10 2026-02-24T16:02:17.628 INFO:teuthology.orchestra.run.trial084.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 8/10 2026-02-24T16:02:17.643 INFO:teuthology.orchestra.run.trial084.stdout: Installing : python3-kubernetes-1:26.1.0-3.el9.noarch 9/10 2026-02-24T16:02:17.644 INFO:teuthology.orchestra.run.trial084.stdout: Installing : ceph-mgr-rook-2:20.3.0-5343.gd26e5c22.el9.noarch 10/10 2026-02-24T16:02:17.757 INFO:teuthology.orchestra.run.trial084.stdout: Running scriptlet: ceph-mgr-rook-2:20.3.0-5343.gd26e5c22.el9.noarch 10/10 2026-02-24T16:02:17.757 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : ceph-mgr-rook-2:20.3.0-5343.gd26e5c22.el9.noarch 1/10 2026-02-24T16:02:17.758 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : python3-pyasn1-0.4.8-7.el9.noarch 2/10 2026-02-24T16:02:17.758 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : python3-pyasn1-modules-0.4.8-7.el9.noarch 3/10 2026-02-24T16:02:17.758 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 4/10 2026-02-24T16:02:17.758 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 5/10 2026-02-24T16:02:17.758 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 6/10 2026-02-24T16:02:17.758 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : python3-google-auth-1:2.45.0-1.el9.noarch 7/10 2026-02-24T16:02:17.758 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : python3-kubernetes-1:26.1.0-3.el9.noarch 8/10 2026-02-24T16:02:17.758 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 9/10 2026-02-24T16:02:17.819 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 10/10 2026-02-24T16:02:17.819 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:02:17.819 INFO:teuthology.orchestra.run.trial084.stdout:Installed: 2026-02-24T16:02:17.819 INFO:teuthology.orchestra.run.trial084.stdout: ceph-mgr-rook-2:20.3.0-5343.gd26e5c22.el9.noarch 2026-02-24T16:02:17.819 INFO:teuthology.orchestra.run.trial084.stdout: python3-cachetools-4.2.4-1.el9.noarch 2026-02-24T16:02:17.819 INFO:teuthology.orchestra.run.trial084.stdout: python3-certifi-2023.05.07-4.el9.noarch 2026-02-24T16:02:17.819 INFO:teuthology.orchestra.run.trial084.stdout: python3-google-auth-1:2.45.0-1.el9.noarch 2026-02-24T16:02:17.819 INFO:teuthology.orchestra.run.trial084.stdout: python3-kubernetes-1:26.1.0-3.el9.noarch 2026-02-24T16:02:17.820 INFO:teuthology.orchestra.run.trial084.stdout: python3-pyasn1-0.4.8-7.el9.noarch 2026-02-24T16:02:17.820 INFO:teuthology.orchestra.run.trial084.stdout: python3-pyasn1-modules-0.4.8-7.el9.noarch 2026-02-24T16:02:17.820 INFO:teuthology.orchestra.run.trial084.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2026-02-24T16:02:17.820 INFO:teuthology.orchestra.run.trial084.stdout: python3-rsa-4.9-2.el9.noarch 2026-02-24T16:02:17.820 INFO:teuthology.orchestra.run.trial084.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2026-02-24T16:02:17.820 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:02:17.820 INFO:teuthology.orchestra.run.trial084.stdout:Complete! 2026-02-24T16:02:17.849 DEBUG:teuthology.orchestra.run.trial084:> sudo yum -y install ceph-mgr-cephadm 2026-02-24T16:02:18.166 INFO:teuthology.orchestra.run.trial084.stdout:Last metadata expiration check: 0:13:55 ago on Tue 24 Feb 2026 03:48:23 PM UTC. 2026-02-24T16:02:18.252 INFO:teuthology.orchestra.run.trial084.stdout:Dependencies resolved. 2026-02-24T16:02:18.252 INFO:teuthology.orchestra.run.trial084.stdout:================================================================================ 2026-02-24T16:02:18.253 INFO:teuthology.orchestra.run.trial084.stdout: Package Arch Version Repository Size 2026-02-24T16:02:18.253 INFO:teuthology.orchestra.run.trial084.stdout:================================================================================ 2026-02-24T16:02:18.253 INFO:teuthology.orchestra.run.trial084.stdout:Installing: 2026-02-24T16:02:18.253 INFO:teuthology.orchestra.run.trial084.stdout: ceph-mgr-cephadm noarch 2:20.3.0-5343.gd26e5c22.el9 ceph-noarch 184 k 2026-02-24T16:02:18.253 INFO:teuthology.orchestra.run.trial084.stdout:Installing dependencies: 2026-02-24T16:02:18.253 INFO:teuthology.orchestra.run.trial084.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2026-02-24T16:02:18.253 INFO:teuthology.orchestra.run.trial084.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2026-02-24T16:02:18.253 INFO:teuthology.orchestra.run.trial084.stdout: python3-jinja2 noarch 2.11.3-8.el9 appstream 249 k 2026-02-24T16:02:18.253 INFO:teuthology.orchestra.run.trial084.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2026-02-24T16:02:18.254 INFO:teuthology.orchestra.run.trial084.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2026-02-24T16:02:18.254 INFO:teuthology.orchestra.run.trial084.stdout: python3-typing-extensions noarch 4.15.0-1.el9 epel 86 k 2026-02-24T16:02:18.254 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:02:18.254 INFO:teuthology.orchestra.run.trial084.stdout:Transaction Summary 2026-02-24T16:02:18.254 INFO:teuthology.orchestra.run.trial084.stdout:================================================================================ 2026-02-24T16:02:18.254 INFO:teuthology.orchestra.run.trial084.stdout:Install 7 Packages 2026-02-24T16:02:18.254 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:02:18.254 INFO:teuthology.orchestra.run.trial084.stdout:Total download size: 7.1 M 2026-02-24T16:02:18.254 INFO:teuthology.orchestra.run.trial084.stdout:Installed size: 34 M 2026-02-24T16:02:18.254 INFO:teuthology.orchestra.run.trial084.stdout:Downloading Packages: 2026-02-24T16:02:18.888 INFO:teuthology.orchestra.run.trial084.stdout:(1/7): python3-jinja2-2.11.3-8.el9.noarch.rpm 501 kB/s | 249 kB 00:00 2026-02-24T16:02:18.974 INFO:teuthology.orchestra.run.trial084.stdout:(2/7): python3-markupsafe-1.1.1-12.el9.x86_64.r 406 kB/s | 35 kB 00:00 2026-02-24T16:02:19.162 INFO:teuthology.orchestra.run.trial084.stdout:(3/7): ceph-mgr-cephadm-20.3.0-5343.gd26e5c22.e 239 kB/s | 184 kB 00:00 2026-02-24T16:02:19.163 INFO:teuthology.orchestra.run.trial084.stdout:(4/7): python3-asyncssh-2.13.2-5.el9.noarch.rpm 2.8 MB/s | 548 kB 00:00 2026-02-24T16:02:19.192 INFO:teuthology.orchestra.run.trial084.stdout:(5/7): python3-typing-extensions-4.15.0-1.el9.n 2.9 MB/s | 86 kB 00:00 2026-02-24T16:02:19.258 INFO:teuthology.orchestra.run.trial084.stdout:(6/7): python3-babel-2.9.1-2.el9.noarch.rpm 6.9 MB/s | 6.0 MB 00:00 2026-02-24T16:02:19.269 INFO:teuthology.orchestra.run.trial084.stdout:(7/7): python3-natsort-7.1.1-5.el9.noarch.rpm 541 kB/s | 58 kB 00:00 2026-02-24T16:02:19.269 INFO:teuthology.orchestra.run.trial084.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:02:19.269 INFO:teuthology.orchestra.run.trial084.stdout:Total 7.0 MB/s | 7.1 MB 00:01 2026-02-24T16:02:19.309 INFO:teuthology.orchestra.run.trial084.stdout:Running transaction check 2026-02-24T16:02:19.315 INFO:teuthology.orchestra.run.trial084.stdout:Transaction check succeeded. 2026-02-24T16:02:19.315 INFO:teuthology.orchestra.run.trial084.stdout:Running transaction test 2026-02-24T16:02:19.388 INFO:teuthology.orchestra.run.trial084.stdout:Transaction test succeeded. 2026-02-24T16:02:19.388 INFO:teuthology.orchestra.run.trial084.stdout:Running transaction 2026-02-24T16:02:19.478 INFO:teuthology.orchestra.run.trial084.stdout: Preparing : 1/1 2026-02-24T16:02:19.527 INFO:teuthology.orchestra.run.trial084.stdout: Installing : python3-typing-extensions-4.15.0-1.el9.noarch 1/7 2026-02-24T16:02:19.537 INFO:teuthology.orchestra.run.trial084.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/7 2026-02-24T16:02:19.543 INFO:teuthology.orchestra.run.trial084.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/7 2026-02-24T16:02:19.765 INFO:teuthology.orchestra.run.trial084.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 4/7 2026-02-24T16:02:19.788 INFO:teuthology.orchestra.run.trial084.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 5/7 2026-02-24T16:02:19.813 INFO:teuthology.orchestra.run.trial084.stdout: Installing : python3-jinja2-2.11.3-8.el9.noarch 6/7 2026-02-24T16:02:19.814 INFO:teuthology.orchestra.run.trial084.stdout: Installing : ceph-mgr-cephadm-2:20.3.0-5343.gd26e5c22.el9.noarch 7/7 2026-02-24T16:02:19.933 INFO:teuthology.orchestra.run.trial084.stdout: Running scriptlet: ceph-mgr-cephadm-2:20.3.0-5343.gd26e5c22.el9.noarch 7/7 2026-02-24T16:02:19.933 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : ceph-mgr-cephadm-2:20.3.0-5343.gd26e5c22.el9.noarch 1/7 2026-02-24T16:02:19.933 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/7 2026-02-24T16:02:19.933 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : python3-jinja2-2.11.3-8.el9.noarch 3/7 2026-02-24T16:02:19.933 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 4/7 2026-02-24T16:02:19.934 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 5/7 2026-02-24T16:02:19.934 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 6/7 2026-02-24T16:02:19.994 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : python3-typing-extensions-4.15.0-1.el9.noarch 7/7 2026-02-24T16:02:19.994 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:02:19.994 INFO:teuthology.orchestra.run.trial084.stdout:Installed: 2026-02-24T16:02:19.995 INFO:teuthology.orchestra.run.trial084.stdout: ceph-mgr-cephadm-2:20.3.0-5343.gd26e5c22.el9.noarch 2026-02-24T16:02:19.995 INFO:teuthology.orchestra.run.trial084.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2026-02-24T16:02:19.995 INFO:teuthology.orchestra.run.trial084.stdout: python3-babel-2.9.1-2.el9.noarch 2026-02-24T16:02:19.995 INFO:teuthology.orchestra.run.trial084.stdout: python3-jinja2-2.11.3-8.el9.noarch 2026-02-24T16:02:19.995 INFO:teuthology.orchestra.run.trial084.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2026-02-24T16:02:19.995 INFO:teuthology.orchestra.run.trial084.stdout: python3-natsort-7.1.1-5.el9.noarch 2026-02-24T16:02:19.995 INFO:teuthology.orchestra.run.trial084.stdout: python3-typing-extensions-4.15.0-1.el9.noarch 2026-02-24T16:02:19.995 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:02:19.995 INFO:teuthology.orchestra.run.trial084.stdout:Complete! 2026-02-24T16:02:20.026 DEBUG:teuthology.orchestra.run.trial084:> sudo yum -y install ceph-osd 2026-02-24T16:02:20.339 INFO:teuthology.orchestra.run.trial084.stdout:Last metadata expiration check: 0:13:57 ago on Tue 24 Feb 2026 03:48:23 PM UTC. 2026-02-24T16:02:20.404 INFO:teuthology.orchestra.run.trial084.stdout:Package ceph-osd-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:02:20.422 INFO:teuthology.orchestra.run.trial084.stdout:Dependencies resolved. 2026-02-24T16:02:20.423 INFO:teuthology.orchestra.run.trial084.stdout:Nothing to do. 2026-02-24T16:02:20.423 INFO:teuthology.orchestra.run.trial084.stdout:Complete! 2026-02-24T16:02:20.444 DEBUG:teuthology.orchestra.run.trial084:> sudo yum -y install ceph-osd-classic 2026-02-24T16:02:20.773 INFO:teuthology.orchestra.run.trial084.stdout:Last metadata expiration check: 0:13:57 ago on Tue 24 Feb 2026 03:48:23 PM UTC. 2026-02-24T16:02:20.838 INFO:teuthology.orchestra.run.trial084.stdout:Package ceph-osd-classic-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:02:20.857 INFO:teuthology.orchestra.run.trial084.stdout:Dependencies resolved. 2026-02-24T16:02:20.858 INFO:teuthology.orchestra.run.trial084.stdout:Nothing to do. 2026-02-24T16:02:20.858 INFO:teuthology.orchestra.run.trial084.stdout:Complete! 2026-02-24T16:02:20.879 DEBUG:teuthology.orchestra.run.trial084:> sudo yum -y install ceph-fuse 2026-02-24T16:02:21.196 INFO:teuthology.orchestra.run.trial084.stdout:Last metadata expiration check: 0:13:58 ago on Tue 24 Feb 2026 03:48:23 PM UTC. 2026-02-24T16:02:21.280 INFO:teuthology.orchestra.run.trial084.stdout:Dependencies resolved. 2026-02-24T16:02:21.280 INFO:teuthology.orchestra.run.trial084.stdout:================================================================================ 2026-02-24T16:02:21.280 INFO:teuthology.orchestra.run.trial084.stdout: Package Arch Version Repo Size 2026-02-24T16:02:21.280 INFO:teuthology.orchestra.run.trial084.stdout:================================================================================ 2026-02-24T16:02:21.281 INFO:teuthology.orchestra.run.trial084.stdout:Installing: 2026-02-24T16:02:21.281 INFO:teuthology.orchestra.run.trial084.stdout: ceph-fuse x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 914 k 2026-02-24T16:02:21.281 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:02:21.281 INFO:teuthology.orchestra.run.trial084.stdout:Transaction Summary 2026-02-24T16:02:21.281 INFO:teuthology.orchestra.run.trial084.stdout:================================================================================ 2026-02-24T16:02:21.281 INFO:teuthology.orchestra.run.trial084.stdout:Install 1 Package 2026-02-24T16:02:21.281 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:02:21.281 INFO:teuthology.orchestra.run.trial084.stdout:Total download size: 914 k 2026-02-24T16:02:21.281 INFO:teuthology.orchestra.run.trial084.stdout:Installed size: 2.9 M 2026-02-24T16:02:21.281 INFO:teuthology.orchestra.run.trial084.stdout:Downloading Packages: 2026-02-24T16:02:23.368 INFO:teuthology.orchestra.run.trial084.stdout:ceph-fuse-20.3.0-5343.gd26e5c22.el9.x86_64.rpm 438 kB/s | 914 kB 00:02 2026-02-24T16:02:23.369 INFO:teuthology.orchestra.run.trial084.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:02:23.369 INFO:teuthology.orchestra.run.trial084.stdout:Total 437 kB/s | 914 kB 00:02 2026-02-24T16:02:23.370 INFO:teuthology.orchestra.run.trial084.stdout:Running transaction check 2026-02-24T16:02:23.375 INFO:teuthology.orchestra.run.trial084.stdout:Transaction check succeeded. 2026-02-24T16:02:23.375 INFO:teuthology.orchestra.run.trial084.stdout:Running transaction test 2026-02-24T16:02:23.407 INFO:teuthology.orchestra.run.trial084.stdout:Transaction test succeeded. 2026-02-24T16:02:23.408 INFO:teuthology.orchestra.run.trial084.stdout:Running transaction 2026-02-24T16:02:23.498 INFO:teuthology.orchestra.run.trial084.stdout: Preparing : 1/1 2026-02-24T16:02:23.515 INFO:teuthology.orchestra.run.trial084.stdout: Installing : ceph-fuse-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/1 2026-02-24T16:02:23.748 INFO:teuthology.orchestra.run.trial084.stdout: Running scriptlet: ceph-fuse-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/1 2026-02-24T16:02:23.809 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : ceph-fuse-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/1 2026-02-24T16:02:23.809 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:02:23.810 INFO:teuthology.orchestra.run.trial084.stdout:Installed: 2026-02-24T16:02:23.810 INFO:teuthology.orchestra.run.trial084.stdout: ceph-fuse-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T16:02:23.810 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:02:23.810 INFO:teuthology.orchestra.run.trial084.stdout:Complete! 2026-02-24T16:02:23.837 DEBUG:teuthology.orchestra.run.trial084:> sudo yum -y install ceph-volume 2026-02-24T16:02:24.155 INFO:teuthology.orchestra.run.trial084.stdout:Last metadata expiration check: 0:14:01 ago on Tue 24 Feb 2026 03:48:23 PM UTC. 2026-02-24T16:02:24.239 INFO:teuthology.orchestra.run.trial084.stdout:Dependencies resolved. 2026-02-24T16:02:24.240 INFO:teuthology.orchestra.run.trial084.stdout:================================================================================ 2026-02-24T16:02:24.240 INFO:teuthology.orchestra.run.trial084.stdout: Package Arch Version Repository Size 2026-02-24T16:02:24.240 INFO:teuthology.orchestra.run.trial084.stdout:================================================================================ 2026-02-24T16:02:24.241 INFO:teuthology.orchestra.run.trial084.stdout:Installing: 2026-02-24T16:02:24.241 INFO:teuthology.orchestra.run.trial084.stdout: ceph-volume noarch 2:20.3.0-5343.gd26e5c22.el9 ceph-noarch 301 k 2026-02-24T16:02:24.241 INFO:teuthology.orchestra.run.trial084.stdout:Installing dependencies: 2026-02-24T16:02:24.241 INFO:teuthology.orchestra.run.trial084.stdout: cryptsetup x86_64 2.8.1-3.el9 baseos 351 k 2026-02-24T16:02:24.241 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:02:24.241 INFO:teuthology.orchestra.run.trial084.stdout:Transaction Summary 2026-02-24T16:02:24.241 INFO:teuthology.orchestra.run.trial084.stdout:================================================================================ 2026-02-24T16:02:24.241 INFO:teuthology.orchestra.run.trial084.stdout:Install 2 Packages 2026-02-24T16:02:24.241 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:02:24.241 INFO:teuthology.orchestra.run.trial084.stdout:Total download size: 652 k 2026-02-24T16:02:24.242 INFO:teuthology.orchestra.run.trial084.stdout:Installed size: 2.2 M 2026-02-24T16:02:24.242 INFO:teuthology.orchestra.run.trial084.stdout:Downloading Packages: 2026-02-24T16:02:24.323 INFO:teuthology.orchestra.run.trial084.stdout:(1/2): cryptsetup-2.8.1-3.el9.x86_64.rpm 16 MB/s | 351 kB 00:00 2026-02-24T16:02:25.560 INFO:teuthology.orchestra.run.trial084.stdout:(2/2): ceph-volume-20.3.0-5343.gd26e5c22.el9.no 240 kB/s | 301 kB 00:01 2026-02-24T16:02:25.560 INFO:teuthology.orchestra.run.trial084.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:02:25.560 INFO:teuthology.orchestra.run.trial084.stdout:Total 494 kB/s | 652 kB 00:01 2026-02-24T16:02:25.570 INFO:teuthology.orchestra.run.trial084.stdout:Running transaction check 2026-02-24T16:02:25.575 INFO:teuthology.orchestra.run.trial084.stdout:Transaction check succeeded. 2026-02-24T16:02:25.575 INFO:teuthology.orchestra.run.trial084.stdout:Running transaction test 2026-02-24T16:02:25.619 INFO:teuthology.orchestra.run.trial084.stdout:Transaction test succeeded. 2026-02-24T16:02:25.619 INFO:teuthology.orchestra.run.trial084.stdout:Running transaction 2026-02-24T16:02:25.701 INFO:teuthology.orchestra.run.trial084.stdout: Preparing : 1/1 2026-02-24T16:02:25.764 INFO:teuthology.orchestra.run.trial084.stdout: Installing : cryptsetup-2.8.1-3.el9.x86_64 1/2 2026-02-24T16:02:25.766 INFO:teuthology.orchestra.run.trial084.stdout: Installing : ceph-volume-2:20.3.0-5343.gd26e5c22.el9.noarch 2/2 2026-02-24T16:02:25.781 INFO:teuthology.orchestra.run.trial084.stdout: Running scriptlet: ceph-volume-2:20.3.0-5343.gd26e5c22.el9.noarch 2/2 2026-02-24T16:02:25.781 INFO:teuthology.orchestra.run.trial084.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T16:02:25.781 INFO:teuthology.orchestra.run.trial084.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2026-02-24T16:02:25.781 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:02:26.006 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : ceph-volume-2:20.3.0-5343.gd26e5c22.el9.noarch 1/2 2026-02-24T16:02:26.069 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : cryptsetup-2.8.1-3.el9.x86_64 2/2 2026-02-24T16:02:26.070 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:02:26.070 INFO:teuthology.orchestra.run.trial084.stdout:Installed: 2026-02-24T16:02:26.070 INFO:teuthology.orchestra.run.trial084.stdout: ceph-volume-2:20.3.0-5343.gd26e5c22.el9.noarch cryptsetup-2.8.1-3.el9.x86_64 2026-02-24T16:02:26.070 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:02:26.070 INFO:teuthology.orchestra.run.trial084.stdout:Complete! 2026-02-24T16:02:26.099 DEBUG:teuthology.orchestra.run.trial084:> sudo yum -y install librados-devel 2026-02-24T16:02:26.419 INFO:teuthology.orchestra.run.trial084.stdout:Last metadata expiration check: 0:14:03 ago on Tue 24 Feb 2026 03:48:23 PM UTC. 2026-02-24T16:02:26.503 INFO:teuthology.orchestra.run.trial084.stdout:Dependencies resolved. 2026-02-24T16:02:26.504 INFO:teuthology.orchestra.run.trial084.stdout:================================================================================ 2026-02-24T16:02:26.504 INFO:teuthology.orchestra.run.trial084.stdout: Package Arch Version Repo Size 2026-02-24T16:02:26.504 INFO:teuthology.orchestra.run.trial084.stdout:================================================================================ 2026-02-24T16:02:26.504 INFO:teuthology.orchestra.run.trial084.stdout:Installing: 2026-02-24T16:02:26.505 INFO:teuthology.orchestra.run.trial084.stdout: librados-devel x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 124 k 2026-02-24T16:02:26.505 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:02:26.505 INFO:teuthology.orchestra.run.trial084.stdout:Transaction Summary 2026-02-24T16:02:26.505 INFO:teuthology.orchestra.run.trial084.stdout:================================================================================ 2026-02-24T16:02:26.505 INFO:teuthology.orchestra.run.trial084.stdout:Install 1 Package 2026-02-24T16:02:26.505 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:02:26.505 INFO:teuthology.orchestra.run.trial084.stdout:Total download size: 124 k 2026-02-24T16:02:26.505 INFO:teuthology.orchestra.run.trial084.stdout:Installed size: 434 k 2026-02-24T16:02:26.505 INFO:teuthology.orchestra.run.trial084.stdout:Downloading Packages: 2026-02-24T16:02:27.141 INFO:teuthology.orchestra.run.trial084.stdout:librados-devel-20.3.0-5343.gd26e5c22.el9.x86_64 194 kB/s | 124 kB 00:00 2026-02-24T16:02:27.142 INFO:teuthology.orchestra.run.trial084.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:02:27.142 INFO:teuthology.orchestra.run.trial084.stdout:Total 194 kB/s | 124 kB 00:00 2026-02-24T16:02:27.142 INFO:teuthology.orchestra.run.trial084.stdout:Running transaction check 2026-02-24T16:02:27.145 INFO:teuthology.orchestra.run.trial084.stdout:Transaction check succeeded. 2026-02-24T16:02:27.145 INFO:teuthology.orchestra.run.trial084.stdout:Running transaction test 2026-02-24T16:02:27.175 INFO:teuthology.orchestra.run.trial084.stdout:Transaction test succeeded. 2026-02-24T16:02:27.176 INFO:teuthology.orchestra.run.trial084.stdout:Running transaction 2026-02-24T16:02:27.230 INFO:teuthology.orchestra.run.trial084.stdout: Preparing : 1/1 2026-02-24T16:02:27.247 INFO:teuthology.orchestra.run.trial084.stdout: Installing : librados-devel-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/1 2026-02-24T16:02:27.331 INFO:teuthology.orchestra.run.trial084.stdout: Running scriptlet: librados-devel-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/1 2026-02-24T16:02:27.394 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : librados-devel-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/1 2026-02-24T16:02:27.394 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:02:27.394 INFO:teuthology.orchestra.run.trial084.stdout:Installed: 2026-02-24T16:02:27.394 INFO:teuthology.orchestra.run.trial084.stdout: librados-devel-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T16:02:27.394 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:02:27.394 INFO:teuthology.orchestra.run.trial084.stdout:Complete! 2026-02-24T16:02:27.423 DEBUG:teuthology.orchestra.run.trial084:> sudo yum -y install libcephfs2 2026-02-24T16:02:27.741 INFO:teuthology.orchestra.run.trial084.stdout:Last metadata expiration check: 0:14:04 ago on Tue 24 Feb 2026 03:48:23 PM UTC. 2026-02-24T16:02:27.808 INFO:teuthology.orchestra.run.trial084.stdout:Package libcephfs2-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:02:27.827 INFO:teuthology.orchestra.run.trial084.stdout:Dependencies resolved. 2026-02-24T16:02:27.827 INFO:teuthology.orchestra.run.trial084.stdout:Nothing to do. 2026-02-24T16:02:27.827 INFO:teuthology.orchestra.run.trial084.stdout:Complete! 2026-02-24T16:02:27.849 DEBUG:teuthology.orchestra.run.trial084:> sudo yum -y install libcephfs-devel 2026-02-24T16:02:28.168 INFO:teuthology.orchestra.run.trial084.stdout:Last metadata expiration check: 0:14:05 ago on Tue 24 Feb 2026 03:48:23 PM UTC. 2026-02-24T16:02:28.252 INFO:teuthology.orchestra.run.trial084.stdout:Dependencies resolved. 2026-02-24T16:02:28.253 INFO:teuthology.orchestra.run.trial084.stdout:================================================================================ 2026-02-24T16:02:28.253 INFO:teuthology.orchestra.run.trial084.stdout: Package Arch Version Repo Size 2026-02-24T16:02:28.253 INFO:teuthology.orchestra.run.trial084.stdout:================================================================================ 2026-02-24T16:02:28.253 INFO:teuthology.orchestra.run.trial084.stdout:Installing: 2026-02-24T16:02:28.253 INFO:teuthology.orchestra.run.trial084.stdout: libcephfs-devel x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 37 k 2026-02-24T16:02:28.253 INFO:teuthology.orchestra.run.trial084.stdout:Installing dependencies: 2026-02-24T16:02:28.253 INFO:teuthology.orchestra.run.trial084.stdout: libcephfs-proxy2 x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 26 k 2026-02-24T16:02:28.253 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:02:28.254 INFO:teuthology.orchestra.run.trial084.stdout:Transaction Summary 2026-02-24T16:02:28.254 INFO:teuthology.orchestra.run.trial084.stdout:================================================================================ 2026-02-24T16:02:28.254 INFO:teuthology.orchestra.run.trial084.stdout:Install 2 Packages 2026-02-24T16:02:28.254 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:02:28.254 INFO:teuthology.orchestra.run.trial084.stdout:Total download size: 63 k 2026-02-24T16:02:28.254 INFO:teuthology.orchestra.run.trial084.stdout:Installed size: 227 k 2026-02-24T16:02:28.254 INFO:teuthology.orchestra.run.trial084.stdout:Downloading Packages: 2026-02-24T16:02:28.614 INFO:teuthology.orchestra.run.trial084.stdout:(1/2): libcephfs-proxy2-20.3.0-5343.gd26e5c22.e 71 kB/s | 26 kB 00:00 2026-02-24T16:02:28.689 INFO:teuthology.orchestra.run.trial084.stdout:(2/2): libcephfs-devel-20.3.0-5343.gd26e5c22.el 86 kB/s | 37 kB 00:00 2026-02-24T16:02:28.689 INFO:teuthology.orchestra.run.trial084.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:02:28.689 INFO:teuthology.orchestra.run.trial084.stdout:Total 144 kB/s | 63 kB 00:00 2026-02-24T16:02:28.689 INFO:teuthology.orchestra.run.trial084.stdout:Running transaction check 2026-02-24T16:02:28.692 INFO:teuthology.orchestra.run.trial084.stdout:Transaction check succeeded. 2026-02-24T16:02:28.692 INFO:teuthology.orchestra.run.trial084.stdout:Running transaction test 2026-02-24T16:02:28.724 INFO:teuthology.orchestra.run.trial084.stdout:Transaction test succeeded. 2026-02-24T16:02:28.724 INFO:teuthology.orchestra.run.trial084.stdout:Running transaction 2026-02-24T16:02:28.773 INFO:teuthology.orchestra.run.trial084.stdout: Preparing : 1/1 2026-02-24T16:02:28.774 INFO:teuthology.orchestra.run.trial084.stdout: Installing : libcephfs-proxy2-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/2 2026-02-24T16:02:28.789 INFO:teuthology.orchestra.run.trial084.stdout: Running scriptlet: libcephfs-proxy2-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/2 2026-02-24T16:02:28.806 INFO:teuthology.orchestra.run.trial084.stdout: Installing : libcephfs-devel-2:20.3.0-5343.gd26e5c22.el9.x86_64 2/2 2026-02-24T16:02:28.863 INFO:teuthology.orchestra.run.trial084.stdout: Running scriptlet: libcephfs-devel-2:20.3.0-5343.gd26e5c22.el9.x86_64 2/2 2026-02-24T16:02:28.863 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : libcephfs-devel-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/2 2026-02-24T16:02:28.926 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : libcephfs-proxy2-2:20.3.0-5343.gd26e5c22.el9.x86_64 2/2 2026-02-24T16:02:28.926 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:02:28.926 INFO:teuthology.orchestra.run.trial084.stdout:Installed: 2026-02-24T16:02:28.926 INFO:teuthology.orchestra.run.trial084.stdout: libcephfs-devel-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T16:02:28.926 INFO:teuthology.orchestra.run.trial084.stdout: libcephfs-proxy2-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T16:02:28.926 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:02:28.926 INFO:teuthology.orchestra.run.trial084.stdout:Complete! 2026-02-24T16:02:28.954 DEBUG:teuthology.orchestra.run.trial084:> sudo yum -y install librados2 2026-02-24T16:02:29.274 INFO:teuthology.orchestra.run.trial084.stdout:Last metadata expiration check: 0:14:06 ago on Tue 24 Feb 2026 03:48:23 PM UTC. 2026-02-24T16:02:29.339 INFO:teuthology.orchestra.run.trial084.stdout:Package librados2-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:02:29.358 INFO:teuthology.orchestra.run.trial084.stdout:Dependencies resolved. 2026-02-24T16:02:29.359 INFO:teuthology.orchestra.run.trial084.stdout:Nothing to do. 2026-02-24T16:02:29.359 INFO:teuthology.orchestra.run.trial084.stdout:Complete! 2026-02-24T16:02:29.380 DEBUG:teuthology.orchestra.run.trial084:> sudo yum -y install librbd1 2026-02-24T16:02:29.696 INFO:teuthology.orchestra.run.trial084.stdout:Last metadata expiration check: 0:14:06 ago on Tue 24 Feb 2026 03:48:23 PM UTC. 2026-02-24T16:02:29.761 INFO:teuthology.orchestra.run.trial084.stdout:Package librbd1-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:02:29.780 INFO:teuthology.orchestra.run.trial084.stdout:Dependencies resolved. 2026-02-24T16:02:29.781 INFO:teuthology.orchestra.run.trial084.stdout:Nothing to do. 2026-02-24T16:02:29.781 INFO:teuthology.orchestra.run.trial084.stdout:Complete! 2026-02-24T16:02:29.802 DEBUG:teuthology.orchestra.run.trial084:> sudo yum -y install python3-rados 2026-02-24T16:02:30.120 INFO:teuthology.orchestra.run.trial084.stdout:Last metadata expiration check: 0:14:07 ago on Tue 24 Feb 2026 03:48:23 PM UTC. 2026-02-24T16:02:30.184 INFO:teuthology.orchestra.run.trial084.stdout:Package python3-rados-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:02:30.203 INFO:teuthology.orchestra.run.trial084.stdout:Dependencies resolved. 2026-02-24T16:02:30.203 INFO:teuthology.orchestra.run.trial084.stdout:Nothing to do. 2026-02-24T16:02:30.203 INFO:teuthology.orchestra.run.trial084.stdout:Complete! 2026-02-24T16:02:30.225 DEBUG:teuthology.orchestra.run.trial084:> sudo yum -y install python3-rgw 2026-02-24T16:02:30.544 INFO:teuthology.orchestra.run.trial084.stdout:Last metadata expiration check: 0:14:07 ago on Tue 24 Feb 2026 03:48:23 PM UTC. 2026-02-24T16:02:30.609 INFO:teuthology.orchestra.run.trial084.stdout:Package python3-rgw-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:02:30.627 INFO:teuthology.orchestra.run.trial084.stdout:Dependencies resolved. 2026-02-24T16:02:30.628 INFO:teuthology.orchestra.run.trial084.stdout:Nothing to do. 2026-02-24T16:02:30.628 INFO:teuthology.orchestra.run.trial084.stdout:Complete! 2026-02-24T16:02:30.650 DEBUG:teuthology.orchestra.run.trial084:> sudo yum -y install python3-cephfs 2026-02-24T16:02:30.969 INFO:teuthology.orchestra.run.trial084.stdout:Last metadata expiration check: 0:14:07 ago on Tue 24 Feb 2026 03:48:23 PM UTC. 2026-02-24T16:02:31.035 INFO:teuthology.orchestra.run.trial084.stdout:Package python3-cephfs-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:02:31.053 INFO:teuthology.orchestra.run.trial084.stdout:Dependencies resolved. 2026-02-24T16:02:31.054 INFO:teuthology.orchestra.run.trial084.stdout:Nothing to do. 2026-02-24T16:02:31.054 INFO:teuthology.orchestra.run.trial084.stdout:Complete! 2026-02-24T16:02:31.075 DEBUG:teuthology.orchestra.run.trial084:> sudo yum -y install python3-rbd 2026-02-24T16:02:31.394 INFO:teuthology.orchestra.run.trial084.stdout:Last metadata expiration check: 0:14:08 ago on Tue 24 Feb 2026 03:48:23 PM UTC. 2026-02-24T16:02:31.459 INFO:teuthology.orchestra.run.trial084.stdout:Package python3-rbd-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:02:31.477 INFO:teuthology.orchestra.run.trial084.stdout:Dependencies resolved. 2026-02-24T16:02:31.478 INFO:teuthology.orchestra.run.trial084.stdout:Nothing to do. 2026-02-24T16:02:31.478 INFO:teuthology.orchestra.run.trial084.stdout:Complete! 2026-02-24T16:02:31.499 DEBUG:teuthology.orchestra.run.trial084:> sudo yum -y install rbd-fuse 2026-02-24T16:02:31.815 INFO:teuthology.orchestra.run.trial084.stdout:Last metadata expiration check: 0:14:08 ago on Tue 24 Feb 2026 03:48:23 PM UTC. 2026-02-24T16:02:31.899 INFO:teuthology.orchestra.run.trial084.stdout:Dependencies resolved. 2026-02-24T16:02:31.899 INFO:teuthology.orchestra.run.trial084.stdout:================================================================================ 2026-02-24T16:02:31.899 INFO:teuthology.orchestra.run.trial084.stdout: Package Architecture Version Repository Size 2026-02-24T16:02:31.899 INFO:teuthology.orchestra.run.trial084.stdout:================================================================================ 2026-02-24T16:02:31.899 INFO:teuthology.orchestra.run.trial084.stdout:Installing: 2026-02-24T16:02:31.899 INFO:teuthology.orchestra.run.trial084.stdout: rbd-fuse x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 86 k 2026-02-24T16:02:31.899 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:02:31.899 INFO:teuthology.orchestra.run.trial084.stdout:Transaction Summary 2026-02-24T16:02:31.899 INFO:teuthology.orchestra.run.trial084.stdout:================================================================================ 2026-02-24T16:02:31.899 INFO:teuthology.orchestra.run.trial084.stdout:Install 1 Package 2026-02-24T16:02:31.899 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:02:31.900 INFO:teuthology.orchestra.run.trial084.stdout:Total download size: 86 k 2026-02-24T16:02:31.900 INFO:teuthology.orchestra.run.trial084.stdout:Installed size: 236 k 2026-02-24T16:02:31.900 INFO:teuthology.orchestra.run.trial084.stdout:Downloading Packages: 2026-02-24T16:02:32.609 INFO:teuthology.orchestra.run.trial084.stdout:rbd-fuse-20.3.0-5343.gd26e5c22.el9.x86_64.rpm 122 kB/s | 86 kB 00:00 2026-02-24T16:02:32.611 INFO:teuthology.orchestra.run.trial084.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:02:32.611 INFO:teuthology.orchestra.run.trial084.stdout:Total 121 kB/s | 86 kB 00:00 2026-02-24T16:02:32.611 INFO:teuthology.orchestra.run.trial084.stdout:Running transaction check 2026-02-24T16:02:32.616 INFO:teuthology.orchestra.run.trial084.stdout:Transaction check succeeded. 2026-02-24T16:02:32.616 INFO:teuthology.orchestra.run.trial084.stdout:Running transaction test 2026-02-24T16:02:32.646 INFO:teuthology.orchestra.run.trial084.stdout:Transaction test succeeded. 2026-02-24T16:02:32.646 INFO:teuthology.orchestra.run.trial084.stdout:Running transaction 2026-02-24T16:02:32.697 INFO:teuthology.orchestra.run.trial084.stdout: Preparing : 1/1 2026-02-24T16:02:32.715 INFO:teuthology.orchestra.run.trial084.stdout: Installing : rbd-fuse-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/1 2026-02-24T16:02:32.806 INFO:teuthology.orchestra.run.trial084.stdout: Running scriptlet: rbd-fuse-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/1 2026-02-24T16:02:32.868 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : rbd-fuse-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/1 2026-02-24T16:02:32.868 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:02:32.868 INFO:teuthology.orchestra.run.trial084.stdout:Installed: 2026-02-24T16:02:32.868 INFO:teuthology.orchestra.run.trial084.stdout: rbd-fuse-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T16:02:32.868 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:02:32.868 INFO:teuthology.orchestra.run.trial084.stdout:Complete! 2026-02-24T16:02:32.896 DEBUG:teuthology.orchestra.run.trial084:> sudo yum -y install rbd-mirror 2026-02-24T16:02:33.213 INFO:teuthology.orchestra.run.trial084.stdout:Last metadata expiration check: 0:14:10 ago on Tue 24 Feb 2026 03:48:23 PM UTC. 2026-02-24T16:02:33.299 INFO:teuthology.orchestra.run.trial084.stdout:Dependencies resolved. 2026-02-24T16:02:33.300 INFO:teuthology.orchestra.run.trial084.stdout:================================================================================ 2026-02-24T16:02:33.300 INFO:teuthology.orchestra.run.trial084.stdout: Package Arch Version Repo Size 2026-02-24T16:02:33.300 INFO:teuthology.orchestra.run.trial084.stdout:================================================================================ 2026-02-24T16:02:33.300 INFO:teuthology.orchestra.run.trial084.stdout:Installing: 2026-02-24T16:02:33.300 INFO:teuthology.orchestra.run.trial084.stdout: rbd-mirror x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 2.6 M 2026-02-24T16:02:33.300 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:02:33.300 INFO:teuthology.orchestra.run.trial084.stdout:Transaction Summary 2026-02-24T16:02:33.300 INFO:teuthology.orchestra.run.trial084.stdout:================================================================================ 2026-02-24T16:02:33.301 INFO:teuthology.orchestra.run.trial084.stdout:Install 1 Package 2026-02-24T16:02:33.301 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:02:33.301 INFO:teuthology.orchestra.run.trial084.stdout:Total download size: 2.6 M 2026-02-24T16:02:33.301 INFO:teuthology.orchestra.run.trial084.stdout:Installed size: 11 M 2026-02-24T16:02:33.301 INFO:teuthology.orchestra.run.trial084.stdout:Downloading Packages: 2026-02-24T16:02:44.321 INFO:teuthology.orchestra.run.trial084.stdout:rbd-mirror-20.3.0-5343.gd26e5c22.el9.x86_64.rpm 241 kB/s | 2.6 MB 00:11 2026-02-24T16:02:44.322 INFO:teuthology.orchestra.run.trial084.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:02:44.322 INFO:teuthology.orchestra.run.trial084.stdout:Total 241 kB/s | 2.6 MB 00:11 2026-02-24T16:02:44.322 INFO:teuthology.orchestra.run.trial084.stdout:Running transaction check 2026-02-24T16:02:44.327 INFO:teuthology.orchestra.run.trial084.stdout:Transaction check succeeded. 2026-02-24T16:02:44.328 INFO:teuthology.orchestra.run.trial084.stdout:Running transaction test 2026-02-24T16:02:44.365 INFO:teuthology.orchestra.run.trial084.stdout:Transaction test succeeded. 2026-02-24T16:02:44.365 INFO:teuthology.orchestra.run.trial084.stdout:Running transaction 2026-02-24T16:02:44.532 INFO:teuthology.orchestra.run.trial084.stdout: Preparing : 1/1 2026-02-24T16:02:44.534 INFO:teuthology.orchestra.run.trial084.stdout: Installing : rbd-mirror-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/1 2026-02-24T16:02:44.558 INFO:teuthology.orchestra.run.trial084.stdout: Running scriptlet: rbd-mirror-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/1 2026-02-24T16:02:44.558 INFO:teuthology.orchestra.run.trial084.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T16:02:44.558 INFO:teuthology.orchestra.run.trial084.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2026-02-24T16:02:44.558 INFO:teuthology.orchestra.run.trial084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-24T16:02:44.559 INFO:teuthology.orchestra.run.trial084.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-24T16:02:44.559 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:02:44.855 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : rbd-mirror-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/1 2026-02-24T16:02:44.855 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:02:44.855 INFO:teuthology.orchestra.run.trial084.stdout:Installed: 2026-02-24T16:02:44.855 INFO:teuthology.orchestra.run.trial084.stdout: rbd-mirror-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T16:02:44.855 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:02:44.855 INFO:teuthology.orchestra.run.trial084.stdout:Complete! 2026-02-24T16:02:44.882 DEBUG:teuthology.orchestra.run.trial084:> sudo yum -y install rbd-nbd 2026-02-24T16:02:45.202 INFO:teuthology.orchestra.run.trial084.stdout:Last metadata expiration check: 0:14:22 ago on Tue 24 Feb 2026 03:48:23 PM UTC. 2026-02-24T16:02:45.285 INFO:teuthology.orchestra.run.trial084.stdout:Dependencies resolved. 2026-02-24T16:02:45.286 INFO:teuthology.orchestra.run.trial084.stdout:================================================================================ 2026-02-24T16:02:45.287 INFO:teuthology.orchestra.run.trial084.stdout: Package Architecture Version Repository Size 2026-02-24T16:02:45.287 INFO:teuthology.orchestra.run.trial084.stdout:================================================================================ 2026-02-24T16:02:45.287 INFO:teuthology.orchestra.run.trial084.stdout:Installing: 2026-02-24T16:02:45.287 INFO:teuthology.orchestra.run.trial084.stdout: rbd-nbd x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 167 k 2026-02-24T16:02:45.287 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:02:45.287 INFO:teuthology.orchestra.run.trial084.stdout:Transaction Summary 2026-02-24T16:02:45.287 INFO:teuthology.orchestra.run.trial084.stdout:================================================================================ 2026-02-24T16:02:45.287 INFO:teuthology.orchestra.run.trial084.stdout:Install 1 Package 2026-02-24T16:02:45.287 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:02:45.287 INFO:teuthology.orchestra.run.trial084.stdout:Total download size: 167 k 2026-02-24T16:02:45.288 INFO:teuthology.orchestra.run.trial084.stdout:Installed size: 496 k 2026-02-24T16:02:45.288 INFO:teuthology.orchestra.run.trial084.stdout:Downloading Packages: 2026-02-24T16:02:46.195 INFO:teuthology.orchestra.run.trial084.stdout:rbd-nbd-20.3.0-5343.gd26e5c22.el9.x86_64.rpm 183 kB/s | 167 kB 00:00 2026-02-24T16:02:46.195 INFO:teuthology.orchestra.run.trial084.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:02:46.195 INFO:teuthology.orchestra.run.trial084.stdout:Total 183 kB/s | 167 kB 00:00 2026-02-24T16:02:46.195 INFO:teuthology.orchestra.run.trial084.stdout:Running transaction check 2026-02-24T16:02:46.200 INFO:teuthology.orchestra.run.trial084.stdout:Transaction check succeeded. 2026-02-24T16:02:46.200 INFO:teuthology.orchestra.run.trial084.stdout:Running transaction test 2026-02-24T16:02:46.231 INFO:teuthology.orchestra.run.trial084.stdout:Transaction test succeeded. 2026-02-24T16:02:46.231 INFO:teuthology.orchestra.run.trial084.stdout:Running transaction 2026-02-24T16:02:46.287 INFO:teuthology.orchestra.run.trial084.stdout: Preparing : 1/1 2026-02-24T16:02:46.304 INFO:teuthology.orchestra.run.trial084.stdout: Installing : rbd-nbd-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/1 2026-02-24T16:02:46.388 INFO:teuthology.orchestra.run.trial084.stdout: Running scriptlet: rbd-nbd-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/1 2026-02-24T16:02:46.450 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : rbd-nbd-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/1 2026-02-24T16:02:46.450 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:02:46.450 INFO:teuthology.orchestra.run.trial084.stdout:Installed: 2026-02-24T16:02:46.450 INFO:teuthology.orchestra.run.trial084.stdout: rbd-nbd-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T16:02:46.450 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:02:46.450 INFO:teuthology.orchestra.run.trial084.stdout:Complete! 2026-02-24T16:02:46.480 DEBUG:teuthology.orchestra.run.trial084:> sudo yum -y install ceph-osd-crimson 2026-02-24T16:02:46.799 INFO:teuthology.orchestra.run.trial084.stdout:Last metadata expiration check: 0:14:23 ago on Tue 24 Feb 2026 03:48:23 PM UTC. 2026-02-24T16:02:46.864 INFO:teuthology.orchestra.run.trial084.stdout:Package ceph-osd-crimson-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:02:46.883 INFO:teuthology.orchestra.run.trial084.stdout:Dependencies resolved. 2026-02-24T16:02:46.883 INFO:teuthology.orchestra.run.trial084.stdout:Nothing to do. 2026-02-24T16:02:46.883 INFO:teuthology.orchestra.run.trial084.stdout:Complete! 2026-02-24T16:02:46.906 DEBUG:teuthology.orchestra.run.trial084:> sudo yum -y install ceph-osd-crimson-debuginfo 2026-02-24T16:02:47.223 INFO:teuthology.orchestra.run.trial084.stdout:Last metadata expiration check: 0:14:24 ago on Tue 24 Feb 2026 03:48:23 PM UTC. 2026-02-24T16:02:47.308 INFO:teuthology.orchestra.run.trial084.stdout:Dependencies resolved. 2026-02-24T16:02:47.309 INFO:teuthology.orchestra.run.trial084.stdout:================================================================================ 2026-02-24T16:02:47.309 INFO:teuthology.orchestra.run.trial084.stdout: Package Arch Version Repo Size 2026-02-24T16:02:47.309 INFO:teuthology.orchestra.run.trial084.stdout:================================================================================ 2026-02-24T16:02:47.309 INFO:teuthology.orchestra.run.trial084.stdout:Installing: 2026-02-24T16:02:47.309 INFO:teuthology.orchestra.run.trial084.stdout: ceph-osd-crimson-debuginfo x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 433 M 2026-02-24T16:02:47.309 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:02:47.310 INFO:teuthology.orchestra.run.trial084.stdout:Transaction Summary 2026-02-24T16:02:47.310 INFO:teuthology.orchestra.run.trial084.stdout:================================================================================ 2026-02-24T16:02:47.310 INFO:teuthology.orchestra.run.trial084.stdout:Install 1 Package 2026-02-24T16:02:47.310 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:02:47.310 INFO:teuthology.orchestra.run.trial084.stdout:Total download size: 433 M 2026-02-24T16:02:47.310 INFO:teuthology.orchestra.run.trial084.stdout:Installed size: 2.6 G 2026-02-24T16:02:47.310 INFO:teuthology.orchestra.run.trial084.stdout:Downloading Packages: 2026-02-24T16:14:34.402 INFO:teuthology.orchestra.run.trial084.stdout:ceph-osd-crimson-debuginfo-20.3.0-5343.gd26e5c2 627 kB/s | 433 MB 11:47 2026-02-24T16:14:34.403 INFO:teuthology.orchestra.run.trial084.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:14:34.403 INFO:teuthology.orchestra.run.trial084.stdout:Total 627 kB/s | 433 MB 11:47 2026-02-24T16:14:34.404 INFO:teuthology.orchestra.run.trial084.stdout:Running transaction check 2026-02-24T16:14:34.405 INFO:teuthology.orchestra.run.trial084.stdout:Transaction check succeeded. 2026-02-24T16:14:34.405 INFO:teuthology.orchestra.run.trial084.stdout:Running transaction test 2026-02-24T16:14:35.701 INFO:teuthology.orchestra.run.trial084.stdout:Transaction test succeeded. 2026-02-24T16:14:35.701 INFO:teuthology.orchestra.run.trial084.stdout:Running transaction 2026-02-24T16:14:59.921 INFO:teuthology.orchestra.run.trial084.stdout: Preparing : 1/1 2026-02-24T16:14:59.939 INFO:teuthology.orchestra.run.trial084.stdout: Installing : ceph-osd-crimson-debuginfo-2:20.3.0-5343.gd26e5c22.e 1/1 2026-02-24T16:15:01.900 INFO:teuthology.orchestra.run.trial084.stdout: Running scriptlet: ceph-osd-crimson-debuginfo-2:20.3.0-5343.gd26e5c22.e 1/1 2026-02-24T16:15:01.962 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : ceph-osd-crimson-debuginfo-2:20.3.0-5343.gd26e5c22.e 1/1 2026-02-24T16:15:01.962 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:15:01.962 INFO:teuthology.orchestra.run.trial084.stdout:Installed: 2026-02-24T16:15:01.962 INFO:teuthology.orchestra.run.trial084.stdout: ceph-osd-crimson-debuginfo-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T16:15:01.962 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:15:01.962 INFO:teuthology.orchestra.run.trial084.stdout:Complete! 2026-02-24T16:15:02.033 DEBUG:teuthology.parallel:result is None 2026-02-24T16:15:02.034 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=d26e5c228e3cdea4d585bcd077f8513a649ab5d1 2026-02-24T16:15:02.175 DEBUG:teuthology.orchestra.run.trial084:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-24T16:15:02.196 INFO:teuthology.orchestra.run.trial084.stdout:20.3.0-5343.gd26e5c22.el9 2026-02-24T16:15:02.196 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5343.gd26e5c22.el9 2026-02-24T16:15:02.196 INFO:teuthology.task.install:The correct ceph version 20.3.0-5343.gd26e5c22 is installed. 2026-02-24T16:15:02.198 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2026-02-24T16:15:02.198 DEBUG:teuthology.orchestra.run.trial084:> set -ex 2026-02-24T16:15:02.198 DEBUG:teuthology.orchestra.run.trial084:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-24T16:15:02.262 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2026-02-24T16:15:02.262 DEBUG:teuthology.orchestra.run.trial084:> set -ex 2026-02-24T16:15:02.263 DEBUG:teuthology.orchestra.run.trial084:> sudo dd of=/usr/bin/daemon-helper 2026-02-24T16:15:02.326 DEBUG:teuthology.orchestra.run.trial084:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-24T16:15:02.390 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2026-02-24T16:15:02.390 DEBUG:teuthology.orchestra.run.trial084:> set -ex 2026-02-24T16:15:02.390 DEBUG:teuthology.orchestra.run.trial084:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-24T16:15:02.455 DEBUG:teuthology.orchestra.run.trial084:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-24T16:15:02.518 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2026-02-24T16:15:02.519 DEBUG:teuthology.orchestra.run.trial084:> set -ex 2026-02-24T16:15:02.519 DEBUG:teuthology.orchestra.run.trial084:> sudo dd of=/usr/bin/stdin-killer 2026-02-24T16:15:02.582 DEBUG:teuthology.orchestra.run.trial084:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-24T16:15:02.651 INFO:teuthology.run_tasks:Running task pexec... 2026-02-24T16:15:02.676 INFO:teuthology.task.pexec:Executing custom commands... 2026-02-24T16:15:02.677 DEBUG:teuthology.orchestra.run.trial084:> TESTDIR=/home/ubuntu/cephtest bash -s 2026-02-24T16:15:02.698 INFO:teuthology.task.pexec:Running commands on host ubuntu@trial084.front.sepia.ceph.com 2026-02-24T16:15:02.698 INFO:teuthology.task.pexec:sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson 2026-02-24T16:15:02.749 DEBUG:teuthology.parallel:result is None 2026-02-24T16:15:02.750 INFO:teuthology.run_tasks:Running task install... 2026-02-24T16:15:02.764 DEBUG:teuthology.task.install:project ceph 2026-02-24T16:15:02.764 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'debug', 'sha1': 'd26e5c228e3cdea4d585bcd077f8513a649ab5d1'}} 2026-02-24T16:15:02.764 DEBUG:teuthology.task.install:config {'flavor': 'debug', 'sha1': 'd26e5c228e3cdea4d585bcd077f8513a649ab5d1'} 2026-02-24T16:15:02.764 INFO:teuthology.task.install:Using flavor: debug 2026-02-24T16:15:02.773 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-osd', 'ceph-osd-classic', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-osd', 'ceph-osd-classic', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-24T16:15:02.773 INFO:teuthology.task.install:extra packages: [] 2026-02-24T16:15:02.774 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'debug', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd26e5c228e3cdea4d585bcd077f8513a649ab5d1', 'tag': None, 'wait_for_package': False} 2026-02-24T16:15:02.774 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=d26e5c228e3cdea4d585bcd077f8513a649ab5d1 2026-02-24T16:15:02.932 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-liucm-shard-redesign-crimson-only/d26e5c228e3cdea4d585bcd077f8513a649ab5d1/centos/9/flavors/debug/ 2026-02-24T16:15:02.932 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5343.gd26e5c22 2026-02-24T16:15:03.220 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-liucm-shard-redesign-crimson-only/d26e5c228e3cdea4d585bcd077f8513a649ab5d1/centos/9/flavors/debug/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-liucm-shard-redesign-crimson-only/d26e5c228e3cdea4d585bcd077f8513a649ab5d1/centos/9/flavors/debug/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-liucm-shard-redesign-crimson-only/d26e5c228e3cdea4d585bcd077f8513a649ab5d1/centos/9/flavors/debug/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-24T16:15:03.220 DEBUG:teuthology.orchestra.run.trial084:> set -ex 2026-02-24T16:15:03.220 DEBUG:teuthology.orchestra.run.trial084:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-24T16:15:03.245 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2026-02-24T16:15:03.245 DEBUG:teuthology.orchestra.run.trial084:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d26e5c228e3cdea4d585bcd077f8513a649ab5d1/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-24T16:15:03.309 DEBUG:teuthology.orchestra.run.trial084:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-24T16:15:03.373 DEBUG:teuthology.orchestra.run.trial084:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-24T16:15:03.428 INFO:teuthology.orchestra.run.trial084.stdout:check_obsoletes = 1 2026-02-24T16:15:03.438 DEBUG:teuthology.orchestra.run.trial084:> sudo yum clean all 2026-02-24T16:15:03.598 INFO:teuthology.orchestra.run.trial084.stdout:56 files removed 2026-02-24T16:15:03.612 DEBUG:teuthology.orchestra.run.trial084:> sudo yum -y install ceph-radosgw 2026-02-24T16:15:04.324 INFO:teuthology.orchestra.run.trial084.stdout:ceph packages for x86_64 164 kB/s | 92 kB 00:00 2026-02-24T16:15:04.882 INFO:teuthology.orchestra.run.trial084.stdout:ceph noarch packages 45 kB/s | 24 kB 00:00 2026-02-24T16:15:05.283 INFO:teuthology.orchestra.run.trial084.stdout:ceph source packages 5.5 kB/s | 2.1 kB 00:00 2026-02-24T16:15:07.497 INFO:teuthology.orchestra.run.trial084.stdout:CentOS Stream 9 - BaseOS 4.1 MB/s | 8.9 MB 00:02 2026-02-24T16:15:10.583 INFO:teuthology.orchestra.run.trial084.stdout:CentOS Stream 9 - AppStream 11 MB/s | 27 MB 00:02 2026-02-24T16:15:13.274 INFO:teuthology.orchestra.run.trial084.stdout:CentOS Stream 9 - CRB 28 MB/s | 8.0 MB 00:00 2026-02-24T16:15:15.152 INFO:teuthology.orchestra.run.trial084.stdout:CentOS Stream 9 - Extras packages 18 kB/s | 20 kB 00:01 2026-02-24T16:15:15.558 INFO:teuthology.orchestra.run.trial084.stdout:Extra Packages for Enterprise Linux 59 MB/s | 20 MB 00:00 2026-02-24T16:15:18.842 INFO:teuthology.orchestra.run.trial084.stdout:lab-extras 2.5 MB/s | 50 kB 00:00 2026-02-24T16:15:20.035 INFO:teuthology.orchestra.run.trial084.stdout:Package ceph-radosgw-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:15:20.054 INFO:teuthology.orchestra.run.trial084.stdout:Dependencies resolved. 2026-02-24T16:15:20.055 INFO:teuthology.orchestra.run.trial084.stdout:Nothing to do. 2026-02-24T16:15:20.055 INFO:teuthology.orchestra.run.trial084.stdout:Complete! 2026-02-24T16:15:20.080 DEBUG:teuthology.orchestra.run.trial084:> sudo yum -y install ceph-test 2026-02-24T16:15:20.402 INFO:teuthology.orchestra.run.trial084.stdout:Last metadata expiration check: 0:00:02 ago on Tue 24 Feb 2026 04:15:18 PM UTC. 2026-02-24T16:15:20.469 INFO:teuthology.orchestra.run.trial084.stdout:Package ceph-test-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:15:20.487 INFO:teuthology.orchestra.run.trial084.stdout:Dependencies resolved. 2026-02-24T16:15:20.488 INFO:teuthology.orchestra.run.trial084.stdout:Nothing to do. 2026-02-24T16:15:20.488 INFO:teuthology.orchestra.run.trial084.stdout:Complete! 2026-02-24T16:15:20.511 DEBUG:teuthology.orchestra.run.trial084:> sudo yum -y install ceph 2026-02-24T16:15:20.830 INFO:teuthology.orchestra.run.trial084.stdout:Last metadata expiration check: 0:00:02 ago on Tue 24 Feb 2026 04:15:18 PM UTC. 2026-02-24T16:15:20.894 INFO:teuthology.orchestra.run.trial084.stdout:Package ceph-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:15:20.912 INFO:teuthology.orchestra.run.trial084.stdout:Dependencies resolved. 2026-02-24T16:15:20.913 INFO:teuthology.orchestra.run.trial084.stdout:Nothing to do. 2026-02-24T16:15:20.913 INFO:teuthology.orchestra.run.trial084.stdout:Complete! 2026-02-24T16:15:20.935 DEBUG:teuthology.orchestra.run.trial084:> sudo yum -y install ceph-base 2026-02-24T16:15:21.254 INFO:teuthology.orchestra.run.trial084.stdout:Last metadata expiration check: 0:00:03 ago on Tue 24 Feb 2026 04:15:18 PM UTC. 2026-02-24T16:15:21.320 INFO:teuthology.orchestra.run.trial084.stdout:Package ceph-base-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:15:21.338 INFO:teuthology.orchestra.run.trial084.stdout:Dependencies resolved. 2026-02-24T16:15:21.339 INFO:teuthology.orchestra.run.trial084.stdout:Nothing to do. 2026-02-24T16:15:21.339 INFO:teuthology.orchestra.run.trial084.stdout:Complete! 2026-02-24T16:15:21.361 DEBUG:teuthology.orchestra.run.trial084:> sudo yum -y install cephadm 2026-02-24T16:15:21.682 INFO:teuthology.orchestra.run.trial084.stdout:Last metadata expiration check: 0:00:03 ago on Tue 24 Feb 2026 04:15:18 PM UTC. 2026-02-24T16:15:21.748 INFO:teuthology.orchestra.run.trial084.stdout:Package cephadm-2:20.3.0-5343.gd26e5c22.el9.noarch is already installed. 2026-02-24T16:15:21.766 INFO:teuthology.orchestra.run.trial084.stdout:Dependencies resolved. 2026-02-24T16:15:21.767 INFO:teuthology.orchestra.run.trial084.stdout:Nothing to do. 2026-02-24T16:15:21.767 INFO:teuthology.orchestra.run.trial084.stdout:Complete! 2026-02-24T16:15:21.789 DEBUG:teuthology.orchestra.run.trial084:> sudo yum -y install ceph-immutable-object-cache 2026-02-24T16:15:22.107 INFO:teuthology.orchestra.run.trial084.stdout:Last metadata expiration check: 0:00:04 ago on Tue 24 Feb 2026 04:15:18 PM UTC. 2026-02-24T16:15:22.172 INFO:teuthology.orchestra.run.trial084.stdout:Package ceph-immutable-object-cache-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:15:22.191 INFO:teuthology.orchestra.run.trial084.stdout:Dependencies resolved. 2026-02-24T16:15:22.192 INFO:teuthology.orchestra.run.trial084.stdout:Nothing to do. 2026-02-24T16:15:22.192 INFO:teuthology.orchestra.run.trial084.stdout:Complete! 2026-02-24T16:15:22.214 DEBUG:teuthology.orchestra.run.trial084:> sudo yum -y install ceph-mgr 2026-02-24T16:15:22.537 INFO:teuthology.orchestra.run.trial084.stdout:Last metadata expiration check: 0:00:04 ago on Tue 24 Feb 2026 04:15:18 PM UTC. 2026-02-24T16:15:22.601 INFO:teuthology.orchestra.run.trial084.stdout:Package ceph-mgr-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:15:22.620 INFO:teuthology.orchestra.run.trial084.stdout:Dependencies resolved. 2026-02-24T16:15:22.621 INFO:teuthology.orchestra.run.trial084.stdout:Nothing to do. 2026-02-24T16:15:22.621 INFO:teuthology.orchestra.run.trial084.stdout:Complete! 2026-02-24T16:15:22.642 DEBUG:teuthology.orchestra.run.trial084:> sudo yum -y install ceph-mgr-dashboard 2026-02-24T16:15:22.964 INFO:teuthology.orchestra.run.trial084.stdout:Last metadata expiration check: 0:00:04 ago on Tue 24 Feb 2026 04:15:18 PM UTC. 2026-02-24T16:15:23.030 INFO:teuthology.orchestra.run.trial084.stdout:Package ceph-mgr-dashboard-2:20.3.0-5343.gd26e5c22.el9.noarch is already installed. 2026-02-24T16:15:23.048 INFO:teuthology.orchestra.run.trial084.stdout:Dependencies resolved. 2026-02-24T16:15:23.049 INFO:teuthology.orchestra.run.trial084.stdout:Nothing to do. 2026-02-24T16:15:23.049 INFO:teuthology.orchestra.run.trial084.stdout:Complete! 2026-02-24T16:15:23.072 DEBUG:teuthology.orchestra.run.trial084:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-24T16:15:23.393 INFO:teuthology.orchestra.run.trial084.stdout:Last metadata expiration check: 0:00:05 ago on Tue 24 Feb 2026 04:15:18 PM UTC. 2026-02-24T16:15:23.459 INFO:teuthology.orchestra.run.trial084.stdout:Package ceph-mgr-diskprediction-local-2:20.3.0-5343.gd26e5c22.el9.noarch is already installed. 2026-02-24T16:15:23.478 INFO:teuthology.orchestra.run.trial084.stdout:Dependencies resolved. 2026-02-24T16:15:23.478 INFO:teuthology.orchestra.run.trial084.stdout:Nothing to do. 2026-02-24T16:15:23.478 INFO:teuthology.orchestra.run.trial084.stdout:Complete! 2026-02-24T16:15:23.499 DEBUG:teuthology.orchestra.run.trial084:> sudo yum -y install ceph-mgr-rook 2026-02-24T16:15:23.818 INFO:teuthology.orchestra.run.trial084.stdout:Last metadata expiration check: 0:00:05 ago on Tue 24 Feb 2026 04:15:18 PM UTC. 2026-02-24T16:15:23.883 INFO:teuthology.orchestra.run.trial084.stdout:Package ceph-mgr-rook-2:20.3.0-5343.gd26e5c22.el9.noarch is already installed. 2026-02-24T16:15:23.902 INFO:teuthology.orchestra.run.trial084.stdout:Dependencies resolved. 2026-02-24T16:15:23.902 INFO:teuthology.orchestra.run.trial084.stdout:Nothing to do. 2026-02-24T16:15:23.902 INFO:teuthology.orchestra.run.trial084.stdout:Complete! 2026-02-24T16:15:23.923 DEBUG:teuthology.orchestra.run.trial084:> sudo yum -y install ceph-mgr-cephadm 2026-02-24T16:15:24.247 INFO:teuthology.orchestra.run.trial084.stdout:Last metadata expiration check: 0:00:06 ago on Tue 24 Feb 2026 04:15:18 PM UTC. 2026-02-24T16:15:24.312 INFO:teuthology.orchestra.run.trial084.stdout:Package ceph-mgr-cephadm-2:20.3.0-5343.gd26e5c22.el9.noarch is already installed. 2026-02-24T16:15:24.331 INFO:teuthology.orchestra.run.trial084.stdout:Dependencies resolved. 2026-02-24T16:15:24.331 INFO:teuthology.orchestra.run.trial084.stdout:Nothing to do. 2026-02-24T16:15:24.332 INFO:teuthology.orchestra.run.trial084.stdout:Complete! 2026-02-24T16:15:24.354 DEBUG:teuthology.orchestra.run.trial084:> sudo yum -y install ceph-osd 2026-02-24T16:15:24.677 INFO:teuthology.orchestra.run.trial084.stdout:Last metadata expiration check: 0:00:06 ago on Tue 24 Feb 2026 04:15:18 PM UTC. 2026-02-24T16:15:24.742 INFO:teuthology.orchestra.run.trial084.stdout:Package ceph-osd-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:15:24.761 INFO:teuthology.orchestra.run.trial084.stdout:Dependencies resolved. 2026-02-24T16:15:24.762 INFO:teuthology.orchestra.run.trial084.stdout:Nothing to do. 2026-02-24T16:15:24.762 INFO:teuthology.orchestra.run.trial084.stdout:Complete! 2026-02-24T16:15:24.783 DEBUG:teuthology.orchestra.run.trial084:> sudo yum -y install ceph-osd-classic 2026-02-24T16:15:25.100 INFO:teuthology.orchestra.run.trial084.stdout:Last metadata expiration check: 0:00:07 ago on Tue 24 Feb 2026 04:15:18 PM UTC. 2026-02-24T16:15:25.165 INFO:teuthology.orchestra.run.trial084.stdout:Package ceph-osd-classic-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:15:25.184 INFO:teuthology.orchestra.run.trial084.stdout:Dependencies resolved. 2026-02-24T16:15:25.184 INFO:teuthology.orchestra.run.trial084.stdout:Nothing to do. 2026-02-24T16:15:25.184 INFO:teuthology.orchestra.run.trial084.stdout:Complete! 2026-02-24T16:15:25.207 DEBUG:teuthology.orchestra.run.trial084:> sudo yum -y install ceph-fuse 2026-02-24T16:15:25.525 INFO:teuthology.orchestra.run.trial084.stdout:Last metadata expiration check: 0:00:07 ago on Tue 24 Feb 2026 04:15:18 PM UTC. 2026-02-24T16:15:25.590 INFO:teuthology.orchestra.run.trial084.stdout:Package ceph-fuse-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:15:25.608 INFO:teuthology.orchestra.run.trial084.stdout:Dependencies resolved. 2026-02-24T16:15:25.609 INFO:teuthology.orchestra.run.trial084.stdout:Nothing to do. 2026-02-24T16:15:25.609 INFO:teuthology.orchestra.run.trial084.stdout:Complete! 2026-02-24T16:15:25.631 DEBUG:teuthology.orchestra.run.trial084:> sudo yum -y install ceph-volume 2026-02-24T16:15:25.954 INFO:teuthology.orchestra.run.trial084.stdout:Last metadata expiration check: 0:00:07 ago on Tue 24 Feb 2026 04:15:18 PM UTC. 2026-02-24T16:15:26.019 INFO:teuthology.orchestra.run.trial084.stdout:Package ceph-volume-2:20.3.0-5343.gd26e5c22.el9.noarch is already installed. 2026-02-24T16:15:26.038 INFO:teuthology.orchestra.run.trial084.stdout:Dependencies resolved. 2026-02-24T16:15:26.038 INFO:teuthology.orchestra.run.trial084.stdout:Nothing to do. 2026-02-24T16:15:26.039 INFO:teuthology.orchestra.run.trial084.stdout:Complete! 2026-02-24T16:15:26.061 DEBUG:teuthology.orchestra.run.trial084:> sudo yum -y install librados-devel 2026-02-24T16:15:26.377 INFO:teuthology.orchestra.run.trial084.stdout:Last metadata expiration check: 0:00:08 ago on Tue 24 Feb 2026 04:15:18 PM UTC. 2026-02-24T16:15:26.443 INFO:teuthology.orchestra.run.trial084.stdout:Package librados-devel-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:15:26.461 INFO:teuthology.orchestra.run.trial084.stdout:Dependencies resolved. 2026-02-24T16:15:26.462 INFO:teuthology.orchestra.run.trial084.stdout:Nothing to do. 2026-02-24T16:15:26.462 INFO:teuthology.orchestra.run.trial084.stdout:Complete! 2026-02-24T16:15:26.484 DEBUG:teuthology.orchestra.run.trial084:> sudo yum -y install libcephfs2 2026-02-24T16:15:26.801 INFO:teuthology.orchestra.run.trial084.stdout:Last metadata expiration check: 0:00:08 ago on Tue 24 Feb 2026 04:15:18 PM UTC. 2026-02-24T16:15:26.866 INFO:teuthology.orchestra.run.trial084.stdout:Package libcephfs2-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:15:26.884 INFO:teuthology.orchestra.run.trial084.stdout:Dependencies resolved. 2026-02-24T16:15:26.885 INFO:teuthology.orchestra.run.trial084.stdout:Nothing to do. 2026-02-24T16:15:26.885 INFO:teuthology.orchestra.run.trial084.stdout:Complete! 2026-02-24T16:15:26.907 DEBUG:teuthology.orchestra.run.trial084:> sudo yum -y install libcephfs-devel 2026-02-24T16:15:27.231 INFO:teuthology.orchestra.run.trial084.stdout:Last metadata expiration check: 0:00:09 ago on Tue 24 Feb 2026 04:15:18 PM UTC. 2026-02-24T16:15:27.296 INFO:teuthology.orchestra.run.trial084.stdout:Package libcephfs-devel-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:15:27.315 INFO:teuthology.orchestra.run.trial084.stdout:Dependencies resolved. 2026-02-24T16:15:27.316 INFO:teuthology.orchestra.run.trial084.stdout:Nothing to do. 2026-02-24T16:15:27.316 INFO:teuthology.orchestra.run.trial084.stdout:Complete! 2026-02-24T16:15:27.338 DEBUG:teuthology.orchestra.run.trial084:> sudo yum -y install librados2 2026-02-24T16:15:27.660 INFO:teuthology.orchestra.run.trial084.stdout:Last metadata expiration check: 0:00:09 ago on Tue 24 Feb 2026 04:15:18 PM UTC. 2026-02-24T16:15:27.725 INFO:teuthology.orchestra.run.trial084.stdout:Package librados2-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:15:27.744 INFO:teuthology.orchestra.run.trial084.stdout:Dependencies resolved. 2026-02-24T16:15:27.745 INFO:teuthology.orchestra.run.trial084.stdout:Nothing to do. 2026-02-24T16:15:27.745 INFO:teuthology.orchestra.run.trial084.stdout:Complete! 2026-02-24T16:15:27.766 DEBUG:teuthology.orchestra.run.trial084:> sudo yum -y install librbd1 2026-02-24T16:15:28.084 INFO:teuthology.orchestra.run.trial084.stdout:Last metadata expiration check: 0:00:10 ago on Tue 24 Feb 2026 04:15:18 PM UTC. 2026-02-24T16:15:28.150 INFO:teuthology.orchestra.run.trial084.stdout:Package librbd1-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:15:28.169 INFO:teuthology.orchestra.run.trial084.stdout:Dependencies resolved. 2026-02-24T16:15:28.169 INFO:teuthology.orchestra.run.trial084.stdout:Nothing to do. 2026-02-24T16:15:28.169 INFO:teuthology.orchestra.run.trial084.stdout:Complete! 2026-02-24T16:15:28.192 DEBUG:teuthology.orchestra.run.trial084:> sudo yum -y install python3-rados 2026-02-24T16:15:28.514 INFO:teuthology.orchestra.run.trial084.stdout:Last metadata expiration check: 0:00:10 ago on Tue 24 Feb 2026 04:15:18 PM UTC. 2026-02-24T16:15:28.580 INFO:teuthology.orchestra.run.trial084.stdout:Package python3-rados-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:15:28.599 INFO:teuthology.orchestra.run.trial084.stdout:Dependencies resolved. 2026-02-24T16:15:28.600 INFO:teuthology.orchestra.run.trial084.stdout:Nothing to do. 2026-02-24T16:15:28.600 INFO:teuthology.orchestra.run.trial084.stdout:Complete! 2026-02-24T16:15:28.621 DEBUG:teuthology.orchestra.run.trial084:> sudo yum -y install python3-rgw 2026-02-24T16:15:28.937 INFO:teuthology.orchestra.run.trial084.stdout:Last metadata expiration check: 0:00:10 ago on Tue 24 Feb 2026 04:15:18 PM UTC. 2026-02-24T16:15:29.003 INFO:teuthology.orchestra.run.trial084.stdout:Package python3-rgw-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:15:29.022 INFO:teuthology.orchestra.run.trial084.stdout:Dependencies resolved. 2026-02-24T16:15:29.023 INFO:teuthology.orchestra.run.trial084.stdout:Nothing to do. 2026-02-24T16:15:29.023 INFO:teuthology.orchestra.run.trial084.stdout:Complete! 2026-02-24T16:15:29.045 DEBUG:teuthology.orchestra.run.trial084:> sudo yum -y install python3-cephfs 2026-02-24T16:15:29.364 INFO:teuthology.orchestra.run.trial084.stdout:Last metadata expiration check: 0:00:11 ago on Tue 24 Feb 2026 04:15:18 PM UTC. 2026-02-24T16:15:29.429 INFO:teuthology.orchestra.run.trial084.stdout:Package python3-cephfs-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:15:29.448 INFO:teuthology.orchestra.run.trial084.stdout:Dependencies resolved. 2026-02-24T16:15:29.449 INFO:teuthology.orchestra.run.trial084.stdout:Nothing to do. 2026-02-24T16:15:29.449 INFO:teuthology.orchestra.run.trial084.stdout:Complete! 2026-02-24T16:15:29.471 DEBUG:teuthology.orchestra.run.trial084:> sudo yum -y install python3-rbd 2026-02-24T16:15:29.791 INFO:teuthology.orchestra.run.trial084.stdout:Last metadata expiration check: 0:00:11 ago on Tue 24 Feb 2026 04:15:18 PM UTC. 2026-02-24T16:15:29.855 INFO:teuthology.orchestra.run.trial084.stdout:Package python3-rbd-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:15:29.875 INFO:teuthology.orchestra.run.trial084.stdout:Dependencies resolved. 2026-02-24T16:15:29.875 INFO:teuthology.orchestra.run.trial084.stdout:Nothing to do. 2026-02-24T16:15:29.875 INFO:teuthology.orchestra.run.trial084.stdout:Complete! 2026-02-24T16:15:29.897 DEBUG:teuthology.orchestra.run.trial084:> sudo yum -y install rbd-fuse 2026-02-24T16:15:30.222 INFO:teuthology.orchestra.run.trial084.stdout:Last metadata expiration check: 0:00:12 ago on Tue 24 Feb 2026 04:15:18 PM UTC. 2026-02-24T16:15:30.288 INFO:teuthology.orchestra.run.trial084.stdout:Package rbd-fuse-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:15:30.306 INFO:teuthology.orchestra.run.trial084.stdout:Dependencies resolved. 2026-02-24T16:15:30.307 INFO:teuthology.orchestra.run.trial084.stdout:Nothing to do. 2026-02-24T16:15:30.307 INFO:teuthology.orchestra.run.trial084.stdout:Complete! 2026-02-24T16:15:30.330 DEBUG:teuthology.orchestra.run.trial084:> sudo yum -y install rbd-mirror 2026-02-24T16:15:30.649 INFO:teuthology.orchestra.run.trial084.stdout:Last metadata expiration check: 0:00:12 ago on Tue 24 Feb 2026 04:15:18 PM UTC. 2026-02-24T16:15:30.714 INFO:teuthology.orchestra.run.trial084.stdout:Package rbd-mirror-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:15:30.733 INFO:teuthology.orchestra.run.trial084.stdout:Dependencies resolved. 2026-02-24T16:15:30.734 INFO:teuthology.orchestra.run.trial084.stdout:Nothing to do. 2026-02-24T16:15:30.734 INFO:teuthology.orchestra.run.trial084.stdout:Complete! 2026-02-24T16:15:30.754 DEBUG:teuthology.orchestra.run.trial084:> sudo yum -y install rbd-nbd 2026-02-24T16:15:31.079 INFO:teuthology.orchestra.run.trial084.stdout:Last metadata expiration check: 0:00:13 ago on Tue 24 Feb 2026 04:15:18 PM UTC. 2026-02-24T16:15:31.146 INFO:teuthology.orchestra.run.trial084.stdout:Package rbd-nbd-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:15:31.165 INFO:teuthology.orchestra.run.trial084.stdout:Dependencies resolved. 2026-02-24T16:15:31.165 INFO:teuthology.orchestra.run.trial084.stdout:Nothing to do. 2026-02-24T16:15:31.165 INFO:teuthology.orchestra.run.trial084.stdout:Complete! 2026-02-24T16:15:31.195 DEBUG:teuthology.parallel:result is None 2026-02-24T16:15:31.196 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=d26e5c228e3cdea4d585bcd077f8513a649ab5d1 2026-02-24T16:15:31.356 DEBUG:teuthology.orchestra.run.trial084:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-24T16:15:31.377 INFO:teuthology.orchestra.run.trial084.stdout:20.3.0-5343.gd26e5c22.el9 2026-02-24T16:15:31.378 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5343.gd26e5c22.el9 2026-02-24T16:15:31.378 INFO:teuthology.task.install:The correct ceph version 20.3.0-5343.gd26e5c22 is installed. 2026-02-24T16:15:31.380 INFO:teuthology.task.install.util:Utilities already shipped, skip it... 2026-02-24T16:15:31.381 INFO:teuthology.run_tasks:Running task ceph... 2026-02-24T16:15:31.479 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2026-02-24T16:15:31.480 DEBUG:teuthology.orchestra.run.trial084:> sudo chmod 777 /var/log/ceph 2026-02-24T16:15:31.505 INFO:tasks.ceph:Disabling ceph logrotate... 2026-02-24T16:15:31.505 DEBUG:teuthology.orchestra.run.trial084:> sudo rm -f -- /etc/logrotate.d/ceph 2026-02-24T16:15:31.568 INFO:tasks.ceph:Creating extra log directories... 2026-02-24T16:15:31.568 DEBUG:teuthology.orchestra.run.trial084:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2026-02-24T16:15:31.641 INFO:tasks.ceph:Creating ceph cluster ceph... 2026-02-24T16:15:31.641 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore main device type': 'RANDOM_BLOCK_SSD', 'seastore max concurrent transactions': 8}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2026-02-24T16:15:31.642 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/mnt/teuthology/liucm-2026-02-24_15:28:57-crimson-rados-wip-liucm-shard-redesign-crimson-only-distro-debug-trial/68238', 'branch': 'wip-liucm-shard-redesign-crimson-only', 'description': 'crimson-rados/perf/{0-crimson_install clusters/{crimson-fixed-1} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/rbm$/{crimson_seastore_rbm} perf_ignorelist settings/optimized workloads/radosbench_4K_seq_read}', 'email': None, 'first_in_suite': False, 'flavor': 'debug', 'job_id': '68238', 'kernel': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'trial', 'name': 'liucm-2026-02-24_15:28:57-crimson-rados-wip-liucm-shard-redesign-crimson-only-distro-debug-trial', 'no_nested_subset': False, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-liucm-shard-redesign-crimson-only'}, 'ceph': {'conf': {'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd': {'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore main device type': 'RANDOM_BLOCK_SSD', 'seastore max concurrent transactions': 8}}, 'crimson_compat': True, 'flavor': 'debug', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH'], 'sha1': 'd26e5c228e3cdea4d585bcd077f8513a649ab5d1'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'global': {'osd crush chooseleaf type': 0, 'osd pool default pg num': 128, 'osd pool default pgp num': 128, 'osd pool default size': 2}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'debug', 'sha1': 'd26e5c228e3cdea4d585bcd077f8513a649ab5d1'}}, 'workunit': {'branch': 'wip-liucm-shard-redesign-crimson-only', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': 'd26e5c228e3cdea4d585bcd077f8513a649ab5d1'}}, 'owner': 'scheduled_liucm@soko04.front.sepia.ceph.com', 'priority': 120, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 9834, 'sha1': 'd26e5c228e3cdea4d585bcd077f8513a649ab5d1', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'wip-liucm-shard-redesign-crimson-only', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_d26e5c228e3cdea4d585bcd077f8513a649ab5d1/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': 'd26e5c228e3cdea4d585bcd077f8513a649ab5d1', 'targets': {'trial084.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK/Xi7BILlGmIASVnslm/6eCVpGkBUUXVms9wstoJQGC+lFMV9gxCD040eOXC3EM8Tp76qDyRRALUATGCeNY+sI='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'flavor': 'debug', 'sha1': 'd26e5c228e3cdea4d585bcd077f8513a649ab5d1'}}, {'pexec': {'all': ['sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson']}}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore main device type': 'RANDOM_BLOCK_SSD', 'seastore max concurrent transactions': 8}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}}, 'wait-for-scrub': False, 'crimson_compat': True, 'flavor': 'debug', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH'], 'sha1': 'd26e5c228e3cdea4d585bcd077f8513a649ab5d1', 'cluster': 'ceph'}}, {'ssh_keys': None}, {'cbt': {'benchmarks': {'radosbench': {'concurrent_ops': 4, 'concurrent_procs': 2, 'op_size': [4096], 'pool_monitoring_list': ['collectl'], 'pool_profile': 'replicated', 'run_monitoring_list': ['collectl'], 'time': 60, 'write_only': False}}, 'cluster': {'iterations': 1, 'osds_per_node': 3, 'pool_profiles': {'replicated': {'pg_size': 256, 'pgp_size': 256, 'replication': 'replicated'}}, 'user': 'ubuntu'}, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'fa17720d0088c3ac28e473468bfc79eeaff5cd38', 'timestamp': '2026-02-24_15:28:57', 'tube': 'trial', 'user': 'liucm', 'verbose': False, 'worker_log': '/home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.904699'} 2026-02-24T16:15:31.643 DEBUG:teuthology.orchestra.run.trial084:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2026-02-24T16:15:31.688 DEBUG:teuthology.orchestra.run.trial084:> sudo install -d -m0777 -- /var/run/ceph 2026-02-24T16:15:31.753 DEBUG:teuthology.orchestra.run.trial084:> set -ex 2026-02-24T16:15:31.753 DEBUG:teuthology.orchestra.run.trial084:> dd if=/scratch_devs of=/dev/stdout 2026-02-24T16:15:31.809 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2026-02-24T16:15:31.810 DEBUG:teuthology.orchestra.run.trial084:> stat /dev/vg_nvme/lv_1 2026-02-24T16:15:31.867 INFO:teuthology.orchestra.run.trial084.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2026-02-24T16:15:31.867 INFO:teuthology.orchestra.run.trial084.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-24T16:15:31.867 INFO:teuthology.orchestra.run.trial084.stdout:Device: 6h/6d Inode: 907 Links: 1 2026-02-24T16:15:31.867 INFO:teuthology.orchestra.run.trial084.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-24T16:15:31.867 INFO:teuthology.orchestra.run.trial084.stdout:Context: system_u:object_r:device_t:s0 2026-02-24T16:15:31.868 INFO:teuthology.orchestra.run.trial084.stdout:Access: 2026-02-24 16:02:44.693317190 +0000 2026-02-24T16:15:31.868 INFO:teuthology.orchestra.run.trial084.stdout:Modify: 2026-02-24 15:47:51.803084653 +0000 2026-02-24T16:15:31.868 INFO:teuthology.orchestra.run.trial084.stdout:Change: 2026-02-24 15:47:51.803084653 +0000 2026-02-24T16:15:31.868 INFO:teuthology.orchestra.run.trial084.stdout: Birth: 2026-02-24 15:47:51.803084653 +0000 2026-02-24T16:15:31.868 DEBUG:teuthology.orchestra.run.trial084:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2026-02-24T16:15:31.929 INFO:teuthology.orchestra.run.trial084.stderr:1+0 records in 2026-02-24T16:15:31.929 INFO:teuthology.orchestra.run.trial084.stderr:1+0 records out 2026-02-24T16:15:31.929 INFO:teuthology.orchestra.run.trial084.stderr:512 bytes copied, 0.000155448 s, 3.3 MB/s 2026-02-24T16:15:31.930 DEBUG:teuthology.orchestra.run.trial084:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2026-02-24T16:15:31.986 DEBUG:teuthology.orchestra.run.trial084:> stat /dev/vg_nvme/lv_2 2026-02-24T16:15:32.043 INFO:teuthology.orchestra.run.trial084.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2026-02-24T16:15:32.043 INFO:teuthology.orchestra.run.trial084.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-24T16:15:32.044 INFO:teuthology.orchestra.run.trial084.stdout:Device: 6h/6d Inode: 937 Links: 1 2026-02-24T16:15:32.044 INFO:teuthology.orchestra.run.trial084.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-24T16:15:32.044 INFO:teuthology.orchestra.run.trial084.stdout:Context: system_u:object_r:device_t:s0 2026-02-24T16:15:32.044 INFO:teuthology.orchestra.run.trial084.stdout:Access: 2026-02-24 16:02:44.693317190 +0000 2026-02-24T16:15:32.044 INFO:teuthology.orchestra.run.trial084.stdout:Modify: 2026-02-24 15:47:52.032075910 +0000 2026-02-24T16:15:32.044 INFO:teuthology.orchestra.run.trial084.stdout:Change: 2026-02-24 15:47:52.032075910 +0000 2026-02-24T16:15:32.044 INFO:teuthology.orchestra.run.trial084.stdout: Birth: 2026-02-24 15:47:52.032075910 +0000 2026-02-24T16:15:32.044 DEBUG:teuthology.orchestra.run.trial084:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2026-02-24T16:15:32.107 INFO:teuthology.orchestra.run.trial084.stderr:1+0 records in 2026-02-24T16:15:32.107 INFO:teuthology.orchestra.run.trial084.stderr:1+0 records out 2026-02-24T16:15:32.107 INFO:teuthology.orchestra.run.trial084.stderr:512 bytes copied, 0.000142675 s, 3.6 MB/s 2026-02-24T16:15:32.108 DEBUG:teuthology.orchestra.run.trial084:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2026-02-24T16:15:32.165 DEBUG:teuthology.orchestra.run.trial084:> stat /dev/vg_nvme/lv_3 2026-02-24T16:15:32.220 INFO:teuthology.orchestra.run.trial084.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2026-02-24T16:15:32.221 INFO:teuthology.orchestra.run.trial084.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-24T16:15:32.221 INFO:teuthology.orchestra.run.trial084.stdout:Device: 6h/6d Inode: 967 Links: 1 2026-02-24T16:15:32.221 INFO:teuthology.orchestra.run.trial084.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-24T16:15:32.221 INFO:teuthology.orchestra.run.trial084.stdout:Context: system_u:object_r:device_t:s0 2026-02-24T16:15:32.221 INFO:teuthology.orchestra.run.trial084.stdout:Access: 2026-02-24 16:02:44.693317190 +0000 2026-02-24T16:15:32.221 INFO:teuthology.orchestra.run.trial084.stdout:Modify: 2026-02-24 15:47:52.291066022 +0000 2026-02-24T16:15:32.221 INFO:teuthology.orchestra.run.trial084.stdout:Change: 2026-02-24 15:47:52.291066022 +0000 2026-02-24T16:15:32.221 INFO:teuthology.orchestra.run.trial084.stdout: Birth: 2026-02-24 15:47:52.291066022 +0000 2026-02-24T16:15:32.221 DEBUG:teuthology.orchestra.run.trial084:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2026-02-24T16:15:32.282 INFO:teuthology.orchestra.run.trial084.stderr:1+0 records in 2026-02-24T16:15:32.282 INFO:teuthology.orchestra.run.trial084.stderr:1+0 records out 2026-02-24T16:15:32.282 INFO:teuthology.orchestra.run.trial084.stderr:512 bytes copied, 0.000158685 s, 3.2 MB/s 2026-02-24T16:15:32.283 DEBUG:teuthology.orchestra.run.trial084:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2026-02-24T16:15:32.340 DEBUG:teuthology.orchestra.run.trial084:> stat /dev/vg_nvme/lv_4 2026-02-24T16:15:32.395 INFO:teuthology.orchestra.run.trial084.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2026-02-24T16:15:32.395 INFO:teuthology.orchestra.run.trial084.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-24T16:15:32.395 INFO:teuthology.orchestra.run.trial084.stdout:Device: 6h/6d Inode: 993 Links: 1 2026-02-24T16:15:32.395 INFO:teuthology.orchestra.run.trial084.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-24T16:15:32.395 INFO:teuthology.orchestra.run.trial084.stdout:Context: system_u:object_r:device_t:s0 2026-02-24T16:15:32.395 INFO:teuthology.orchestra.run.trial084.stdout:Access: 2026-02-24 16:02:44.694317155 +0000 2026-02-24T16:15:32.395 INFO:teuthology.orchestra.run.trial084.stdout:Modify: 2026-02-24 15:47:52.525057088 +0000 2026-02-24T16:15:32.395 INFO:teuthology.orchestra.run.trial084.stdout:Change: 2026-02-24 15:47:52.525057088 +0000 2026-02-24T16:15:32.395 INFO:teuthology.orchestra.run.trial084.stdout: Birth: 2026-02-24 15:47:52.525057088 +0000 2026-02-24T16:15:32.395 DEBUG:teuthology.orchestra.run.trial084:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2026-02-24T16:15:32.456 INFO:teuthology.orchestra.run.trial084.stderr:1+0 records in 2026-02-24T16:15:32.456 INFO:teuthology.orchestra.run.trial084.stderr:1+0 records out 2026-02-24T16:15:32.457 INFO:teuthology.orchestra.run.trial084.stderr:512 bytes copied, 0.000144027 s, 3.6 MB/s 2026-02-24T16:15:32.458 DEBUG:teuthology.orchestra.run.trial084:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2026-02-24T16:15:32.514 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-24T16:15:32.514 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@trial084.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2026-02-24T16:15:32.515 INFO:tasks.ceph:Generating config... 2026-02-24T16:15:32.517 INFO:tasks.ceph:[osd] debug monc = 20 2026-02-24T16:15:32.517 INFO:tasks.ceph:[osd] crimson cpu num = 4 2026-02-24T16:15:32.518 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2026-02-24T16:15:32.518 INFO:tasks.ceph:[osd] debug filestore = 0/0 2026-02-24T16:15:32.518 INFO:tasks.ceph:[osd] debug journal = 0/0 2026-02-24T16:15:32.518 INFO:tasks.ceph:[osd] debug ms = 0/0 2026-02-24T16:15:32.518 INFO:tasks.ceph:[osd] debug osd = 0/0 2026-02-24T16:15:32.518 INFO:tasks.ceph:[osd] debug seastore = 20 2026-02-24T16:15:32.518 INFO:tasks.ceph:[osd] debug seastore backref = 20 2026-02-24T16:15:32.518 INFO:tasks.ceph:[osd] debug seastore cache = 20 2026-02-24T16:15:32.518 INFO:tasks.ceph:[osd] debug seastore cleaner = 20 2026-02-24T16:15:32.518 INFO:tasks.ceph:[osd] debug seastore device = 20 2026-02-24T16:15:32.518 INFO:tasks.ceph:[osd] debug seastore epm = 20 2026-02-24T16:15:32.518 INFO:tasks.ceph:[osd] debug seastore fixedkv tree = 20 2026-02-24T16:15:32.519 INFO:tasks.ceph:[osd] debug seastore journal = 20 2026-02-24T16:15:32.519 INFO:tasks.ceph:[osd] debug seastore lba = 20 2026-02-24T16:15:32.519 INFO:tasks.ceph:[osd] debug seastore odata = 20 2026-02-24T16:15:32.519 INFO:tasks.ceph:[osd] debug seastore omap = 20 2026-02-24T16:15:32.519 INFO:tasks.ceph:[osd] debug seastore onode = 20 2026-02-24T16:15:32.519 INFO:tasks.ceph:[osd] debug seastore t = 20 2026-02-24T16:15:32.519 INFO:tasks.ceph:[osd] debug seastore tm = 20 2026-02-24T16:15:32.519 INFO:tasks.ceph:[osd] osd objectstore = seastore 2026-02-24T16:15:32.519 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2026-02-24T16:15:32.519 INFO:tasks.ceph:[osd] seastore cachepin size pershard = 64M 2026-02-24T16:15:32.519 INFO:tasks.ceph:[osd] seastore main device type = RANDOM_BLOCK_SSD 2026-02-24T16:15:32.519 INFO:tasks.ceph:[osd] seastore max concurrent transactions = 8 2026-02-24T16:15:32.520 INFO:tasks.ceph:[global] auth client required = none 2026-02-24T16:15:32.520 INFO:tasks.ceph:[global] auth cluster required = none 2026-02-24T16:15:32.520 INFO:tasks.ceph:[global] auth service required = none 2026-02-24T16:15:32.520 INFO:tasks.ceph:[global] auth supported = none 2026-02-24T16:15:32.520 INFO:tasks.ceph:[global] debug asok = 0/0 2026-02-24T16:15:32.520 INFO:tasks.ceph:[global] debug auth = 0/0 2026-02-24T16:15:32.520 INFO:tasks.ceph:[global] debug bdev = 0/0 2026-02-24T16:15:32.520 INFO:tasks.ceph:[global] debug bluefs = 0/0 2026-02-24T16:15:32.520 INFO:tasks.ceph:[global] debug bluestore = 0/0 2026-02-24T16:15:32.520 INFO:tasks.ceph:[global] debug buffer = 0/0 2026-02-24T16:15:32.520 INFO:tasks.ceph:[global] debug civetweb = 0/0 2026-02-24T16:15:32.520 INFO:tasks.ceph:[global] debug client = 0/0 2026-02-24T16:15:32.521 INFO:tasks.ceph:[global] debug compressor = 0/0 2026-02-24T16:15:32.521 INFO:tasks.ceph:[global] debug context = 0/0 2026-02-24T16:15:32.521 INFO:tasks.ceph:[global] debug crush = 0/0 2026-02-24T16:15:32.521 INFO:tasks.ceph:[global] debug crypto = 0/0 2026-02-24T16:15:32.521 INFO:tasks.ceph:[global] debug dpdk = 0/0 2026-02-24T16:15:32.521 INFO:tasks.ceph:[global] debug eventtrace = 0/0 2026-02-24T16:15:32.521 INFO:tasks.ceph:[global] debug filer = 0/0 2026-02-24T16:15:32.521 INFO:tasks.ceph:[global] debug filestore = 0/0 2026-02-24T16:15:32.521 INFO:tasks.ceph:[global] debug finisher = 0/0 2026-02-24T16:15:32.521 INFO:tasks.ceph:[global] debug fuse = 0/0 2026-02-24T16:15:32.521 INFO:tasks.ceph:[global] debug heartbeatmap = 0/0 2026-02-24T16:15:32.521 INFO:tasks.ceph:[global] debug javaclient = 0/0 2026-02-24T16:15:32.521 INFO:tasks.ceph:[global] debug journal = 0/0 2026-02-24T16:15:32.522 INFO:tasks.ceph:[global] debug journaler = 0/0 2026-02-24T16:15:32.522 INFO:tasks.ceph:[global] debug kstore = 0/0 2026-02-24T16:15:32.522 INFO:tasks.ceph:[global] debug leveldb = 0/0 2026-02-24T16:15:32.522 INFO:tasks.ceph:[global] debug lockdep = 0/0 2026-02-24T16:15:32.522 INFO:tasks.ceph:[global] debug mds = 0/0 2026-02-24T16:15:32.522 INFO:tasks.ceph:[global] debug mds balancer = 0/0 2026-02-24T16:15:32.522 INFO:tasks.ceph:[global] debug mds locker = 0/0 2026-02-24T16:15:32.522 INFO:tasks.ceph:[global] debug mds log = 0/0 2026-02-24T16:15:32.522 INFO:tasks.ceph:[global] debug mds log expire = 0/0 2026-02-24T16:15:32.522 INFO:tasks.ceph:[global] debug mds migrator = 0/0 2026-02-24T16:15:32.522 INFO:tasks.ceph:[global] debug memdb = 0/0 2026-02-24T16:15:32.522 INFO:tasks.ceph:[global] debug mgr = 0/0 2026-02-24T16:15:32.522 INFO:tasks.ceph:[global] debug mgrc = 0/0 2026-02-24T16:15:32.523 INFO:tasks.ceph:[global] debug mon = 0/0 2026-02-24T16:15:32.523 INFO:tasks.ceph:[global] debug monc = 0/0 2026-02-24T16:15:32.523 INFO:tasks.ceph:[global] debug ms = 0/0 2026-02-24T16:15:32.523 INFO:tasks.ceph:[global] debug objclass = 0/0 2026-02-24T16:15:32.523 INFO:tasks.ceph:[global] debug objectcacher = 0/0 2026-02-24T16:15:32.523 INFO:tasks.ceph:[global] debug objecter = 0/0 2026-02-24T16:15:32.523 INFO:tasks.ceph:[global] debug optracker = 0/0 2026-02-24T16:15:32.523 INFO:tasks.ceph:[global] debug osd = 0/0 2026-02-24T16:15:32.523 INFO:tasks.ceph:[global] debug paxos = 0/0 2026-02-24T16:15:32.523 INFO:tasks.ceph:[global] debug perfcounter = 0/0 2026-02-24T16:15:32.523 INFO:tasks.ceph:[global] debug rados = 0/0 2026-02-24T16:15:32.523 INFO:tasks.ceph:[global] debug rbd = 0/0 2026-02-24T16:15:32.523 INFO:tasks.ceph:[global] debug rbd mirror = 0/0 2026-02-24T16:15:32.524 INFO:tasks.ceph:[global] debug rbd replay = 0/0 2026-02-24T16:15:32.524 INFO:tasks.ceph:[global] debug refs = 0/0 2026-02-24T16:15:32.524 INFO:tasks.ceph:[global] debug rgw = 0/0 2026-02-24T16:15:32.524 INFO:tasks.ceph:[global] debug rgw lifecycle = 0/0 2026-02-24T16:15:32.524 INFO:tasks.ceph:[global] debug rgw notification = 0/0 2026-02-24T16:15:32.524 INFO:tasks.ceph:[global] debug rgw sync = 0/0 2026-02-24T16:15:32.524 INFO:tasks.ceph:[global] debug rocksdb = 0/0 2026-02-24T16:15:32.524 INFO:tasks.ceph:[global] debug striper = 0/0 2026-02-24T16:15:32.524 INFO:tasks.ceph:[global] debug throttle = 0/0 2026-02-24T16:15:32.524 INFO:tasks.ceph:[global] debug timer = 0/0 2026-02-24T16:15:32.524 INFO:tasks.ceph:[global] debug tp = 0/0 2026-02-24T16:15:32.524 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2026-02-24T16:15:32.524 INFO:tasks.ceph:[global] ms client mode = crc 2026-02-24T16:15:32.525 INFO:tasks.ceph:[global] ms cluster mode = crc 2026-02-24T16:15:32.525 INFO:tasks.ceph:[global] ms mon client mode = crc 2026-02-24T16:15:32.525 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2026-02-24T16:15:32.525 INFO:tasks.ceph:[global] ms mon service mode = crc 2026-02-24T16:15:32.525 INFO:tasks.ceph:[global] ms service mode = crc 2026-02-24T16:15:32.525 INFO:tasks.ceph:[mgr] debug mgr = 20 2026-02-24T16:15:32.525 INFO:tasks.ceph:[mgr] debug ms = 1 2026-02-24T16:15:32.525 INFO:tasks.ceph:[mon] debug mon = 0/0 2026-02-24T16:15:32.525 INFO:tasks.ceph:[mon] debug ms = 0/0 2026-02-24T16:15:32.525 INFO:tasks.ceph:[mon] debug paxos = 0/0 2026-02-24T16:15:32.525 INFO:tasks.ceph:[mon] osd pool default crimson = True 2026-02-24T16:15:32.526 INFO:tasks.ceph:Setting up mon.a... 2026-02-24T16:15:32.526 DEBUG:teuthology.orchestra.run.trial084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2026-02-24T16:15:32.595 INFO:teuthology.orchestra.run.trial084.stdout:creating /etc/ceph/ceph.keyring 2026-02-24T16:15:32.598 DEBUG:teuthology.orchestra.run.trial084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2026-02-24T16:15:32.682 DEBUG:teuthology.orchestra.run.trial084:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-24T16:15:32.706 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '10.20.193.84')] 2026-02-24T16:15:32.707 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'none', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '10.20.193.84', 'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': 'true', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore main device type': 'RANDOM_BLOCK_SSD', 'seastore max concurrent transactions': 8}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': '0/0', 'debug mon': '0/0', 'debug paxos': '0/0', 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}} 2026-02-24T16:15:32.707 DEBUG:teuthology.orchestra.run.trial084:> set -ex 2026-02-24T16:15:32.708 DEBUG:teuthology.orchestra.run.trial084:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2026-02-24T16:15:32.763 DEBUG:teuthology.orchestra.run.trial084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 10.20.193.84 --print /home/ubuntu/cephtest/ceph.monmap 2026-02-24T16:15:32.842 INFO:teuthology.orchestra.run.trial084.stderr:2026-02-24T16:15:32.839+0000 7ff778ee2f40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:15:32.842 INFO:teuthology.orchestra.run.trial084.stderr:2026-02-24T16:15:32.840+0000 7ff778ee2f40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:15:32.842 INFO:teuthology.orchestra.run.trial084.stderr:2026-02-24T16:15:32.840+0000 7ff778ee2f40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:15:32.844 INFO:teuthology.orchestra.run.trial084.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2026-02-24T16:15:32.844 INFO:teuthology.orchestra.run.trial084.stdout:monmaptool: generated fsid a1ef0711-bcd7-4e13-9432-5701b02e2426 2026-02-24T16:15:32.844 INFO:teuthology.orchestra.run.trial084.stdout:setting min_mon_release = tentacle 2026-02-24T16:15:32.844 INFO:teuthology.orchestra.run.trial084.stdout:epoch 0 2026-02-24T16:15:32.844 INFO:teuthology.orchestra.run.trial084.stdout:fsid a1ef0711-bcd7-4e13-9432-5701b02e2426 2026-02-24T16:15:32.844 INFO:teuthology.orchestra.run.trial084.stdout:last_changed 2026-02-24T16:15:32.841585+0000 2026-02-24T16:15:32.844 INFO:teuthology.orchestra.run.trial084.stdout:created 2026-02-24T16:15:32.841585+0000 2026-02-24T16:15:32.844 INFO:teuthology.orchestra.run.trial084.stdout:min_mon_release 20 (tentacle) 2026-02-24T16:15:32.844 INFO:teuthology.orchestra.run.trial084.stdout:election_strategy: 1 2026-02-24T16:15:32.844 INFO:teuthology.orchestra.run.trial084.stdout:0: [v2:10.20.193.84:3300/0,v1:10.20.193.84:6789/0] mon.a 2026-02-24T16:15:32.845 INFO:teuthology.orchestra.run.trial084.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2026-02-24T16:15:32.845 DEBUG:teuthology.orchestra.run.trial084:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2026-02-24T16:15:32.900 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID a1ef0711-bcd7-4e13-9432-5701b02e2426... 2026-02-24T16:15:32.902 DEBUG:teuthology.orchestra.run.trial084:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2026-02-24T16:15:32.980 INFO:teuthology.orchestra.run.trial084.stdout:[global] 2026-02-24T16:15:32.980 INFO:teuthology.orchestra.run.trial084.stdout: chdir = "" 2026-02-24T16:15:32.980 INFO:teuthology.orchestra.run.trial084.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2026-02-24T16:15:32.980 INFO:teuthology.orchestra.run.trial084.stdout: auth supported = none 2026-02-24T16:15:32.980 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:15:32.981 INFO:teuthology.orchestra.run.trial084.stdout: filestore xattr use omap = true 2026-02-24T16:15:32.981 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:15:32.981 INFO:teuthology.orchestra.run.trial084.stdout: mon clock drift allowed = 1.000 2026-02-24T16:15:32.981 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:15:32.981 INFO:teuthology.orchestra.run.trial084.stdout: osd crush chooseleaf type = 0 2026-02-24T16:15:32.981 INFO:teuthology.orchestra.run.trial084.stdout: auth debug = true 2026-02-24T16:15:32.981 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:15:32.981 INFO:teuthology.orchestra.run.trial084.stdout: ms die on old message = true 2026-02-24T16:15:32.981 INFO:teuthology.orchestra.run.trial084.stdout: ms die on bug = true 2026-02-24T16:15:32.981 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:15:32.981 INFO:teuthology.orchestra.run.trial084.stdout: mon max pg per osd = 10000 # >= luminous 2026-02-24T16:15:32.981 INFO:teuthology.orchestra.run.trial084.stdout: mon pg warn max object skew = 0 2026-02-24T16:15:32.982 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:15:32.982 INFO:teuthology.orchestra.run.trial084.stdout: # disable pg_autoscaler by default for new pools 2026-02-24T16:15:32.982 INFO:teuthology.orchestra.run.trial084.stdout: osd_pool_default_pg_autoscale_mode = off 2026-02-24T16:15:32.982 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:15:32.982 INFO:teuthology.orchestra.run.trial084.stdout: osd pool default size = 2 2026-02-24T16:15:32.982 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:15:32.982 INFO:teuthology.orchestra.run.trial084.stdout: mon osd allow primary affinity = true 2026-02-24T16:15:32.982 INFO:teuthology.orchestra.run.trial084.stdout: mon osd allow pg remap = true 2026-02-24T16:15:32.982 INFO:teuthology.orchestra.run.trial084.stdout: mon warn on legacy crush tunables = false 2026-02-24T16:15:32.982 INFO:teuthology.orchestra.run.trial084.stdout: mon warn on crush straw calc version zero = false 2026-02-24T16:15:32.982 INFO:teuthology.orchestra.run.trial084.stdout: mon warn on no sortbitwise = false 2026-02-24T16:15:32.982 INFO:teuthology.orchestra.run.trial084.stdout: mon warn on osd down out interval zero = false 2026-02-24T16:15:32.983 INFO:teuthology.orchestra.run.trial084.stdout: mon warn on too few osds = false 2026-02-24T16:15:32.983 INFO:teuthology.orchestra.run.trial084.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2026-02-24T16:15:32.983 INFO:teuthology.orchestra.run.trial084.stdout: mon_warn_on_pool_no_redundancy = false 2026-02-24T16:15:32.983 INFO:teuthology.orchestra.run.trial084.stdout: mon_allow_pool_size_one = true 2026-02-24T16:15:32.983 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:15:32.983 INFO:teuthology.orchestra.run.trial084.stdout: osd pool default erasure code profile = plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2026-02-24T16:15:32.983 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:15:32.983 INFO:teuthology.orchestra.run.trial084.stdout: osd default data pool replay window = 5 2026-02-24T16:15:32.983 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:15:32.983 INFO:teuthology.orchestra.run.trial084.stdout: mon allow pool delete = true 2026-02-24T16:15:32.983 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:15:32.983 INFO:teuthology.orchestra.run.trial084.stdout: mon cluster log file level = debug 2026-02-24T16:15:32.984 INFO:teuthology.orchestra.run.trial084.stdout: debug asserts on shutdown = true 2026-02-24T16:15:32.984 INFO:teuthology.orchestra.run.trial084.stdout: mon health detail to clog = false 2026-02-24T16:15:32.984 INFO:teuthology.orchestra.run.trial084.stdout: mon host = 10.20.193.84 2026-02-24T16:15:32.984 INFO:teuthology.orchestra.run.trial084.stdout: auth client required = none 2026-02-24T16:15:32.984 INFO:teuthology.orchestra.run.trial084.stdout: auth cluster required = none 2026-02-24T16:15:32.984 INFO:teuthology.orchestra.run.trial084.stdout: auth service required = none 2026-02-24T16:15:32.984 INFO:teuthology.orchestra.run.trial084.stdout: debug asok = 0/0 2026-02-24T16:15:32.984 INFO:teuthology.orchestra.run.trial084.stdout: debug auth = 0/0 2026-02-24T16:15:32.984 INFO:teuthology.orchestra.run.trial084.stdout: debug bdev = 0/0 2026-02-24T16:15:32.984 INFO:teuthology.orchestra.run.trial084.stdout: debug bluefs = 0/0 2026-02-24T16:15:32.984 INFO:teuthology.orchestra.run.trial084.stdout: debug bluestore = 0/0 2026-02-24T16:15:32.984 INFO:teuthology.orchestra.run.trial084.stdout: debug buffer = 0/0 2026-02-24T16:15:32.985 INFO:teuthology.orchestra.run.trial084.stdout: debug civetweb = 0/0 2026-02-24T16:15:32.985 INFO:teuthology.orchestra.run.trial084.stdout: debug client = 0/0 2026-02-24T16:15:32.985 INFO:teuthology.orchestra.run.trial084.stdout: debug compressor = 0/0 2026-02-24T16:15:32.985 INFO:teuthology.orchestra.run.trial084.stdout: debug context = 0/0 2026-02-24T16:15:32.985 INFO:teuthology.orchestra.run.trial084.stdout: debug crush = 0/0 2026-02-24T16:15:32.985 INFO:teuthology.orchestra.run.trial084.stdout: debug crypto = 0/0 2026-02-24T16:15:32.985 INFO:teuthology.orchestra.run.trial084.stdout: debug dpdk = 0/0 2026-02-24T16:15:32.985 INFO:teuthology.orchestra.run.trial084.stdout: debug eventtrace = 0/0 2026-02-24T16:15:32.985 INFO:teuthology.orchestra.run.trial084.stdout: debug filer = 0/0 2026-02-24T16:15:32.985 INFO:teuthology.orchestra.run.trial084.stdout: debug filestore = 0/0 2026-02-24T16:15:32.985 INFO:teuthology.orchestra.run.trial084.stdout: debug finisher = 0/0 2026-02-24T16:15:32.985 INFO:teuthology.orchestra.run.trial084.stdout: debug fuse = 0/0 2026-02-24T16:15:32.985 INFO:teuthology.orchestra.run.trial084.stdout: debug heartbeatmap = 0/0 2026-02-24T16:15:32.986 INFO:teuthology.orchestra.run.trial084.stdout: debug javaclient = 0/0 2026-02-24T16:15:32.986 INFO:teuthology.orchestra.run.trial084.stdout: debug journal = 0/0 2026-02-24T16:15:32.986 INFO:teuthology.orchestra.run.trial084.stdout: debug journaler = 0/0 2026-02-24T16:15:32.986 INFO:teuthology.orchestra.run.trial084.stdout: debug kstore = 0/0 2026-02-24T16:15:32.986 INFO:teuthology.orchestra.run.trial084.stdout: debug leveldb = 0/0 2026-02-24T16:15:32.986 INFO:teuthology.orchestra.run.trial084.stdout: debug lockdep = 0/0 2026-02-24T16:15:32.986 INFO:teuthology.orchestra.run.trial084.stdout: debug mds = 0/0 2026-02-24T16:15:32.986 INFO:teuthology.orchestra.run.trial084.stdout: debug mds balancer = 0/0 2026-02-24T16:15:32.986 INFO:teuthology.orchestra.run.trial084.stdout: debug mds locker = 0/0 2026-02-24T16:15:32.986 INFO:teuthology.orchestra.run.trial084.stdout: debug mds log = 0/0 2026-02-24T16:15:32.986 INFO:teuthology.orchestra.run.trial084.stdout: debug mds log expire = 0/0 2026-02-24T16:15:32.986 INFO:teuthology.orchestra.run.trial084.stdout: debug mds migrator = 0/0 2026-02-24T16:15:32.986 INFO:teuthology.orchestra.run.trial084.stdout: debug memdb = 0/0 2026-02-24T16:15:32.986 INFO:teuthology.orchestra.run.trial084.stdout: debug mgr = 0/0 2026-02-24T16:15:32.987 INFO:teuthology.orchestra.run.trial084.stdout: debug mgrc = 0/0 2026-02-24T16:15:32.987 INFO:teuthology.orchestra.run.trial084.stdout: debug mon = 0/0 2026-02-24T16:15:32.987 INFO:teuthology.orchestra.run.trial084.stdout: debug monc = 0/0 2026-02-24T16:15:32.987 INFO:teuthology.orchestra.run.trial084.stdout: debug ms = 0/0 2026-02-24T16:15:32.987 INFO:teuthology.orchestra.run.trial084.stdout: debug objclass = 0/0 2026-02-24T16:15:32.987 INFO:teuthology.orchestra.run.trial084.stdout: debug objectcacher = 0/0 2026-02-24T16:15:32.987 INFO:teuthology.orchestra.run.trial084.stdout: debug objecter = 0/0 2026-02-24T16:15:32.987 INFO:teuthology.orchestra.run.trial084.stdout: debug optracker = 0/0 2026-02-24T16:15:32.987 INFO:teuthology.orchestra.run.trial084.stdout: debug osd = 0/0 2026-02-24T16:15:32.987 INFO:teuthology.orchestra.run.trial084.stdout: debug paxos = 0/0 2026-02-24T16:15:32.987 INFO:teuthology.orchestra.run.trial084.stdout: debug perfcounter = 0/0 2026-02-24T16:15:32.987 INFO:teuthology.orchestra.run.trial084.stdout: debug rados = 0/0 2026-02-24T16:15:32.987 INFO:teuthology.orchestra.run.trial084.stdout: debug rbd = 0/0 2026-02-24T16:15:32.987 INFO:teuthology.orchestra.run.trial084.stdout: debug rbd mirror = 0/0 2026-02-24T16:15:32.988 INFO:teuthology.orchestra.run.trial084.stdout: debug rbd replay = 0/0 2026-02-24T16:15:32.988 INFO:teuthology.orchestra.run.trial084.stdout: debug refs = 0/0 2026-02-24T16:15:32.988 INFO:teuthology.orchestra.run.trial084.stdout: debug rgw = 0/0 2026-02-24T16:15:32.988 INFO:teuthology.orchestra.run.trial084.stdout: debug rgw lifecycle = 0/0 2026-02-24T16:15:32.988 INFO:teuthology.orchestra.run.trial084.stdout: debug rgw notification = 0/0 2026-02-24T16:15:32.988 INFO:teuthology.orchestra.run.trial084.stdout: debug rgw sync = 0/0 2026-02-24T16:15:32.988 INFO:teuthology.orchestra.run.trial084.stdout: debug rocksdb = 0/0 2026-02-24T16:15:32.988 INFO:teuthology.orchestra.run.trial084.stdout: debug striper = 0/0 2026-02-24T16:15:32.988 INFO:teuthology.orchestra.run.trial084.stdout: debug throttle = 0/0 2026-02-24T16:15:32.988 INFO:teuthology.orchestra.run.trial084.stdout: debug timer = 0/0 2026-02-24T16:15:32.988 INFO:teuthology.orchestra.run.trial084.stdout: debug tp = 0/0 2026-02-24T16:15:32.988 INFO:teuthology.orchestra.run.trial084.stdout: enable experimental unrecoverable data corrupting features = crimson 2026-02-24T16:15:32.989 INFO:teuthology.orchestra.run.trial084.stdout: ms client mode = crc 2026-02-24T16:15:32.989 INFO:teuthology.orchestra.run.trial084.stdout: ms cluster mode = crc 2026-02-24T16:15:32.989 INFO:teuthology.orchestra.run.trial084.stdout: ms mon client mode = crc 2026-02-24T16:15:32.989 INFO:teuthology.orchestra.run.trial084.stdout: ms mon cluster mode = crc 2026-02-24T16:15:32.989 INFO:teuthology.orchestra.run.trial084.stdout: ms mon service mode = crc 2026-02-24T16:15:32.989 INFO:teuthology.orchestra.run.trial084.stdout: ms service mode = crc 2026-02-24T16:15:32.989 INFO:teuthology.orchestra.run.trial084.stdout: fsid = a1ef0711-bcd7-4e13-9432-5701b02e2426 2026-02-24T16:15:32.989 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:15:32.989 INFO:teuthology.orchestra.run.trial084.stdout:[osd] 2026-02-24T16:15:32.989 INFO:teuthology.orchestra.run.trial084.stdout: osd journal size = 100 2026-02-24T16:15:32.989 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:15:32.989 INFO:teuthology.orchestra.run.trial084.stdout: osd scrub load threshold = 5.0 2026-02-24T16:15:32.989 INFO:teuthology.orchestra.run.trial084.stdout: osd scrub max interval = 600 2026-02-24T16:15:32.990 INFO:teuthology.orchestra.run.trial084.stdout: osd mclock profile = high_recovery_ops 2026-02-24T16:15:32.990 INFO:teuthology.orchestra.run.trial084.stdout: osd mclock skip benchmark = true 2026-02-24T16:15:32.990 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:15:32.990 INFO:teuthology.orchestra.run.trial084.stdout: osd recover clone overlap = true 2026-02-24T16:15:32.990 INFO:teuthology.orchestra.run.trial084.stdout: osd recovery max chunk = 1048576 2026-02-24T16:15:32.990 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:15:32.990 INFO:teuthology.orchestra.run.trial084.stdout: osd debug shutdown = true 2026-02-24T16:15:32.990 INFO:teuthology.orchestra.run.trial084.stdout: osd debug op order = true 2026-02-24T16:15:32.990 INFO:teuthology.orchestra.run.trial084.stdout: osd debug verify stray on activate = true 2026-02-24T16:15:32.990 INFO:teuthology.orchestra.run.trial084.stdout: osd debug trim objects = true 2026-02-24T16:15:32.990 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:15:32.990 INFO:teuthology.orchestra.run.trial084.stdout: osd open classes on start = true 2026-02-24T16:15:32.990 INFO:teuthology.orchestra.run.trial084.stdout: osd debug pg log writeout = true 2026-02-24T16:15:32.991 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:15:32.991 INFO:teuthology.orchestra.run.trial084.stdout: osd deep scrub update digest min age = 30 2026-02-24T16:15:32.991 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:15:32.991 INFO:teuthology.orchestra.run.trial084.stdout: osd map max advance = 10 2026-02-24T16:15:32.991 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:15:32.991 INFO:teuthology.orchestra.run.trial084.stdout: journal zero on create = true 2026-02-24T16:15:32.991 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:15:32.991 INFO:teuthology.orchestra.run.trial084.stdout: filestore ondisk finisher threads = 3 2026-02-24T16:15:32.991 INFO:teuthology.orchestra.run.trial084.stdout: filestore apply finisher threads = 3 2026-02-24T16:15:32.991 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:15:32.991 INFO:teuthology.orchestra.run.trial084.stdout: bdev debug aio = true 2026-02-24T16:15:32.991 INFO:teuthology.orchestra.run.trial084.stdout: osd debug misdirected ops = true 2026-02-24T16:15:32.991 INFO:teuthology.orchestra.run.trial084.stdout: debug monc = 20 2026-02-24T16:15:32.992 INFO:teuthology.orchestra.run.trial084.stdout: crimson cpu num = 4 2026-02-24T16:15:32.992 INFO:teuthology.orchestra.run.trial084.stdout: crimson osd obc lru size = 10 2026-02-24T16:15:32.992 INFO:teuthology.orchestra.run.trial084.stdout: debug filestore = 0/0 2026-02-24T16:15:32.992 INFO:teuthology.orchestra.run.trial084.stdout: debug journal = 0/0 2026-02-24T16:15:32.992 INFO:teuthology.orchestra.run.trial084.stdout: debug ms = 0/0 2026-02-24T16:15:32.992 INFO:teuthology.orchestra.run.trial084.stdout: debug osd = 0/0 2026-02-24T16:15:32.992 INFO:teuthology.orchestra.run.trial084.stdout: debug seastore = 20 2026-02-24T16:15:32.992 INFO:teuthology.orchestra.run.trial084.stdout: debug seastore backref = 20 2026-02-24T16:15:32.992 INFO:teuthology.orchestra.run.trial084.stdout: debug seastore cache = 20 2026-02-24T16:15:32.992 INFO:teuthology.orchestra.run.trial084.stdout: debug seastore cleaner = 20 2026-02-24T16:15:32.992 INFO:teuthology.orchestra.run.trial084.stdout: debug seastore device = 20 2026-02-24T16:15:32.992 INFO:teuthology.orchestra.run.trial084.stdout: debug seastore epm = 20 2026-02-24T16:15:32.992 INFO:teuthology.orchestra.run.trial084.stdout: debug seastore fixedkv tree = 20 2026-02-24T16:15:32.993 INFO:teuthology.orchestra.run.trial084.stdout: debug seastore journal = 20 2026-02-24T16:15:32.993 INFO:teuthology.orchestra.run.trial084.stdout: debug seastore lba = 20 2026-02-24T16:15:32.993 INFO:teuthology.orchestra.run.trial084.stdout: debug seastore odata = 20 2026-02-24T16:15:32.993 INFO:teuthology.orchestra.run.trial084.stdout: debug seastore omap = 20 2026-02-24T16:15:32.993 INFO:teuthology.orchestra.run.trial084.stdout: debug seastore onode = 20 2026-02-24T16:15:32.993 INFO:teuthology.orchestra.run.trial084.stdout: debug seastore t = 20 2026-02-24T16:15:32.993 INFO:teuthology.orchestra.run.trial084.stdout: debug seastore tm = 20 2026-02-24T16:15:32.993 INFO:teuthology.orchestra.run.trial084.stdout: osd objectstore = seastore 2026-02-24T16:15:32.993 INFO:teuthology.orchestra.run.trial084.stdout: osd shutdown pgref assert = True 2026-02-24T16:15:32.993 INFO:teuthology.orchestra.run.trial084.stdout: seastore cachepin size pershard = 64M 2026-02-24T16:15:32.993 INFO:teuthology.orchestra.run.trial084.stdout: seastore main device type = RANDOM_BLOCK_SSD 2026-02-24T16:15:32.993 INFO:teuthology.orchestra.run.trial084.stdout: seastore max concurrent transactions = 8 2026-02-24T16:15:32.993 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:15:32.994 INFO:teuthology.orchestra.run.trial084.stdout:[mgr] 2026-02-24T16:15:32.994 INFO:teuthology.orchestra.run.trial084.stdout: debug ms = 1 2026-02-24T16:15:32.994 INFO:teuthology.orchestra.run.trial084.stdout: debug mgr = 20 2026-02-24T16:15:32.994 INFO:teuthology.orchestra.run.trial084.stdout: debug mon = 20 2026-02-24T16:15:32.994 INFO:teuthology.orchestra.run.trial084.stdout: debug auth = 20 2026-02-24T16:15:32.994 INFO:teuthology.orchestra.run.trial084.stdout: mon reweight min pgs per osd = 4 2026-02-24T16:15:32.994 INFO:teuthology.orchestra.run.trial084.stdout: mon reweight min bytes per osd = 10 2026-02-24T16:15:32.994 INFO:teuthology.orchestra.run.trial084.stdout: mgr/telemetry/nag = false 2026-02-24T16:15:32.994 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:15:32.994 INFO:teuthology.orchestra.run.trial084.stdout:[mon] 2026-02-24T16:15:32.994 INFO:teuthology.orchestra.run.trial084.stdout: debug ms = 0/0 2026-02-24T16:15:32.994 INFO:teuthology.orchestra.run.trial084.stdout: debug mon = 0/0 2026-02-24T16:15:32.994 INFO:teuthology.orchestra.run.trial084.stdout: debug paxos = 0/0 2026-02-24T16:15:32.995 INFO:teuthology.orchestra.run.trial084.stdout: debug auth = 20 2026-02-24T16:15:32.995 INFO:teuthology.orchestra.run.trial084.stdout: mon data avail warn = 5 2026-02-24T16:15:32.995 INFO:teuthology.orchestra.run.trial084.stdout: mon mgr mkfs grace = 240 2026-02-24T16:15:32.995 INFO:teuthology.orchestra.run.trial084.stdout: mon reweight min pgs per osd = 4 2026-02-24T16:15:32.995 INFO:teuthology.orchestra.run.trial084.stdout: mon osd reporter subtree level = osd 2026-02-24T16:15:32.995 INFO:teuthology.orchestra.run.trial084.stdout: mon osd prime pg temp = true 2026-02-24T16:15:32.995 INFO:teuthology.orchestra.run.trial084.stdout: mon reweight min bytes per osd = 10 2026-02-24T16:15:32.995 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:15:32.995 INFO:teuthology.orchestra.run.trial084.stdout: # rotate auth tickets quickly to exercise renewal paths 2026-02-24T16:15:32.995 INFO:teuthology.orchestra.run.trial084.stdout: auth mon ticket ttl = 660 # 11m 2026-02-24T16:15:32.995 INFO:teuthology.orchestra.run.trial084.stdout: auth service ticket ttl = 240 # 4m 2026-02-24T16:15:32.995 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:15:32.995 INFO:teuthology.orchestra.run.trial084.stdout: # don't complain about insecure global_id in the test suite 2026-02-24T16:15:32.996 INFO:teuthology.orchestra.run.trial084.stdout: mon_warn_on_insecure_global_id_reclaim = false 2026-02-24T16:15:32.996 INFO:teuthology.orchestra.run.trial084.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2026-02-24T16:15:32.996 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:15:32.996 INFO:teuthology.orchestra.run.trial084.stdout: # 1m isn't quite enough 2026-02-24T16:15:32.996 INFO:teuthology.orchestra.run.trial084.stdout: mon_down_mkfs_grace = 2m 2026-02-24T16:15:32.996 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:15:32.996 INFO:teuthology.orchestra.run.trial084.stdout: mon_warn_on_filestore_osds = false 2026-02-24T16:15:32.996 INFO:teuthology.orchestra.run.trial084.stdout: osd pool default crimson = True 2026-02-24T16:15:32.996 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:15:32.996 INFO:teuthology.orchestra.run.trial084.stdout:[client] 2026-02-24T16:15:32.996 INFO:teuthology.orchestra.run.trial084.stdout: rgw cache enabled = true 2026-02-24T16:15:32.996 INFO:teuthology.orchestra.run.trial084.stdout: rgw enable ops log = true 2026-02-24T16:15:32.996 INFO:teuthology.orchestra.run.trial084.stdout: rgw enable usage log = true 2026-02-24T16:15:32.997 INFO:teuthology.orchestra.run.trial084.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2026-02-24T16:15:32.997 INFO:teuthology.orchestra.run.trial084.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2026-02-24T16:15:32.997 INFO:teuthology.orchestra.run.trial084.stdout:[mon.a] 2026-02-24T16:15:32.998 INFO:tasks.ceph:Creating admin key on mon.a... 2026-02-24T16:15:32.998 DEBUG:teuthology.orchestra.run.trial084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2026-02-24T16:15:33.089 INFO:tasks.ceph:Copying monmap to all nodes... 2026-02-24T16:15:33.090 DEBUG:teuthology.orchestra.run.trial084:> set -ex 2026-02-24T16:15:33.090 DEBUG:teuthology.orchestra.run.trial084:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2026-02-24T16:15:33.107 DEBUG:teuthology.orchestra.run.trial084:> set -ex 2026-02-24T16:15:33.107 DEBUG:teuthology.orchestra.run.trial084:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2026-02-24T16:15:33.164 INFO:tasks.ceph:Sending monmap to node ubuntu@trial084.front.sepia.ceph.com 2026-02-24T16:15:33.164 DEBUG:teuthology.orchestra.run.trial084:> set -ex 2026-02-24T16:15:33.164 DEBUG:teuthology.orchestra.run.trial084:> sudo dd of=/etc/ceph/ceph.keyring 2026-02-24T16:15:33.164 DEBUG:teuthology.orchestra.run.trial084:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-24T16:15:33.236 DEBUG:teuthology.orchestra.run.trial084:> set -ex 2026-02-24T16:15:33.237 DEBUG:teuthology.orchestra.run.trial084:> dd of=/home/ubuntu/cephtest/ceph.monmap 2026-02-24T16:15:33.292 INFO:tasks.ceph:Setting up mon nodes... 2026-02-24T16:15:33.293 INFO:tasks.ceph:Setting up mgr nodes... 2026-02-24T16:15:33.293 DEBUG:teuthology.orchestra.run.trial084:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2026-02-24T16:15:33.382 INFO:teuthology.orchestra.run.trial084.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2026-02-24T16:15:33.385 INFO:tasks.ceph:Setting up mds nodes... 2026-02-24T16:15:33.386 INFO:tasks.ceph_client:Setting up client nodes... 2026-02-24T16:15:33.386 DEBUG:teuthology.orchestra.run.trial084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2026-02-24T16:15:33.427 INFO:teuthology.orchestra.run.trial084.stdout:creating /etc/ceph/ceph.client.0.keyring 2026-02-24T16:15:33.439 INFO:tasks.ceph:Running mkfs on osd nodes... 2026-02-24T16:15:33.439 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@trial084.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2026-02-24T16:15:33.440 DEBUG:teuthology.orchestra.run.trial084:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2026-02-24T16:15:33.502 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-24T16:15:33.503 INFO:tasks.ceph:role: osd.0 2026-02-24T16:15:33.503 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@trial084.front.sepia.ceph.com 2026-02-24T16:15:33.503 DEBUG:teuthology.orchestra.run.trial084:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2026-02-24T16:15:33.568 INFO:teuthology.orchestra.run.trial084.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=45875200 blks 2026-02-24T16:15:33.568 INFO:teuthology.orchestra.run.trial084.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-24T16:15:33.568 INFO:teuthology.orchestra.run.trial084.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-24T16:15:33.569 INFO:teuthology.orchestra.run.trial084.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-24T16:15:33.569 INFO:teuthology.orchestra.run.trial084.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-24T16:15:33.569 INFO:teuthology.orchestra.run.trial084.stdout: = sunit=0 swidth=0 blks 2026-02-24T16:15:33.569 INFO:teuthology.orchestra.run.trial084.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-24T16:15:33.569 INFO:teuthology.orchestra.run.trial084.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-24T16:15:33.569 INFO:teuthology.orchestra.run.trial084.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-24T16:15:33.569 INFO:teuthology.orchestra.run.trial084.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-24T16:15:34.143 INFO:teuthology.orchestra.run.trial084.stdout:Discarding blocks...Done. 2026-02-24T16:15:34.145 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@trial084.front.sepia.ceph.com -o noatime 2026-02-24T16:15:34.145 DEBUG:teuthology.orchestra.run.trial084:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2026-02-24T16:15:34.213 DEBUG:teuthology.orchestra.run.trial084:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2026-02-24T16:15:34.285 DEBUG:teuthology.orchestra.run.trial084:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2026-02-24T16:15:34.348 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-24T16:15:34.348 INFO:tasks.ceph:role: osd.1 2026-02-24T16:15:34.348 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@trial084.front.sepia.ceph.com 2026-02-24T16:15:34.348 DEBUG:teuthology.orchestra.run.trial084:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2026-02-24T16:15:34.413 INFO:teuthology.orchestra.run.trial084.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=45875200 blks 2026-02-24T16:15:34.413 INFO:teuthology.orchestra.run.trial084.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-24T16:15:34.413 INFO:teuthology.orchestra.run.trial084.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-24T16:15:34.413 INFO:teuthology.orchestra.run.trial084.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-24T16:15:34.413 INFO:teuthology.orchestra.run.trial084.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-24T16:15:34.413 INFO:teuthology.orchestra.run.trial084.stdout: = sunit=0 swidth=0 blks 2026-02-24T16:15:34.413 INFO:teuthology.orchestra.run.trial084.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-24T16:15:34.413 INFO:teuthology.orchestra.run.trial084.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-24T16:15:34.413 INFO:teuthology.orchestra.run.trial084.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-24T16:15:34.413 INFO:teuthology.orchestra.run.trial084.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-24T16:15:34.972 INFO:teuthology.orchestra.run.trial084.stdout:Discarding blocks...Done. 2026-02-24T16:15:34.974 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@trial084.front.sepia.ceph.com -o noatime 2026-02-24T16:15:34.975 DEBUG:teuthology.orchestra.run.trial084:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2026-02-24T16:15:35.047 DEBUG:teuthology.orchestra.run.trial084:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2026-02-24T16:15:35.117 DEBUG:teuthology.orchestra.run.trial084:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2026-02-24T16:15:35.180 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-24T16:15:35.180 INFO:tasks.ceph:role: osd.2 2026-02-24T16:15:35.180 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@trial084.front.sepia.ceph.com 2026-02-24T16:15:35.180 DEBUG:teuthology.orchestra.run.trial084:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2026-02-24T16:15:35.245 INFO:teuthology.orchestra.run.trial084.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=45875200 blks 2026-02-24T16:15:35.245 INFO:teuthology.orchestra.run.trial084.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-24T16:15:35.245 INFO:teuthology.orchestra.run.trial084.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-24T16:15:35.245 INFO:teuthology.orchestra.run.trial084.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-24T16:15:35.245 INFO:teuthology.orchestra.run.trial084.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-24T16:15:35.245 INFO:teuthology.orchestra.run.trial084.stdout: = sunit=0 swidth=0 blks 2026-02-24T16:15:35.245 INFO:teuthology.orchestra.run.trial084.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-24T16:15:35.245 INFO:teuthology.orchestra.run.trial084.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-24T16:15:35.245 INFO:teuthology.orchestra.run.trial084.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-24T16:15:35.245 INFO:teuthology.orchestra.run.trial084.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-24T16:15:35.800 INFO:teuthology.orchestra.run.trial084.stdout:Discarding blocks...Done. 2026-02-24T16:15:35.803 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@trial084.front.sepia.ceph.com -o noatime 2026-02-24T16:15:35.803 DEBUG:teuthology.orchestra.run.trial084:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2026-02-24T16:15:35.871 DEBUG:teuthology.orchestra.run.trial084:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2026-02-24T16:15:35.942 DEBUG:teuthology.orchestra.run.trial084:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-24T16:15:36.094 INFO:teuthology.orchestra.run.trial084.stderr:INFO  osd - OSD::main: parsing early config 2026-02-24T16:15:36.146 INFO:teuthology.orchestra.run.trial084.stderr:INFO 2026-02-24 16:15:36,145 seastar - Reactor backend: linux-aio 2026-02-24T16:15:36.155 INFO:teuthology.orchestra.run.trial084.stderr:INFO 2026-02-24 16:15:36,155 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-24T16:15:36.155 INFO:teuthology.orchestra.run.trial084.stderr:INFO 2026-02-24 16:15:36,155 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-24T16:15:36.157 INFO:teuthology.orchestra.run.trial084.stderr:==39231==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-24T16:15:36.217 INFO:teuthology.orchestra.run.trial084.stderr:DEBUG 2026-02-24 16:15:36,216 [shard 0:main] seastar - reactor::drain 2026-02-24T16:15:38.237 INFO:teuthology.orchestra.run.trial084.stderr:----------------------------------------------------- 2026-02-24T16:15:38.237 INFO:teuthology.orchestra.run.trial084.stderr:Suppressions used: 2026-02-24T16:15:38.237 INFO:teuthology.orchestra.run.trial084.stderr: count bytes template 2026-02-24T16:15:38.238 INFO:teuthology.orchestra.run.trial084.stderr: 1 8 MallocExtension::Register 2026-02-24T16:15:38.238 INFO:teuthology.orchestra.run.trial084.stderr:----------------------------------------------------- 2026-02-24T16:15:38.238 INFO:teuthology.orchestra.run.trial084.stderr: 2026-02-24T16:15:38.257 INFO:teuthology.orchestra.run.trial084.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-24T16:15:38.257 INFO:teuthology.orchestra.run.trial084.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-24T16:15:38.302 INFO:teuthology.orchestra.run.trial084.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-24T16:15:38.302 INFO:teuthology.orchestra.run.trial084.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-24T16:15:38.309 INFO:teuthology.orchestra.run.trial084.stderr:INFO 2026-02-24 16:15:38,309 seastar - Reactor backend: linux-aio 2026-02-24T16:15:38.320 INFO:teuthology.orchestra.run.trial084.stderr:INFO 2026-02-24 16:15:38,319 [shard 2:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-24T16:15:38.320 INFO:teuthology.orchestra.run.trial084.stderr:INFO 2026-02-24 16:15:38,319 [shard 2:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-24T16:15:38.324 INFO:teuthology.orchestra.run.trial084.stderr:==39229==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-24T16:15:38.324 INFO:teuthology.orchestra.run.trial084.stderr:INFO 2026-02-24 16:15:38,324 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-24T16:15:38.482 INFO:teuthology.orchestra.run.trial084.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x191c1b /lib64/libasan.so.8+0xc748 /lib64/libasan.so.8+0x91c1 /lib64/libasan.so.8+0xee7dc 0x9ec9adb 0x9f2f2d6 0x9f2f3ec 0x9f2f4c8 0x9f2f744 0x9f2fceb 0x9f300ef 0x9f303ac 0xa0d4a95 0xa096e39 0xa095e39 0x7eff1a0 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7272fae 0x7276b31 0x72f0839 0x7747f18 0x755fef8 0x7585796 0x752dc84 0x72cd4a6 0x7632ed7 0x7534252 0x72cd53c 0x76d5c2f 0x7747fa1 0x755fffa 0x75600f2 0x7560178 0x75601fd 0x758594e 0x7748022 0x756027a 0x75602ff 0x7585b55 0x76e815e 0x76e81e3 0x76e82c1 0x77428e2 0x7699a68 0x784409e 0x787bf26 0x76e6210 0x77b2de2 0x78d6e12 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:15:38.482 INFO:teuthology.orchestra.run.trial084.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffa29ccb77 0xffffffffa29d4bab 0xffffffffa29d500b 0xffffffffa26910d2 0xffffffffa3350451 0xffffffffa3400ba2 2026-02-24T16:15:38.482 INFO:teuthology.orchestra.run.trial084.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc250 /lib64/libasan.so.8+0xc553 /lib64/libasan.so.8+0x8eea /lib64/libasan.so.8+0xedc45 0xe6ca27 0xe6cd08 0x77b1595 0x77b1696 0x763e27c 0x7bc7238 0x7efedf5 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7272fae 0x7276b31 0x72f0839 0x7747f18 0x755fef8 0x7585796 0x752dc84 0x72cd4a6 0x7632ed7 0x7534252 0x72cd53c 0x76d5c2f 0x7747fa1 0x755fffa 0x75600f2 0x7560178 0x75601fd 0x758594e 0x7748022 0x756027a 0x75602ff 0x7585b55 0x76e815e 0x76e81e3 0x76e82c1 0x77428e2 0x7699a68 0x784409e 0x787bf26 0x76e6210 0x77b2de2 0x78d6e12 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:15:38.482 INFO:teuthology.orchestra.run.trial084.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xfeef0 /lib64/libasan.so.8+0xfd214 /lib64/libasan.so.8+0xedcb5 0xe5abd5 0xe5aef8 0xa0b497c 0xa0d1bdf 0xa0d68c7 0xa096e39 0xa095e39 0x7eff1a0 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7272fae 0x7276b31 0x72f0839 0x7747f18 0x755fef8 0x7585796 0x752dc84 0x72cd4a6 0x7632ed7 0x7534252 0x72cd53c 0x76d5c2f 0x7747fa1 0x755fffa 0x75600f2 0x7560178 0x75601fd 0x758594e 0x7748022 0x756027a 0x75602ff 0x7585b55 0x76e815e 0x76e81e3 0x76e82c1 0x77428e2 0x7699a68 0x784409e 0x787bf26 0x76e6210 0x77b2de2 0x78d6e12 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:15:38.514 INFO:teuthology.orchestra.run.trial084.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x15f26 0xa0ce427 0xa0d87eb 0x7bc72d4 0x7eff0d5 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7273c89 0x781ca8c 0x781cbe0 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fbc5e 0x9f82d83 0x9f835be 0xf2905b /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb83e94 2026-02-24T16:15:38.573 INFO:teuthology.orchestra.run.trial084.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x191c62 /lib64/libasan.so.8+0xc748 /lib64/libasan.so.8+0x8eea /lib64/libasan.so.8+0xedc45 0xe6ca27 0xe6cd08 0x77b1595 0x77b1696 0x763e27c 0x7bc7238 0x7eff0d5 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7273c89 0x781ca8c 0x781cbe0 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:15:38.574 INFO:teuthology.orchestra.run.trial084.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f 0x9fe8eb3 0xa0d8a91 0x7bc72d4 0x7eff0d5 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7273c89 0x781ca8c 0x781cbe0 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:15:38.574 INFO:teuthology.orchestra.run.trial084.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xeab04 0xd1c39c 0xe5ad37 0xe5ae0b 0x7eeb463 0x7f0412c 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7273c89 0x781ca8c 0x781cbe0 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:15:38.685 INFO:teuthology.orchestra.run.trial084.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f 0xd1c39c 0xe5aa9b 0xe5ae0b 0x7bc7158 0x7efff65 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x722be9f 0x722c8e3 0x77424df 0x755e078 0x757f093 0x752da6e 0x71c9494 0x763159f 0x7533811 0x71c952a 0x76d525e 0x77426e1 0x755e1fe 0x755e2f6 0x755e37c 0x755e401 0x757f24a 0x7742762 0x755e47e 0x755e503 0x757f451 0x76e5fc1 0x76e6046 0x76e6124 0x77428e2 0x7699a68 0x784409e 0x787bf26 0x76e6210 0x77b2de2 0x78d6e12 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:15:38.685 INFO:teuthology.orchestra.run.trial084.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f 0xa0c2888 0xa0c2f8a 0xa0c330d 0xa0d3464 0xa0d68c7 0xa096e39 0xa095e39 0x7eff1a0 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x722be9f 0x722c8e3 0x77424df 0x755e078 0x757f093 0x752da6e 0x71c9494 0x763159f 0x7533811 0x71c952a 0x76d525e 0x77426e1 0x755e1fe 0x755e2f6 0x755e37c 0x755e401 0x757f24a 0x7742762 0x755e47e 0x755e503 0x757f451 0x76e5fc1 0x76e6046 0x76e6124 0x77428e2 0x7699a68 0x784409e 0x787bf26 0x76e6210 0x77b2de2 0x78d6e12 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:15:38.685 INFO:teuthology.orchestra.run.trial084.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xb13ef 0xba66a7 0x10d1d1e 0xf741d0 0x9fe4d11 0x9fe500e 0x9fe5281 0x9fe5537 0x9fe5fe5 0x9fe64e3 0x9fe6b13 0x9fe6c7d 0xa0c203d 0xa0d4e40 0xa096e39 0xa095e39 0x7eff1a0 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x722be9f 0x722c8e3 0x77424df 0x755e078 0x757f093 0x752da6e 0x71c9494 0x763159f 0x7533811 0x71c952a 0x76d525e 0x77426e1 0x755e1fe 0x755e2f6 0x755e37c 0x755e401 0x757f24a 0x7742762 0x755e47e 0x755e503 0x757f451 0x76e5fc1 0x76e6046 0x76e6124 0x77428e2 0x7699a68 0x784409e 0x787bf26 0x76e6210 0x77b2de2 0x78d6e12 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:15:40.822 INFO:teuthology.orchestra.run.trial084.stderr:----------------------------------------------------- 2026-02-24T16:15:40.823 INFO:teuthology.orchestra.run.trial084.stderr:Suppressions used: 2026-02-24T16:15:40.823 INFO:teuthology.orchestra.run.trial084.stderr: count bytes template 2026-02-24T16:15:40.823 INFO:teuthology.orchestra.run.trial084.stderr: 1 8 MallocExtension::Register 2026-02-24T16:15:40.823 INFO:teuthology.orchestra.run.trial084.stderr:----------------------------------------------------- 2026-02-24T16:15:40.823 INFO:teuthology.orchestra.run.trial084.stderr: 2026-02-24T16:15:40.859 DEBUG:teuthology.orchestra.run.trial084:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2026-02-24T16:15:40.885 DEBUG:teuthology.orchestra.run.trial084:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-24T16:15:41.038 INFO:teuthology.orchestra.run.trial084.stderr:INFO  osd - OSD::main: parsing early config 2026-02-24T16:15:41.094 INFO:teuthology.orchestra.run.trial084.stderr:INFO 2026-02-24 16:15:41,093 seastar - Reactor backend: linux-aio 2026-02-24T16:15:41.106 INFO:teuthology.orchestra.run.trial084.stderr:INFO 2026-02-24 16:15:41,105 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-24T16:15:41.106 INFO:teuthology.orchestra.run.trial084.stderr:INFO 2026-02-24 16:15:41,105 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-24T16:15:41.108 INFO:teuthology.orchestra.run.trial084.stderr:==39299==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-24T16:15:41.167 INFO:teuthology.orchestra.run.trial084.stderr:DEBUG 2026-02-24 16:15:41,166 [shard 0:main] seastar - reactor::drain 2026-02-24T16:15:43.234 INFO:teuthology.orchestra.run.trial084.stderr:----------------------------------------------------- 2026-02-24T16:15:43.234 INFO:teuthology.orchestra.run.trial084.stderr:Suppressions used: 2026-02-24T16:15:43.235 INFO:teuthology.orchestra.run.trial084.stderr: count bytes template 2026-02-24T16:15:43.235 INFO:teuthology.orchestra.run.trial084.stderr: 1 8 MallocExtension::Register 2026-02-24T16:15:43.235 INFO:teuthology.orchestra.run.trial084.stderr:----------------------------------------------------- 2026-02-24T16:15:43.235 INFO:teuthology.orchestra.run.trial084.stderr: 2026-02-24T16:15:43.254 INFO:teuthology.orchestra.run.trial084.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-24T16:15:43.254 INFO:teuthology.orchestra.run.trial084.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-24T16:15:43.299 INFO:teuthology.orchestra.run.trial084.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-24T16:15:43.300 INFO:teuthology.orchestra.run.trial084.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-24T16:15:43.306 INFO:teuthology.orchestra.run.trial084.stderr:INFO 2026-02-24 16:15:43,305 seastar - Reactor backend: linux-aio 2026-02-24T16:15:43.317 INFO:teuthology.orchestra.run.trial084.stderr:INFO 2026-02-24 16:15:43,317 [shard 3:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-24T16:15:43.318 INFO:teuthology.orchestra.run.trial084.stderr:INFO 2026-02-24 16:15:43,317 [shard 3:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-24T16:15:43.321 INFO:teuthology.orchestra.run.trial084.stderr:==39297==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-24T16:15:43.321 INFO:teuthology.orchestra.run.trial084.stderr:INFO 2026-02-24 16:15:43,320 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-24T16:15:43.478 INFO:teuthology.orchestra.run.trial084.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f 0xa0c1a3b 0xa0c1e26 0xa0c21b9 0xa0d4e40 0xa096e39 0xa095e39 0x7eff1a0 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7272fae 0x7276b31 0x72f0839 0x7747f18 0x755fef8 0x7585796 0x752dc84 0x72cd4a6 0x7632ed7 0x7534252 0x72cd53c 0x76d5c2f 0x7747fa1 0x755fffa 0x75600f2 0x7560178 0x75601fd 0x758594e 0x7748022 0x756027a 0x75602ff 0x7585b55 0x76e815e 0x76e81e3 0x76e82c1 0x77428e2 0x7699a68 0x784409e 0x787bf26 0x76e6210 0x77b2de2 0x78d6e12 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:15:43.478 INFO:teuthology.orchestra.run.trial084.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f 0x38d43ee 0x38d45db 0x38f0458 0x9ff8673 0x9ff877b 0xa0b171c 0xa0b18b1 0xa0b1c43 0xa0b3229 0xa0b4544 0xa0b4964 0xa0d1bdf 0xa0d68c7 0xa096e39 0xa095e39 0x7eff1a0 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7272fae 0x7276b31 0x72f0839 0x7747f18 0x755fef8 0x7585796 0x752dc84 0x72cd4a6 0x7632ed7 0x7534252 0x72cd53c 0x76d5c2f 0x7747fa1 0x755fffa 0x75600f2 0x7560178 0x75601fd 0x758594e 0x7748022 0x756027a 0x75602ff 0x7585b55 0x76e815e 0x76e81e3 0x76e82c1 0x77428e2 0x7699a68 0x784409e 0x787bf26 0x76e6210 0x77b2de2 0x78d6e12 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:15:43.479 INFO:teuthology.orchestra.run.trial084.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x16b36a /lib64/libasan.so.8+0xb0541 /lib64/libasan.so.8+0xb1437 0xba66a7 0x10d1d1e 0xf741d0 0x9fe4d11 0x9fe500e 0x9fe5281 0x9fe5537 0x9fe5fe5 0x9fe64e3 0x9fe6b13 0x9fe6c7d 0xa0c203d 0xa0d4e40 0xa096e39 0xa095e39 0x7eff1a0 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7272fae 0x7276b31 0x72f0839 0x7747f18 0x755fef8 0x7585796 0x752dc84 0x72cd4a6 0x7632ed7 0x7534252 0x72cd53c 0x76d5c2f 0x7747fa1 0x755fffa 0x75600f2 0x7560178 0x75601fd 0x758594e 0x7748022 0x756027a 0x75602ff 0x7585b55 0x76e815e 0x76e81e3 0x76e82c1 0x77428e2 0x7699a68 0x784409e 0x787bf26 0x76e6210 0x77b2de2 0x78d6e12 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:15:43.511 INFO:teuthology.orchestra.run.trial084.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x361f /lib64/libasan.so.8+0x15e37 0x9fe4ecf 0x9fe5281 0x9fe5537 0x9fe5fe5 0x9fe64e3 0x9fe6b13 0x9fe6c7d 0xa0c203d 0xa0d4e40 0xa096e39 0xa095e39 0x7eff1a0 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7273c89 0x781ca8c 0x781cbe0 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fbc5e 0x9f82d83 0x9f835be 0xf2905b /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb83e94 2026-02-24T16:15:43.537 INFO:teuthology.orchestra.run.trial084.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f 0xe6cd08 0xa0d88c9 0x7bc72d4 0x7efedf5 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7273c89 0x781ca8c 0x781cbe0 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:15:43.571 INFO:teuthology.orchestra.run.trial084.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xeb84 /lib64/libasan.so.8+0xec776 0xcee0e1 0x773f96f 0x75cae6c 0x74261b4 0x7eff288 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7273c89 0x781ca8c 0x781cbe0 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:15:43.572 INFO:teuthology.orchestra.run.trial084.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x12976b /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x15197 0x116816a 0xf741c5 0x9fe4d11 0x9fe500e 0x9fe5281 0x9fe6fa7 0x9fe74f4 0x9fe76f3 0xa0c7a65 0xa0c81c0 0xa0d244e 0xa0d68c7 0xa096e39 0xa095e39 0x7eff1a0 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7273c89 0x781ca8c 0x781cbe0 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:15:43.687 INFO:teuthology.orchestra.run.trial084.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x16b357 /lib64/libasan.so.8+0xb0541 /lib64/libasan.so.8+0xb1437 0xba66a7 0x10d1d1e 0xf741d0 0x9fe4b44 0x9fe500e 0x9fe5281 0x9fe6280 0x9fe67ee 0x9fe6b13 0x9fe6c7d 0xa0c203d 0xa0d4e40 0xa096e39 0xa095e39 0x7eff1a0 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x722be9f 0x722c8e3 0x77424df 0x755e078 0x757f093 0x752da6e 0x71c9494 0x763159f 0x7533811 0x71c952a 0x76d525e 0x77426e1 0x755e1fe 0x755e2f6 0x755e37c 0x755e401 0x757f24a 0x7742762 0x755e47e 0x755e503 0x757f451 0x76e5fc1 0x76e6046 0x76e6124 0x77428e2 0x7699a68 0x784409e 0x787bf26 0x76e6210 0x77b2de2 0x78d6e12 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:15:43.687 INFO:teuthology.orchestra.run.trial084.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x91b5 /lib64/libasan.so.8+0xee7dc 0x9ec9adb 0x9f2f2d6 0x9f2f3ec 0x9f2f4c8 0x9f2f744 0x9f2fa47 0x9f2fceb 0x9f300ef 0x9f303ac 0x9f311f4 0xa0a9b3b 0xa0a9c35 0x7efee32 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x722be9f 0x722c8e3 0x77424df 0x755e078 0x757f093 0x752da6e 0x71c9494 0x763159f 0x7533811 0x71c952a 0x76d525e 0x77426e1 0x755e1fe 0x755e2f6 0x755e37c 0x755e401 0x757f24a 0x7742762 0x755e47e 0x755e503 0x757f451 0x76e5fc1 0x76e6046 0x76e6124 0x77428e2 0x7699a68 0x784409e 0x787bf26 0x76e6210 0x77b2de2 0x78d6e12 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:15:43.688 INFO:teuthology.orchestra.run.trial084.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc597 /lib64/libasan.so.8+0x8eea /lib64/libasan.so.8+0xedc45 0xe6ca27 0xe6cd08 0x77b1595 0x77b1696 0x763e27c 0x7bc7238 0x7eff0d5 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x722be9f 0x722c8e3 0x77424df 0x755e078 0x757f093 0x752da6e 0x71c9494 0x763159f 0x7533811 0x71c952a 0x76d525e 0x77426e1 0x755e1fe 0x755e2f6 0x755e37c 0x755e401 0x757f24a 0x7742762 0x755e47e 0x755e503 0x757f451 0x76e5fc1 0x76e6046 0x76e6124 0x77428e2 0x7699a68 0x784409e 0x787bf26 0x76e6210 0x77b2de2 0x78d6e12 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:15:43.688 INFO:teuthology.orchestra.run.trial084.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffa281fee1 0xffffffffa28203b0 0xffffffffa269525e 0xffffffffa29e7269 0xffffffffa29ca7d5 0xffffffffa29d4bab 0xffffffffa29d500b 0xffffffffa26910d2 0xffffffffa3350451 0xffffffffa3400ba2 2026-02-24T16:15:45.876 INFO:teuthology.orchestra.run.trial084.stderr:----------------------------------------------------- 2026-02-24T16:15:45.876 INFO:teuthology.orchestra.run.trial084.stderr:Suppressions used: 2026-02-24T16:15:45.877 INFO:teuthology.orchestra.run.trial084.stderr: count bytes template 2026-02-24T16:15:45.877 INFO:teuthology.orchestra.run.trial084.stderr: 1 8 MallocExtension::Register 2026-02-24T16:15:45.877 INFO:teuthology.orchestra.run.trial084.stderr:----------------------------------------------------- 2026-02-24T16:15:45.877 INFO:teuthology.orchestra.run.trial084.stderr: 2026-02-24T16:15:45.911 DEBUG:teuthology.orchestra.run.trial084:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2026-02-24T16:15:45.935 DEBUG:teuthology.orchestra.run.trial084:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-24T16:15:46.087 INFO:teuthology.orchestra.run.trial084.stderr:INFO  osd - OSD::main: parsing early config 2026-02-24T16:15:46.146 INFO:teuthology.orchestra.run.trial084.stderr:INFO 2026-02-24 16:15:46,145 seastar - Reactor backend: linux-aio 2026-02-24T16:15:46.155 INFO:teuthology.orchestra.run.trial084.stderr:INFO 2026-02-24 16:15:46,155 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-24T16:15:46.155 INFO:teuthology.orchestra.run.trial084.stderr:INFO 2026-02-24 16:15:46,155 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-24T16:15:46.157 INFO:teuthology.orchestra.run.trial084.stderr:==39366==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-24T16:15:46.217 INFO:teuthology.orchestra.run.trial084.stderr:DEBUG 2026-02-24 16:15:46,216 [shard 0:main] seastar - reactor::drain 2026-02-24T16:15:48.236 INFO:teuthology.orchestra.run.trial084.stderr:----------------------------------------------------- 2026-02-24T16:15:48.236 INFO:teuthology.orchestra.run.trial084.stderr:Suppressions used: 2026-02-24T16:15:48.237 INFO:teuthology.orchestra.run.trial084.stderr: count bytes template 2026-02-24T16:15:48.237 INFO:teuthology.orchestra.run.trial084.stderr: 1 8 MallocExtension::Register 2026-02-24T16:15:48.237 INFO:teuthology.orchestra.run.trial084.stderr:----------------------------------------------------- 2026-02-24T16:15:48.237 INFO:teuthology.orchestra.run.trial084.stderr: 2026-02-24T16:15:48.256 INFO:teuthology.orchestra.run.trial084.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-24T16:15:48.256 INFO:teuthology.orchestra.run.trial084.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-24T16:15:48.297 INFO:teuthology.orchestra.run.trial084.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-24T16:15:48.298 INFO:teuthology.orchestra.run.trial084.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-24T16:15:48.304 INFO:teuthology.orchestra.run.trial084.stderr:INFO 2026-02-24 16:15:48,304 seastar - Reactor backend: linux-aio 2026-02-24T16:15:48.314 INFO:teuthology.orchestra.run.trial084.stderr:INFO 2026-02-24 16:15:48,314 [shard 2:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-24T16:15:48.315 INFO:teuthology.orchestra.run.trial084.stderr:INFO 2026-02-24 16:15:48,314 [shard 2:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-24T16:15:48.320 INFO:teuthology.orchestra.run.trial084.stderr:==39364==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-24T16:15:48.320 INFO:teuthology.orchestra.run.trial084.stderr:INFO 2026-02-24 16:15:48,319 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-24T16:15:48.477 INFO:teuthology.orchestra.run.trial084.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f 0xcee072 0xa0d34df 0xa0d68c7 0xa096e39 0xa095e39 0x7eff1a0 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7272fae 0x7276b31 0x72f0839 0x7747f18 0x755fef8 0x7585796 0x752dc84 0x72cd4a6 0x7632ed7 0x7534252 0x72cd53c 0x76d5c2f 0x7747fa1 0x755fffa 0x75600f2 0x7560178 0x75601fd 0x758594e 0x7748022 0x756027a 0x75602ff 0x7585b55 0x76e815e 0x76e81e3 0x76e82c1 0x77428e2 0x7699a68 0x784409e 0x787bf26 0x76e6210 0x77b2de2 0x78d6e12 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:15:48.477 INFO:teuthology.orchestra.run.trial084.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x11978f /lib64/libasan.so.8+0xfd239 /lib64/libasan.so.8+0xef979 0x3422b88 0x3422d62 0x3423019 0x342310e 0x7242ba8 0x7eff220 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7272fae 0x7276b31 0x72f0839 0x7747f18 0x755fef8 0x7585796 0x752dc84 0x72cd4a6 0x7632ed7 0x7534252 0x72cd53c 0x76d5c2f 0x7747fa1 0x755fffa 0x75600f2 0x7560178 0x75601fd 0x758594e 0x7748022 0x756027a 0x75602ff 0x7585b55 0x76e815e 0x76e81e3 0x76e82c1 0x77428e2 0x7699a68 0x784409e 0x787bf26 0x76e6210 0x77b2de2 0x78d6e12 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:15:48.477 INFO:teuthology.orchestra.run.trial084.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f 0x3422b97 0x3423019 0x342310e 0x7242ba8 0xa0b74d2 0xa0b76ea 0xa0b79bc 0x7eff1de 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7272fae 0x7276b31 0x72f0839 0x7747f18 0x755fef8 0x7585796 0x752dc84 0x72cd4a6 0x7632ed7 0x7534252 0x72cd53c 0x76d5c2f 0x7747fa1 0x755fffa 0x75600f2 0x7560178 0x75601fd 0x758594e 0x7748022 0x756027a 0x75602ff 0x7585b55 0x76e815e 0x76e81e3 0x76e82c1 0x77428e2 0x7699a68 0x784409e 0x787bf26 0x76e6210 0x77b2de2 0x78d6e12 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:15:48.509 INFO:teuthology.orchestra.run.trial084.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x8cc0f /lib64/libasan.so.8+0x10069c /lib64/libasan.so.8+0xc65b9 /lib64/libasan.so.8+0x15197 0x9fa8665 0xa0c464a 0xa0c8dbc 0xa0cdce0 0xa0ce31e 0xa0d87eb 0x7bc72d4 0x7efedf5 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7273c89 0x781ca8c 0x781cbe0 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fbc5e 0x9f82d83 0x9f835be 0xf2905b /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb83e94 2026-02-24T16:15:48.568 INFO:teuthology.orchestra.run.trial084.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x129787 /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x15197 0x116816a 0xf741c5 0x9fe4d11 0x9fe500e 0x9fe5281 0x9fe6fa7 0x9fe74f4 0x9fe76f3 0xa0c7a65 0xa0c81c0 0xa0d244e 0xa0d68c7 0xa096e39 0xa095e39 0x7eff1a0 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7273c89 0x781ca8c 0x781cbe0 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:15:48.569 INFO:teuthology.orchestra.run.trial084.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc580 /lib64/libasan.so.8+0x91c1 /lib64/libasan.so.8+0xee7dc 0x9ec9adb 0xa09aab6 0xa09abe4 0xa0cdb7b 0xa0ce31e 0xa0d87eb 0x7bc72d4 0x7eff0d5 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7273c89 0x781ca8c 0x781cbe0 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:15:48.570 INFO:teuthology.orchestra.run.trial084.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f 0xe55928 0x1168113 0xf741c5 0x9fe4d11 0x9fe500e 0x9fe5281 0x9fe5537 0x9fe5fe5 0x9fe64e3 0x9fe6b13 0x9fe6c7d 0xa0c203d 0xa0d4e40 0xa096e39 0xa095e39 0x7eff1a0 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7273c89 0x781ca8c 0x781cbe0 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:15:48.613 INFO:teuthology.orchestra.run.trial084.stderr:Reactor stalled for 32 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x8cc25 /lib64/libasan.so.8+0x10069c /lib64/libasan.so.8+0xc65b9 /lib64/libasan.so.8+0x15197 0x116816a 0xf741c5 0x9fe4b44 0x9fe500e 0x9fe5281 0x9fe5537 0x9fe5fe5 0x9fe64e3 0x9fe6b13 0x9fe6c7d 0xa0c203d 0xa0d4e40 0xa096e39 0xa095e39 0x7eff1a0 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x722be9f 0x722c8e3 0x77424df 0x755e078 0x757f093 0x752da6e 0x71c9494 0x763159f 0x7533811 0x71c952a 0x76d525e 0x77426e1 0x755e1fe 0x755e2f6 0x755e37c 0x755e401 0x757f24a 0x7742762 0x755e47e 0x755e503 0x757f451 0x76e5fc1 0x76e6046 0x76e6124 0x77428e2 0x7699a68 0x784409e 0x787bf26 0x76e6210 0x77b2de2 0x77b6764 0x76973b7 0x787c3c0 0x76e6d14 0x76e6e2d 0xa3a2446 0xa3a2649 0xa3a26e7 0xa3a276c 0xa3f5b6e 0xa3f5c27 0xa427569 0xa427910 0xa32dc97 0xa32dd50 0xa32dddf 0xa3f56d6 0xa3f5799 0xa3f58a6 0xa3f5a61 0x763eb45 0x7697625 0x769785c 0x7538bad 0x726516c 0x781cf48 0x781d09c 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fbc5e 0x9f82d83 0x9f835be 0xf2905b /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb83e94 2026-02-24T16:15:48.687 INFO:teuthology.orchestra.run.trial084.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc65bc /lib64/libasan.so.8+0x157c7 0x10d1d95 0xf741d0 0x9fe882e 0x9fe916d 0xa0ce105 0xa0d87eb 0x7bc72d4 0x7efedf5 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x722be9f 0x722c8e3 0x77424df 0x755e078 0x757f093 0x752da6e 0x71c9494 0x763159f 0x7533811 0x71c952a 0x76d525e 0x77426e1 0x755e1fe 0x755e2f6 0x755e37c 0x755e401 0x757f24a 0x7742762 0x755e47e 0x755e503 0x757f451 0x76e5fc1 0x76e6046 0x76e6124 0x77428e2 0x7699a68 0x784409e 0x787bf26 0x76e6210 0x77b2de2 0x78d6e12 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:15:48.687 INFO:teuthology.orchestra.run.trial084.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x16b364 /lib64/libasan.so.8+0xb0541 /lib64/libasan.so.8+0xb1437 0xba66a7 0x10d1d1e 0xf741d0 0x9fe4b44 0x9fe500e 0x9fe5281 0x9fe5537 0x9fe5af8 0x9fec307 0x9fec9e7 0xa0d221c 0xa0d68c7 0xa096e39 0xa095e39 0x7eff1a0 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x722be9f 0x722c8e3 0x77424df 0x755e078 0x757f093 0x752da6e 0x71c9494 0x763159f 0x7533811 0x71c952a 0x76d525e 0x77426e1 0x755e1fe 0x755e2f6 0x755e37c 0x755e401 0x757f24a 0x7742762 0x755e47e 0x755e503 0x757f451 0x76e5fc1 0x76e6046 0x76e6124 0x77428e2 0x7699a68 0x784409e 0x787bf26 0x76e6210 0x77b2de2 0x78d6e12 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:15:48.688 INFO:teuthology.orchestra.run.trial084.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x11916f /lib64/libasan.so.8+0xebbf /lib64/libasan.so.8+0xec776 0xcee0e1 0x773f994 0x75cae6c 0x74261b4 0x7bc7311 0x7efedf5 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x722be9f 0x722c8e3 0x77424df 0x755e078 0x757f093 0x752da6e 0x71c9494 0x763159f 0x7533811 0x71c952a 0x76d525e 0x77426e1 0x755e1fe 0x755e2f6 0x755e37c 0x755e401 0x757f24a 0x7742762 0x755e47e 0x755e503 0x757f451 0x76e5fc1 0x76e6046 0x76e6124 0x77428e2 0x7699a68 0x784409e 0x787bf26 0x76e6210 0x77b2de2 0x78d6e12 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:15:50.824 INFO:teuthology.orchestra.run.trial084.stderr:----------------------------------------------------- 2026-02-24T16:15:50.824 INFO:teuthology.orchestra.run.trial084.stderr:Suppressions used: 2026-02-24T16:15:50.824 INFO:teuthology.orchestra.run.trial084.stderr: count bytes template 2026-02-24T16:15:50.824 INFO:teuthology.orchestra.run.trial084.stderr: 1 8 MallocExtension::Register 2026-02-24T16:15:50.824 INFO:teuthology.orchestra.run.trial084.stderr:----------------------------------------------------- 2026-02-24T16:15:50.824 INFO:teuthology.orchestra.run.trial084.stderr: 2026-02-24T16:15:50.860 DEBUG:teuthology.orchestra.run.trial084:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2026-02-24T16:15:50.884 INFO:tasks.ceph:Reading keys from all nodes... 2026-02-24T16:15:50.884 DEBUG:teuthology.orchestra.run.trial084:> set -ex 2026-02-24T16:15:50.885 DEBUG:teuthology.orchestra.run.trial084:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2026-02-24T16:15:50.950 DEBUG:teuthology.orchestra.run.trial084:> set -ex 2026-02-24T16:15:50.950 DEBUG:teuthology.orchestra.run.trial084:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2026-02-24T16:15:51.014 DEBUG:teuthology.orchestra.run.trial084:> set -ex 2026-02-24T16:15:51.014 DEBUG:teuthology.orchestra.run.trial084:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2026-02-24T16:15:51.078 DEBUG:teuthology.orchestra.run.trial084:> set -ex 2026-02-24T16:15:51.079 DEBUG:teuthology.orchestra.run.trial084:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2026-02-24T16:15:51.142 DEBUG:teuthology.orchestra.run.trial084:> set -ex 2026-02-24T16:15:51.142 DEBUG:teuthology.orchestra.run.trial084:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2026-02-24T16:15:51.198 INFO:tasks.ceph:Adding keys to all mons... 2026-02-24T16:15:51.199 DEBUG:teuthology.orchestra.run.trial084:> sudo tee -a /etc/ceph/ceph.keyring 2026-02-24T16:15:51.262 INFO:teuthology.orchestra.run.trial084.stdout:[mgr.x] 2026-02-24T16:15:51.262 INFO:teuthology.orchestra.run.trial084.stdout: key = AQClzp1ptInEFhAA0tzMiLUDJDkMHGcpRcmcmQ== 2026-02-24T16:15:51.262 INFO:teuthology.orchestra.run.trial084.stdout:[osd.0] 2026-02-24T16:15:51.262 INFO:teuthology.orchestra.run.trial084.stdout: key = AQCqzp1p/hoeFxAAKsNhrKzx1XZl9caJmqZuyQ== 2026-02-24T16:15:51.262 INFO:teuthology.orchestra.run.trial084.stdout:[osd.1] 2026-02-24T16:15:51.263 INFO:teuthology.orchestra.run.trial084.stdout: key = AQCvzp1pQKP3FhAAyBZOUop05AyxwtGethDUJw== 2026-02-24T16:15:51.263 INFO:teuthology.orchestra.run.trial084.stdout:[osd.2] 2026-02-24T16:15:51.263 INFO:teuthology.orchestra.run.trial084.stdout: key = AQC0zp1p8mviFhAA7F3h1Ym/3L3lvHJgygC8PQ== 2026-02-24T16:15:51.263 INFO:teuthology.orchestra.run.trial084.stdout:[client.0] 2026-02-24T16:15:51.263 INFO:teuthology.orchestra.run.trial084.stdout: key = AQClzp1ppPt1GRAALI+8zEq2809TzmeNF1hTfg== 2026-02-24T16:15:51.265 DEBUG:teuthology.orchestra.run.trial084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2026-02-24T16:15:51.354 DEBUG:teuthology.orchestra.run.trial084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-24T16:15:51.403 DEBUG:teuthology.orchestra.run.trial084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-24T16:15:51.451 DEBUG:teuthology.orchestra.run.trial084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-24T16:15:51.500 DEBUG:teuthology.orchestra.run.trial084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-24T16:15:51.551 INFO:tasks.ceph:Running mkfs on mon nodes... 2026-02-24T16:15:51.551 DEBUG:teuthology.orchestra.run.trial084:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2026-02-24T16:15:51.577 DEBUG:teuthology.orchestra.run.trial084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2026-02-24T16:15:51.663 INFO:teuthology.orchestra.run.trial084.stderr:2026-02-24T16:15:51.661+0000 7ffa450eac00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:15:51.663 INFO:teuthology.orchestra.run.trial084.stderr:2026-02-24T16:15:51.661+0000 7ffa450eac00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:15:51.671 DEBUG:teuthology.orchestra.run.trial084:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2026-02-24T16:15:51.696 DEBUG:teuthology.orchestra.run.trial084:> rm -- /home/ubuntu/cephtest/ceph.monmap 2026-02-24T16:15:51.752 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2026-02-24T16:15:51.753 INFO:tasks.ceph.mon.a:Restarting daemon 2026-02-24T16:15:51.753 DEBUG:teuthology.orchestra.run.trial084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2026-02-24T16:15:51.794 INFO:tasks.ceph.mon.a:Started 2026-02-24T16:15:51.794 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2026-02-24T16:15:51.795 INFO:tasks.ceph.mgr.x:Restarting daemon 2026-02-24T16:15:51.795 DEBUG:teuthology.orchestra.run.trial084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2026-02-24T16:15:51.797 INFO:tasks.ceph.mgr.x:Started 2026-02-24T16:15:51.797 DEBUG:tasks.ceph:set 0 configs 2026-02-24T16:15:51.798 DEBUG:teuthology.orchestra.run.trial084:> sudo ceph --cluster ceph config dump 2026-02-24T16:15:51.859 INFO:tasks.ceph.mgr.x.trial084.stderr:2026-02-24T16:15:51.857+0000 7f60a5687fc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:15:51.859 INFO:tasks.ceph.mon.a.trial084.stderr:2026-02-24T16:15:51.857+0000 7f8504c2ec00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:15:51.860 INFO:tasks.ceph.mon.a.trial084.stderr:2026-02-24T16:15:51.857+0000 7f8504c2ec00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:15:51.860 INFO:tasks.ceph.mon.a.trial084.stderr:2026-02-24T16:15:51.857+0000 7f8504c2ec00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:15:51.871 INFO:teuthology.orchestra.run.trial084.stderr:2026-02-24T16:15:51.867+0000 7f81a80cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:15:51.873 INFO:teuthology.orchestra.run.trial084.stderr:2026-02-24T16:15:51.871+0000 7f81a80cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:15:51.905 INFO:teuthology.orchestra.run.trial084.stdout:WHO MASK LEVEL OPTION VALUE RO 2026-02-24T16:15:51.914 INFO:tasks.ceph:Setting crush tunables to default 2026-02-24T16:15:51.914 DEBUG:teuthology.orchestra.run.trial084:> sudo ceph --cluster ceph osd crush tunables default 2026-02-24T16:15:51.989 INFO:teuthology.orchestra.run.trial084.stderr:2026-02-24T16:15:51.984+0000 7f25f6c5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:15:51.990 INFO:teuthology.orchestra.run.trial084.stderr:2026-02-24T16:15:51.988+0000 7f25f6c5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:15:52.023 INFO:teuthology.orchestra.run.trial084.stderr:adjusted tunables profile to default 2026-02-24T16:15:52.031 INFO:tasks.ceph:check_enable_crimson: True 2026-02-24T16:15:52.031 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2026-02-24T16:15:52.031 DEBUG:teuthology.orchestra.run.trial084:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2026-02-24T16:15:52.062 INFO:tasks.ceph.mgr.x.trial084.stderr:2026-02-24T16:15:52.060+0000 7f60a5687fc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:15:52.063 INFO:tasks.ceph.mgr.x.trial084.stderr:2026-02-24T16:15:52.060+0000 7f60a5687fc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:15:52.106 INFO:teuthology.orchestra.run.trial084.stderr:2026-02-24T16:15:52.101+0000 7f9a4a28b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:15:52.107 INFO:teuthology.orchestra.run.trial084.stderr:2026-02-24T16:15:52.105+0000 7f9a4a28b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:15:52.144 INFO:tasks.ceph.mon.a.trial084.stderr:2026-02-24T16:15:52.142+0000 7f84fbff1640 -1 WARNING: experimental feature 'crimson' is enabled 2026-02-24T16:15:52.144 INFO:tasks.ceph.mon.a.trial084.stderr:Please be aware that this feature is experimental, untested, 2026-02-24T16:15:52.144 INFO:tasks.ceph.mon.a.trial084.stderr:unsupported, and may result in data corruption, data loss, 2026-02-24T16:15:52.144 INFO:tasks.ceph.mon.a.trial084.stderr:and/or irreparable damage to your cluster. Do not use 2026-02-24T16:15:52.144 INFO:tasks.ceph.mon.a.trial084.stderr:feature with important data. 2026-02-24T16:15:52.144 INFO:tasks.ceph.mon.a.trial084.stderr: 2026-02-24T16:15:52.887 INFO:tasks.ceph.mgr.x.trial084.stderr:/usr/lib64/python3.9/site-packages/scipy/__init__.py:73: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2026-02-24T16:15:52.887 INFO:tasks.ceph.mgr.x.trial084.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2026-02-24T16:15:52.887 INFO:tasks.ceph.mgr.x.trial084.stderr: from numpy import show_config as show_numpy_config 2026-02-24T16:15:53.034 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2026-02-24T16:15:53.034 DEBUG:teuthology.orchestra.run.trial084:> set -ex 2026-02-24T16:15:53.034 DEBUG:teuthology.orchestra.run.trial084:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2026-02-24T16:15:53.098 DEBUG:teuthology.orchestra.run.trial084:> set -ex 2026-02-24T16:15:53.098 DEBUG:teuthology.orchestra.run.trial084:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2026-02-24T16:15:53.162 DEBUG:teuthology.orchestra.run.trial084:> set -ex 2026-02-24T16:15:53.162 DEBUG:teuthology.orchestra.run.trial084:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2026-02-24T16:15:53.225 DEBUG:teuthology.orchestra.run.trial084:> sudo ceph --cluster ceph osd new b7189ed7-5d3c-4fc3-aae4-14731c0a3caf 0 2026-02-24T16:15:53.338 INFO:teuthology.orchestra.run.trial084.stderr:2026-02-24T16:15:53.333+0000 7f9701ec2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:15:53.339 INFO:teuthology.orchestra.run.trial084.stderr:2026-02-24T16:15:53.337+0000 7f9701ec2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:15:53.375 INFO:teuthology.orchestra.run.trial084.stdout:0 2026-02-24T16:15:53.384 DEBUG:teuthology.orchestra.run.trial084:> sudo ceph --cluster ceph osd new 5b8bf083-03a3-4693-a593-3f96c1ec1ae0 1 2026-02-24T16:15:53.459 INFO:teuthology.orchestra.run.trial084.stderr:2026-02-24T16:15:53.454+0000 7f470a774640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:15:53.460 INFO:teuthology.orchestra.run.trial084.stderr:2026-02-24T16:15:53.458+0000 7f470a774640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:15:53.496 INFO:teuthology.orchestra.run.trial084.stdout:1 2026-02-24T16:15:53.506 DEBUG:teuthology.orchestra.run.trial084:> sudo ceph --cluster ceph osd new 4ba2d14d-310a-4c2b-9b43-c7cd6b84889a 2 2026-02-24T16:15:53.581 INFO:teuthology.orchestra.run.trial084.stderr:2026-02-24T16:15:53.576+0000 7fdd703ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:15:53.582 INFO:teuthology.orchestra.run.trial084.stderr:2026-02-24T16:15:53.580+0000 7fdd703ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:15:53.618 INFO:teuthology.orchestra.run.trial084.stdout:2 2026-02-24T16:15:53.627 DEBUG:teuthology.orchestra.run.trial084:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2026-02-24T16:15:53.655 INFO:tasks.ceph.osd.0:Restarting daemon 2026-02-24T16:15:53.655 DEBUG:teuthology.orchestra.run.trial084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2026-02-24T16:15:53.697 INFO:tasks.ceph.osd.0:Started 2026-02-24T16:15:53.697 DEBUG:teuthology.orchestra.run.trial084:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2026-02-24T16:15:53.725 INFO:tasks.ceph.osd.1:Restarting daemon 2026-02-24T16:15:53.725 DEBUG:teuthology.orchestra.run.trial084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2026-02-24T16:15:53.769 INFO:tasks.ceph.osd.1:Started 2026-02-24T16:15:53.770 DEBUG:teuthology.orchestra.run.trial084:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2026-02-24T16:15:53.799 INFO:tasks.ceph.osd.2:Restarting daemon 2026-02-24T16:15:53.799 DEBUG:teuthology.orchestra.run.trial084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2026-02-24T16:15:53.843 INFO:tasks.ceph.osd.2:Started 2026-02-24T16:15:53.844 DEBUG:teuthology.orchestra.run.trial084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-24T16:15:53.920 INFO:teuthology.orchestra.run.trial084.stderr:2026-02-24T16:15:53.917+0000 7ffaba77e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:15:53.921 INFO:teuthology.orchestra.run.trial084.stderr:2026-02-24T16:15:53.919+0000 7ffaba77e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:15:53.953 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:15:53.953 INFO:teuthology.orchestra.run.trial084.stdout:{"epoch":6,"fsid":"a1ef0711-bcd7-4e13-9432-5701b02e2426","created":"2026-02-24T16:15:51.868373+0000","modified":"2026-02-24T16:15:53.617322+0000","last_up_change":"0.000000","last_in_change":"2026-02-24T16:15:53.617322+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"b7189ed7-5d3c-4fc3-aae4-14731c0a3caf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5b8bf083-03a3-4693-a593-3f96c1ec1ae0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4ba2d14d-310a-4c2b-9b43-c7cd6b84889a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-24T16:15:53.963 INFO:tasks.ceph.ceph_manager.ceph:[] 2026-02-24T16:15:53.963 INFO:tasks.ceph:Waiting for OSDs to come up 2026-02-24T16:15:54.265 DEBUG:teuthology.orchestra.run.trial084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-24T16:15:54.333 INFO:teuthology.misc.health.trial084.stderr:2026-02-24T16:15:54.328+0000 7f1b3230b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:15:54.334 INFO:teuthology.misc.health.trial084.stderr:2026-02-24T16:15:54.332+0000 7f1b3230b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:15:54.365 INFO:teuthology.misc.health.trial084.stdout: 2026-02-24T16:15:54.365 INFO:teuthology.misc.health.trial084.stdout:{"epoch":6,"fsid":"a1ef0711-bcd7-4e13-9432-5701b02e2426","created":"2026-02-24T16:15:51.868373+0000","modified":"2026-02-24T16:15:53.617322+0000","last_up_change":"0.000000","last_in_change":"2026-02-24T16:15:53.617322+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"b7189ed7-5d3c-4fc3-aae4-14731c0a3caf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5b8bf083-03a3-4693-a593-3f96c1ec1ae0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4ba2d14d-310a-4c2b-9b43-c7cd6b84889a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-24T16:15:54.373 DEBUG:teuthology.misc:0 of 3 OSDs are up 2026-02-24T16:15:55.097 INFO:tasks.ceph.mgr.x.trial084.stderr:2026-02-24T16:15:55.095+0000 7f5fc2b72640 -1 mgr.server handle_report got status from non-daemon mon.a 2026-02-24T16:16:00.675 DEBUG:teuthology.orchestra.run.trial084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-24T16:16:00.745 INFO:teuthology.misc.health.trial084.stderr:2026-02-24T16:16:00.741+0000 7f3ea02b7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:16:00.746 INFO:teuthology.misc.health.trial084.stderr:2026-02-24T16:16:00.744+0000 7f3ea02b7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:16:00.855 INFO:teuthology.misc.health.trial084.stdout: 2026-02-24T16:16:00.855 INFO:teuthology.misc.health.trial084.stdout:{"epoch":7,"fsid":"a1ef0711-bcd7-4e13-9432-5701b02e2426","created":"2026-02-24T16:15:51.868373+0000","modified":"2026-02-24T16:15:59.944969+0000","last_up_change":"0.000000","last_in_change":"2026-02-24T16:15:53.617322+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"b7189ed7-5d3c-4fc3-aae4-14731c0a3caf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5b8bf083-03a3-4693-a593-3f96c1ec1ae0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4ba2d14d-310a-4c2b-9b43-c7cd6b84889a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-24T16:16:00.863 DEBUG:teuthology.misc:0 of 3 OSDs are up 2026-02-24T16:16:07.165 DEBUG:teuthology.orchestra.run.trial084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-24T16:16:07.233 INFO:teuthology.misc.health.trial084.stderr:2026-02-24T16:16:07.230+0000 7f5c9f15c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:16:07.234 INFO:teuthology.misc.health.trial084.stderr:2026-02-24T16:16:07.233+0000 7f5c9f15c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:16:07.346 INFO:teuthology.misc.health.trial084.stdout: 2026-02-24T16:16:07.347 INFO:teuthology.misc.health.trial084.stdout:{"epoch":13,"fsid":"a1ef0711-bcd7-4e13-9432-5701b02e2426","created":"2026-02-24T16:15:51.868373+0000","modified":"2026-02-24T16:16:06.969497+0000","last_up_change":"2026-02-24T16:16:02.953143+0000","last_in_change":"2026-02-24T16:15:53.617322+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-24T16:16:04.105671+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b7189ed7-5d3c-4fc3-aae4-14731c0a3caf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.84:6805","nonce":3005800773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.84:6804","nonce":3005800773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.84:6810","nonce":3005800773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.84:6811","nonce":3005800773}]},"public_addr":"10.20.193.84:6805/3005800773","cluster_addr":"10.20.193.84:6804/3005800773","heartbeat_back_addr":"10.20.193.84:6810/3005800773","heartbeat_front_addr":"10.20.193.84:6811/3005800773","state":["exists","up"]},{"osd":1,"uuid":"5b8bf083-03a3-4693-a593-3f96c1ec1ae0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.84:6803","nonce":169668144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.84:6802","nonce":169668144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.84:6809","nonce":169668144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.84:6808","nonce":169668144}]},"public_addr":"10.20.193.84:6803/169668144","cluster_addr":"10.20.193.84:6802/169668144","heartbeat_back_addr":"10.20.193.84:6809/169668144","heartbeat_front_addr":"10.20.193.84:6808/169668144","state":["exists","up"]},{"osd":2,"uuid":"4ba2d14d-310a-4c2b-9b43-c7cd6b84889a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.84:6807","nonce":790563364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.84:6806","nonce":790563364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.84:6812","nonce":790563364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.84:6813","nonce":790563364}]},"public_addr":"10.20.193.84:6807/790563364","cluster_addr":"10.20.193.84:6806/790563364","heartbeat_back_addr":"10.20.193.84:6812/790563364","heartbeat_front_addr":"10.20.193.84:6813/790563364","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-24T16:16:07.355 DEBUG:teuthology.misc:3 of 3 OSDs are up 2026-02-24T16:16:07.355 INFO:tasks.ceph:Creating RBD pool 2026-02-24T16:16:07.355 DEBUG:teuthology.orchestra.run.trial084:> sudo ceph --cluster ceph osd pool create rbd 8 2026-02-24T16:16:07.429 INFO:teuthology.orchestra.run.trial084.stderr:2026-02-24T16:16:07.425+0000 7fb0eb6cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:16:07.430 INFO:teuthology.orchestra.run.trial084.stderr:2026-02-24T16:16:07.429+0000 7fb0eb6cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:16:07.975 INFO:teuthology.orchestra.run.trial084.stderr:pool 'rbd' created 2026-02-24T16:16:07.990 DEBUG:teuthology.orchestra.run.trial084:> rbd --cluster ceph pool init rbd 2026-02-24T16:16:08.036 INFO:teuthology.orchestra.run.trial084.stderr:2026-02-24T16:16:08.035+0000 7f0ddcefe240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:16:08.036 INFO:teuthology.orchestra.run.trial084.stderr:2026-02-24T16:16:08.035+0000 7f0ddcefe240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:16:08.037 INFO:teuthology.orchestra.run.trial084.stderr:2026-02-24T16:16:08.036+0000 7f0ddcefe240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:16:11.016 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2026-02-24T16:16:11.017 DEBUG:teuthology.orchestra.run.trial084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2026-02-24T16:16:11.018 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2026-02-24T16:16:11.096 INFO:teuthology.orchestra.run.trial084.stderr:2026-02-24T16:16:11.091+0000 7f578fe8b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:16:11.097 INFO:teuthology.orchestra.run.trial084.stderr:2026-02-24T16:16:11.095+0000 7f578fe8b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:16:11.206 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:16:11.215 INFO:teuthology.orchestra.run.trial084.stdout:[{"version":1,"timestamp":"0.000000","name":"","changes":[]}] 2026-02-24T16:16:11.215 INFO:tasks.ceph_manager:config epoch is 1 2026-02-24T16:16:11.215 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2026-02-24T16:16:11.216 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2026-02-24T16:16:11.216 DEBUG:teuthology.orchestra.run.trial084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2026-02-24T16:16:11.291 INFO:teuthology.orchestra.run.trial084.stderr:2026-02-24T16:16:11.287+0000 7f3ddcc3d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:16:11.292 INFO:teuthology.orchestra.run.trial084.stderr:2026-02-24T16:16:11.290+0000 7f3ddcc3d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:16:11.434 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:16:11.442 INFO:teuthology.orchestra.run.trial084.stdout:{"epoch":5,"flags":0,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.84:6800","nonce":64268235},{"type":"v1","addr":"10.20.193.84:6801","nonce":64268235}]},"active_addr":"10.20.193.84:6801/64268235","active_change":"2026-02-24T16:15:54.091001+0000","active_mgr_features":4544132024016699391,"available":true,"standbys":[],"modules":["iostat","nfs"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_pg_upmap_activity":{"name":"update_pg_upmap_activity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Updates pg_upmap activity stats to be used in `balancer status detail`","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often the agent on each host will gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"The first TCP port the agent will try to bind to. Up to 1000 subsequent ports will be attempted if this port cannot be bound","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this option can facilitate debugging daemons that encounter runtime problems.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"How frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"Fraction of total system memory to divide among autotuned daemons. Converged systems hosting both Ceph and compute should set a lower value; systems with more than 10 GiB of memory per OSD may benefit from a slightly larger value so that the abundance of RAM can be fully utilized","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command persistent logging","long_desc":"","tags":[],"see_also":[]},"certificate_automated_rotation_enabled":{"name":"certificate_automated_rotation_enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"This flag controls whether cephadm automatically rotates certificates upon expiration.","long_desc":"","tags":[],"see_also":[]},"certificate_check_debug_mode":{"name":"certificate_check_debug_mode","type":"bool","level":"dev","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"FOR TESTING ONLY: This flag forces the certificate check instead of waiting for certificate_check_period.","long_desc":"","tags":[],"see_also":[]},"certificate_check_period":{"name":"certificate_check_period","type":"int","level":"advanced","flags":0,"default_value":"1","min":"0","max":"30","enum_allowed":[],"desc":"Specifies how often (in days) the certificate should be checked for validity.","long_desc":"","tags":[],"see_also":[]},"certificate_duration_days":{"name":"certificate_duration_days","type":"int","level":"advanced","flags":0,"default_value":"1095","min":"90","max":"3650","enum_allowed":[],"desc":"Specifies the duration of self-signed certificates generated and signed by the cephadm root CA","long_desc":"","tags":[],"see_also":[]},"certificate_renewal_threshold_days":{"name":"certificate_renewal_threshold_days","type":"int","level":"advanced","flags":0,"default_value":"30","min":"10","max":"90","enum_allowed":[],"desc":"Specifies the lead time in days to initiate certificate renewal before expiration.","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently Podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.28.1","min":"","max":"","enum_allowed":[],"desc":"Alertmanager container image","long_desc":"","tags":[],"see_also":[]},"container_image_alloy":{"name":"container_image_alloy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/alloy:latest","min":"","max":"","enum_allowed":[],"desc":"Alloy container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"Elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:12.3.1","min":"","max":"","enum_allowed":[],"desc":"Grafana container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"Haproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.9.1","min":"","max":"","enum_allowed":[],"desc":"Node exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.6","min":"","max":"","enum_allowed":[],"desc":"Nvmeof container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"Oauth2 proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v3.6.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba metrics container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"Snmp gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"quay.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"Location of dashboards to include in Grafana","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"How frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether the hardware monitoring daemon be deployed on every host?","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at the debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"Max number of daemons per service per host: guard against mistakes and runaways","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"Max number of OSDs that will be drained simultaneously when OSDs are removed","long_desc":"","tags":[],"see_also":[]},"max_parallel_osd_upgrades":{"name":"max_parallel_osd_upgrades","type":"int","level":"advanced","flags":0,"default_value":"16","min":"","max":"","enum_allowed":[],"desc":"Maximum number of OSD daemons upgraded in parallel.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"Mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default IP address for RedFish API (OOB management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"Location of alerts to include in Prometheus Alertmanager","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry URL for login purposes. This is not the default registry.","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"Cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Customized SSH config file to distribute to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many sequential SSH connection liveness check failures before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often SSH connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"stray_daemon_check_interval":{"name":"stray_daemon_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"How frequently cephadm should check for the presence of stray daemons","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use the cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest to ensure that all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROM_ALERT_CREDENTIAL_CACHE_TTL":{"name":"PROM_ALERT_CREDENTIAL_CACHE_TTL","type":"int","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_HOSTNAME_PER_DAEMON":{"name":"RGW_HOSTNAME_PER_DAEMON","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crypto_caller":{"name":"crypto_caller","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"healthcheck_history_max_entries":{"name":"healthcheck_history_max_entries","type":"int","level":"advanced","flags":1,"default_value":"1000","min":"","max":"","enum_allowed":[],"desc":"Maximum number of health check history entries to keep","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"prometheus_tls_secret_name":{"name":"prometheus_tls_secret_name","type":"str","level":"advanced","flags":0,"default_value":"rook-ceph-prometheus-server-tls","min":"","max":"","enum_allowed":[],"desc":"name of tls secret in k8s for prometheus","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_cloning":{"name":"pause_cloning","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_purging":{"name":"pause_purging","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous subvolume purge threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"tentacle":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"force_disabled_modules":{},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"10.20.193.84:0","nonce":937362397}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"10.20.193.84:0","nonce":2248671224}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"10.20.193.84:0","nonce":1583786203}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"10.20.193.84:0","nonce":3787756789}]}]} 2026-02-24T16:16:11.446 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2026-02-24T16:16:11.447 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2026-02-24T16:16:11.447 DEBUG:teuthology.orchestra.run.trial084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-24T16:16:11.523 INFO:teuthology.orchestra.run.trial084.stderr:2026-02-24T16:16:11.519+0000 7ff5cd10b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:16:11.524 INFO:teuthology.orchestra.run.trial084.stderr:2026-02-24T16:16:11.522+0000 7ff5cd10b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:16:11.633 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:16:11.633 INFO:teuthology.orchestra.run.trial084.stdout:{"epoch":17,"fsid":"a1ef0711-bcd7-4e13-9432-5701b02e2426","created":"2026-02-24T16:15:51.868373+0000","modified":"2026-02-24T16:16:10.985473+0000","last_up_change":"2026-02-24T16:16:02.953143+0000","last_in_change":"2026-02-24T16:15:53.617322+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-24T16:16:04.105671+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-24T16:16:07.540349+0000","flags":270337,"flags_names":"hashpspool,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b7189ed7-5d3c-4fc3-aae4-14731c0a3caf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.84:6805","nonce":3005800773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.84:6804","nonce":3005800773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.84:6810","nonce":3005800773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.84:6811","nonce":3005800773}]},"public_addr":"10.20.193.84:6805/3005800773","cluster_addr":"10.20.193.84:6804/3005800773","heartbeat_back_addr":"10.20.193.84:6810/3005800773","heartbeat_front_addr":"10.20.193.84:6811/3005800773","state":["exists","up"]},{"osd":1,"uuid":"5b8bf083-03a3-4693-a593-3f96c1ec1ae0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.84:6803","nonce":169668144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.84:6802","nonce":169668144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.84:6809","nonce":169668144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.84:6808","nonce":169668144}]},"public_addr":"10.20.193.84:6803/169668144","cluster_addr":"10.20.193.84:6802/169668144","heartbeat_back_addr":"10.20.193.84:6809/169668144","heartbeat_front_addr":"10.20.193.84:6808/169668144","state":["exists","up"]},{"osd":2,"uuid":"4ba2d14d-310a-4c2b-9b43-c7cd6b84889a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.84:6807","nonce":790563364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.84:6806","nonce":790563364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.84:6812","nonce":790563364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.84:6813","nonce":790563364}]},"public_addr":"10.20.193.84:6807/790563364","cluster_addr":"10.20.193.84:6806/790563364","heartbeat_back_addr":"10.20.193.84:6812/790563364","heartbeat_front_addr":"10.20.193.84:6813/790563364","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-24T16:16:11.643 INFO:tasks.ceph.ceph_manager.ceph:all up! 2026-02-24T16:16:11.643 DEBUG:teuthology.orchestra.run.trial084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-24T16:16:11.721 INFO:teuthology.orchestra.run.trial084.stderr:2026-02-24T16:16:11.717+0000 7f31fb6cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:16:11.722 INFO:teuthology.orchestra.run.trial084.stderr:2026-02-24T16:16:11.720+0000 7f31fb6cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:16:11.833 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:16:11.834 INFO:teuthology.orchestra.run.trial084.stdout:{"epoch":17,"fsid":"a1ef0711-bcd7-4e13-9432-5701b02e2426","created":"2026-02-24T16:15:51.868373+0000","modified":"2026-02-24T16:16:10.985473+0000","last_up_change":"2026-02-24T16:16:02.953143+0000","last_in_change":"2026-02-24T16:15:53.617322+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-24T16:16:04.105671+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-24T16:16:07.540349+0000","flags":270337,"flags_names":"hashpspool,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b7189ed7-5d3c-4fc3-aae4-14731c0a3caf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.84:6805","nonce":3005800773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.84:6804","nonce":3005800773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.84:6810","nonce":3005800773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.84:6811","nonce":3005800773}]},"public_addr":"10.20.193.84:6805/3005800773","cluster_addr":"10.20.193.84:6804/3005800773","heartbeat_back_addr":"10.20.193.84:6810/3005800773","heartbeat_front_addr":"10.20.193.84:6811/3005800773","state":["exists","up"]},{"osd":1,"uuid":"5b8bf083-03a3-4693-a593-3f96c1ec1ae0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.84:6803","nonce":169668144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.84:6802","nonce":169668144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.84:6809","nonce":169668144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.84:6808","nonce":169668144}]},"public_addr":"10.20.193.84:6803/169668144","cluster_addr":"10.20.193.84:6802/169668144","heartbeat_back_addr":"10.20.193.84:6809/169668144","heartbeat_front_addr":"10.20.193.84:6808/169668144","state":["exists","up"]},{"osd":2,"uuid":"4ba2d14d-310a-4c2b-9b43-c7cd6b84889a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.84:6807","nonce":790563364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.84:6806","nonce":790563364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.84:6812","nonce":790563364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.84:6813","nonce":790563364}]},"public_addr":"10.20.193.84:6807/790563364","cluster_addr":"10.20.193.84:6806/790563364","heartbeat_back_addr":"10.20.193.84:6812/790563364","heartbeat_front_addr":"10.20.193.84:6813/790563364","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-24T16:16:11.843 DEBUG:teuthology.orchestra.run.trial084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-24T16:16:11.844 DEBUG:teuthology.orchestra.run.trial084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-24T16:16:11.844 DEBUG:teuthology.orchestra.run.trial084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-24T16:16:11.923 INFO:teuthology.orchestra.run.trial084.stderr:2026-02-24T16:16:11.918+0000 7fabcd65a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:16:11.923 INFO:teuthology.orchestra.run.trial084.stderr:2026-02-24T16:16:11.918+0000 7fe75c25a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:16:11.924 INFO:teuthology.orchestra.run.trial084.stderr:2026-02-24T16:16:11.922+0000 7fabcd65a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:16:11.924 INFO:teuthology.orchestra.run.trial084.stderr:2026-02-24T16:16:11.919+0000 7fce5770b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:16:11.925 INFO:teuthology.orchestra.run.trial084.stderr:2026-02-24T16:16:11.922+0000 7fe75c25a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:16:11.925 INFO:teuthology.orchestra.run.trial084.stderr:2026-02-24T16:16:11.923+0000 7fce5770b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:16:11.938 INFO:teuthology.orchestra.run.trial084.stdout:38654705673 2026-02-24T16:16:11.938 DEBUG:teuthology.orchestra.run.trial084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-24T16:16:11.940 INFO:teuthology.orchestra.run.trial084.stdout:42949672968 2026-02-24T16:16:11.940 DEBUG:teuthology.orchestra.run.trial084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-24T16:16:11.941 INFO:teuthology.orchestra.run.trial084.stdout:42949672968 2026-02-24T16:16:11.942 DEBUG:teuthology.orchestra.run.trial084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-24T16:16:12.061 INFO:teuthology.orchestra.run.trial084.stderr:2026-02-24T16:16:12.056+0000 7f610f651640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:16:12.061 INFO:teuthology.orchestra.run.trial084.stderr:2026-02-24T16:16:12.056+0000 7f0485acb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:16:12.062 INFO:teuthology.orchestra.run.trial084.stderr:2026-02-24T16:16:12.057+0000 7fb46ce5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:16:12.062 INFO:teuthology.orchestra.run.trial084.stderr:2026-02-24T16:16:12.060+0000 7f610f651640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:16:12.063 INFO:teuthology.orchestra.run.trial084.stderr:2026-02-24T16:16:12.060+0000 7f0485acb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:16:12.063 INFO:teuthology.orchestra.run.trial084.stderr:2026-02-24T16:16:12.061+0000 7fb46ce5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:16:12.172 INFO:teuthology.orchestra.run.trial084.stdout:38654705668 2026-02-24T16:16:12.178 INFO:teuthology.orchestra.run.trial084.stdout:42949672964 2026-02-24T16:16:12.180 INFO:teuthology.orchestra.run.trial084.stdout:42949672964 2026-02-24T16:16:12.183 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705673 got 38654705668 for osd.1 2026-02-24T16:16:12.189 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672968 got 42949672964 for osd.2 2026-02-24T16:16:12.190 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672968 got 42949672964 for osd.0 2026-02-24T16:16:13.184 DEBUG:teuthology.orchestra.run.trial084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-24T16:16:13.190 DEBUG:teuthology.orchestra.run.trial084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-24T16:16:13.192 DEBUG:teuthology.orchestra.run.trial084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-24T16:16:13.265 INFO:teuthology.orchestra.run.trial084.stderr:2026-02-24T16:16:13.260+0000 7f27bacb7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:16:13.266 INFO:teuthology.orchestra.run.trial084.stderr:2026-02-24T16:16:13.264+0000 7f27bacb7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:16:13.307 INFO:teuthology.orchestra.run.trial084.stderr:2026-02-24T16:16:13.302+0000 7fa2dec5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:16:13.307 INFO:teuthology.orchestra.run.trial084.stderr:2026-02-24T16:16:13.302+0000 7faf5110b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:16:13.308 INFO:teuthology.orchestra.run.trial084.stderr:2026-02-24T16:16:13.306+0000 7fa2dec5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:16:13.308 INFO:teuthology.orchestra.run.trial084.stderr:2026-02-24T16:16:13.307+0000 7faf5110b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:16:13.378 INFO:teuthology.orchestra.run.trial084.stdout:38654705673 2026-02-24T16:16:13.389 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705673 got 38654705673 for osd.1 2026-02-24T16:16:13.390 DEBUG:teuthology.parallel:result is None 2026-02-24T16:16:13.418 INFO:teuthology.orchestra.run.trial084.stdout:42949672968 2026-02-24T16:16:13.422 INFO:teuthology.orchestra.run.trial084.stdout:42949672968 2026-02-24T16:16:13.428 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672968 got 42949672968 for osd.0 2026-02-24T16:16:13.429 DEBUG:teuthology.parallel:result is None 2026-02-24T16:16:13.433 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672968 got 42949672968 for osd.2 2026-02-24T16:16:13.434 DEBUG:teuthology.parallel:result is None 2026-02-24T16:16:13.434 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-24T16:16:13.434 DEBUG:teuthology.orchestra.run.trial084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-24T16:16:13.549 INFO:teuthology.orchestra.run.trial084.stderr:2026-02-24T16:16:13.545+0000 7fac45ddc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:16:13.549 INFO:teuthology.orchestra.run.trial084.stderr:2026-02-24T16:16:13.548+0000 7fac45ddc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:16:13.660 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:16:13.660 INFO:teuthology.orchestra.run.trial084.stderr:dumped all 2026-02-24T16:16:13.669 INFO:teuthology.orchestra.run.trial084.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2026-02-24T16:16:12.095172+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":157286352,"kb_used":62917004,"kb_used_data":62917004,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":94369348,"statfs":{"total":161061224448,"available":96634212352,"internally_reserved":0,"allocated":64427012096,"data_stored":64427012096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.120144"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-24T16:16:11.000094+0000","last_change":"2026-02-24T16:16:11.005492+0000","last_active":"2026-02-24T16:16:11.000094+0000","last_peered":"2026-02-24T16:16:11.000094+0000","last_clean":"2026-02-24T16:16:11.000094+0000","last_became_active":"2026-02-24T16:16:09.021958+0000","last_became_peered":"2026-02-24T16:16:09.021958+0000","last_unstale":"2026-02-24T16:16:11.000094+0000","last_undegraded":"2026-02-24T16:16:11.000094+0000","last_fullsized":"2026-02-24T16:16:11.000094+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T16:16:07.974206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T16:16:07.974206+0000","last_clean_scrub_stamp":"2026-02-24T16:16:07.974206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-24T16:16:11.002344+0000","last_change":"2026-02-24T16:16:11.006531+0000","last_active":"2026-02-24T16:16:11.002344+0000","last_peered":"2026-02-24T16:16:11.002344+0000","last_clean":"2026-02-24T16:16:11.002344+0000","last_became_active":"2026-02-24T16:16:09.018343+0000","last_became_peered":"2026-02-24T16:16:09.018343+0000","last_unstale":"2026-02-24T16:16:11.002344+0000","last_undegraded":"2026-02-24T16:16:11.002344+0000","last_fullsized":"2026-02-24T16:16:11.002344+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T16:16:07.974206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T16:16:07.974206+0000","last_clean_scrub_stamp":"2026-02-24T16:16:07.974206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-24T16:16:11.003024+0000","last_change":"2026-02-24T16:16:11.006594+0000","last_active":"2026-02-24T16:16:11.003024+0000","last_peered":"2026-02-24T16:16:11.003024+0000","last_clean":"2026-02-24T16:16:11.003024+0000","last_became_active":"2026-02-24T16:16:09.021641+0000","last_became_peered":"2026-02-24T16:16:09.021641+0000","last_unstale":"2026-02-24T16:16:11.003024+0000","last_undegraded":"2026-02-24T16:16:11.003024+0000","last_fullsized":"2026-02-24T16:16:11.003024+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T16:16:07.974206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T16:16:07.974206+0000","last_clean_scrub_stamp":"2026-02-24T16:16:07.974206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-24T16:16:11.003129+0000","last_change":"2026-02-24T16:16:11.006505+0000","last_active":"2026-02-24T16:16:11.003129+0000","last_peered":"2026-02-24T16:16:11.003129+0000","last_clean":"2026-02-24T16:16:11.003129+0000","last_became_active":"2026-02-24T16:16:09.031831+0000","last_became_peered":"2026-02-24T16:16:09.031831+0000","last_unstale":"2026-02-24T16:16:11.003129+0000","last_undegraded":"2026-02-24T16:16:11.003129+0000","last_fullsized":"2026-02-24T16:16:11.003129+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T16:16:07.974206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T16:16:07.974206+0000","last_clean_scrub_stamp":"2026-02-24T16:16:07.974206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'1","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-24T16:16:11.002615+0000","last_change":"2026-02-24T16:16:11.002615+0000","last_active":"2026-02-24T16:16:11.002615+0000","last_peered":"2026-02-24T16:16:11.002615+0000","last_clean":"2026-02-24T16:16:11.002615+0000","last_became_active":"2026-02-24T16:16:09.047997+0000","last_became_peered":"2026-02-24T16:16:09.047997+0000","last_unstale":"2026-02-24T16:16:11.002615+0000","last_undegraded":"2026-02-24T16:16:11.002615+0000","last_fullsized":"2026-02-24T16:16:11.002615+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T16:16:07.974206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T16:16:07.974206+0000","last_clean_scrub_stamp":"2026-02-24T16:16:07.974206+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-24T16:16:10.999312+0000","last_change":"2026-02-24T16:16:11.002272+0000","last_active":"2026-02-24T16:16:10.999312+0000","last_peered":"2026-02-24T16:16:10.999312+0000","last_clean":"2026-02-24T16:16:10.999312+0000","last_became_active":"2026-02-24T16:16:09.029178+0000","last_became_peered":"2026-02-24T16:16:09.029178+0000","last_unstale":"2026-02-24T16:16:10.999312+0000","last_undegraded":"2026-02-24T16:16:10.999312+0000","last_fullsized":"2026-02-24T16:16:10.999312+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T16:16:07.974206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T16:16:07.974206+0000","last_clean_scrub_stamp":"2026-02-24T16:16:07.974206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-24T16:16:10.997763+0000","last_change":"2026-02-24T16:16:11.001016+0000","last_active":"2026-02-24T16:16:10.997763+0000","last_peered":"2026-02-24T16:16:10.997763+0000","last_clean":"2026-02-24T16:16:10.997763+0000","last_became_active":"2026-02-24T16:16:09.031358+0000","last_became_peered":"2026-02-24T16:16:09.031358+0000","last_unstale":"2026-02-24T16:16:10.997763+0000","last_undegraded":"2026-02-24T16:16:10.997763+0000","last_fullsized":"2026-02-24T16:16:10.997763+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T16:16:07.974206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T16:16:07.974206+0000","last_clean_scrub_stamp":"2026-02-24T16:16:07.974206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":16,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-24T16:16:11.002311+0000","last_change":"2026-02-24T16:16:11.006507+0000","last_active":"2026-02-24T16:16:11.002311+0000","last_peered":"2026-02-24T16:16:11.002311+0000","last_clean":"2026-02-24T16:16:11.002311+0000","last_became_active":"2026-02-24T16:16:09.016983+0000","last_became_peered":"2026-02-24T16:16:09.016983+0000","last_unstale":"2026-02-24T16:16:11.002311+0000","last_undegraded":"2026-02-24T16:16:11.002311+0000","last_fullsized":"2026-02-24T16:16:11.002311+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T16:16:07.974206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T16:16:07.974206+0000","last_clean_scrub_stamp":"2026-02-24T16:16:07.974206+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":90,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-24T16:16:11.001731+0000","last_change":"2026-02-24T16:16:05.995499+0000","last_active":"2026-02-24T16:16:11.001731+0000","last_peered":"2026-02-24T16:16:11.001731+0000","last_clean":"2026-02-24T16:16:11.001731+0000","last_became_active":"2026-02-24T16:16:05.988859+0000","last_became_peered":"2026-02-24T16:16:05.988859+0000","last_unstale":"2026-02-24T16:16:11.001731+0000","last_undegraded":"2026-02-24T16:16:11.001731+0000","last_fullsized":"2026-02-24T16:16:11.001731+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T16:16:04.957640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T16:16:04.957640+0000","last_clean_scrub_stamp":"2026-02-24T16:16:04.957640+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":161061224448,"available":96634163200,"internally_reserved":0,"allocated":64427061248,"data_stored":64427061248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53687074816,"available":32211079168,"internally_reserved":0,"allocated":21475995648,"data_stored":21475995648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949672968,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20971812,"kb_used_data":20971812,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456972,"statfs":{"total":53687074816,"available":32211939328,"internally_reserved":0,"allocated":21475135488,"data_stored":21475135488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949672968,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972572,"kb_used_data":20972572,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456212,"statfs":{"total":53687074816,"available":32211161088,"internally_reserved":0,"allocated":21475913728,"data_stored":21475913728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705673,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972620,"kb_used_data":20972620,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456164,"statfs":{"total":53687074816,"available":32211111936,"internally_reserved":0,"allocated":21475962880,"data_stored":21475962880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53687074816,"available":32211079168,"internally_reserved":0,"allocated":21475995648,"data_stored":21475995648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53687074816,"available":32211152896,"internally_reserved":0,"allocated":21475921920,"data_stored":21475921920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53687074816,"available":32211079168,"internally_reserved":0,"allocated":21475995648,"data_stored":21475995648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53687074816,"available":32211931136,"internally_reserved":0,"allocated":21475143680,"data_stored":21475143680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-24T16:16:13.671 DEBUG:teuthology.orchestra.run.trial084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-24T16:16:13.746 INFO:teuthology.orchestra.run.trial084.stderr:2026-02-24T16:16:13.743+0000 7f074e05a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:16:13.747 INFO:teuthology.orchestra.run.trial084.stderr:2026-02-24T16:16:13.745+0000 7f074e05a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:16:13.856 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:16:13.857 INFO:teuthology.orchestra.run.trial084.stderr:dumped all 2026-02-24T16:16:13.865 INFO:teuthology.orchestra.run.trial084.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2026-02-24T16:16:12.095172+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":157286352,"kb_used":62917004,"kb_used_data":62917004,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":94369348,"statfs":{"total":161061224448,"available":96634212352,"internally_reserved":0,"allocated":64427012096,"data_stored":64427012096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.120144"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-24T16:16:11.000094+0000","last_change":"2026-02-24T16:16:11.005492+0000","last_active":"2026-02-24T16:16:11.000094+0000","last_peered":"2026-02-24T16:16:11.000094+0000","last_clean":"2026-02-24T16:16:11.000094+0000","last_became_active":"2026-02-24T16:16:09.021958+0000","last_became_peered":"2026-02-24T16:16:09.021958+0000","last_unstale":"2026-02-24T16:16:11.000094+0000","last_undegraded":"2026-02-24T16:16:11.000094+0000","last_fullsized":"2026-02-24T16:16:11.000094+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T16:16:07.974206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T16:16:07.974206+0000","last_clean_scrub_stamp":"2026-02-24T16:16:07.974206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-24T16:16:11.002344+0000","last_change":"2026-02-24T16:16:11.006531+0000","last_active":"2026-02-24T16:16:11.002344+0000","last_peered":"2026-02-24T16:16:11.002344+0000","last_clean":"2026-02-24T16:16:11.002344+0000","last_became_active":"2026-02-24T16:16:09.018343+0000","last_became_peered":"2026-02-24T16:16:09.018343+0000","last_unstale":"2026-02-24T16:16:11.002344+0000","last_undegraded":"2026-02-24T16:16:11.002344+0000","last_fullsized":"2026-02-24T16:16:11.002344+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T16:16:07.974206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T16:16:07.974206+0000","last_clean_scrub_stamp":"2026-02-24T16:16:07.974206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-24T16:16:11.003024+0000","last_change":"2026-02-24T16:16:11.006594+0000","last_active":"2026-02-24T16:16:11.003024+0000","last_peered":"2026-02-24T16:16:11.003024+0000","last_clean":"2026-02-24T16:16:11.003024+0000","last_became_active":"2026-02-24T16:16:09.021641+0000","last_became_peered":"2026-02-24T16:16:09.021641+0000","last_unstale":"2026-02-24T16:16:11.003024+0000","last_undegraded":"2026-02-24T16:16:11.003024+0000","last_fullsized":"2026-02-24T16:16:11.003024+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T16:16:07.974206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T16:16:07.974206+0000","last_clean_scrub_stamp":"2026-02-24T16:16:07.974206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-24T16:16:11.003129+0000","last_change":"2026-02-24T16:16:11.006505+0000","last_active":"2026-02-24T16:16:11.003129+0000","last_peered":"2026-02-24T16:16:11.003129+0000","last_clean":"2026-02-24T16:16:11.003129+0000","last_became_active":"2026-02-24T16:16:09.031831+0000","last_became_peered":"2026-02-24T16:16:09.031831+0000","last_unstale":"2026-02-24T16:16:11.003129+0000","last_undegraded":"2026-02-24T16:16:11.003129+0000","last_fullsized":"2026-02-24T16:16:11.003129+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T16:16:07.974206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T16:16:07.974206+0000","last_clean_scrub_stamp":"2026-02-24T16:16:07.974206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'1","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-24T16:16:11.002615+0000","last_change":"2026-02-24T16:16:11.002615+0000","last_active":"2026-02-24T16:16:11.002615+0000","last_peered":"2026-02-24T16:16:11.002615+0000","last_clean":"2026-02-24T16:16:11.002615+0000","last_became_active":"2026-02-24T16:16:09.047997+0000","last_became_peered":"2026-02-24T16:16:09.047997+0000","last_unstale":"2026-02-24T16:16:11.002615+0000","last_undegraded":"2026-02-24T16:16:11.002615+0000","last_fullsized":"2026-02-24T16:16:11.002615+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T16:16:07.974206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T16:16:07.974206+0000","last_clean_scrub_stamp":"2026-02-24T16:16:07.974206+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-24T16:16:10.999312+0000","last_change":"2026-02-24T16:16:11.002272+0000","last_active":"2026-02-24T16:16:10.999312+0000","last_peered":"2026-02-24T16:16:10.999312+0000","last_clean":"2026-02-24T16:16:10.999312+0000","last_became_active":"2026-02-24T16:16:09.029178+0000","last_became_peered":"2026-02-24T16:16:09.029178+0000","last_unstale":"2026-02-24T16:16:10.999312+0000","last_undegraded":"2026-02-24T16:16:10.999312+0000","last_fullsized":"2026-02-24T16:16:10.999312+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T16:16:07.974206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T16:16:07.974206+0000","last_clean_scrub_stamp":"2026-02-24T16:16:07.974206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-24T16:16:10.997763+0000","last_change":"2026-02-24T16:16:11.001016+0000","last_active":"2026-02-24T16:16:10.997763+0000","last_peered":"2026-02-24T16:16:10.997763+0000","last_clean":"2026-02-24T16:16:10.997763+0000","last_became_active":"2026-02-24T16:16:09.031358+0000","last_became_peered":"2026-02-24T16:16:09.031358+0000","last_unstale":"2026-02-24T16:16:10.997763+0000","last_undegraded":"2026-02-24T16:16:10.997763+0000","last_fullsized":"2026-02-24T16:16:10.997763+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T16:16:07.974206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T16:16:07.974206+0000","last_clean_scrub_stamp":"2026-02-24T16:16:07.974206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":16,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-24T16:16:11.002311+0000","last_change":"2026-02-24T16:16:11.006507+0000","last_active":"2026-02-24T16:16:11.002311+0000","last_peered":"2026-02-24T16:16:11.002311+0000","last_clean":"2026-02-24T16:16:11.002311+0000","last_became_active":"2026-02-24T16:16:09.016983+0000","last_became_peered":"2026-02-24T16:16:09.016983+0000","last_unstale":"2026-02-24T16:16:11.002311+0000","last_undegraded":"2026-02-24T16:16:11.002311+0000","last_fullsized":"2026-02-24T16:16:11.002311+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T16:16:07.974206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T16:16:07.974206+0000","last_clean_scrub_stamp":"2026-02-24T16:16:07.974206+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":90,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-24T16:16:11.001731+0000","last_change":"2026-02-24T16:16:05.995499+0000","last_active":"2026-02-24T16:16:11.001731+0000","last_peered":"2026-02-24T16:16:11.001731+0000","last_clean":"2026-02-24T16:16:11.001731+0000","last_became_active":"2026-02-24T16:16:05.988859+0000","last_became_peered":"2026-02-24T16:16:05.988859+0000","last_unstale":"2026-02-24T16:16:11.001731+0000","last_undegraded":"2026-02-24T16:16:11.001731+0000","last_fullsized":"2026-02-24T16:16:11.001731+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T16:16:04.957640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T16:16:04.957640+0000","last_clean_scrub_stamp":"2026-02-24T16:16:04.957640+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":161061224448,"available":96634163200,"internally_reserved":0,"allocated":64427061248,"data_stored":64427061248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53687074816,"available":32211079168,"internally_reserved":0,"allocated":21475995648,"data_stored":21475995648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949672968,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20971812,"kb_used_data":20971812,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456972,"statfs":{"total":53687074816,"available":32211939328,"internally_reserved":0,"allocated":21475135488,"data_stored":21475135488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949672968,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972572,"kb_used_data":20972572,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456212,"statfs":{"total":53687074816,"available":32211161088,"internally_reserved":0,"allocated":21475913728,"data_stored":21475913728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705673,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972620,"kb_used_data":20972620,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456164,"statfs":{"total":53687074816,"available":32211111936,"internally_reserved":0,"allocated":21475962880,"data_stored":21475962880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53687074816,"available":32211079168,"internally_reserved":0,"allocated":21475995648,"data_stored":21475995648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53687074816,"available":32211152896,"internally_reserved":0,"allocated":21475921920,"data_stored":21475921920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53687074816,"available":32211079168,"internally_reserved":0,"allocated":21475995648,"data_stored":21475995648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53687074816,"available":32211931136,"internally_reserved":0,"allocated":21475143680,"data_stored":21475143680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-24T16:16:13.867 INFO:tasks.ceph.ceph_manager.ceph:clean! 2026-02-24T16:16:13.867 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2026-02-24T16:16:13.867 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2026-02-24T16:16:13.867 DEBUG:teuthology.orchestra.run.trial084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-24T16:16:13.944 INFO:teuthology.orchestra.run.trial084.stderr:2026-02-24T16:16:13.940+0000 7f5a8db5c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:16:13.945 INFO:teuthology.orchestra.run.trial084.stderr:2026-02-24T16:16:13.943+0000 7f5a8db5c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:16:14.083 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:16:14.083 INFO:teuthology.orchestra.run.trial084.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2026-02-24T16:16:14.094 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2026-02-24T16:16:14.094 INFO:teuthology.run_tasks:Running task ssh_keys... 2026-02-24T16:16:14.225 DEBUG:teuthology.orchestra.run.trial084:> echo 'StrictHostKeyChecking no 2026-02-24T16:16:14.225 DEBUG:teuthology.orchestra.run.trial084:> ' > /home/ubuntu/.ssh/config && echo 'UserKnownHostsFile ' /dev/null >> /home/ubuntu/.ssh/config && chmod 600 /home/ubuntu/.ssh/config 2026-02-24T16:16:14.241 INFO:teuthology.task.ssh_keys:generated public key AAAAB3NzaC1yc2EAAAADAQABAAABAQDKxgPeC8SNVlmr76V5iPMNUDhKHF+cxqZTH3behlXDi+6XWHkLAm4UIg9L32q5zTJc6D4+aTgH0fB6mopWwpCpIrel+ejDw7D4MzTOIShNcF2H8iwdlkL7+5H3pYa8s2Cxg1+iWknb110DIiB6x2Kjq+AY47QBray8BkhIYH5feeKkQo4zBHxYBE7V8zI7/BVWUvCLkN30Qy7rfURttVVquDn7PMzdnfBDz1fY0bFNX8uD6EUmvhJwG3QoWvXFCjh8K+ldpjjNSr84wPiFiPskMycUcTBpQEVMgXPMaN1L8THReoOXTS5kh/kp7uVUDvLgB7h4nIX5R3TX9WwWH51D 2026-02-24T16:16:14.242 INFO:teuthology.task.ssh_keys:pushing keys to trial084.front.sepia.ceph.com for ubuntu 2026-02-24T16:16:14.242 DEBUG:teuthology.orchestra.run.trial084:> rm -f -- /home/ubuntu/.ssh/id_rsa 2026-02-24T16:16:14.298 DEBUG:teuthology.orchestra.run.trial084:> set -ex 2026-02-24T16:16:14.298 DEBUG:teuthology.orchestra.run.trial084:> dd of=/home/ubuntu/.ssh/id_rsa 2026-02-24T16:16:14.298 DEBUG:teuthology.orchestra.run.trial084:> chmod 0500 /home/ubuntu/.ssh/id_rsa 2026-02-24T16:16:14.354 DEBUG:teuthology.orchestra.run.trial084:> rm -f -- /home/ubuntu/.ssh/id_rsa.pub 2026-02-24T16:16:14.411 DEBUG:teuthology.orchestra.run.trial084:> set -ex 2026-02-24T16:16:14.411 DEBUG:teuthology.orchestra.run.trial084:> dd of=/home/ubuntu/.ssh/id_rsa.pub 2026-02-24T16:16:14.467 DEBUG:teuthology.orchestra.run.trial084:> cp -v -a /home/ubuntu/.ssh/authorized_keys /home/ubuntu/.ssh/authorized_keys_2026-02-24_16:16:14:467589 2026-02-24T16:16:14.524 INFO:teuthology.orchestra.run.trial084.stdout:'/home/ubuntu/.ssh/authorized_keys' -> '/home/ubuntu/.ssh/authorized_keys_2026-02-24_16:16:14:467589' 2026-02-24T16:16:14.524 DEBUG:teuthology.orchestra.run.trial084:> set -ex 2026-02-24T16:16:14.524 DEBUG:teuthology.orchestra.run.trial084:> dd of=/home/ubuntu/.ssh/authorized_keys conv=notrunc oflag=append 2026-02-24T16:16:14.580 INFO:teuthology.run_tasks:Running task cbt... 2026-02-24T16:16:14.608 INFO:tasks.cbt:cbt configuration is {'cluster': {'user': 'ubuntu', 'head': 'ubuntu@trial084.front.sepia.ceph.com', 'osds': ['ubuntu@trial084.front.sepia.ceph.com'], 'mons': ['ubuntu@trial084.front.sepia.ceph.com'], 'clients': ['ubuntu@trial084.front.sepia.ceph.com'], 'rgws': {'ubuntu@trial084.front.sepia.ceph.com': None}, 'osds_per_node': 3, 'rebuild_every_test': False, 'use_existing': True, 'is_teuthology': True, 'iterations': 1, 'tmp_dir': '/tmp/cbt', 'pool_profiles': {'replicated': {'pg_size': 256, 'pgp_size': 256, 'replication': 'replicated'}}, 'pid_dir': '/var/run/ceph'}, 'benchmarks': {'radosbench': {'concurrent_ops': 4, 'concurrent_procs': 2, 'op_size': [4096], 'pool_monitoring_list': ['collectl'], 'pool_profile': 'replicated', 'run_monitoring_list': ['collectl'], 'time': 60, 'write_only': False}}, 'client_endpoints': None, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}} 2026-02-24T16:16:14.608 DEBUG:teuthology.orchestra.run.trial084:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/cbt 2026-02-24T16:16:14.640 DEBUG:teuthology.orchestra.run.trial084:> set -ex 2026-02-24T16:16:14.641 DEBUG:teuthology.orchestra.run.trial084:> dd of=/home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2026-02-24T16:16:14.693 DEBUG:teuthology.orchestra.run.trial084:> git clone --depth 1 -b master https://github.com/ceph/cbt.git /home/ubuntu/cephtest/cbt 2026-02-24T16:16:14.755 INFO:teuthology.orchestra.run.trial084.stderr:Cloning into '/home/ubuntu/cephtest/cbt'... 2026-02-24T16:16:15.009 INFO:tasks.cbt:Installing collectl 2026-02-24T16:16:15.010 DEBUG:teuthology.orchestra.run.trial084:> sudo mkdir collectl && cd collectl && sudo wget https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download -O collectl.tar.gz && sudo tar -xvf collectl.tar.gz && cd collectl-4.3.1 && sudo ./INSTALL 2026-02-24T16:16:15.084 INFO:teuthology.orchestra.run.trial084.stderr:--2026-02-24 16:16:15-- https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download 2026-02-24T16:16:15.133 INFO:teuthology.orchestra.run.trial084.stderr:Resolving sourceforge.net (sourceforge.net)... 104.18.13.149, 104.18.12.149, 2606:4700::6812:d95, ... 2026-02-24T16:16:15.136 INFO:teuthology.orchestra.run.trial084.stderr:Connecting to sourceforge.net (sourceforge.net)|104.18.13.149|:443... connected. 2026-02-24T16:16:15.334 INFO:teuthology.orchestra.run.trial084.stderr:HTTP request sent, awaiting response... 302 Found 2026-02-24T16:16:15.335 INFO:teuthology.orchestra.run.trial084.stderr:Location: https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABpnc7P98gWNiwydKlE1McGSpNL_bdqIePdalVhqKEgEECij8E4zN4UEkOduoiBemUf0JFCf2Z__zdnRcmYB_DS0rZOUQ%3D%3D&use_mirror=psychz&r= [following] 2026-02-24T16:16:15.335 INFO:teuthology.orchestra.run.trial084.stderr:--2026-02-24 16:16:15-- https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABpnc7P98gWNiwydKlE1McGSpNL_bdqIePdalVhqKEgEECij8E4zN4UEkOduoiBemUf0JFCf2Z__zdnRcmYB_DS0rZOUQ%3D%3D&use_mirror=psychz&r= 2026-02-24T16:16:15.336 INFO:teuthology.orchestra.run.trial084.stderr:Resolving downloads.sourceforge.net (downloads.sourceforge.net)... 104.18.13.149, 104.18.12.149, 2606:4700::6812:d95, ... 2026-02-24T16:16:15.338 INFO:teuthology.orchestra.run.trial084.stderr:Connecting to downloads.sourceforge.net (downloads.sourceforge.net)|104.18.13.149|:443... connected. 2026-02-24T16:16:15.518 INFO:teuthology.orchestra.run.trial084.stderr:HTTP request sent, awaiting response... 302 Found 2026-02-24T16:16:15.518 INFO:teuthology.orchestra.run.trial084.stderr:Location: https://psychz.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?viasf=1 [following] 2026-02-24T16:16:15.518 INFO:teuthology.orchestra.run.trial084.stderr:--2026-02-24 16:16:15-- https://psychz.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?viasf=1 2026-02-24T16:16:15.526 INFO:teuthology.orchestra.run.trial084.stderr:Resolving psychz.dl.sourceforge.net (psychz.dl.sourceforge.net)... 208.87.241.191 2026-02-24T16:16:15.600 INFO:teuthology.orchestra.run.trial084.stderr:Connecting to psychz.dl.sourceforge.net (psychz.dl.sourceforge.net)|208.87.241.191|:443... connected. 2026-02-24T16:16:15.947 INFO:teuthology.orchestra.run.trial084.stderr:HTTP request sent, awaiting response... 200 OK 2026-02-24T16:16:15.947 INFO:teuthology.orchestra.run.trial084.stderr:Length: 620693 (606K) [application/x-gzip] 2026-02-24T16:16:15.947 INFO:teuthology.orchestra.run.trial084.stderr:Saving to: ‘collectl.tar.gz’ 2026-02-24T16:16:15.947 INFO:teuthology.orchestra.run.trial084.stderr: 2026-02-24T16:16:16.050 INFO:teuthology.orchestra.run.trial084.stderr: 0K .......... .......... .......... .......... .......... 8% 487K 1s 2026-02-24T16:16:16.094 INFO:teuthology.orchestra.run.trial084.stderr: 50K .......... .......... .......... .......... .......... 16% 1.07M 1s 2026-02-24T16:16:16.123 INFO:teuthology.orchestra.run.trial084.stderr: 100K .......... .......... .......... .......... .......... 24% 1.69M 1s 2026-02-24T16:16:16.124 INFO:teuthology.orchestra.run.trial084.stderr: 150K .......... .......... .......... .......... .......... 32% 81.9M 0s 2026-02-24T16:16:16.201 INFO:teuthology.orchestra.run.trial084.stderr: 200K .......... .......... .......... .......... .......... 41% 680K 0s 2026-02-24T16:16:16.201 INFO:teuthology.orchestra.run.trial084.stderr: 250K .......... .......... .......... .......... .......... 49% 91.5M 0s 2026-02-24T16:16:16.201 INFO:teuthology.orchestra.run.trial084.stderr: 300K .......... .......... .......... .......... .......... 57% 142M 0s 2026-02-24T16:16:16.201 INFO:teuthology.orchestra.run.trial084.stderr: 350K .......... .......... .......... .......... .......... 65% 134M 0s 2026-02-24T16:16:16.250 INFO:teuthology.orchestra.run.trial084.stderr: 400K .......... .......... .......... .......... .......... 74% 998K 0s 2026-02-24T16:16:16.250 INFO:teuthology.orchestra.run.trial084.stderr: 450K .......... .......... .......... .......... .......... 82% 387M 0s 2026-02-24T16:16:16.274 INFO:teuthology.orchestra.run.trial084.stderr: 500K .......... .......... .......... .......... .......... 90% 2.13M 0s 2026-02-24T16:16:16.274 INFO:teuthology.orchestra.run.trial084.stderr: 550K .......... .......... .......... .......... .......... 98% 189M 0s 2026-02-24T16:16:16.274 INFO:teuthology.orchestra.run.trial084.stderr: 600K ...... 100% 11.4T=0.3s 2026-02-24T16:16:16.274 INFO:teuthology.orchestra.run.trial084.stderr: 2026-02-24T16:16:16.274 INFO:teuthology.orchestra.run.trial084.stderr:2026-02-24 16:16:16 (1.82 MB/s) - ‘collectl.tar.gz’ saved [620693/620693] 2026-02-24T16:16:16.275 INFO:teuthology.orchestra.run.trial084.stderr: 2026-02-24T16:16:16.295 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/ 2026-02-24T16:16:16.295 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/collectl.conf 2026-02-24T16:16:16.295 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/docs/ 2026-02-24T16:16:16.295 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/docs/SlowProc.html 2026-02-24T16:16:16.295 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/docs/BuddyInfo.html 2026-02-24T16:16:16.295 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/docs/WhySummary.html 2026-02-24T16:16:16.295 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/docs/Socket.html 2026-02-24T16:16:16.295 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/docs/colmux.html 2026-02-24T16:16:16.295 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/docs/Tutorial.html 2026-02-24T16:16:16.295 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/docs/Gexpr.html 2026-02-24T16:16:16.296 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/docs/Import.html 2026-02-24T16:16:16.296 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/docs/style.css 2026-02-24T16:16:16.296 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/docs/Ganglia3.jpg 2026-02-24T16:16:16.296 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/docs/Ganglia2.jpg 2026-02-24T16:16:16.296 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/docs/FileNaming.html 2026-02-24T16:16:16.296 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/docs/TheMath.html 2026-02-24T16:16:16.296 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/docs/Environmental.html 2026-02-24T16:16:16.296 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/docs/Process.html 2026-02-24T16:16:16.296 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/docs/OutputFiles.html 2026-02-24T16:16:16.296 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/docs/HiResTime.html 2026-02-24T16:16:16.296 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/docs/OutputFormats.html 2026-02-24T16:16:16.296 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/docs/slub.jpg 2026-02-24T16:16:16.296 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/docs/Lustre.html 2026-02-24T16:16:16.297 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/docs/Documentation.html 2026-02-24T16:16:16.297 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/docs/Inodes.html 2026-02-24T16:16:16.297 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/docs/Graphite.html 2026-02-24T16:16:16.297 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/docs/InputFiles.html 2026-02-24T16:16:16.297 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/docs/Export.html 2026-02-24T16:16:16.297 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/docs/Interrupts.html 2026-02-24T16:16:16.297 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/docs/Startup.html 2026-02-24T16:16:16.297 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/docs/Performance.html 2026-02-24T16:16:16.297 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/docs/Messages.html 2026-02-24T16:16:16.297 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/docs/Logging.html 2026-02-24T16:16:16.297 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/docs/Data-detail.html 2026-02-24T16:16:16.297 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/docs/CPUs.html 2026-02-24T16:16:16.297 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/docs/Matrix.html 2026-02-24T16:16:16.298 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/docs/Data-brief.html 2026-02-24T16:16:16.298 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/docs/SlabInfo.html 2026-02-24T16:16:16.298 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/docs/Misc.html 2026-02-24T16:16:16.298 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/docs/Examples.html 2026-02-24T16:16:16.298 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/docs/OpenStack.html 2026-02-24T16:16:16.298 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/docs/Plotfiles.html 2026-02-24T16:16:16.298 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/docs/ColmuxCPU.jpg 2026-02-24T16:16:16.298 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/docs/Tutorial-Lustre.html 2026-02-24T16:16:16.298 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/docs/Data-verbose.html 2026-02-24T16:16:16.298 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/docs/Data.html 2026-02-24T16:16:16.298 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/docs/Exceptions.html 2026-02-24T16:16:16.298 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/docs/RunningAsAService.html 2026-02-24T16:16:16.298 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/docs/FAQ-collectl.html 2026-02-24T16:16:16.299 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/docs/Ganglia.jpg 2026-02-24T16:16:16.299 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/docs/OperationalModes.html 2026-02-24T16:16:16.299 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/docs/Playback.html 2026-02-24T16:16:16.299 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/docs/Hello.html 2026-02-24T16:16:16.299 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/docs/Sockets.html 2026-02-24T16:16:16.299 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/docs/Network.html 2026-02-24T16:16:16.299 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/docs/Infiniband.html 2026-02-24T16:16:16.299 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/docs/Memory.html 2026-02-24T16:16:16.299 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/docs/ColmuxLustre.jpg 2026-02-24T16:16:16.299 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/docs/Disks.html 2026-02-24T16:16:16.299 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/docs/NfsInfo.html 2026-02-24T16:16:16.299 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/service/ 2026-02-24T16:16:16.299 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/service/collectl.service 2026-02-24T16:16:16.300 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/initd/ 2026-02-24T16:16:16.300 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/initd/collectl-suse 2026-02-24T16:16:16.300 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/initd/collectl-debian 2026-02-24T16:16:16.300 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/initd/collectl-generic 2026-02-24T16:16:16.300 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/initd/collectl 2026-02-24T16:16:16.300 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/misc.ph 2026-02-24T16:16:16.300 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/gexpr.ph 2026-02-24T16:16:16.300 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/statsd.ph 2026-02-24T16:16:16.300 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/envrules.std 2026-02-24T16:16:16.300 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/hello.ph 2026-02-24T16:16:16.300 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/vmsum.ph 2026-02-24T16:16:16.300 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/proctree.ph 2026-02-24T16:16:16.300 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/README-WINDOWS 2026-02-24T16:16:16.301 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/formatit.ph 2026-02-24T16:16:16.301 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/colmux 2026-02-24T16:16:16.301 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/INSTALL 2026-02-24T16:16:16.301 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/graphite.ph 2026-02-24T16:16:16.301 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/RELEASE-collectl 2026-02-24T16:16:16.301 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/vmstat.ph 2026-02-24T16:16:16.301 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/README 2026-02-24T16:16:16.301 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/man1/ 2026-02-24T16:16:16.301 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/man1/collectl.1 2026-02-24T16:16:16.301 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/man1/colmux.1 2026-02-24T16:16:16.301 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/lexpr.ph 2026-02-24T16:16:16.301 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/client.pl 2026-02-24T16:16:16.301 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/collectl 2026-02-24T16:16:16.302 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/COPYING 2026-02-24T16:16:16.302 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/vnet.ph 2026-02-24T16:16:16.302 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/GPL 2026-02-24T16:16:16.302 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/UNINSTALL 2026-02-24T16:16:16.302 INFO:teuthology.orchestra.run.trial084.stdout:collectl-4.3.1/ARTISTIC 2026-02-24T16:16:16.338 INFO:teuthology.orchestra.run.trial084.stderr:./INSTALL: line 90: chkconfig: command not found 2026-02-24T16:16:16.339 DEBUG:teuthology.orchestra.run.trial084:> sudo yum -y install librbd-devel pdsh pdsh-rcmd-ssh perf 2026-02-24T16:16:16.700 INFO:teuthology.orchestra.run.trial084.stdout:Last metadata expiration check: 0:00:58 ago on Tue 24 Feb 2026 04:15:18 PM UTC. 2026-02-24T16:16:16.786 INFO:teuthology.orchestra.run.trial084.stdout:Dependencies resolved. 2026-02-24T16:16:16.787 INFO:teuthology.orchestra.run.trial084.stdout:================================================================================ 2026-02-24T16:16:16.788 INFO:teuthology.orchestra.run.trial084.stdout: Package Arch Version Repository Size 2026-02-24T16:16:16.788 INFO:teuthology.orchestra.run.trial084.stdout:================================================================================ 2026-02-24T16:16:16.788 INFO:teuthology.orchestra.run.trial084.stdout:Installing: 2026-02-24T16:16:16.788 INFO:teuthology.orchestra.run.trial084.stdout: librbd-devel x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 24 k 2026-02-24T16:16:16.788 INFO:teuthology.orchestra.run.trial084.stdout: pdsh x86_64 2.34-7.el9 epel 118 k 2026-02-24T16:16:16.788 INFO:teuthology.orchestra.run.trial084.stdout: pdsh-rcmd-ssh x86_64 2.34-7.el9 epel 13 k 2026-02-24T16:16:16.788 INFO:teuthology.orchestra.run.trial084.stdout: perf x86_64 5.14.0-686.el9 appstream 3.3 M 2026-02-24T16:16:16.788 INFO:teuthology.orchestra.run.trial084.stdout:Installing dependencies: 2026-02-24T16:16:16.788 INFO:teuthology.orchestra.run.trial084.stdout: libradospp-devel x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 31 k 2026-02-24T16:16:16.788 INFO:teuthology.orchestra.run.trial084.stdout: libtraceevent x86_64 1.8.4-2.el9 baseos 290 k 2026-02-24T16:16:16.788 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:16:16.789 INFO:teuthology.orchestra.run.trial084.stdout:Transaction Summary 2026-02-24T16:16:16.789 INFO:teuthology.orchestra.run.trial084.stdout:================================================================================ 2026-02-24T16:16:16.789 INFO:teuthology.orchestra.run.trial084.stdout:Install 6 Packages 2026-02-24T16:16:16.789 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:16:16.789 INFO:teuthology.orchestra.run.trial084.stdout:Total download size: 3.8 M 2026-02-24T16:16:16.789 INFO:teuthology.orchestra.run.trial084.stdout:Installed size: 14 M 2026-02-24T16:16:16.789 INFO:teuthology.orchestra.run.trial084.stdout:Downloading Packages: 2026-02-24T16:16:17.419 INFO:teuthology.orchestra.run.trial084.stdout:(1/6): libradospp-devel-20.3.0-5343.gd26e5c22.e 94 kB/s | 31 kB 00:00 2026-02-24T16:16:17.419 INFO:teuthology.orchestra.run.trial084.stdout:(2/6): librbd-devel-20.3.0-5343.gd26e5c22.el9.x 72 kB/s | 24 kB 00:00 2026-02-24T16:16:17.538 INFO:teuthology.orchestra.run.trial084.stdout:(3/6): perf-5.14.0-686.el9.x86_64.rpm 28 MB/s | 3.3 MB 00:00 2026-02-24T16:16:17.684 INFO:teuthology.orchestra.run.trial084.stdout:(4/6): pdsh-rcmd-ssh-2.34-7.el9.x86_64.rpm 90 kB/s | 13 kB 00:00 2026-02-24T16:16:17.693 INFO:teuthology.orchestra.run.trial084.stdout:(5/6): pdsh-2.34-7.el9.x86_64.rpm 433 kB/s | 118 kB 00:00 2026-02-24T16:16:17.973 INFO:teuthology.orchestra.run.trial084.stdout:(6/6): libtraceevent-1.8.4-2.el9.x86_64.rpm 327 kB/s | 290 kB 00:00 2026-02-24T16:16:17.978 INFO:teuthology.orchestra.run.trial084.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:16:17.978 INFO:teuthology.orchestra.run.trial084.stdout:Total 3.2 MB/s | 3.8 MB 00:01 2026-02-24T16:16:18.001 INFO:teuthology.orchestra.run.trial084.stdout:Running transaction check 2026-02-24T16:16:18.012 INFO:teuthology.orchestra.run.trial084.stdout:Transaction check succeeded. 2026-02-24T16:16:18.012 INFO:teuthology.orchestra.run.trial084.stdout:Running transaction test 2026-02-24T16:16:18.069 INFO:teuthology.orchestra.run.trial084.stdout:Transaction test succeeded. 2026-02-24T16:16:18.069 INFO:teuthology.orchestra.run.trial084.stdout:Running transaction 2026-02-24T16:16:18.143 INFO:teuthology.orchestra.run.trial084.stdout: Preparing : 1/1 2026-02-24T16:16:18.149 INFO:teuthology.orchestra.run.trial084.stdout: Installing : pdsh-rcmd-ssh-2.34-7.el9.x86_64 1/6 2026-02-24T16:16:18.188 INFO:teuthology.orchestra.run.trial084.stdout: Installing : pdsh-2.34-7.el9.x86_64 2/6 2026-02-24T16:16:18.193 INFO:teuthology.orchestra.run.trial084.stdout: Installing : libtraceevent-1.8.4-2.el9.x86_64 3/6 2026-02-24T16:16:18.195 INFO:teuthology.orchestra.run.trial084.stdout: Installing : libradospp-devel-2:20.3.0-5343.gd26e5c22.el9.x86_64 4/6 2026-02-24T16:16:18.357 INFO:teuthology.orchestra.run.trial084.stdout: Installing : librbd-devel-2:20.3.0-5343.gd26e5c22.el9.x86_64 5/6 2026-02-24T16:16:18.378 INFO:teuthology.orchestra.run.trial084.stdout: Installing : perf-5.14.0-686.el9.x86_64 6/6 2026-02-24T16:16:18.494 INFO:teuthology.orchestra.run.trial084.stdout: Running scriptlet: perf-5.14.0-686.el9.x86_64 6/6 2026-02-24T16:16:18.494 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : libradospp-devel-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/6 2026-02-24T16:16:18.494 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : librbd-devel-2:20.3.0-5343.gd26e5c22.el9.x86_64 2/6 2026-02-24T16:16:18.494 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : libtraceevent-1.8.4-2.el9.x86_64 3/6 2026-02-24T16:16:18.494 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : perf-5.14.0-686.el9.x86_64 4/6 2026-02-24T16:16:18.495 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : pdsh-2.34-7.el9.x86_64 5/6 2026-02-24T16:16:18.558 INFO:teuthology.orchestra.run.trial084.stdout: Verifying : pdsh-rcmd-ssh-2.34-7.el9.x86_64 6/6 2026-02-24T16:16:18.558 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:16:18.558 INFO:teuthology.orchestra.run.trial084.stdout:Installed: 2026-02-24T16:16:18.558 INFO:teuthology.orchestra.run.trial084.stdout: libradospp-devel-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T16:16:18.558 INFO:teuthology.orchestra.run.trial084.stdout: librbd-devel-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T16:16:18.558 INFO:teuthology.orchestra.run.trial084.stdout: libtraceevent-1.8.4-2.el9.x86_64 2026-02-24T16:16:18.558 INFO:teuthology.orchestra.run.trial084.stdout: pdsh-2.34-7.el9.x86_64 2026-02-24T16:16:18.559 INFO:teuthology.orchestra.run.trial084.stdout: pdsh-rcmd-ssh-2.34-7.el9.x86_64 2026-02-24T16:16:18.559 INFO:teuthology.orchestra.run.trial084.stdout: perf-5.14.0-686.el9.x86_64 2026-02-24T16:16:18.559 INFO:teuthology.orchestra.run.trial084.stdout: 2026-02-24T16:16:18.559 INFO:teuthology.orchestra.run.trial084.stdout:Complete! 2026-02-24T16:16:18.588 DEBUG:teuthology.orchestra.run.trial084:> python3 -m pip install -r /home/ubuntu/cephtest/cbt/requirements.txt 2026-02-24T16:16:18.727 INFO:teuthology.orchestra.run.trial084.stdout:Defaulting to user installation because normal site-packages is not writeable 2026-02-24T16:16:18.738 INFO:teuthology.orchestra.run.trial084.stdout:Requirement already satisfied: pyyaml in /usr/lib64/python3.9/site-packages (from -r /home/ubuntu/cephtest/cbt/requirements.txt (line 1)) (5.4.1) 2026-02-24T16:16:19.193 INFO:teuthology.orchestra.run.trial084.stdout:Collecting lxml 2026-02-24T16:16:19.217 INFO:teuthology.orchestra.run.trial084.stdout: Downloading lxml-6.0.2-cp39-cp39-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl (5.3 MB) 2026-02-24T16:16:19.586 INFO:teuthology.orchestra.run.trial084.stdout:Collecting matplotlib 2026-02-24T16:16:19.592 INFO:teuthology.orchestra.run.trial084.stdout: Downloading matplotlib-3.9.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (8.3 MB) 2026-02-24T16:16:19.667 INFO:teuthology.orchestra.run.trial084.stdout:Requirement already satisfied: packaging>=20.0 in /usr/lib/python3.9/site-packages (from matplotlib->-r /home/ubuntu/cephtest/cbt/requirements.txt (line 3)) (20.9) 2026-02-24T16:16:19.667 INFO:teuthology.orchestra.run.trial084.stdout:Requirement already satisfied: pyparsing>=2.3.1 in /usr/lib/python3.9/site-packages (from matplotlib->-r /home/ubuntu/cephtest/cbt/requirements.txt (line 3)) (2.4.7) 2026-02-24T16:16:20.002 INFO:teuthology.orchestra.run.trial084.stdout:Collecting fonttools>=4.22.0 2026-02-24T16:16:20.009 INFO:teuthology.orchestra.run.trial084.stdout: Downloading fonttools-4.60.2-cp39-cp39-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (4.8 MB) 2026-02-24T16:16:20.045 INFO:teuthology.orchestra.run.trial084.stdout:Requirement already satisfied: python-dateutil>=2.7 in /usr/lib/python3.9/site-packages (from matplotlib->-r /home/ubuntu/cephtest/cbt/requirements.txt (line 3)) (2.9.0.post0) 2026-02-24T16:16:20.078 INFO:teuthology.orchestra.run.trial084.stdout:Collecting importlib-resources>=3.2.0 2026-02-24T16:16:20.082 INFO:teuthology.orchestra.run.trial084.stdout: Downloading importlib_resources-6.5.2-py3-none-any.whl (37 kB) 2026-02-24T16:16:20.083 INFO:teuthology.orchestra.run.trial084.stdout:Requirement already satisfied: numpy>=1.23 in /usr/lib64/python3.9/site-packages (from matplotlib->-r /home/ubuntu/cephtest/cbt/requirements.txt (line 3)) (1.23.5) 2026-02-24T16:16:20.223 INFO:teuthology.orchestra.run.trial084.stdout:Collecting kiwisolver>=1.3.1 2026-02-24T16:16:20.234 INFO:teuthology.orchestra.run.trial084.stdout: Downloading kiwisolver-1.4.7-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.6 MB) 2026-02-24T16:16:20.257 INFO:teuthology.orchestra.run.trial084.stdout:Collecting cycler>=0.10 2026-02-24T16:16:20.261 INFO:teuthology.orchestra.run.trial084.stdout: Downloading cycler-0.12.1-py3-none-any.whl (8.3 kB) 2026-02-24T16:16:20.429 INFO:teuthology.orchestra.run.trial084.stdout:Collecting contourpy>=1.0.1 2026-02-24T16:16:20.433 INFO:teuthology.orchestra.run.trial084.stdout: Downloading contourpy-1.3.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (321 kB) 2026-02-24T16:16:20.841 INFO:teuthology.orchestra.run.trial084.stdout:Collecting pillow>=8 2026-02-24T16:16:20.849 INFO:teuthology.orchestra.run.trial084.stdout: Downloading pillow-11.3.0-cp39-cp39-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (6.6 MB) 2026-02-24T16:16:21.001 INFO:teuthology.orchestra.run.trial084.stdout:Collecting zipp>=3.1.0 2026-02-24T16:16:21.007 INFO:teuthology.orchestra.run.trial084.stdout: Downloading zipp-3.23.0-py3-none-any.whl (10 kB) 2026-02-24T16:16:21.018 INFO:teuthology.orchestra.run.trial084.stdout:Requirement already satisfied: six>=1.5 in /usr/lib/python3.9/site-packages (from python-dateutil>=2.7->matplotlib->-r /home/ubuntu/cephtest/cbt/requirements.txt (line 3)) (1.15.0) 2026-02-24T16:16:21.132 INFO:teuthology.orchestra.run.trial084.stdout:Installing collected packages: zipp, pillow, kiwisolver, importlib-resources, fonttools, cycler, contourpy, matplotlib, lxml 2026-02-24T16:16:22.260 INFO:teuthology.orchestra.run.trial084.stdout:Successfully installed contourpy-1.3.0 cycler-0.12.1 fonttools-4.60.2 importlib-resources-6.5.2 kiwisolver-1.4.7 lxml-6.0.2 matplotlib-3.9.4 pillow-11.3.0 zipp-3.23.0 2026-02-24T16:16:22.376 INFO:tasks.cbt:benchmark: radosbench 2026-02-24T16:16:22.377 DEBUG:teuthology.orchestra.run.trial084:> sudo /sbin/sysctl -q -w kernel.perf_event_paranoid=0 2026-02-24T16:16:22.403 DEBUG:teuthology.orchestra.run.trial084:> /home/ubuntu/cephtest/cbt/cbt.py -a /home/ubuntu/cephtest/archive/cbt /home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2026-02-24T16:16:22.515 INFO:teuthology.orchestra.run.trial084.stderr:16:16:22 - DEBUG - cbt - Settings.cluster: 2026-02-24T16:16:22.515 INFO:teuthology.orchestra.run.trial084.stderr: {'archive_dir': '/home/ubuntu/cephtest/archive/cbt', 2026-02-24T16:16:22.515 INFO:teuthology.orchestra.run.trial084.stderr: 'clients': ['ubuntu@trial084.front.sepia.ceph.com'], 2026-02-24T16:16:22.515 INFO:teuthology.orchestra.run.trial084.stderr: 'conf_file': '/etc/ceph/ceph.conf', 2026-02-24T16:16:22.515 INFO:teuthology.orchestra.run.trial084.stderr: 'head': 'ubuntu@trial084.front.sepia.ceph.com', 2026-02-24T16:16:22.515 INFO:teuthology.orchestra.run.trial084.stderr: 'is_teuthology': True, 2026-02-24T16:16:22.515 INFO:teuthology.orchestra.run.trial084.stderr: 'iterations': 1, 2026-02-24T16:16:22.516 INFO:teuthology.orchestra.run.trial084.stderr: 'mons': ['ubuntu@trial084.front.sepia.ceph.com'], 2026-02-24T16:16:22.516 INFO:teuthology.orchestra.run.trial084.stderr: 'osds': ['ubuntu@trial084.front.sepia.ceph.com'], 2026-02-24T16:16:22.516 INFO:teuthology.orchestra.run.trial084.stderr: 'osds_per_node': 3, 2026-02-24T16:16:22.516 INFO:teuthology.orchestra.run.trial084.stderr: 'pid_dir': '/var/run/ceph', 2026-02-24T16:16:22.516 INFO:teuthology.orchestra.run.trial084.stderr: 'pool_profiles': {'replicated': {'pg_size': 256, 2026-02-24T16:16:22.516 INFO:teuthology.orchestra.run.trial084.stderr: 'pgp_size': 256, 2026-02-24T16:16:22.516 INFO:teuthology.orchestra.run.trial084.stderr: 'replication': 'replicated'}}, 2026-02-24T16:16:22.516 INFO:teuthology.orchestra.run.trial084.stderr: 'rebuild_every_test': False, 2026-02-24T16:16:22.516 INFO:teuthology.orchestra.run.trial084.stderr: 'rgws': {'ubuntu@trial084.front.sepia.ceph.com': None}, 2026-02-24T16:16:22.516 INFO:teuthology.orchestra.run.trial084.stderr: 'tmp_dir': '/tmp/cbt', 2026-02-24T16:16:22.516 INFO:teuthology.orchestra.run.trial084.stderr: 'use_existing': True, 2026-02-24T16:16:22.516 INFO:teuthology.orchestra.run.trial084.stderr: 'user': 'ubuntu'} 2026-02-24T16:16:22.517 INFO:teuthology.orchestra.run.trial084.stderr:16:16:22 - INFO - cbt - Results dir: /home/ubuntu/cephtest/archive/cbt/results/00000000/id-31cd68af 2026-02-24T16:16:22.517 INFO:teuthology.orchestra.run.trial084.stderr:16:16:22 - INFO - cbt - Pausing for 60s for idle monitoring. 2026-02-24T16:16:22.517 INFO:teuthology.orchestra.run.trial084.stderr:16:16:22 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial084.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/idle_monitoring/collectl 2026-02-24T16:16:22.682 INFO:teuthology.orchestra.run.trial084.stderr:16:16:22 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial084.front.sepia.ceph.com collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/idle_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2026-02-24T16:16:22.683 INFO:teuthology.orchestra.run.trial084.stderr:16:16:22 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial084.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/idle_monitoring/perf 2026-02-24T16:16:22.843 INFO:teuthology.orchestra.run.trial084.stderr:16:16:22 - DEBUG - cbt - PerfMonitoring: remote_node 2026-02-24T16:16:22.843 INFO:teuthology.orchestra.run.trial084.stderr:16:16:22 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial084.front.sepia.ceph.com for pid in `cat /var/run/ceph/crimson-osd.*.pid`; do perf stat -p ${pid} -o /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/idle_monitoring/perf/perf_stat.${pid} & done 2026-02-24T16:17:22.889 INFO:teuthology.orchestra.run.trial084.stderr:16:17:22 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial084.front.sepia.ceph.com pkill -SIGINT -f collectl 2026-02-24T16:17:23.066 INFO:teuthology.orchestra.run.trial084.stderr:16:17:23 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial084.front.sepia.ceph.com sudo pkill -SIGINT -f perf\ 2026-02-24T16:17:23.241 INFO:teuthology.orchestra.run.trial084.stderr:16:17:23 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial084.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/idle_monitoring/perf/perf.data 2026-02-24T16:17:23.243 INFO:teuthology.orchestra.run.trial084.stderr:16:17:23 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial084.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/idle_monitoring/perf/perf_stat.* 2026-02-24T16:17:23.245 INFO:teuthology.orchestra.run.trial084.stderr:16:17:23 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial084.front.sepia.ceph.com sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/* 2026-02-24T16:17:23.418 INFO:teuthology.orchestra.run.trial084.stderr:16:17:23 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=rpdcp -f 10 -R ssh -w ubuntu@trial084.front.sepia.ceph.com -r /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/* /home/ubuntu/cephtest/archive/cbt/results/00000000/id-31cd68af 2026-02-24T16:17:23.581 INFO:teuthology.orchestra.run.trial084.stderr:16:17:23 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial084.front.sepia.ceph.com sudo killall -9 rados 2026-02-24T16:17:23.756 INFO:teuthology.orchestra.run.trial084.stderr:16:17:23 - INFO - cbt - Results dir: /home/ubuntu/cephtest/archive/cbt/results/00000000/id-31cd68af 2026-02-24T16:17:23.756 INFO:teuthology.orchestra.run.trial084.stderr:16:17:23 - INFO - cbt - Skipping existing test in /home/ubuntu/cephtest/archive/cbt/results/00000000/id-31cd68af. 2026-02-24T16:17:23.756 INFO:teuthology.orchestra.run.trial084.stderr:16:17:23 - INFO - cbt - Running benchmark /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004 2026-02-24T16:17:23.756 INFO:teuthology.orchestra.run.trial084.stderr:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-31cd68af 2026-02-24T16:17:23.756 INFO:teuthology.orchestra.run.trial084.stderr:{'concurrent_ops': 4, 'concurrent_procs': 2, 'pool_profile': 'replicated', 'time': 60, 'write_only': False, 'op_size': 4096, 'pool_monitoring_list': 'collectl', 'run_monitoring_list': 'collectl', 'benchmark': 'radosbench', 'iteration': 0} == iteration 0 == 2026-02-24T16:17:23.756 INFO:teuthology.orchestra.run.trial084.stderr:16:17:23 - INFO - cbt - Setting OSD Read Ahead to: 0 2026-02-24T16:17:23.756 INFO:teuthology.orchestra.run.trial084.stderr:16:17:23 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial084.front.sepia.ceph.com find /dev/disk/by-partlabel/osd-device-*data -exec readlink {} \; | cut -d"/" -f 3 | sed "s/[0-9]$//" | xargs -I{} sudo sh -c "echo 0 > /sys/block/'{}'/queue/read_ahead_kb" 2026-02-24T16:17:23.758 INFO:teuthology.orchestra.run.trial084.stderr:16:17:23 - DEBUG - cbt - Cleaning existing temporary run directory: /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004 2026-02-24T16:17:23.758 INFO:teuthology.orchestra.run.trial084.stderr:16:17:23 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial084.front.sepia.ceph.com sudo rm -rf /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004 2026-02-24T16:17:23.929 INFO:teuthology.orchestra.run.trial084.stderr:16:17:23 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial084.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/pool_monitoring/collectl 2026-02-24T16:17:24.091 INFO:teuthology.orchestra.run.trial084.stderr:16:17:24 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial084.front.sepia.ceph.com collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/pool_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2026-02-24T16:17:24.092 INFO:teuthology.orchestra.run.trial084.stderr:16:17:24 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial084.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/pool_monitoring/perf 2026-02-24T16:17:24.256 INFO:teuthology.orchestra.run.trial084.stderr:16:17:24 - DEBUG - cbt - PerfMonitoring: remote_node 2026-02-24T16:17:24.256 INFO:teuthology.orchestra.run.trial084.stderr:16:17:24 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial084.front.sepia.ceph.com for pid in `cat /var/run/ceph/crimson-osd.*.pid`; do perf stat -p ${pid} -o /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/pool_monitoring/perf/perf_stat.${pid} & done 2026-02-24T16:17:24.257 INFO:teuthology.orchestra.run.trial084.stderr:16:17:24 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial084.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool delete rados-bench-cbt rados-bench-cbt --yes-i-really-really-mean-it 2026-02-24T16:17:24.599 INFO:teuthology.orchestra.run.trial084.stderr:16:17:24 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial084.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool create rados-bench-cbt 256 256 2026-02-24T16:17:25.051 INFO:teuthology.orchestra.run.trial084.stderr:16:17:25 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial084.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool application enable rados-bench-cbt radosbench 2026-02-24T16:17:26.051 INFO:teuthology.orchestra.run.trial084.stderr:16:17:26 - INFO - cbt - Checking Health after pool creation. 2026-02-24T16:17:31.057 INFO:teuthology.orchestra.run.trial084.stderr:16:17:31 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial084.front.sepia.ceph.com /usr/bin/ceph -c /etc/ceph/ceph.conf health 2026-02-24T16:17:31.413 INFO:teuthology.orchestra.run.trial084.stderr:16:17:31 - INFO - cbt - Final Pool Health Check. 2026-02-24T16:17:36.418 INFO:teuthology.orchestra.run.trial084.stderr:16:17:36 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial084.front.sepia.ceph.com /usr/bin/ceph -c /etc/ceph/ceph.conf health 2026-02-24T16:17:36.777 INFO:teuthology.orchestra.run.trial084.stderr:16:17:36 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial084.front.sepia.ceph.com pkill -SIGINT -f collectl 2026-02-24T16:17:36.943 INFO:teuthology.orchestra.run.trial084.stderr:16:17:36 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial084.front.sepia.ceph.com sudo pkill -SIGINT -f perf\ 2026-02-24T16:17:37.125 INFO:teuthology.orchestra.run.trial084.stderr:16:17:37 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial084.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/pool_monitoring/perf/perf.data 2026-02-24T16:17:37.127 INFO:teuthology.orchestra.run.trial084.stderr:16:17:37 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial084.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/pool_monitoring/perf/perf_stat.* 2026-02-24T16:17:37.129 INFO:teuthology.orchestra.run.trial084.stderr:16:17:37 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial084.front.sepia.ceph.com sync 2026-02-24T16:17:37.371 INFO:teuthology.orchestra.run.trial084.stderr:16:17:37 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial084.front.sepia.ceph.com echo 3 | sudo tee /proc/sys/vm/drop_caches 2026-02-24T16:17:38.350 INFO:teuthology.orchestra.run.trial084.stderr:16:17:38 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial084.front.sepia.ceph.com /usr/bin/rados -c /etc/ceph/ceph.conf -v 2026-02-24T16:17:38.559 INFO:teuthology.orchestra.run.trial084.stderr:16:17:38 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial084.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/write 2026-02-24T16:17:38.717 INFO:teuthology.orchestra.run.trial084.stderr:16:17:38 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial084.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf daemon osd.0 config show > /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/write/ceph_settings.out 2026-02-24T16:17:38.998 INFO:teuthology.orchestra.run.trial084.stderr:16:17:38 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial084.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/write/collectl 2026-02-24T16:17:39.156 INFO:teuthology.orchestra.run.trial084.stderr:16:17:39 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial084.front.sepia.ceph.com collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/write/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2026-02-24T16:17:39.158 INFO:teuthology.orchestra.run.trial084.stderr:16:17:39 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial084.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/write/perf 2026-02-24T16:17:39.318 INFO:teuthology.orchestra.run.trial084.stderr:16:17:39 - DEBUG - cbt - PerfMonitoring: remote_node 2026-02-24T16:17:39.319 INFO:teuthology.orchestra.run.trial084.stderr:16:17:39 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial084.front.sepia.ceph.com for pid in `cat /var/run/ceph/crimson-osd.*.pid`; do perf stat -p ${pid} -o /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/write/perf/perf_stat.${pid} & done 2026-02-24T16:17:39.320 INFO:teuthology.orchestra.run.trial084.stderr:16:17:39 - INFO - cbt - Running radosbench write test. 2026-02-24T16:17:39.320 INFO:teuthology.orchestra.run.trial084.stderr:16:17:39 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial084.front.sepia.ceph.com /usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench -b 4096 60 write --concurrent-ios 4 --run-name `hostname -f`-0 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/write/objecter.0.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/write/output.0 2026-02-24T16:17:39.322 INFO:teuthology.orchestra.run.trial084.stderr:16:17:39 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial084.front.sepia.ceph.com /usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench -b 4096 60 write --concurrent-ios 4 --run-name `hostname -f`-1 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/write/objecter.1.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/write/output.1 2026-02-24T16:18:40.536 INFO:teuthology.orchestra.run.trial084.stderr:16:18:40 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial084.front.sepia.ceph.com pkill -SIGINT -f collectl 2026-02-24T16:18:40.720 INFO:teuthology.orchestra.run.trial084.stderr:16:18:40 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial084.front.sepia.ceph.com sudo pkill -SIGINT -f perf\ 2026-02-24T16:18:40.893 INFO:teuthology.orchestra.run.trial084.stderr:16:18:40 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial084.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/write/perf/perf.data 2026-02-24T16:18:40.894 INFO:teuthology.orchestra.run.trial084.stderr:16:18:40 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial084.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/write/perf/perf_stat.* 2026-02-24T16:18:40.896 INFO:teuthology.orchestra.run.trial084.stderr:16:18:40 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial084.front.sepia.ceph.com find /var/run/ceph/ceph-osd*.asok -maxdepth 1 -exec sudo /usr/bin/ceph --admin-daemon {} dump_historic_ops \; > /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/write/historic_ops.out 2026-02-24T16:18:41.259 INFO:teuthology.orchestra.run.trial084.stderr:16:18:41 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial084.front.sepia.ceph.com sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/write/* 2026-02-24T16:18:41.428 INFO:teuthology.orchestra.run.trial084.stderr:16:18:41 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=rpdcp -f 10 -R ssh -w ubuntu@trial084.front.sepia.ceph.com -r /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/write/* /home/ubuntu/cephtest/archive/cbt/results/00000000/id-31cd68af/write 2026-02-24T16:18:41.591 INFO:teuthology.orchestra.run.trial084.stderr:16:18:41 - INFO - cbt - Convert results to json format. 2026-02-24T16:18:41.592 INFO:teuthology.orchestra.run.trial084.stderr:16:18:41 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial084.front.sepia.ceph.com sync 2026-02-24T16:18:41.749 INFO:teuthology.orchestra.run.trial084.stderr:16:18:41 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial084.front.sepia.ceph.com echo 3 | sudo tee /proc/sys/vm/drop_caches 2026-02-24T16:18:42.246 INFO:teuthology.orchestra.run.trial084.stderr:16:18:42 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial084.front.sepia.ceph.com /usr/bin/rados -c /etc/ceph/ceph.conf -v 2026-02-24T16:18:42.454 INFO:teuthology.orchestra.run.trial084.stderr:16:18:42 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial084.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/seq 2026-02-24T16:18:42.612 INFO:teuthology.orchestra.run.trial084.stderr:16:18:42 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial084.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf daemon osd.0 config show > /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/seq/ceph_settings.out 2026-02-24T16:18:42.896 INFO:teuthology.orchestra.run.trial084.stderr:16:18:42 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial084.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/seq/collectl 2026-02-24T16:18:43.056 INFO:teuthology.orchestra.run.trial084.stderr:16:18:43 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial084.front.sepia.ceph.com collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/seq/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2026-02-24T16:18:43.057 INFO:teuthology.orchestra.run.trial084.stderr:16:18:43 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial084.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/seq/perf 2026-02-24T16:18:43.228 INFO:teuthology.orchestra.run.trial084.stderr:16:18:43 - DEBUG - cbt - PerfMonitoring: remote_node 2026-02-24T16:18:43.228 INFO:teuthology.orchestra.run.trial084.stderr:16:18:43 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial084.front.sepia.ceph.com for pid in `cat /var/run/ceph/crimson-osd.*.pid`; do perf stat -p ${pid} -o /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/seq/perf/perf_stat.${pid} & done 2026-02-24T16:18:43.229 INFO:teuthology.orchestra.run.trial084.stderr:16:18:43 - INFO - cbt - Running radosbench seq test. 2026-02-24T16:18:43.230 INFO:teuthology.orchestra.run.trial084.stderr:16:18:43 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial084.front.sepia.ceph.com /usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench 60 seq --concurrent-ios 4 --run-name `hostname -f`-0 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/seq/objecter.0.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/seq/output.0 2026-02-24T16:18:43.231 INFO:teuthology.orchestra.run.trial084.stderr:16:18:43 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial084.front.sepia.ceph.com /usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench 60 seq --concurrent-ios 4 --run-name `hostname -f`-1 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/seq/objecter.1.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/seq/output.1 2026-02-24T16:18:59.437 INFO:teuthology.orchestra.run.trial084.stderr:16:18:59 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial084.front.sepia.ceph.com pkill -SIGINT -f collectl 2026-02-24T16:18:59.610 INFO:teuthology.orchestra.run.trial084.stderr:16:18:59 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial084.front.sepia.ceph.com sudo pkill -SIGINT -f perf\ 2026-02-24T16:18:59.783 INFO:teuthology.orchestra.run.trial084.stderr:16:18:59 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial084.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/seq/perf/perf.data 2026-02-24T16:18:59.784 INFO:teuthology.orchestra.run.trial084.stderr:16:18:59 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial084.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/seq/perf/perf_stat.* 2026-02-24T16:18:59.786 INFO:teuthology.orchestra.run.trial084.stderr:16:18:59 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial084.front.sepia.ceph.com find /var/run/ceph/ceph-osd*.asok -maxdepth 1 -exec sudo /usr/bin/ceph --admin-daemon {} dump_historic_ops \; > /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/seq/historic_ops.out 2026-02-24T16:19:00.143 INFO:teuthology.orchestra.run.trial084.stderr:16:19:00 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial084.front.sepia.ceph.com sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/seq/* 2026-02-24T16:19:00.311 INFO:teuthology.orchestra.run.trial084.stderr:16:19:00 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=rpdcp -f 10 -R ssh -w ubuntu@trial084.front.sepia.ceph.com -r /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/seq/* /home/ubuntu/cephtest/archive/cbt/results/00000000/id-31cd68af/seq 2026-02-24T16:19:00.468 INFO:teuthology.orchestra.run.trial084.stderr:16:19:00 - INFO - cbt - Convert results to json format. 2026-02-24T16:19:00.481 DEBUG:teuthology.run_tasks:Unwinding manager cbt 2026-02-24T16:19:00.496 DEBUG:teuthology.orchestra.run.trial084:> rm --one-file-system -rf -- /home/ubuntu/cephtest/cbt 2026-02-24T16:19:00.523 INFO:tasks.cbt_performance:Collecting CBT performance data config 2026-02-24T16:19:00.523 INFO:tasks.cbt_performance:reading cbt preferences 2026-02-24T16:19:00.523 DEBUG:teuthology.orchestra.run.trial084:> find /home/ubuntu/cephtest -name 'json_output.*' 2026-02-24T16:19:00.580 INFO:teuthology.orchestra.run.trial084.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-31cd68af/write/json_output.0.trial084.front.sepia.ceph.com 2026-02-24T16:19:00.581 INFO:teuthology.orchestra.run.trial084.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-31cd68af/write/json_output.1.trial084.front.sepia.ceph.com 2026-02-24T16:19:00.581 INFO:teuthology.orchestra.run.trial084.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-31cd68af/seq/json_output.0.trial084.front.sepia.ceph.com 2026-02-24T16:19:00.581 INFO:teuthology.orchestra.run.trial084.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-31cd68af/seq/json_output.1.trial084.front.sepia.ceph.com 2026-02-24T16:19:00.582 DEBUG:teuthology.orchestra.run.trial084:> set -ex 2026-02-24T16:19:00.582 DEBUG:teuthology.orchestra.run.trial084:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-31cd68af/write/json_output.0.trial084.front.sepia.ceph.com of=/dev/stdout 2026-02-24T16:19:00.637 INFO:tasks.cbt_performance:reading total cpu cycles 2026-02-24T16:19:00.637 DEBUG:teuthology.orchestra.run.trial084:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-31cd68af/write -name 'perf_stat.*' 2026-02-24T16:19:00.694 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2026-02-24T16:19:00.694 INFO:tasks.cbt_performance:total cpu cycles: 0 2026-02-24T16:19:00.694 DEBUG:teuthology.orchestra.run.trial084:> set -ex 2026-02-24T16:19:00.695 DEBUG:teuthology.orchestra.run.trial084:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-31cd68af/write/json_output.1.trial084.front.sepia.ceph.com of=/dev/stdout 2026-02-24T16:19:00.750 INFO:tasks.cbt_performance:reading total cpu cycles 2026-02-24T16:19:00.751 DEBUG:teuthology.orchestra.run.trial084:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-31cd68af/write -name 'perf_stat.*' 2026-02-24T16:19:00.806 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2026-02-24T16:19:00.806 INFO:tasks.cbt_performance:total cpu cycles: 0 2026-02-24T16:19:00.807 DEBUG:teuthology.orchestra.run.trial084:> set -ex 2026-02-24T16:19:00.807 DEBUG:teuthology.orchestra.run.trial084:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-31cd68af/seq/json_output.0.trial084.front.sepia.ceph.com of=/dev/stdout 2026-02-24T16:19:00.862 INFO:tasks.cbt_performance:reading total cpu cycles 2026-02-24T16:19:00.863 DEBUG:teuthology.orchestra.run.trial084:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-31cd68af/seq -name 'perf_stat.*' 2026-02-24T16:19:00.918 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2026-02-24T16:19:00.919 INFO:tasks.cbt_performance:total cpu cycles: 0 2026-02-24T16:19:00.919 DEBUG:teuthology.orchestra.run.trial084:> set -ex 2026-02-24T16:19:00.919 DEBUG:teuthology.orchestra.run.trial084:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-31cd68af/seq/json_output.1.trial084.front.sepia.ceph.com of=/dev/stdout 2026-02-24T16:19:00.976 INFO:tasks.cbt_performance:reading total cpu cycles 2026-02-24T16:19:00.976 DEBUG:teuthology.orchestra.run.trial084:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-31cd68af/seq -name 'perf_stat.*' 2026-02-24T16:19:01.032 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2026-02-24T16:19:01.033 INFO:tasks.cbt_performance:total cpu cycles: 0 2026-02-24T16:19:01.038 WARNING:tasks.cbt_performance:Could not send performance data to http://mira118.front.sepia.ceph.com:4000/cbt_performance: HTTPConnectionPool(host='mira118.front.sepia.ceph.com', port=4000): Max retries exceeded with url: /cbt_performance (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -2] Name or service not known')) 2026-02-24T16:19:01.038 WARNING:tasks.cbt_performance:Continuing without performance data collection 2026-02-24T16:19:01.039 DEBUG:teuthology.run_tasks:Unwinding manager ssh_keys 2026-02-24T16:19:01.054 INFO:teuthology.task.ssh_keys:Cleaning up SSH keys 2026-02-24T16:19:01.054 INFO:teuthology.task.ssh_keys:cleaning up keys added for testing 2026-02-24T16:19:01.054 INFO:teuthology.task.ssh_keys: cleaning up keys for user ubuntu on trial084.front.sepia.ceph.com 2026-02-24T16:19:01.054 DEBUG:teuthology.orchestra.run.trial084:> rm -- /home/ubuntu/.ssh/id_rsa 2026-02-24T16:19:01.089 DEBUG:teuthology.orchestra.run.trial084:> rm -- /home/ubuntu/.ssh/id_rsa.pub 2026-02-24T16:19:01.145 DEBUG:teuthology.orchestra.run.trial084:> stat -c '"%a"' /home/ubuntu/.ssh/authorized_keys 2026-02-24T16:19:01.201 INFO:teuthology.orchestra.run.trial084.stdout:"600" 2026-02-24T16:19:01.202 DEBUG:teuthology.orchestra.run.trial084:> mv -- /home/ubuntu/.ssh/authorized_keys_2026-02-24_16:16:14:467589 /home/ubuntu/.ssh/authorized_keys 2026-02-24T16:19:01.259 DEBUG:teuthology.orchestra.run.trial084:> chmod 600 /home/ubuntu/.ssh/authorized_keys 2026-02-24T16:19:01.315 DEBUG:teuthology.orchestra.run.trial084:> rm /home/ubuntu/.ssh/config 2026-02-24T16:19:01.371 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2026-02-24T16:19:01.387 DEBUG:teuthology.orchestra.run.trial084:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2026-02-24T16:19:01.485 INFO:teuthology.orchestra.run.trial084.stderr:2026-02-24T16:19:01.480+0000 7f548e1fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:19:01.486 INFO:teuthology.orchestra.run.trial084.stderr:2026-02-24T16:19:01.485+0000 7f548e1fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:19:01.610 INFO:teuthology.misc:Shutting down mds daemons... 2026-02-24T16:19:01.610 INFO:teuthology.misc:Shutting down osd daemons... 2026-02-24T16:19:01.610 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2026-02-24T16:19:01.611 INFO:teuthology.orchestra.run:waiting for 300 2026-02-24T16:19:01.697 INFO:tasks.ceph.osd.0:Stopped 2026-02-24T16:19:01.697 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2026-02-24T16:19:01.698 INFO:teuthology.orchestra.run:waiting for 300 2026-02-24T16:19:01.796 INFO:tasks.ceph.osd.1:Stopped 2026-02-24T16:19:01.797 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2026-02-24T16:19:01.797 INFO:teuthology.orchestra.run:waiting for 300 2026-02-24T16:19:02.099 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2026-02-24T16:19:02.099 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2026-02-24T16:19:07.301 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2026-02-24T16:19:07.302 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2026-02-24T16:19:07.302 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2026-02-24T16:19:07.898 INFO:tasks.ceph.osd.2:Stopped 2026-02-24T16:19:07.898 INFO:teuthology.misc:Shutting down mgr daemons... 2026-02-24T16:19:07.898 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2026-02-24T16:19:07.899 INFO:teuthology.orchestra.run:waiting for 300 2026-02-24T16:19:07.934 INFO:tasks.ceph.mgr.x:Stopped 2026-02-24T16:19:07.934 INFO:teuthology.misc:Shutting down mon daemons... 2026-02-24T16:19:07.934 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2026-02-24T16:19:07.934 INFO:teuthology.orchestra.run:waiting for 300 2026-02-24T16:19:07.983 INFO:tasks.ceph.mon.a:Stopped 2026-02-24T16:19:07.984 INFO:tasks.ceph:Checking cluster log for badness... 2026-02-24T16:19:07.984 DEBUG:teuthology.orchestra.run.trial084:> sudo grep -E '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | grep -E -v '\(MDS_ALL_DOWN\)' | grep -E -v '\(MDS_UP_LESS_THAN_MAX\)' | grep -E -v '\(POOL_APP_NOT_ENABLED\)' | grep -E -v '\(PG_' | grep -E -v '\(OSD_' | grep -E -v '\(OBJECT_' | grep -E -v 'overall HEALTH' | head -n 1 2026-02-24T16:19:08.049 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@trial084.front.sepia.ceph.com 2026-02-24T16:19:08.049 DEBUG:teuthology.orchestra.run.trial084:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2026-02-24T16:19:08.313 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@trial084.front.sepia.ceph.com 2026-02-24T16:19:08.314 DEBUG:teuthology.orchestra.run.trial084:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2026-02-24T16:19:08.399 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@trial084.front.sepia.ceph.com 2026-02-24T16:19:08.400 DEBUG:teuthology.orchestra.run.trial084:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2026-02-24T16:19:08.431 INFO:tasks.ceph:Archiving mon data... 2026-02-24T16:19:08.432 DEBUG:teuthology.misc:Transferring archived files from trial084:/var/lib/ceph/mon/ceph-a to /home/teuthworker/mnt/teuthology/liucm-2026-02-24_15:28:57-crimson-rados-wip-liucm-shard-redesign-crimson-only-distro-debug-trial/68238/data/mon.a.tgz 2026-02-24T16:19:08.433 DEBUG:teuthology.orchestra.run.trial084:> mktemp 2026-02-24T16:19:08.487 INFO:teuthology.orchestra.run.trial084.stdout:/tmp/tmp.VIHMIV0NeN 2026-02-24T16:19:08.488 DEBUG:teuthology.orchestra.run.trial084:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.VIHMIV0NeN 2026-02-24T16:19:08.577 DEBUG:teuthology.orchestra.run.trial084:> sudo chmod 0666 /tmp/tmp.VIHMIV0NeN 2026-02-24T16:19:08.658 DEBUG:teuthology.orchestra.remote:trial084:/tmp/tmp.VIHMIV0NeN is 201KB 2026-02-24T16:19:08.711 DEBUG:teuthology.orchestra.run.trial084:> rm -fr /tmp/tmp.VIHMIV0NeN 2026-02-24T16:19:08.728 INFO:tasks.ceph:Cleaning ceph cluster... 2026-02-24T16:19:08.728 DEBUG:teuthology.orchestra.run.trial084:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2026-02-24T16:19:08.847 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2026-02-24T16:19:08.847 INFO:tasks.ceph:Archiving crash dumps... 2026-02-24T16:19:08.850 DEBUG:teuthology.misc:Transferring archived files from trial084:/var/lib/ceph/crash to /home/teuthworker/mnt/teuthology/liucm-2026-02-24_15:28:57-crimson-rados-wip-liucm-shard-redesign-crimson-only-distro-debug-trial/68238/remote/trial084/crash 2026-02-24T16:19:08.851 DEBUG:teuthology.orchestra.run.trial084:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2026-02-24T16:19:08.911 INFO:tasks.ceph:Compressing logs... 2026-02-24T16:19:08.911 DEBUG:teuthology.orchestra.run.trial084:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2026-02-24T16:19:08.974 INFO:teuthology.orchestra.run.trial084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2026-02-24T16:19:08.987 INFO:teuthology.orchestra.run.trial084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2026-02-24T16:19:08.987 INFO:teuthology.orchestra.run.trial084.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40984.log 2026-02-24T16:19:08.987 INFO:teuthology.orchestra.run.trial084.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2026-02-24T16:19:08.987 INFO:teuthology.orchestra.run.trial084.stderr:/var/log/ceph/ceph-client.admin.40984.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45025.log 2026-02-24T16:19:08.988 INFO:teuthology.orchestra.run.trial084.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.40984.log.gz 2026-02-24T16:19:08.988 INFO:teuthology.orchestra.run.trial084.stderr:gzip -5 --verbose/var/log/ceph/ceph-osd.0.log: -- /var/log/ceph/ceph.tmp-client.admin.38596.log 2026-02-24T16:19:08.988 INFO:teuthology.orchestra.run.trial084.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2026-02-24T16:19:08.988 INFO:teuthology.orchestra.run.trial084.stderr:gzip/var/log/ceph/ceph-client.admin.45025.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.41033.log 2026-02-24T16:19:08.988 INFO:teuthology.orchestra.run.trial084.stderr: 66.7% -- replaced with /var/log/ceph/ceph-client.admin.45025.log.gz 2026-02-24T16:19:08.988 INFO:teuthology.orchestra.run.trial084.stderr:/var/log/ceph/ceph.tmp-client.admin.38596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40657.log 2026-02-24T16:19:08.988 INFO:teuthology.orchestra.run.trial084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40214.log 2026-02-24T16:19:08.988 INFO:teuthology.orchestra.run.trial084.stderr:/var/log/ceph/ceph-client.admin.41033.log: 66.4% -- replaced with /var/log/ceph/ceph.tmp-client.admin.38596.log.gz 2026-02-24T16:19:08.988 INFO:teuthology.orchestra.run.trial084.stderr:gzip/var/log/ceph/ceph-client.admin.40657.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.41523.log 2026-02-24T16:19:08.988 INFO:teuthology.orchestra.run.trial084.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41033.log.gz 2026-02-24T16:19:08.988 INFO:teuthology.orchestra.run.trial084.stderr:/var/log/ceph/ceph-client.admin.40214.log: gzip 50.8% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.40657.log.gz -- 2026-02-24T16:19:08.988 INFO:teuthology.orchestra.run.trial084.stderr: /var/log/ceph/ceph-client.admin.44370.log 2026-02-24T16:19:08.989 INFO:teuthology.orchestra.run.trial084.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.40214.log.gz 2026-02-24T16:19:08.989 INFO:teuthology.orchestra.run.trial084.stderr:/var/log/ceph/ceph-client.admin.41523.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39983.log 2026-02-24T16:19:08.989 INFO:teuthology.orchestra.run.trial084.stderr: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.41523.log.gz 2026-02-24T16:19:08.989 INFO:teuthology.orchestra.run.trial084.stderr:/var/log/ceph/ceph-client.admin.44370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41135.log 2026-02-24T16:19:08.989 INFO:teuthology.orchestra.run.trial084.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.44370.log.gz 2026-02-24T16:19:08.989 INFO:teuthology.orchestra.run.trial084.stderr:/var/log/ceph/ceph-client.admin.39983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41572.log 2026-02-24T16:19:08.989 INFO:teuthology.orchestra.run.trial084.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39983.log.gz 2026-02-24T16:19:08.989 INFO:teuthology.orchestra.run.trial084.stderr:/var/log/ceph/ceph-client.admin.41135.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41136.log 2026-02-24T16:19:08.989 INFO:teuthology.orchestra.run.trial084.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41135.log.gz 2026-02-24T16:19:08.989 INFO:teuthology.orchestra.run.trial084.stderr:/var/log/ceph/ceph-client.admin.41572.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41134.log 2026-02-24T16:19:08.989 INFO:teuthology.orchestra.run.trial084.stderr: 50.8%/var/log/ceph/ceph-client.admin.41136.log: -- replaced with /var/log/ceph/ceph-client.admin.41572.log.gz 2026-02-24T16:19:08.989 INFO:teuthology.orchestra.run.trial084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41283.log 2026-02-24T16:19:08.990 INFO:teuthology.orchestra.run.trial084.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41136.log.gz 2026-02-24T16:19:08.990 INFO:teuthology.orchestra.run.trial084.stderr:/var/log/ceph/ceph-client.admin.41134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39888.log 2026-02-24T16:19:08.990 INFO:teuthology.orchestra.run.trial084.stderr:/var/log/ceph/ceph-client.admin.41283.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41134.log.gz 2026-02-24T16:19:08.990 INFO:teuthology.orchestra.run.trial084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40167.log 2026-02-24T16:19:08.990 INFO:teuthology.orchestra.run.trial084.stderr: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.41283.log.gz 2026-02-24T16:19:08.990 INFO:teuthology.orchestra.run.trial084.stderr:/var/log/ceph/ceph-client.admin.39888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41281.log 2026-02-24T16:19:08.990 INFO:teuthology.orchestra.run.trial084.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.39888.log.gz 2026-02-24T16:19:08.990 INFO:teuthology.orchestra.run.trial084.stderr:/var/log/ceph/ceph-client.admin.40167.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41430.log 2026-02-24T16:19:08.990 INFO:teuthology.orchestra.run.trial084.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40167.log.gz 2026-02-24T16:19:08.990 INFO:teuthology.orchestra.run.trial084.stderr:/var/log/ceph/ceph-client.admin.41281.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44311.log 2026-02-24T16:19:08.990 INFO:teuthology.orchestra.run.trial084.stderr: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.41281.log.gz 2026-02-24T16:19:08.990 INFO:teuthology.orchestra.run.trial084.stderr:/var/log/ceph/ceph-client.admin.41430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40030.log 2026-02-24T16:19:08.990 INFO:teuthology.orchestra.run.trial084.stderr:gzip 51.2% -5 -- replaced with /var/log/ceph/ceph-client.admin.41430.log.gz --verbose 2026-02-24T16:19:08.991 INFO:teuthology.orchestra.run.trial084.stderr: -- /var/log/ceph/ceph.audit.log 2026-02-24T16:19:08.991 INFO:teuthology.orchestra.run.trial084.stderr:/var/log/ceph/ceph-client.admin.44311.log: /var/log/ceph/ceph-client.admin.40030.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.44311.log.gz 2026-02-24T16:19:08.991 INFO:teuthology.orchestra.run.trial084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46391.log 2026-02-24T16:19:08.991 INFO:teuthology.orchestra.run.trial084.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40030.log.gz 2026-02-24T16:19:08.991 INFO:teuthology.orchestra.run.trial084.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40480.log 2026-02-24T16:19:08.991 INFO:teuthology.orchestra.run.trial084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2026-02-24T16:19:08.991 INFO:teuthology.orchestra.run.trial084.stderr:/var/log/ceph/ceph-client.admin.46391.log: 87.5% 50.0% -- replaced with /var/log/ceph/ceph.audit.log.gz 2026-02-24T16:19:08.991 INFO:teuthology.orchestra.run.trial084.stderr: -- replaced with /var/log/ceph/ceph-client.admin.46391.log.gz 2026-02-24T16:19:08.991 INFO:teuthology.orchestra.run.trial084.stderr:gzip -5/var/log/ceph/ceph-client.admin.40480.log: --verbose -- /var/log/ceph/ceph-client.admin.40725.log 2026-02-24T16:19:08.991 INFO:teuthology.orchestra.run.trial084.stderr:/var/log/ceph/ceph-osd.1.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40480.log.gz 2026-02-24T16:19:08.991 INFO:teuthology.orchestra.run.trial084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40261.log 2026-02-24T16:19:08.991 INFO:teuthology.orchestra.run.trial084.stderr:/var/log/ceph/ceph-client.admin.40725.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40588.log 2026-02-24T16:19:08.991 INFO:teuthology.orchestra.run.trial084.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40725.log.gz 2026-02-24T16:19:08.992 INFO:teuthology.orchestra.run.trial084.stderr:/var/log/ceph/ceph-client.admin.40261.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40935.log 2026-02-24T16:19:08.992 INFO:teuthology.orchestra.run.trial084.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40261.log.gz 2026-02-24T16:19:08.992 INFO:teuthology.orchestra.run.trial084.stderr:/var/log/ceph/ceph-client.admin.40588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40886.log 2026-02-24T16:19:08.992 INFO:teuthology.orchestra.run.trial084.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40588.log.gz 2026-02-24T16:19:08.992 INFO:teuthology.orchestra.run.trial084.stderr:/var/log/ceph/ceph-client.admin.40935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45023.log 2026-02-24T16:19:08.992 INFO:teuthology.orchestra.run.trial084.stderr:/var/log/ceph/ceph-client.admin.40886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44255.log 2026-02-24T16:19:08.992 INFO:teuthology.orchestra.run.trial084.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.40935.log.gz 2026-02-24T16:19:08.992 INFO:teuthology.orchestra.run.trial084.stderr: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.40886.log.gz 2026-02-24T16:19:08.992 INFO:teuthology.orchestra.run.trial084.stderr:gzip -5/var/log/ceph/ceph-client.admin.45023.log: --verbose -- /var/log/ceph/ceph-client.admin.41621.log 2026-02-24T16:19:08.992 INFO:teuthology.orchestra.run.trial084.stderr:/var/log/ceph/ceph-client.admin.44255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45712.log 2026-02-24T16:19:08.992 INFO:teuthology.orchestra.run.trial084.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.44255.log.gz 2026-02-24T16:19:08.992 INFO:teuthology.orchestra.run.trial084.stderr:/var/log/ceph/ceph-client.admin.41621.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2026-02-24T16:19:08.992 INFO:teuthology.orchestra.run.trial084.stderr: 66.4% -- replaced with /var/log/ceph/ceph-client.admin.45023.log.gz 2026-02-24T16:19:08.993 INFO:teuthology.orchestra.run.trial084.stderr:/var/log/ceph/ceph-client.admin.45712.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41621.log.gz 2026-02-24T16:19:08.993 INFO:teuthology.orchestra.run.trial084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40795.log 2026-02-24T16:19:08.993 INFO:teuthology.orchestra.run.trial084.stderr: 66.4% -- replaced with /var/log/ceph/ceph-client.admin.45712.log.gz 2026-02-24T16:19:08.993 INFO:teuthology.orchestra.run.trial084.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41429.log 2026-02-24T16:19:08.993 INFO:teuthology.orchestra.run.trial084.stderr:/var/log/ceph/ceph-client.admin.40795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45714.log 2026-02-24T16:19:08.993 INFO:teuthology.orchestra.run.trial084.stderr:/var/log/ceph/ceph-client.admin.41429.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40820.log 2026-02-24T16:19:08.993 INFO:teuthology.orchestra.run.trial084.stderr: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.41429.log.gz 2026-02-24T16:19:08.993 INFO:teuthology.orchestra.run.trial084.stderr:/var/log/ceph/ceph-client.admin.45714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44415.log 2026-02-24T16:19:08.993 INFO:teuthology.orchestra.run.trial084.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.40795.log.gz 2026-02-24T16:19:08.993 INFO:teuthology.orchestra.run.trial084.stderr: 66.4% -- replaced with /var/log/ceph/ceph-client.admin.45714.log.gz 2026-02-24T16:19:08.993 INFO:teuthology.orchestra.run.trial084.stderr:/var/log/ceph/ceph-client.admin.40820.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2026-02-24T16:19:08.993 INFO:teuthology.orchestra.run.trial084.stderr: 67.2% -- replaced with /var/log/ceph/ceph-client.admin.40820.log.gz 2026-02-24T16:19:08.993 INFO:teuthology.orchestra.run.trial084.stderr:gzip/var/log/ceph/ceph-client.admin.44415.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.41282.log 2026-02-24T16:19:08.994 INFO:teuthology.orchestra.run.trial084.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44469.log 2026-02-24T16:19:08.994 INFO:teuthology.orchestra.run.trial084.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44415.log.gz 2026-02-24T16:19:08.994 INFO:teuthology.orchestra.run.trial084.stderr:/var/log/ceph/ceph-client.admin.41282.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41376.log 2026-02-24T16:19:08.994 INFO:teuthology.orchestra.run.trial084.stderr: 85.3% -- replaced with /var/log/ceph/ceph.log.gz 2026-02-24T16:19:08.994 INFO:teuthology.orchestra.run.trial084.stderr: 51.2% -- replaced with /var/log/ceph/ceph-client.admin.41282.log.gz 2026-02-24T16:19:08.994 INFO:teuthology.orchestra.run.trial084.stderr:/var/log/ceph/ceph-client.admin.44469.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44469.log.gz 2026-02-24T16:19:08.994 INFO:teuthology.orchestra.run.trial084.stderr:/var/log/ceph/ceph-client.admin.41376.log: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.41376.log.gz 2026-02-24T16:19:08.994 INFO:teuthology.orchestra.run.trial084.stderr: 91.8% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2026-02-24T16:19:12.303 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2026-02-24T16:19:13.678 INFO:teuthology.orchestra.run.trial084.stderr: 92.2% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2026-02-24T16:19:13.822 INFO:teuthology.orchestra.run.trial084.stderr: 92.2% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2026-02-24T16:19:14.224 INFO:teuthology.orchestra.run.trial084.stderr: 92.2% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2026-02-24T16:19:14.227 INFO:teuthology.orchestra.run.trial084.stderr: 2026-02-24T16:19:14.227 INFO:teuthology.orchestra.run.trial084.stderr:real 0m5.260s 2026-02-24T16:19:14.228 INFO:teuthology.orchestra.run.trial084.stderr:user 0m13.864s 2026-02-24T16:19:14.228 INFO:teuthology.orchestra.run.trial084.stderr:sys 0m0.923s 2026-02-24T16:19:14.228 INFO:tasks.ceph:Archiving logs... 2026-02-24T16:19:14.229 DEBUG:teuthology.misc:Transferring archived files from trial084:/var/log/ceph to /home/teuthworker/mnt/teuthology/liucm-2026-02-24_15:28:57-crimson-rados-wip-liucm-shard-redesign-crimson-only-distro-debug-trial/68238/remote/trial084/log 2026-02-24T16:19:14.230 DEBUG:teuthology.orchestra.run.trial084:> sudo tar c -f - -C /var/log/ceph -- . 2026-02-24T16:19:15.504 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-24T16:19:15.575 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-24T16:19:15.586 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2026-02-24T16:19:15.586 DEBUG:teuthology.orchestra.run.trial084:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-24T16:19:15.614 DEBUG:teuthology.run_tasks:Unwinding manager clock 2026-02-24T16:19:15.624 INFO:teuthology.task.clock:Checking final clock skew... 2026-02-24T16:19:15.624 DEBUG:teuthology.orchestra.run.trial084:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-24T16:19:15.669 INFO:teuthology.orchestra.run.trial084.stderr:bash: line 1: ntpq: command not found 2026-02-24T16:19:15.699 INFO:teuthology.orchestra.run.trial084.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-24T16:19:15.699 INFO:teuthology.orchestra.run.trial084.stdout:=============================================================================== 2026-02-24T16:19:15.700 INFO:teuthology.orchestra.run.trial084.stdout:^* static.190.111.161.5.cli> 4 8 377 181 -60us[ -91us] +/- 11ms 2026-02-24T16:19:15.700 INFO:teuthology.orchestra.run.trial084.stdout:^- time-ord.far.net 2 8 377 246 -3303us[-3332us] +/- 55ms 2026-02-24T16:19:15.700 INFO:teuthology.orchestra.run.trial084.stdout:^- ip216-250-115-174.pbiaas> 2 8 377 253 -175us[ -204us] +/- 122ms 2026-02-24T16:19:15.700 INFO:teuthology.orchestra.run.trial084.stdout:^- ntp.shastacoe.net 2 8 377 248 -360us[ -390us] +/- 70ms 2026-02-24T16:19:15.702 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2026-02-24T16:19:15.715 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2026-02-24T16:19:15.715 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2026-02-24T16:19:15.734 DEBUG:teuthology.orchestra.run.trial084:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2026-02-24T16:19:15.801 DEBUG:teuthology.orchestra.run.trial084:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-24T16:19:15.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-24T16:19:15.826 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2026-02-24T16:19:15.837 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2026-02-24T16:19:15.851 INFO:teuthology.task.internal:Duration was 2035.370251 seconds 2026-02-24T16:19:15.852 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2026-02-24T16:19:15.867 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2026-02-24T16:19:15.868 DEBUG:teuthology.orchestra.run.trial084:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-24T16:19:15.906 INFO:teuthology.orchestra.run.trial084.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-24T16:19:16.228 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2026-02-24T16:19:16.229 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial084.front.sepia.ceph.com 2026-02-24T16:19:16.229 DEBUG:teuthology.orchestra.run.trial084:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-24T16:19:16.287 INFO:teuthology.task.internal.syslog:Gathering journactl... 2026-02-24T16:19:16.287 DEBUG:teuthology.orchestra.run.trial084:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-24T16:19:16.392 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2026-02-24T16:19:16.392 DEBUG:teuthology.orchestra.run.trial084:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-24T16:19:16.415 INFO:teuthology.orchestra.run.trial084.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-24T16:19:16.416 INFO:teuthology.orchestra.run.trial084.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-24T16:19:16.416 INFO:teuthology.orchestra.run.trial084.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-24T16:19:16.416 INFO:teuthology.orchestra.run.trial084.stderr: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-02-24T16:19:16.416 INFO:teuthology.orchestra.run.trial084.stderr:/home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-24T16:19:16.427 INFO:teuthology.orchestra.run.trial084.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: 83.1% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-24T16:19:16.429 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2026-02-24T16:19:16.446 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2026-02-24T16:19:16.447 DEBUG:teuthology.orchestra.run.trial084:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-24T16:19:16.493 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2026-02-24T16:19:16.511 DEBUG:teuthology.orchestra.run.trial084:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-24T16:19:16.557 INFO:teuthology.orchestra.run.trial084.stdout:kernel.core_pattern = core 2026-02-24T16:19:16.576 DEBUG:teuthology.orchestra.run.trial084:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-24T16:19:16.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-24T16:19:16.632 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2026-02-24T16:19:16.650 INFO:teuthology.task.internal:Transferring archived files... 2026-02-24T16:19:16.652 DEBUG:teuthology.misc:Transferring archived files from trial084:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/liucm-2026-02-24_15:28:57-crimson-rados-wip-liucm-shard-redesign-crimson-only-distro-debug-trial/68238/remote/trial084 2026-02-24T16:19:16.652 DEBUG:teuthology.orchestra.run.trial084:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-24T16:19:16.758 INFO:teuthology.task.internal:Removing archive directory... 2026-02-24T16:19:16.759 DEBUG:teuthology.orchestra.run.trial084:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-24T16:19:16.775 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2026-02-24T16:19:16.802 INFO:teuthology.task.internal:Not uploading archives. 2026-02-24T16:19:16.802 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2026-02-24T16:19:16.824 INFO:teuthology.task.internal:Tidying up after the test... 2026-02-24T16:19:16.824 DEBUG:teuthology.orchestra.run.trial084:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-24T16:19:16.840 INFO:teuthology.orchestra.run.trial084.stdout: 920834 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 24 16:19 /home/ubuntu/cephtest 2026-02-24T16:19:16.841 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2026-02-24T16:19:16.860 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2026-02-24T16:19:16.898 INFO:teuthology.run:Summary data: description: crimson-rados/perf/{0-crimson_install clusters/{crimson-fixed-1} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/rbm$/{crimson_seastore_rbm} perf_ignorelist settings/optimized workloads/radosbench_4K_seq_read} duration: 2035.3702509403229 flavor: debug owner: scheduled_liucm@soko04.front.sepia.ceph.com success: true 2026-02-24T16:19:16.899 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-24T16:19:17.018 INFO:teuthology.run:pass