2026-02-12T12:21:50.746 INFO:root:teuthology version: 0.0.1.dev29+gc433f1062 2026-02-12T12:21:50.763 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-12T12:21:50.847 INFO:teuthology.run:Config: archive_path: /home/teuthworker/mnt/teuthology/moagrawa-2026-02-10_06:35:43-crimson-rados-main-distro-debug-trial/42515 branch: main description: crimson-rados/perf/{0-crimson_install clusters/{crimson-fixed-1} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/rbm$/{crimson_seastore_rbm} perf_ignorelist settings/optimized workloads/sample_radosbench} email: null first_in_suite: false flavor: debug job_id: '42515' kernel: branch: distro kdb: 1 sha1: distro last_in_suite: false machine_type: trial name: moagrawa-2026-02-10_06:35:43-crimson-rados-main-distro-debug-trial no_nested_subset: false os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: global: auth client required: none auth cluster required: none auth service required: none auth supported: none debug asok: 0/0 debug auth: 0/0 debug bdev: 0/0 debug bluefs: 0/0 debug bluestore: 0/0 debug buffer: 0/0 debug civetweb: 0/0 debug client: 0/0 debug compressor: 0/0 debug context: 0/0 debug crush: 0/0 debug crypto: 0/0 debug dpdk: 0/0 debug eventtrace: 0/0 debug filer: 0/0 debug filestore: 0/0 debug finisher: 0/0 debug fuse: 0/0 debug heartbeatmap: 0/0 debug javaclient: 0/0 debug journal: 0/0 debug journaler: 0/0 debug kstore: 0/0 debug leveldb: 0/0 debug lockdep: 0/0 debug mds: 0/0 debug mds balancer: 0/0 debug mds locker: 0/0 debug mds log: 0/0 debug mds log expire: 0/0 debug mds migrator: 0/0 debug memdb: 0/0 debug mgr: 0/0 debug mgrc: 0/0 debug mon: 0/0 debug monc: 0/0 debug ms: 0/0 debug objclass: 0/0 debug objectcacher: 0/0 debug objecter: 0/0 debug optracker: 0/0 debug osd: 0/0 debug paxos: 0/0 debug perfcounter: 0/0 debug rados: 0/0 debug rbd: 0/0 debug rbd mirror: 0/0 debug rbd replay: 0/0 debug refs: 0/0 debug rgw: 0/0 debug rgw lifecycle: 0/0 debug rgw notification: 0/0 debug rgw sync: 0/0 debug rocksdb: 0/0 debug striper: 0/0 debug throttle: 0/0 debug timer: 0/0 debug tp: 0/0 enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 0/0 debug ms: 0/0 debug paxos: 0/0 osd pool default crimson: true osd: crimson cpu num: 4 crimson osd obc lru size: 10 debug filestore: 0/0 debug journal: 0/0 debug ms: 0/0 debug osd: 0/0 debug seastore: 20 debug seastore backref: 20 debug seastore cache: 20 debug seastore cleaner: 20 debug seastore device: 20 debug seastore epm: 20 debug seastore fixedkv tree: 20 debug seastore journal: 20 debug seastore lba: 20 debug seastore odata: 20 debug seastore omap: 20 debug seastore onode: 20 debug seastore t: 20 debug seastore tm: 20 osd objectstore: seastore osd shutdown pgref assert: true seastore cachepin size pershard: 64M seastore main device type: RANDOM_BLOCK_SSD seastore max concurrent transactions: 8 crimson_compat: true flavor: debug log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) - \(PG_ - \(OSD_ - \(OBJECT_ - overall HEALTH sha1: 5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log global: osd crush chooseleaf type: 0 osd pool default pg num: 128 osd pool default pgp num: 128 osd pool default size: 2 mon: {} install: ceph: flavor: debug sha1: 5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0 workunit: branch: seastore_io_uring env: CRIMSON_COMPAT: '1' sha1: 5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0 owner: scheduled_moagrawa@soko04.front.sepia.ceph.com priority: 1000 repo: https://github.com/ceph/ceph.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 2425 sha1: 5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0 sleep_before_teardown: 0 suite: crimson-rados suite_branch: seastore_io_uring suite_path: /home/teuthworker/src/github.com_mohit84_ceph_5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0/qa suite_relpath: qa suite_repo: https://github.com/mohit84/ceph.git suite_sha1: 5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0 targets: trial063.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHAvkLfvTmZ12iefE8b9mikT5HhlKlAe8pMet91IgN5SYYpWssChVxyc7J5+xOcjW+ECQpbWmh3P1eoeCRBpbYM= tasks: - install: extra_packages: deb: - ceph-osd-crimson - ceph-osd-crimson-dbg rpm: - ceph-osd-crimson - ceph-osd-crimson-debuginfo - pexec: all: - sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson - install: null - ceph: conf: osd: debug monc: 20 wait-for-scrub: false - ssh_keys: null - cbt: benchmarks: radosbench: concurrent_ops: 4 concurrent_procs: 2 op_size: - 4096 pool_monitoring_list: - collectl pool_profile: replicated run_monitoring_list: - collectl time: 60 write_only: true cluster: iterations: 1 osds_per_node: 3 pool_profiles: replicated: pg_size: 256 pgp_size: 256 replication: replicated user: ubuntu monitoring_profiles: perf: args: stat -p {pid} -o {perf_dir}/perf_stat.{pid} nodes: - osds perf_cmd: perf pid_glob: crimson-osd.*.pid teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: c433f1062990a0488dc29a553589bc609a460691 timestamp: 2026-02-10_06:35:43 tube: trial user: moagrawa verbose: false worker_log: /home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.2161924 2026-02-12T12:21:50.848 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_mohit84_ceph_5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0/qa; will attempt to use it 2026-02-12T12:21:50.848 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_mohit84_ceph_5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0/qa/tasks 2026-02-12T12:21:50.848 INFO:teuthology.run_tasks:Running task internal.check_packages... 2026-02-12T12:21:50.850 INFO:teuthology.task.internal:Checking packages... 2026-02-12T12:21:50.850 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'debug' and ceph hash '5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0' 2026-02-12T12:21:50.850 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2026-02-12T12:21:50.850 INFO:teuthology.packaging:ref: None 2026-02-12T12:21:50.850 INFO:teuthology.packaging:tag: None 2026-02-12T12:21:50.850 INFO:teuthology.packaging:branch: main 2026-02-12T12:21:50.850 INFO:teuthology.packaging:sha1: 5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0 2026-02-12T12:21:50.850 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&ref=main 2026-02-12T12:21:50.995 INFO:teuthology.task.internal:Found packages for ceph version 20.3.0-5273.g2bf58db4 2026-02-12T12:21:50.996 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2026-02-12T12:21:51.008 INFO:teuthology.task.internal:no buildpackages task found 2026-02-12T12:21:51.008 INFO:teuthology.run_tasks:Running task internal.save_config... 2026-02-12T12:21:51.023 INFO:teuthology.task.internal:Saving configuration 2026-02-12T12:21:51.042 INFO:teuthology.run_tasks:Running task internal.check_lock... 2026-02-12T12:21:51.055 INFO:teuthology.task.internal.check_lock:Checking locks... 2026-02-12T12:21:51.140 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial063.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/moagrawa-2026-02-10_06:35:43-crimson-rados-main-distro-debug-trial/42515', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-12 12:18:38.452062', 'locked_by': 'scheduled_moagrawa@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHAvkLfvTmZ12iefE8b9mikT5HhlKlAe8pMet91IgN5SYYpWssChVxyc7J5+xOcjW+ECQpbWmh3P1eoeCRBpbYM='} 2026-02-12T12:21:51.140 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2026-02-12T12:21:51.151 INFO:teuthology.task.internal:roles: ubuntu@trial063.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2026-02-12T12:21:51.151 INFO:teuthology.run_tasks:Running task console_log... 2026-02-12T12:21:51.239 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x70a34a51b600>, signals=[15]) 2026-02-12T12:21:51.239 INFO:teuthology.run_tasks:Running task internal.connect... 2026-02-12T12:21:51.250 INFO:teuthology.task.internal:Opening connections... 2026-02-12T12:21:51.250 DEBUG:teuthology.task.internal:connecting to ubuntu@trial063.front.sepia.ceph.com 2026-02-12T12:21:51.251 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-12T12:21:51.310 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2026-02-12T12:21:51.321 DEBUG:teuthology.orchestra.run.trial063:> uname -m 2026-02-12T12:21:51.383 INFO:teuthology.orchestra.run.trial063.stdout:x86_64 2026-02-12T12:21:51.383 DEBUG:teuthology.orchestra.run.trial063:> cat /etc/os-release 2026-02-12T12:21:51.439 INFO:teuthology.orchestra.run.trial063.stdout:NAME="CentOS Stream" 2026-02-12T12:21:51.439 INFO:teuthology.orchestra.run.trial063.stdout:VERSION="9" 2026-02-12T12:21:51.439 INFO:teuthology.orchestra.run.trial063.stdout:ID="centos" 2026-02-12T12:21:51.440 INFO:teuthology.orchestra.run.trial063.stdout:ID_LIKE="rhel fedora" 2026-02-12T12:21:51.440 INFO:teuthology.orchestra.run.trial063.stdout:VERSION_ID="9" 2026-02-12T12:21:51.440 INFO:teuthology.orchestra.run.trial063.stdout:PLATFORM_ID="platform:el9" 2026-02-12T12:21:51.440 INFO:teuthology.orchestra.run.trial063.stdout:PRETTY_NAME="CentOS Stream 9" 2026-02-12T12:21:51.440 INFO:teuthology.orchestra.run.trial063.stdout:ANSI_COLOR="0;31" 2026-02-12T12:21:51.440 INFO:teuthology.orchestra.run.trial063.stdout:LOGO="fedora-logo-icon" 2026-02-12T12:21:51.440 INFO:teuthology.orchestra.run.trial063.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2026-02-12T12:21:51.440 INFO:teuthology.orchestra.run.trial063.stdout:HOME_URL="https://centos.org/" 2026-02-12T12:21:51.440 INFO:teuthology.orchestra.run.trial063.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2026-02-12T12:21:51.440 INFO:teuthology.orchestra.run.trial063.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2026-02-12T12:21:51.440 INFO:teuthology.orchestra.run.trial063.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2026-02-12T12:21:51.441 INFO:teuthology.lock.ops:Updating trial063.front.sepia.ceph.com on lock server 2026-02-12T12:21:51.530 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2026-02-12T12:21:51.543 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2026-02-12T12:21:51.555 INFO:teuthology.task.internal:Checking for old test directory... 2026-02-12T12:21:51.555 DEBUG:teuthology.orchestra.run.trial063:> test '!' -e /home/ubuntu/cephtest 2026-02-12T12:21:51.570 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2026-02-12T12:21:51.582 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2026-02-12T12:21:51.582 DEBUG:teuthology.orchestra.run.trial063:> test -z $(ls -A /var/lib/ceph) 2026-02-12T12:21:51.628 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2026-02-12T12:21:51.736 INFO:teuthology.run_tasks:Running task kernel... 2026-02-12T12:21:51.757 INFO:teuthology.task.kernel:normalize config orig: {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'} 2026-02-12T12:21:51.757 INFO:teuthology.task.kernel:config {'mon.a': {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'}}, timeout 300 2026-02-12T12:21:51.757 DEBUG:teuthology.orchestra.run.trial063:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-12T12:21:51.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-12T12:21:51.772 DEBUG:teuthology.orchestra.run.trial063:> uname -r 2026-02-12T12:21:51.828 INFO:teuthology.orchestra.run.trial063.stdout:5.14.0-645.el9.x86_64 2026-02-12T12:21:51.828 INFO:teuthology.task.kernel:Running kernel on trial063: 5.14.0-645.el9.x86_64 2026-02-12T12:21:51.828 DEBUG:teuthology.orchestra.run.trial063:> sudo yum install -y kernel 2026-02-12T12:22:01.014 INFO:teuthology.orchestra.run.trial063.stdout:CentOS Stream 9 - BaseOS 1.0 MB/s | 8.9 MB 00:08 2026-02-12T12:22:22.639 INFO:teuthology.orchestra.run.trial063.stdout:CentOS Stream 9 - AppStream 1.3 MB/s | 27 MB 00:21 2026-02-12T12:22:26.152 INFO:teuthology.orchestra.run.trial063.stdout:CentOS Stream 9 - CRB 6.7 MB/s | 7.8 MB 00:01 2026-02-12T12:22:27.742 INFO:teuthology.orchestra.run.trial063.stdout:CentOS Stream 9 - Extras packages 23 kB/s | 20 kB 00:00 2026-02-12T12:22:28.303 INFO:teuthology.orchestra.run.trial063.stdout:Extra Packages for Enterprise Linux 42 MB/s | 20 MB 00:00 2026-02-12T12:22:31.455 INFO:teuthology.orchestra.run.trial063.stdout:lab-extras 2.7 MB/s | 50 kB 00:00 2026-02-12T12:22:32.521 INFO:teuthology.orchestra.run.trial063.stdout:Package kernel-5.14.0-645.el9.x86_64 is already installed. 2026-02-12T12:22:32.541 INFO:teuthology.orchestra.run.trial063.stdout:Dependencies resolved. 2026-02-12T12:22:32.542 INFO:teuthology.orchestra.run.trial063.stdout:================================================================================ 2026-02-12T12:22:32.542 INFO:teuthology.orchestra.run.trial063.stdout: Package Architecture Version Repository Size 2026-02-12T12:22:32.542 INFO:teuthology.orchestra.run.trial063.stdout:================================================================================ 2026-02-12T12:22:32.542 INFO:teuthology.orchestra.run.trial063.stdout:Installing: 2026-02-12T12:22:32.543 INFO:teuthology.orchestra.run.trial063.stdout: kernel x86_64 5.14.0-677.el9 baseos 768 k 2026-02-12T12:22:32.543 INFO:teuthology.orchestra.run.trial063.stdout:Installing dependencies: 2026-02-12T12:22:32.543 INFO:teuthology.orchestra.run.trial063.stdout: kernel-core x86_64 5.14.0-677.el9 baseos 17 M 2026-02-12T12:22:32.543 INFO:teuthology.orchestra.run.trial063.stdout: kernel-modules x86_64 5.14.0-677.el9 baseos 40 M 2026-02-12T12:22:32.543 INFO:teuthology.orchestra.run.trial063.stdout: kernel-modules-core x86_64 5.14.0-677.el9 baseos 31 M 2026-02-12T12:22:32.543 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:22:32.543 INFO:teuthology.orchestra.run.trial063.stdout:Transaction Summary 2026-02-12T12:22:32.543 INFO:teuthology.orchestra.run.trial063.stdout:================================================================================ 2026-02-12T12:22:32.543 INFO:teuthology.orchestra.run.trial063.stdout:Install 4 Packages 2026-02-12T12:22:32.543 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:22:32.543 INFO:teuthology.orchestra.run.trial063.stdout:Total download size: 88 M 2026-02-12T12:22:32.544 INFO:teuthology.orchestra.run.trial063.stdout:Installed size: 135 M 2026-02-12T12:22:32.544 INFO:teuthology.orchestra.run.trial063.stdout:Downloading Packages: 2026-02-12T12:22:33.021 INFO:teuthology.orchestra.run.trial063.stdout:(1/4): kernel-5.14.0-677.el9.x86_64.rpm 3.2 MB/s | 768 kB 00:00 2026-02-12T12:22:33.749 INFO:teuthology.orchestra.run.trial063.stdout:(2/4): kernel-core-5.14.0-677.el9.x86_64.rpm 18 MB/s | 17 MB 00:00 2026-02-12T12:22:34.820 INFO:teuthology.orchestra.run.trial063.stdout:(3/4): kernel-modules-core-5.14.0-677.el9.x86_6 17 MB/s | 31 MB 00:01 2026-02-12T12:22:35.280 INFO:teuthology.orchestra.run.trial063.stdout:(4/4): kernel-modules-5.14.0-677.el9.x86_64.rpm 16 MB/s | 40 MB 00:02 2026-02-12T12:22:35.280 INFO:teuthology.orchestra.run.trial063.stdout:-------------------------------------------------------------------------------- 2026-02-12T12:22:35.280 INFO:teuthology.orchestra.run.trial063.stdout:Total 32 MB/s | 88 MB 00:02 2026-02-12T12:22:35.471 INFO:teuthology.orchestra.run.trial063.stdout:Running transaction check 2026-02-12T12:22:35.679 INFO:teuthology.orchestra.run.trial063.stdout:Transaction check succeeded. 2026-02-12T12:22:35.679 INFO:teuthology.orchestra.run.trial063.stdout:Running transaction test 2026-02-12T12:22:36.023 INFO:teuthology.orchestra.run.trial063.stdout:Transaction test succeeded. 2026-02-12T12:22:36.023 INFO:teuthology.orchestra.run.trial063.stdout:Running transaction 2026-02-12T12:22:37.118 INFO:teuthology.orchestra.run.trial063.stdout: Preparing : 1/1 2026-02-12T12:22:37.405 INFO:teuthology.orchestra.run.trial063.stdout: Installing : kernel-modules-core-5.14.0-677.el9.x86_64 1/4 2026-02-12T12:22:37.424 INFO:teuthology.orchestra.run.trial063.stdout: Installing : kernel-core-5.14.0-677.el9.x86_64 2/4 2026-02-12T12:22:39.181 INFO:teuthology.orchestra.run.trial063.stdout: Running scriptlet: kernel-core-5.14.0-677.el9.x86_64 2/4 2026-02-12T12:22:39.210 INFO:teuthology.orchestra.run.trial063.stdout: Installing : kernel-modules-5.14.0-677.el9.x86_64 3/4 2026-02-12T12:22:42.231 INFO:teuthology.orchestra.run.trial063.stdout: Running scriptlet: kernel-modules-5.14.0-677.el9.x86_64 3/4 2026-02-12T12:22:42.237 INFO:teuthology.orchestra.run.trial063.stdout: Installing : kernel-5.14.0-677.el9.x86_64 4/4 2026-02-12T12:22:45.339 INFO:teuthology.orchestra.run.trial063.stdout: Running scriptlet: kernel-modules-core-5.14.0-677.el9.x86_64 4/4 2026-02-12T12:23:15.828 INFO:teuthology.orchestra.run.trial063.stdout: Running scriptlet: kernel-core-5.14.0-677.el9.x86_64 4/4 2026-02-12T12:23:15.849 INFO:teuthology.orchestra.run.trial063.stdout: Running scriptlet: kernel-modules-5.14.0-677.el9.x86_64 4/4 2026-02-12T12:23:15.913 INFO:teuthology.orchestra.run.trial063.stdout: Running scriptlet: kernel-5.14.0-677.el9.x86_64 4/4 2026-02-12T12:23:15.913 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : kernel-5.14.0-677.el9.x86_64 1/4 2026-02-12T12:23:15.913 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : kernel-core-5.14.0-677.el9.x86_64 2/4 2026-02-12T12:23:15.914 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : kernel-modules-5.14.0-677.el9.x86_64 3/4 2026-02-12T12:23:15.965 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : kernel-modules-core-5.14.0-677.el9.x86_64 4/4 2026-02-12T12:23:15.965 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:23:15.965 INFO:teuthology.orchestra.run.trial063.stdout:Installed: 2026-02-12T12:23:15.965 INFO:teuthology.orchestra.run.trial063.stdout: kernel-5.14.0-677.el9.x86_64 2026-02-12T12:23:15.966 INFO:teuthology.orchestra.run.trial063.stdout: kernel-core-5.14.0-677.el9.x86_64 2026-02-12T12:23:15.966 INFO:teuthology.orchestra.run.trial063.stdout: kernel-modules-5.14.0-677.el9.x86_64 2026-02-12T12:23:15.966 INFO:teuthology.orchestra.run.trial063.stdout: kernel-modules-core-5.14.0-677.el9.x86_64 2026-02-12T12:23:15.966 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:23:15.966 INFO:teuthology.orchestra.run.trial063.stdout:Complete! 2026-02-12T12:23:16.023 DEBUG:teuthology.orchestra.run.trial063:> rpm -q kernel | sort -rV | head -n 1 2026-02-12T12:23:16.088 INFO:teuthology.orchestra.run.trial063.stdout:kernel-5.14.0-677.el9.x86_64 2026-02-12T12:23:16.089 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-677.el9.x86_64 2026-02-12T12:23:16.089 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-645.el9.x86_64 Expected: 5.14.0-677.el9.x86_64 2026-02-12T12:23:16.090 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2026-02-12T12:23:16.090 DEBUG:teuthology.task.kernel:src is distro, skipping download 2026-02-12T12:23:16.090 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2026-02-12T12:23:16.090 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@trial063.front.sepia.ceph.com, path=None, version=distro) 2026-02-12T12:23:16.090 DEBUG:teuthology.orchestra.run.trial063:> rpm -q kernel | sort -rV | head -n 1 2026-02-12T12:23:16.155 INFO:teuthology.orchestra.run.trial063.stdout:kernel-5.14.0-677.el9.x86_64 2026-02-12T12:23:16.156 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-677.el9.x86_64 2026-02-12T12:23:16.157 DEBUG:teuthology.orchestra.run.trial063:> sudo rpm -qi grub2-tools 2026-02-12T12:23:16.225 INFO:teuthology.orchestra.run.trial063.stdout:Name : grub2-tools 2026-02-12T12:23:16.226 INFO:teuthology.orchestra.run.trial063.stdout:Epoch : 1 2026-02-12T12:23:16.226 INFO:teuthology.orchestra.run.trial063.stdout:Version : 2.06 2026-02-12T12:23:16.226 INFO:teuthology.orchestra.run.trial063.stdout:Release : 118.el9 2026-02-12T12:23:16.226 INFO:teuthology.orchestra.run.trial063.stdout:Architecture: x86_64 2026-02-12T12:23:16.226 INFO:teuthology.orchestra.run.trial063.stdout:Install Date: Thu 04 Dec 2025 11:54:21 PM UTC 2026-02-12T12:23:16.226 INFO:teuthology.orchestra.run.trial063.stdout:Group : Unspecified 2026-02-12T12:23:16.226 INFO:teuthology.orchestra.run.trial063.stdout:Size : 8393909 2026-02-12T12:23:16.226 INFO:teuthology.orchestra.run.trial063.stdout:License : GPLv3+ 2026-02-12T12:23:16.226 INFO:teuthology.orchestra.run.trial063.stdout:Signature : RSA/SHA256, Wed 05 Nov 2025 09:13:15 AM UTC, Key ID 05b555b38483c65d 2026-02-12T12:23:16.226 INFO:teuthology.orchestra.run.trial063.stdout:Source RPM : grub2-2.06-118.el9.src.rpm 2026-02-12T12:23:16.227 INFO:teuthology.orchestra.run.trial063.stdout:Build Date : Tue 04 Nov 2025 03:44:14 PM UTC 2026-02-12T12:23:16.227 INFO:teuthology.orchestra.run.trial063.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2026-02-12T12:23:16.227 INFO:teuthology.orchestra.run.trial063.stdout:Packager : builder@centos.org 2026-02-12T12:23:16.227 INFO:teuthology.orchestra.run.trial063.stdout:Vendor : CentOS 2026-02-12T12:23:16.227 INFO:teuthology.orchestra.run.trial063.stdout:URL : http://www.gnu.org/software/grub/ 2026-02-12T12:23:16.227 INFO:teuthology.orchestra.run.trial063.stdout:Summary : Support tools for GRUB. 2026-02-12T12:23:16.227 INFO:teuthology.orchestra.run.trial063.stdout:Description : 2026-02-12T12:23:16.227 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:23:16.227 INFO:teuthology.orchestra.run.trial063.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2026-02-12T12:23:16.227 INFO:teuthology.orchestra.run.trial063.stdout:customizable bootloader with modular architecture. It supports a rich 2026-02-12T12:23:16.227 INFO:teuthology.orchestra.run.trial063.stdout:variety of kernel formats, file systems, computer architectures and 2026-02-12T12:23:16.227 INFO:teuthology.orchestra.run.trial063.stdout:hardware devices. 2026-02-12T12:23:16.228 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:23:16.228 INFO:teuthology.orchestra.run.trial063.stdout:This subpackage provides tools for support of all platforms. 2026-02-12T12:23:16.229 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2026-02-12T12:23:16.229 INFO:teuthology.task.kernel:Updating grub on trial063 to boot 5.14.0-677.el9.x86_64 2026-02-12T12:23:16.229 DEBUG:teuthology.orchestra.run.trial063:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f -print -quit 2026-02-12T12:23:16.292 INFO:teuthology.orchestra.run.trial063.stdout:/boot/loader/entries/2734456005d743dca7f3dc77c35a2d0d-0-rescue.conf 2026-02-12T12:23:16.293 DEBUG:teuthology.orchestra.run.trial063:> sudo command -v grubby && echo yes || echo no 2026-02-12T12:23:16.356 INFO:teuthology.orchestra.run.trial063.stdout:/sbin/grubby 2026-02-12T12:23:16.358 INFO:teuthology.orchestra.run.trial063.stdout:yes 2026-02-12T12:23:16.358 DEBUG:teuthology.orchestra.run.trial063:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f | grep -F "5.14.0-677.el9.x86_64" | head -n 1 2026-02-12T12:23:16.422 INFO:teuthology.orchestra.run.trial063.stdout:/boot/loader/entries/b0373be789df49cdafedf423703d2566-5.14.0-677.el9.x86_64.conf 2026-02-12T12:23:16.423 DEBUG:teuthology.orchestra.run.trial063:> sudo grub2-set-default b0373be789df49cdafedf423703d2566-5.14.0-677.el9.x86_64 2026-02-12T12:23:16.495 DEBUG:teuthology.orchestra.run.trial063:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2026-02-12T12:23:16.705 INFO:teuthology.orchestra.run.trial063.stderr:Generating grub configuration file ... 2026-02-12T12:23:16.767 INFO:teuthology.orchestra.run.trial063.stderr:Found linux image: /boot/vmlinuz-5.14.0-677.el9.x86_64 2026-02-12T12:23:16.774 INFO:teuthology.orchestra.run.trial063.stderr:Found initrd image: /boot/initramfs-5.14.0-677.el9.x86_64.img 2026-02-12T12:23:17.022 INFO:teuthology.orchestra.run.trial063.stderr:Found linux image: /boot/vmlinuz-5.14.0-645.el9.x86_64 2026-02-12T12:23:17.029 INFO:teuthology.orchestra.run.trial063.stderr:Found initrd image: /boot/initramfs-5.14.0-645.el9.x86_64.img 2026-02-12T12:23:17.061 INFO:teuthology.orchestra.run.trial063.stderr:Found linux image: /boot/vmlinuz-0-rescue-b0373be789df49cdafedf423703d2566 2026-02-12T12:23:17.067 INFO:teuthology.orchestra.run.trial063.stderr:Found initrd image: /boot/initramfs-0-rescue-b0373be789df49cdafedf423703d2566.img 2026-02-12T12:23:17.090 INFO:teuthology.orchestra.run.trial063.stderr:Found linux image: /boot/vmlinuz-0-rescue-2734456005d743dca7f3dc77c35a2d0d 2026-02-12T12:23:17.097 INFO:teuthology.orchestra.run.trial063.stderr:Found initrd image: /boot/initramfs-0-rescue-2734456005d743dca7f3dc77c35a2d0d.img 2026-02-12T12:23:17.179 INFO:teuthology.orchestra.run.trial063.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2026-02-12T12:23:17.188 INFO:teuthology.orchestra.run.trial063.stderr:done 2026-02-12T12:23:17.190 DEBUG:teuthology.orchestra.run.trial063:> test -d /sys/firmware/efi 2026-02-12T12:23:17.206 DEBUG:teuthology.orchestra.run.trial063:> sudo find /boot/efi/EFI -mindepth 1 -maxdepth 1 -type d || true 2026-02-12T12:23:17.270 INFO:teuthology.orchestra.run.trial063.stdout:/boot/efi/EFI/BOOT 2026-02-12T12:23:17.270 INFO:teuthology.orchestra.run.trial063.stdout:/boot/efi/EFI/centos 2026-02-12T12:23:17.271 DEBUG:teuthology.orchestra.run.trial063:> sudo test -f /boot/efi/EFI/BOOT/grub.cfg && echo yes || echo no 2026-02-12T12:23:17.337 INFO:teuthology.orchestra.run.trial063.stdout:no 2026-02-12T12:23:17.337 DEBUG:teuthology.orchestra.run.trial063:> sudo test -f /boot/efi/EFI/centos/grub.cfg && echo yes || echo no 2026-02-12T12:23:17.402 INFO:teuthology.orchestra.run.trial063.stdout:yes 2026-02-12T12:23:17.403 DEBUG:teuthology.orchestra.run.trial063:> sudo cp -f /boot/grub2/grub.cfg /boot/efi/EFI/centos/grub.cfg 2026-02-12T12:23:17.467 DEBUG:teuthology.orchestra.run.trial063:> sudo shutdown -r now 2026-02-12T12:23:17.540 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2026-02-12T12:23:47.539 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2026-02-12T12:23:47.539 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial063.front.sepia.ceph.com' 2026-02-12T12:23:47.540 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-12T12:24:47.602 DEBUG:teuthology.orchestra.remote:timed out 2026-02-12T12:24:56.611 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial063.front.sepia.ceph.com' 2026-02-12T12:24:56.612 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-12T12:24:56.740 DEBUG:teuthology.orchestra.run.trial063:> true 2026-02-12T12:24:56.896 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial063.front.sepia.ceph.com' 2026-02-12T12:24:56.896 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-677.el9.x86_64"... 2026-02-12T12:24:56.896 DEBUG:teuthology.orchestra.run.trial063:> uname -r 2026-02-12T12:24:56.952 INFO:teuthology.orchestra.run.trial063.stdout:5.14.0-677.el9.x86_64 2026-02-12T12:24:56.953 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-677.el9.x86_64 vs 5.14.0-677.el9.x86_64 2026-02-12T12:24:56.953 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2026-02-12T12:24:56.953 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2026-02-12T12:24:57.954 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2026-02-12T12:24:57.954 DEBUG:teuthology.orchestra.run.trial063:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-12T12:24:57.983 INFO:teuthology.orchestra.run.trial063.stdout:ttyS1 2026-02-12T12:24:58.006 DEBUG:teuthology.parallel:result is None 2026-02-12T12:24:58.006 INFO:teuthology.run_tasks:Running task internal.base... 2026-02-12T12:24:58.017 INFO:teuthology.task.internal:Creating test directory... 2026-02-12T12:24:58.017 DEBUG:teuthology.orchestra.run.trial063:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-12T12:24:58.064 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2026-02-12T12:24:58.077 INFO:teuthology.run_tasks:Running task internal.archive... 2026-02-12T12:24:58.090 INFO:teuthology.task.internal:Creating archive directory... 2026-02-12T12:24:58.091 DEBUG:teuthology.orchestra.run.trial063:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-12T12:24:58.127 INFO:teuthology.run_tasks:Running task internal.coredump... 2026-02-12T12:24:58.139 INFO:teuthology.task.internal:Enabling coredump saving... 2026-02-12T12:24:58.139 DEBUG:teuthology.orchestra.run.trial063:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-12T12:24:58.186 INFO:teuthology.orchestra.run.trial063.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-12T12:24:58.194 INFO:teuthology.orchestra.run.trial063.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-12T12:24:58.195 INFO:teuthology.run_tasks:Running task internal.sudo... 2026-02-12T12:24:58.208 INFO:teuthology.task.internal:Configuring sudo... 2026-02-12T12:24:58.208 DEBUG:teuthology.orchestra.run.trial063:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-12T12:24:58.259 INFO:teuthology.run_tasks:Running task internal.syslog... 2026-02-12T12:24:58.276 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2026-02-12T12:24:58.276 DEBUG:teuthology.orchestra.run.trial063:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-12T12:24:58.314 DEBUG:teuthology.orchestra.run.trial063:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-12T12:24:58.476 DEBUG:teuthology.orchestra.run.trial063:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-12T12:24:58.501 DEBUG:teuthology.orchestra.run.trial063:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-12T12:24:58.649 DEBUG:teuthology.orchestra.run.trial063:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-12T12:24:58.674 DEBUG:teuthology.orchestra.run.trial063:> set -ex 2026-02-12T12:24:58.674 DEBUG:teuthology.orchestra.run.trial063:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-12T12:24:58.736 DEBUG:teuthology.orchestra.run.trial063:> sudo service rsyslog restart 2026-02-12T12:24:58.804 INFO:teuthology.orchestra.run.trial063.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-12T12:24:59.193 INFO:teuthology.run_tasks:Running task internal.timer... 2026-02-12T12:24:59.211 INFO:teuthology.task.internal:Starting timer... 2026-02-12T12:24:59.212 INFO:teuthology.run_tasks:Running task pcp... 2026-02-12T12:24:59.235 INFO:teuthology.run_tasks:Running task selinux... 2026-02-12T12:24:59.254 DEBUG:teuthology.orchestra.run.trial063:> sudo service auditd rotate 2026-02-12T12:24:59.284 INFO:teuthology.orchestra.run.trial063.stdout:Rotating logs: 2026-02-12T12:24:59.286 DEBUG:teuthology.task.selinux:Getting current SELinux state 2026-02-12T12:24:59.286 DEBUG:teuthology.orchestra.run.trial063:> /usr/sbin/getenforce 2026-02-12T12:24:59.342 INFO:teuthology.orchestra.run.trial063.stdout:Permissive 2026-02-12T12:24:59.342 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@trial063.front.sepia.ceph.com': 'permissive'} 2026-02-12T12:24:59.342 DEBUG:teuthology.orchestra.run.trial063:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-12T12:24:59.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-12T12:24:59.407 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2026-02-12T12:24:59.407 DEBUG:teuthology.orchestra.run.trial063:> sudo /usr/sbin/setenforce permissive 2026-02-12T12:24:59.470 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2026-02-12T12:24:59.486 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2026-02-12T12:24:59.499 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2026-02-12T12:24:59.501 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit trial063.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2026-02-12T12:26:04.605 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@trial063.front.sepia.ceph.com')] 2026-02-12T12:26:04.607 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial063.front.sepia.ceph.com' 2026-02-12T12:26:04.609 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-12T12:26:04.713 DEBUG:teuthology.orchestra.run.trial063:> true 2026-02-12T12:26:04.789 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial063.front.sepia.ceph.com' 2026-02-12T12:26:04.790 INFO:teuthology.run_tasks:Running task clock... 2026-02-12T12:26:04.804 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2026-02-12T12:26:04.805 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-12T12:26:04.805 DEBUG:teuthology.orchestra.run.trial063:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-12T12:26:04.857 INFO:teuthology.orchestra.run.trial063.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2026-02-12T12:26:04.868 INFO:teuthology.orchestra.run.trial063.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2026-02-12T12:26:04.888 INFO:teuthology.orchestra.run.trial063.stderr:sudo: ntpd: command not found 2026-02-12T12:26:04.898 INFO:teuthology.orchestra.run.trial063.stdout:506 Cannot talk to daemon 2026-02-12T12:26:04.910 INFO:teuthology.orchestra.run.trial063.stderr:Failed to start ntp.service: Unit ntp.service not found. 2026-02-12T12:26:04.921 INFO:teuthology.orchestra.run.trial063.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2026-02-12T12:26:04.965 INFO:teuthology.orchestra.run.trial063.stderr:bash: line 1: ntpq: command not found 2026-02-12T12:26:04.984 INFO:teuthology.orchestra.run.trial063.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-12T12:26:04.984 INFO:teuthology.orchestra.run.trial063.stdout:=============================================================================== 2026-02-12T12:26:04.984 INFO:teuthology.orchestra.run.trial063.stdout:^? ntp11.kernfusion.at 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-12T12:26:04.984 INFO:teuthology.orchestra.run.trial063.stdout:^? 216-144-228-179-host.col> 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-12T12:26:04.984 INFO:teuthology.orchestra.run.trial063.stdout:^? 172-104-28-175.ip.linode> 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-12T12:26:04.984 INFO:teuthology.orchestra.run.trial063.stdout:^? 96-19-94-82.cpe.sparklig> 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-12T12:26:04.986 INFO:teuthology.run_tasks:Running task install... 2026-02-12T12:26:04.999 DEBUG:teuthology.task.install:project ceph 2026-02-12T12:26:05.000 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'debug', 'sha1': '5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0'}} 2026-02-12T12:26:05.000 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'flavor': 'debug', 'sha1': '5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0'} 2026-02-12T12:26:05.000 INFO:teuthology.task.install:Using flavor: debug 2026-02-12T12:26:05.010 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-osd', 'ceph-osd-classic', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-osd', 'ceph-osd-classic', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-12T12:26:05.010 INFO:teuthology.task.install:extra packages: {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']} 2026-02-12T12:26:05.010 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'debug', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0', 'tag': None, 'wait_for_package': False} 2026-02-12T12:26:05.011 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0 2026-02-12T12:26:05.171 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-moagrawa-seastore-io-uring-again-crimson-only/5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0/centos/9/flavors/debug/ 2026-02-12T12:26:05.171 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5145.g5fdb7d8f 2026-02-12T12:26:05.551 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-moagrawa-seastore-io-uring-again-crimson-only/5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0/centos/9/flavors/debug/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-moagrawa-seastore-io-uring-again-crimson-only/5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0/centos/9/flavors/debug/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-moagrawa-seastore-io-uring-again-crimson-only/5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0/centos/9/flavors/debug/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-12T12:26:05.551 DEBUG:teuthology.orchestra.run.trial063:> set -ex 2026-02-12T12:26:05.551 DEBUG:teuthology.orchestra.run.trial063:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-12T12:26:05.577 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-osd-crimson, ceph-osd-crimson-debuginfo on remote rpm x86_64 2026-02-12T12:26:05.578 DEBUG:teuthology.orchestra.run.trial063:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-12T12:26:05.643 DEBUG:teuthology.orchestra.run.trial063:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-12T12:26:05.714 DEBUG:teuthology.orchestra.run.trial063:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-12T12:26:05.778 INFO:teuthology.orchestra.run.trial063.stdout:check_obsoletes = 1 2026-02-12T12:26:05.779 DEBUG:teuthology.orchestra.run.trial063:> sudo yum clean all 2026-02-12T12:26:05.928 INFO:teuthology.orchestra.run.trial063.stdout:41 files removed 2026-02-12T12:26:05.942 DEBUG:teuthology.orchestra.run.trial063:> sudo yum -y install ceph-radosgw 2026-02-12T12:26:06.782 INFO:teuthology.orchestra.run.trial063.stdout:ceph packages for x86_64 135 kB/s | 94 kB 00:00 2026-02-12T12:26:07.455 INFO:teuthology.orchestra.run.trial063.stdout:ceph noarch packages 38 kB/s | 24 kB 00:00 2026-02-12T12:26:08.044 INFO:teuthology.orchestra.run.trial063.stdout:ceph source packages 3.7 kB/s | 2.1 kB 00:00 2026-02-12T12:26:16.342 INFO:teuthology.orchestra.run.trial063.stdout:CentOS Stream 9 - BaseOS 1.1 MB/s | 8.9 MB 00:08 2026-02-12T12:26:17.286 INFO:teuthology.orchestra.run.trial063.stdout:CentOS Stream 9 - AppStream 67 MB/s | 27 MB 00:00 2026-02-12T12:26:21.358 INFO:teuthology.orchestra.run.trial063.stdout:CentOS Stream 9 - CRB 4.7 MB/s | 7.8 MB 00:01 2026-02-12T12:26:22.476 INFO:teuthology.orchestra.run.trial063.stdout:CentOS Stream 9 - Extras packages 51 kB/s | 20 kB 00:00 2026-02-12T12:26:23.125 INFO:teuthology.orchestra.run.trial063.stdout:Extra Packages for Enterprise Linux 35 MB/s | 20 MB 00:00 2026-02-12T12:26:26.373 INFO:teuthology.orchestra.run.trial063.stdout:lab-extras 2.3 MB/s | 50 kB 00:00 2026-02-12T12:26:27.668 INFO:teuthology.orchestra.run.trial063.stdout:Dependencies resolved. 2026-02-12T12:26:27.669 INFO:teuthology.orchestra.run.trial063.stdout:================================================================================ 2026-02-12T12:26:27.669 INFO:teuthology.orchestra.run.trial063.stdout: Package Arch Version Repository Size 2026-02-12T12:26:27.670 INFO:teuthology.orchestra.run.trial063.stdout:================================================================================ 2026-02-12T12:26:27.670 INFO:teuthology.orchestra.run.trial063.stdout:Installing: 2026-02-12T12:26:27.670 INFO:teuthology.orchestra.run.trial063.stdout: ceph-radosgw x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 11 M 2026-02-12T12:26:27.670 INFO:teuthology.orchestra.run.trial063.stdout:Upgrading: 2026-02-12T12:26:27.670 INFO:teuthology.orchestra.run.trial063.stdout: librados2 x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 3.3 M 2026-02-12T12:26:27.670 INFO:teuthology.orchestra.run.trial063.stdout: librbd1 x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 2.7 M 2026-02-12T12:26:27.670 INFO:teuthology.orchestra.run.trial063.stdout:Installing dependencies: 2026-02-12T12:26:27.670 INFO:teuthology.orchestra.run.trial063.stdout: boost-program-options x86_64 1.75.0-13.el9 appstream 104 k 2026-02-12T12:26:27.670 INFO:teuthology.orchestra.run.trial063.stdout: ceph-base x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 5.1 M 2026-02-12T12:26:27.670 INFO:teuthology.orchestra.run.trial063.stdout: ceph-common x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 21 M 2026-02-12T12:26:27.670 INFO:teuthology.orchestra.run.trial063.stdout: ceph-selinux x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 25 k 2026-02-12T12:26:27.671 INFO:teuthology.orchestra.run.trial063.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2026-02-12T12:26:27.671 INFO:teuthology.orchestra.run.trial063.stdout: libarrow x86_64 9.0.0-15.el9 epel 4.4 M 2026-02-12T12:26:27.671 INFO:teuthology.orchestra.run.trial063.stdout: libarrow-doc noarch 9.0.0-15.el9 epel 25 k 2026-02-12T12:26:27.671 INFO:teuthology.orchestra.run.trial063.stdout: libcephfs2 x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 860 k 2026-02-12T12:26:27.671 INFO:teuthology.orchestra.run.trial063.stdout: libnbd x86_64 1.20.3-4.el9 appstream 164 k 2026-02-12T12:26:27.671 INFO:teuthology.orchestra.run.trial063.stdout: liboath x86_64 2.6.12-1.el9 epel 49 k 2026-02-12T12:26:27.671 INFO:teuthology.orchestra.run.trial063.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2026-02-12T12:26:27.671 INFO:teuthology.orchestra.run.trial063.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2026-02-12T12:26:27.671 INFO:teuthology.orchestra.run.trial063.stdout: libradosstriper1 x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 263 k 2026-02-12T12:26:27.671 INFO:teuthology.orchestra.run.trial063.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2026-02-12T12:26:27.671 INFO:teuthology.orchestra.run.trial063.stdout: librgw2 x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 5.6 M 2026-02-12T12:26:27.671 INFO:teuthology.orchestra.run.trial063.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2026-02-12T12:26:27.672 INFO:teuthology.orchestra.run.trial063.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2026-02-12T12:26:27.672 INFO:teuthology.orchestra.run.trial063.stdout: mailcap noarch 2.1.49-5.el9 baseos 33 k 2026-02-12T12:26:27.672 INFO:teuthology.orchestra.run.trial063.stdout: parquet-libs x86_64 9.0.0-15.el9 epel 838 k 2026-02-12T12:26:27.672 INFO:teuthology.orchestra.run.trial063.stdout: pciutils x86_64 3.7.0-7.el9 baseos 93 k 2026-02-12T12:26:27.672 INFO:teuthology.orchestra.run.trial063.stdout: python3-ceph-argparse x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 45 k 2026-02-12T12:26:27.672 INFO:teuthology.orchestra.run.trial063.stdout: python3-ceph-common x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 184 k 2026-02-12T12:26:27.672 INFO:teuthology.orchestra.run.trial063.stdout: python3-cephfs x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 203 k 2026-02-12T12:26:27.672 INFO:teuthology.orchestra.run.trial063.stdout: python3-rados x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 324 k 2026-02-12T12:26:27.672 INFO:teuthology.orchestra.run.trial063.stdout: python3-rbd x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 305 k 2026-02-12T12:26:27.672 INFO:teuthology.orchestra.run.trial063.stdout: python3-rgw x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 100 k 2026-02-12T12:26:27.672 INFO:teuthology.orchestra.run.trial063.stdout: qatlib x86_64 25.08.0-2.el9 appstream 240 k 2026-02-12T12:26:27.672 INFO:teuthology.orchestra.run.trial063.stdout: qatzip-libs x86_64 1.3.1-1.el9 appstream 66 k 2026-02-12T12:26:27.672 INFO:teuthology.orchestra.run.trial063.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2026-02-12T12:26:27.672 INFO:teuthology.orchestra.run.trial063.stdout: thrift x86_64 0.15.0-4.el9 epel 1.6 M 2026-02-12T12:26:27.673 INFO:teuthology.orchestra.run.trial063.stdout:Installing weak dependencies: 2026-02-12T12:26:27.673 INFO:teuthology.orchestra.run.trial063.stdout: libcephfs-daemon x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 35 k 2026-02-12T12:26:27.673 INFO:teuthology.orchestra.run.trial063.stdout: libcephfs-proxy2 x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 24 k 2026-02-12T12:26:27.673 INFO:teuthology.orchestra.run.trial063.stdout: nvme-cli x86_64 2.13-1.el9 baseos 1.1 M 2026-02-12T12:26:27.673 INFO:teuthology.orchestra.run.trial063.stdout: qatlib-service x86_64 25.08.0-2.el9 appstream 37 k 2026-02-12T12:26:27.673 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:26:27.673 INFO:teuthology.orchestra.run.trial063.stdout:Transaction Summary 2026-02-12T12:26:27.673 INFO:teuthology.orchestra.run.trial063.stdout:================================================================================ 2026-02-12T12:26:27.673 INFO:teuthology.orchestra.run.trial063.stdout:Install 35 Packages 2026-02-12T12:26:27.673 INFO:teuthology.orchestra.run.trial063.stdout:Upgrade 2 Packages 2026-02-12T12:26:27.673 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:26:27.674 INFO:teuthology.orchestra.run.trial063.stdout:Total download size: 61 M 2026-02-12T12:26:27.674 INFO:teuthology.orchestra.run.trial063.stdout:Downloading Packages: 2026-02-12T12:26:37.649 INFO:teuthology.orchestra.run.trial063.stdout:(1/37): ceph-base-20.3.0-5145.g5fdb7d8f.el9.x86 534 kB/s | 5.1 MB 00:09 2026-02-12T12:26:37.713 INFO:teuthology.orchestra.run.trial063.stdout:(2/37): ceph-selinux-20.3.0-5145.g5fdb7d8f.el9. 390 kB/s | 25 kB 00:00 2026-02-12T12:26:37.779 INFO:teuthology.orchestra.run.trial063.stdout:(3/37): libcephfs-daemon-20.3.0-5145.g5fdb7d8f. 533 kB/s | 35 kB 00:00 2026-02-12T12:26:37.848 INFO:teuthology.orchestra.run.trial063.stdout:(4/37): libcephfs-proxy2-20.3.0-5145.g5fdb7d8f. 353 kB/s | 24 kB 00:00 2026-02-12T12:26:40.276 INFO:teuthology.orchestra.run.trial063.stdout:(5/37): libcephfs2-20.3.0-5145.g5fdb7d8f.el9.x8 354 kB/s | 860 kB 00:02 2026-02-12T12:26:40.899 INFO:teuthology.orchestra.run.trial063.stdout:(6/37): libradosstriper1-20.3.0-5145.g5fdb7d8f. 422 kB/s | 263 kB 00:00 2026-02-12T12:26:47.046 INFO:teuthology.orchestra.run.trial063.stdout:(7/37): ceph-radosgw-20.3.0-5145.g5fdb7d8f.el9. 597 kB/s | 11 MB 00:19 2026-02-12T12:26:47.183 INFO:teuthology.orchestra.run.trial063.stdout:(8/37): python3-ceph-argparse-20.3.0-5145.g5fdb 331 kB/s | 45 kB 00:00 2026-02-12T12:26:47.631 INFO:teuthology.orchestra.run.trial063.stdout:(9/37): python3-ceph-common-20.3.0-5145.g5fdb7d 411 kB/s | 184 kB 00:00 2026-02-12T12:26:48.044 INFO:teuthology.orchestra.run.trial063.stdout:(10/37): python3-cephfs-20.3.0-5145.g5fdb7d8f.e 493 kB/s | 203 kB 00:00 2026-02-12T12:26:48.662 INFO:teuthology.orchestra.run.trial063.stdout:(11/37): python3-rados-20.3.0-5145.g5fdb7d8f.el 524 kB/s | 324 kB 00:00 2026-02-12T12:26:49.286 INFO:teuthology.orchestra.run.trial063.stdout:(12/37): python3-rbd-20.3.0-5145.g5fdb7d8f.el9. 488 kB/s | 305 kB 00:00 2026-02-12T12:26:49.492 INFO:teuthology.orchestra.run.trial063.stdout:(13/37): python3-rgw-20.3.0-5145.g5fdb7d8f.el9. 487 kB/s | 100 kB 00:00 2026-02-12T12:26:49.901 INFO:teuthology.orchestra.run.trial063.stdout:(14/37): mailcap-2.1.49-5.el9.noarch.rpm 81 kB/s | 33 kB 00:00 2026-02-12T12:26:51.180 INFO:teuthology.orchestra.run.trial063.stdout:(15/37): nvme-cli-2.13-1.el9.x86_64.rpm 878 kB/s | 1.1 MB 00:01 2026-02-12T12:26:51.317 INFO:teuthology.orchestra.run.trial063.stdout:(16/37): librgw2-20.3.0-5145.g5fdb7d8f.el9.x86_ 553 kB/s | 5.6 MB 00:10 2026-02-12T12:26:51.335 INFO:teuthology.orchestra.run.trial063.stdout:(17/37): pciutils-3.7.0-7.el9.x86_64.rpm 603 kB/s | 93 kB 00:00 2026-02-12T12:26:51.364 INFO:teuthology.orchestra.run.trial063.stdout:(18/37): boost-program-options-1.75.0-13.el9.x8 2.2 MB/s | 104 kB 00:00 2026-02-12T12:26:51.375 INFO:teuthology.orchestra.run.trial063.stdout:(19/37): libnbd-1.20.3-4.el9.x86_64.rpm 4.0 MB/s | 164 kB 00:00 2026-02-12T12:26:51.381 INFO:teuthology.orchestra.run.trial063.stdout:(20/37): libpmemobj-1.12.1-1.el9.x86_64.rpm 9.3 MB/s | 160 kB 00:00 2026-02-12T12:26:51.384 INFO:teuthology.orchestra.run.trial063.stdout:(21/37): librabbitmq-0.11.0-7.el9.x86_64.rpm 4.9 MB/s | 45 kB 00:00 2026-02-12T12:26:51.402 INFO:teuthology.orchestra.run.trial063.stdout:(22/37): lttng-ust-2.12.0-6.el9.x86_64.rpm 16 MB/s | 292 kB 00:00 2026-02-12T12:26:51.406 INFO:teuthology.orchestra.run.trial063.stdout:(23/37): librdkafka-1.6.1-102.el9.x86_64.rpm 26 MB/s | 662 kB 00:00 2026-02-12T12:26:51.413 INFO:teuthology.orchestra.run.trial063.stdout:(24/37): qatlib-25.08.0-2.el9.x86_64.rpm 22 MB/s | 240 kB 00:00 2026-02-12T12:26:51.414 INFO:teuthology.orchestra.run.trial063.stdout:(25/37): qatlib-service-25.08.0-2.el9.x86_64.rp 4.2 MB/s | 37 kB 00:00 2026-02-12T12:26:51.422 INFO:teuthology.orchestra.run.trial063.stdout:(26/37): qatzip-libs-1.3.1-1.el9.x86_64.rpm 7.0 MB/s | 66 kB 00:00 2026-02-12T12:26:51.661 INFO:teuthology.orchestra.run.trial063.stdout:(27/37): gperftools-libs-2.9.1-3.el9.x86_64.rpm 1.2 MB/s | 308 kB 00:00 2026-02-12T12:26:51.698 INFO:teuthology.orchestra.run.trial063.stdout:(28/37): libarrow-doc-9.0.0-15.el9.noarch.rpm 677 kB/s | 25 kB 00:00 2026-02-12T12:26:51.736 INFO:teuthology.orchestra.run.trial063.stdout:(29/37): liboath-2.6.12-1.el9.x86_64.rpm 1.2 MB/s | 49 kB 00:00 2026-02-12T12:26:51.776 INFO:teuthology.orchestra.run.trial063.stdout:(30/37): libunwind-1.6.2-1.el9.x86_64.rpm 1.7 MB/s | 67 kB 00:00 2026-02-12T12:26:51.805 INFO:teuthology.orchestra.run.trial063.stdout:(31/37): libarrow-9.0.0-15.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2026-02-12T12:26:51.844 INFO:teuthology.orchestra.run.trial063.stdout:(32/37): re2-20211101-20.el9.x86_64.rpm 4.9 MB/s | 191 kB 00:00 2026-02-12T12:26:51.876 INFO:teuthology.orchestra.run.trial063.stdout:(33/37): parquet-libs-9.0.0-15.el9.x86_64.rpm 8.2 MB/s | 838 kB 00:00 2026-02-12T12:26:51.890 INFO:teuthology.orchestra.run.trial063.stdout:(34/37): thrift-0.15.0-4.el9.x86_64.rpm 34 MB/s | 1.6 MB 00:00 2026-02-12T12:26:57.671 INFO:teuthology.orchestra.run.trial063.stdout:(35/37): librbd1-20.3.0-5145.g5fdb7d8f.el9.x86_ 476 kB/s | 2.7 MB 00:05 2026-02-12T12:26:57.939 INFO:teuthology.orchestra.run.trial063.stdout:(36/37): ceph-common-20.3.0-5145.g5fdb7d8f.el9. 708 kB/s | 21 MB 00:30 2026-02-12T12:26:58.165 INFO:teuthology.orchestra.run.trial063.stdout:(37/37): librados2-20.3.0-5145.g5fdb7d8f.el9.x8 535 kB/s | 3.3 MB 00:06 2026-02-12T12:26:58.169 INFO:teuthology.orchestra.run.trial063.stdout:-------------------------------------------------------------------------------- 2026-02-12T12:26:58.169 INFO:teuthology.orchestra.run.trial063.stdout:Total 2.0 MB/s | 61 MB 00:30 2026-02-12T12:26:58.264 INFO:teuthology.orchestra.run.trial063.stdout:Running transaction check 2026-02-12T12:26:58.281 INFO:teuthology.orchestra.run.trial063.stdout:Transaction check succeeded. 2026-02-12T12:26:58.282 INFO:teuthology.orchestra.run.trial063.stdout:Running transaction test 2026-02-12T12:26:58.528 INFO:teuthology.orchestra.run.trial063.stdout:Transaction test succeeded. 2026-02-12T12:26:58.528 INFO:teuthology.orchestra.run.trial063.stdout:Running transaction 2026-02-12T12:26:58.817 INFO:teuthology.orchestra.run.trial063.stdout: Preparing : 1/1 2026-02-12T12:26:58.959 INFO:teuthology.orchestra.run.trial063.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/39 2026-02-12T12:26:58.964 INFO:teuthology.orchestra.run.trial063.stdout: Upgrading : librados2-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2/39 2026-02-12T12:26:59.011 INFO:teuthology.orchestra.run.trial063.stdout: Running scriptlet: librados2-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2/39 2026-02-12T12:26:59.012 INFO:teuthology.orchestra.run.trial063.stdout: Installing : libcephfs2-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 3/39 2026-02-12T12:26:59.024 INFO:teuthology.orchestra.run.trial063.stdout: Running scriptlet: libcephfs2-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 3/39 2026-02-12T12:26:59.027 INFO:teuthology.orchestra.run.trial063.stdout: Installing : libcephfs-daemon-2:20.3.0-5145.g5fdb7d8f.el9.x86_6 4/39 2026-02-12T12:26:59.027 INFO:teuthology.orchestra.run.trial063.stdout: Installing : libcephfs-proxy2-2:20.3.0-5145.g5fdb7d8f.el9.x86_6 5/39 2026-02-12T12:26:59.053 INFO:teuthology.orchestra.run.trial063.stdout: Running scriptlet: libcephfs-proxy2-2:20.3.0-5145.g5fdb7d8f.el9.x86_6 5/39 2026-02-12T12:26:59.055 INFO:teuthology.orchestra.run.trial063.stdout: Installing : python3-rados-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 6/39 2026-02-12T12:26:59.063 INFO:teuthology.orchestra.run.trial063.stdout: Installing : liboath-2.6.12-1.el9.x86_64 7/39 2026-02-12T12:26:59.065 INFO:teuthology.orchestra.run.trial063.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/39 2026-02-12T12:26:59.068 INFO:teuthology.orchestra.run.trial063.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/39 2026-02-12T12:26:59.072 INFO:teuthology.orchestra.run.trial063.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/39 2026-02-12T12:26:59.083 INFO:teuthology.orchestra.run.trial063.stdout: Installing : python3-ceph-argparse-2:20.3.0-5145.g5fdb7d8f.el9. 11/39 2026-02-12T12:26:59.098 INFO:teuthology.orchestra.run.trial063.stdout: Installing : python3-cephfs-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 12/39 2026-02-12T12:26:59.099 INFO:teuthology.orchestra.run.trial063.stdout: Installing : libradosstriper1-2:20.3.0-5145.g5fdb7d8f.el9.x86_6 13/39 2026-02-12T12:26:59.119 INFO:teuthology.orchestra.run.trial063.stdout: Running scriptlet: libradosstriper1-2:20.3.0-5145.g5fdb7d8f.el9.x86_6 13/39 2026-02-12T12:26:59.123 INFO:teuthology.orchestra.run.trial063.stdout: Installing : thrift-0.15.0-4.el9.x86_64 14/39 2026-02-12T12:26:59.152 INFO:teuthology.orchestra.run.trial063.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/39 2026-02-12T12:26:59.156 INFO:teuthology.orchestra.run.trial063.stdout: Installing : libarrow-9.0.0-15.el9.x86_64 16/39 2026-02-12T12:26:59.162 INFO:teuthology.orchestra.run.trial063.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/39 2026-02-12T12:26:59.164 INFO:teuthology.orchestra.run.trial063.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/39 2026-02-12T12:26:59.168 INFO:teuthology.orchestra.run.trial063.stdout: Installing : libarrow-doc-9.0.0-15.el9.noarch 19/39 2026-02-12T12:26:59.282 INFO:teuthology.orchestra.run.trial063.stdout: Installing : libnbd-1.20.3-4.el9.x86_64 20/39 2026-02-12T12:26:59.283 INFO:teuthology.orchestra.run.trial063.stdout: Upgrading : librbd1-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 21/39 2026-02-12T12:26:59.309 INFO:teuthology.orchestra.run.trial063.stdout: Running scriptlet: librbd1-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 21/39 2026-02-12T12:26:59.311 INFO:teuthology.orchestra.run.trial063.stdout: Installing : python3-rbd-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 22/39 2026-02-12T12:26:59.319 INFO:teuthology.orchestra.run.trial063.stdout: Installing : boost-program-options-1.75.0-13.el9.x86_64 23/39 2026-02-12T12:26:59.567 INFO:teuthology.orchestra.run.trial063.stdout: Installing : parquet-libs-9.0.0-15.el9.x86_64 24/39 2026-02-12T12:26:59.568 INFO:teuthology.orchestra.run.trial063.stdout: Installing : librgw2-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 25/39 2026-02-12T12:26:59.584 INFO:teuthology.orchestra.run.trial063.stdout: Running scriptlet: librgw2-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 25/39 2026-02-12T12:26:59.587 INFO:teuthology.orchestra.run.trial063.stdout: Installing : python3-rgw-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 26/39 2026-02-12T12:26:59.592 INFO:teuthology.orchestra.run.trial063.stdout: Installing : pciutils-3.7.0-7.el9.x86_64 27/39 2026-02-12T12:26:59.594 INFO:teuthology.orchestra.run.trial063.stdout: Installing : qatlib-25.08.0-2.el9.x86_64 28/39 2026-02-12T12:26:59.595 INFO:teuthology.orchestra.run.trial063.stdout: Installing : qatlib-service-25.08.0-2.el9.x86_64 29/39 2026-02-12T12:26:59.613 INFO:teuthology.orchestra.run.trial063.stdout: Running scriptlet: qatlib-service-25.08.0-2.el9.x86_64 29/39 2026-02-12T12:26:59.725 INFO:teuthology.orchestra.run.trial063.stdout: Installing : qatzip-libs-1.3.1-1.el9.x86_64 30/39 2026-02-12T12:26:59.727 INFO:teuthology.orchestra.run.trial063.stdout: Installing : nvme-cli-2.13-1.el9.x86_64 31/39 2026-02-12T12:27:00.000 INFO:teuthology.orchestra.run.trial063.stdout: Running scriptlet: nvme-cli-2.13-1.el9.x86_64 31/39 2026-02-12T12:27:00.000 INFO:teuthology.orchestra.run.trial063.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service → /usr/lib/systemd/system/nvmefc-boot-connections.service. 2026-02-12T12:27:00.000 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:27:00.043 INFO:teuthology.orchestra.run.trial063.stdout: Installing : mailcap-2.1.49-5.el9.noarch 32/39 2026-02-12T12:27:00.046 INFO:teuthology.orchestra.run.trial063.stdout: Installing : python3-ceph-common-2:20.3.0-5145.g5fdb7d8f.el9.x8 33/39 2026-02-12T12:27:00.986 INFO:teuthology.orchestra.run.trial063.stdout: Running scriptlet: ceph-common-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 34/39 2026-02-12T12:27:00.988 INFO:teuthology.orchestra.run.trial063.stdout: Installing : ceph-common-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 34/39 2026-02-12T12:27:01.250 INFO:teuthology.orchestra.run.trial063.stdout: Running scriptlet: ceph-common-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 34/39 2026-02-12T12:27:01.253 INFO:teuthology.orchestra.run.trial063.stdout: Installing : ceph-base-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 35/39 2026-02-12T12:27:01.291 INFO:teuthology.orchestra.run.trial063.stdout: Running scriptlet: ceph-base-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 35/39 2026-02-12T12:27:01.291 INFO:teuthology.orchestra.run.trial063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2026-02-12T12:27:01.291 INFO:teuthology.orchestra.run.trial063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2026-02-12T12:27:01.291 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:27:01.294 INFO:teuthology.orchestra.run.trial063.stdout: Installing : ceph-selinux-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 36/39 2026-02-12T12:27:05.785 INFO:teuthology.orchestra.run.trial063.stdout: Running scriptlet: ceph-selinux-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 36/39 2026-02-12T12:27:05.786 INFO:teuthology.orchestra.run.trial063.stdout:skipping the directory /sys 2026-02-12T12:27:05.786 INFO:teuthology.orchestra.run.trial063.stdout:skipping the directory /proc 2026-02-12T12:27:05.786 INFO:teuthology.orchestra.run.trial063.stdout:skipping the directory /mnt 2026-02-12T12:27:05.786 INFO:teuthology.orchestra.run.trial063.stdout:skipping the directory /var/tmp 2026-02-12T12:27:05.786 INFO:teuthology.orchestra.run.trial063.stdout:skipping the directory /home 2026-02-12T12:27:05.786 INFO:teuthology.orchestra.run.trial063.stdout:skipping the directory /root 2026-02-12T12:27:05.786 INFO:teuthology.orchestra.run.trial063.stdout:skipping the directory /tmp 2026-02-12T12:27:05.786 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:27:06.277 INFO:teuthology.orchestra.run.trial063.stdout: Installing : ceph-radosgw-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 37/39 2026-02-12T12:27:06.301 INFO:teuthology.orchestra.run.trial063.stdout: Running scriptlet: ceph-radosgw-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 37/39 2026-02-12T12:27:06.301 INFO:teuthology.orchestra.run.trial063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-12T12:27:06.301 INFO:teuthology.orchestra.run.trial063.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2026-02-12T12:27:06.301 INFO:teuthology.orchestra.run.trial063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-12T12:27:06.301 INFO:teuthology.orchestra.run.trial063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-12T12:27:06.301 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:27:06.303 INFO:teuthology.orchestra.run.trial063.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 38/39 2026-02-12T12:27:06.315 INFO:teuthology.orchestra.run.trial063.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 38/39 2026-02-12T12:27:06.315 INFO:teuthology.orchestra.run.trial063.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 39/39 2026-02-12T12:27:06.369 INFO:teuthology.orchestra.run.trial063.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 39/39 2026-02-12T12:27:06.369 INFO:teuthology.orchestra.run.trial063.stdout:Creating group 'qat' with GID 992. 2026-02-12T12:27:06.369 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:27:06.656 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : ceph-base-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 1/39 2026-02-12T12:27:06.656 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : ceph-common-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2/39 2026-02-12T12:27:06.656 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : ceph-radosgw-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 3/39 2026-02-12T12:27:06.657 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : ceph-selinux-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 4/39 2026-02-12T12:27:06.657 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : libcephfs-daemon-2:20.3.0-5145.g5fdb7d8f.el9.x86_6 5/39 2026-02-12T12:27:06.657 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : libcephfs-proxy2-2:20.3.0-5145.g5fdb7d8f.el9.x86_6 6/39 2026-02-12T12:27:06.657 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : libcephfs2-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 7/39 2026-02-12T12:27:06.657 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : libradosstriper1-2:20.3.0-5145.g5fdb7d8f.el9.x86_6 8/39 2026-02-12T12:27:06.657 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : librgw2-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 9/39 2026-02-12T12:27:06.657 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : python3-ceph-argparse-2:20.3.0-5145.g5fdb7d8f.el9. 10/39 2026-02-12T12:27:06.657 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : python3-ceph-common-2:20.3.0-5145.g5fdb7d8f.el9.x8 11/39 2026-02-12T12:27:06.657 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : python3-cephfs-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 12/39 2026-02-12T12:27:06.657 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : python3-rados-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 13/39 2026-02-12T12:27:06.657 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : python3-rbd-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 14/39 2026-02-12T12:27:06.658 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : python3-rgw-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 15/39 2026-02-12T12:27:06.658 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : mailcap-2.1.49-5.el9.noarch 16/39 2026-02-12T12:27:06.658 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : nvme-cli-2.13-1.el9.x86_64 17/39 2026-02-12T12:27:06.658 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : pciutils-3.7.0-7.el9.x86_64 18/39 2026-02-12T12:27:06.658 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : boost-program-options-1.75.0-13.el9.x86_64 19/39 2026-02-12T12:27:06.658 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : libnbd-1.20.3-4.el9.x86_64 20/39 2026-02-12T12:27:06.658 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 21/39 2026-02-12T12:27:06.658 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 22/39 2026-02-12T12:27:06.658 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 23/39 2026-02-12T12:27:06.658 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 24/39 2026-02-12T12:27:06.658 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : qatlib-25.08.0-2.el9.x86_64 25/39 2026-02-12T12:27:06.659 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : qatlib-service-25.08.0-2.el9.x86_64 26/39 2026-02-12T12:27:06.659 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : qatzip-libs-1.3.1-1.el9.x86_64 27/39 2026-02-12T12:27:06.659 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 28/39 2026-02-12T12:27:06.659 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : libarrow-9.0.0-15.el9.x86_64 29/39 2026-02-12T12:27:06.659 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : libarrow-doc-9.0.0-15.el9.noarch 30/39 2026-02-12T12:27:06.659 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : liboath-2.6.12-1.el9.x86_64 31/39 2026-02-12T12:27:06.659 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 32/39 2026-02-12T12:27:06.659 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : parquet-libs-9.0.0-15.el9.x86_64 33/39 2026-02-12T12:27:06.659 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : re2-1:20211101-20.el9.x86_64 34/39 2026-02-12T12:27:06.659 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : thrift-0.15.0-4.el9.x86_64 35/39 2026-02-12T12:27:06.659 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : librados2-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 36/39 2026-02-12T12:27:06.659 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 37/39 2026-02-12T12:27:06.659 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : librbd1-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 38/39 2026-02-12T12:27:06.716 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 39/39 2026-02-12T12:27:06.716 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:27:06.716 INFO:teuthology.orchestra.run.trial063.stdout:Upgraded: 2026-02-12T12:27:06.716 INFO:teuthology.orchestra.run.trial063.stdout: librados2-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:27:06.717 INFO:teuthology.orchestra.run.trial063.stdout: librbd1-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:27:06.717 INFO:teuthology.orchestra.run.trial063.stdout:Installed: 2026-02-12T12:27:06.717 INFO:teuthology.orchestra.run.trial063.stdout: boost-program-options-1.75.0-13.el9.x86_64 2026-02-12T12:27:06.717 INFO:teuthology.orchestra.run.trial063.stdout: ceph-base-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:27:06.717 INFO:teuthology.orchestra.run.trial063.stdout: ceph-common-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:27:06.717 INFO:teuthology.orchestra.run.trial063.stdout: ceph-radosgw-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:27:06.717 INFO:teuthology.orchestra.run.trial063.stdout: ceph-selinux-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:27:06.717 INFO:teuthology.orchestra.run.trial063.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2026-02-12T12:27:06.717 INFO:teuthology.orchestra.run.trial063.stdout: libarrow-9.0.0-15.el9.x86_64 2026-02-12T12:27:06.717 INFO:teuthology.orchestra.run.trial063.stdout: libarrow-doc-9.0.0-15.el9.noarch 2026-02-12T12:27:06.717 INFO:teuthology.orchestra.run.trial063.stdout: libcephfs-daemon-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:27:06.718 INFO:teuthology.orchestra.run.trial063.stdout: libcephfs-proxy2-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:27:06.718 INFO:teuthology.orchestra.run.trial063.stdout: libcephfs2-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:27:06.718 INFO:teuthology.orchestra.run.trial063.stdout: libnbd-1.20.3-4.el9.x86_64 2026-02-12T12:27:06.718 INFO:teuthology.orchestra.run.trial063.stdout: liboath-2.6.12-1.el9.x86_64 2026-02-12T12:27:06.718 INFO:teuthology.orchestra.run.trial063.stdout: libpmemobj-1.12.1-1.el9.x86_64 2026-02-12T12:27:06.718 INFO:teuthology.orchestra.run.trial063.stdout: librabbitmq-0.11.0-7.el9.x86_64 2026-02-12T12:27:06.718 INFO:teuthology.orchestra.run.trial063.stdout: libradosstriper1-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:27:06.718 INFO:teuthology.orchestra.run.trial063.stdout: librdkafka-1.6.1-102.el9.x86_64 2026-02-12T12:27:06.718 INFO:teuthology.orchestra.run.trial063.stdout: librgw2-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:27:06.718 INFO:teuthology.orchestra.run.trial063.stdout: libunwind-1.6.2-1.el9.x86_64 2026-02-12T12:27:06.718 INFO:teuthology.orchestra.run.trial063.stdout: lttng-ust-2.12.0-6.el9.x86_64 2026-02-12T12:27:06.718 INFO:teuthology.orchestra.run.trial063.stdout: mailcap-2.1.49-5.el9.noarch 2026-02-12T12:27:06.718 INFO:teuthology.orchestra.run.trial063.stdout: nvme-cli-2.13-1.el9.x86_64 2026-02-12T12:27:06.719 INFO:teuthology.orchestra.run.trial063.stdout: parquet-libs-9.0.0-15.el9.x86_64 2026-02-12T12:27:06.719 INFO:teuthology.orchestra.run.trial063.stdout: pciutils-3.7.0-7.el9.x86_64 2026-02-12T12:27:06.719 INFO:teuthology.orchestra.run.trial063.stdout: python3-ceph-argparse-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:27:06.719 INFO:teuthology.orchestra.run.trial063.stdout: python3-ceph-common-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:27:06.719 INFO:teuthology.orchestra.run.trial063.stdout: python3-cephfs-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:27:06.719 INFO:teuthology.orchestra.run.trial063.stdout: python3-rados-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:27:06.719 INFO:teuthology.orchestra.run.trial063.stdout: python3-rbd-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:27:06.719 INFO:teuthology.orchestra.run.trial063.stdout: python3-rgw-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:27:06.719 INFO:teuthology.orchestra.run.trial063.stdout: qatlib-25.08.0-2.el9.x86_64 2026-02-12T12:27:06.719 INFO:teuthology.orchestra.run.trial063.stdout: qatlib-service-25.08.0-2.el9.x86_64 2026-02-12T12:27:06.719 INFO:teuthology.orchestra.run.trial063.stdout: qatzip-libs-1.3.1-1.el9.x86_64 2026-02-12T12:27:06.719 INFO:teuthology.orchestra.run.trial063.stdout: re2-1:20211101-20.el9.x86_64 2026-02-12T12:27:06.719 INFO:teuthology.orchestra.run.trial063.stdout: thrift-0.15.0-4.el9.x86_64 2026-02-12T12:27:06.720 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:27:06.720 INFO:teuthology.orchestra.run.trial063.stdout:Complete! 2026-02-12T12:27:06.758 DEBUG:teuthology.orchestra.run.trial063:> sudo yum -y install ceph-test 2026-02-12T12:27:07.062 INFO:teuthology.orchestra.run.trial063.stdout:Last metadata expiration check: 0:00:41 ago on Thu 12 Feb 2026 12:26:26 PM UTC. 2026-02-12T12:27:07.147 INFO:teuthology.orchestra.run.trial063.stdout:Dependencies resolved. 2026-02-12T12:27:07.148 INFO:teuthology.orchestra.run.trial063.stdout:================================================================================ 2026-02-12T12:27:07.148 INFO:teuthology.orchestra.run.trial063.stdout: Package Arch Version Repository Size 2026-02-12T12:27:07.148 INFO:teuthology.orchestra.run.trial063.stdout:================================================================================ 2026-02-12T12:27:07.148 INFO:teuthology.orchestra.run.trial063.stdout:Installing: 2026-02-12T12:27:07.148 INFO:teuthology.orchestra.run.trial063.stdout: ceph-test x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 126 M 2026-02-12T12:27:07.148 INFO:teuthology.orchestra.run.trial063.stdout:Installing dependencies: 2026-02-12T12:27:07.148 INFO:teuthology.orchestra.run.trial063.stdout: hwloc-libs x86_64 2.4.1-6.el9 baseos 2.1 M 2026-02-12T12:27:07.148 INFO:teuthology.orchestra.run.trial063.stdout: libasan8 x86_64 15.1.1-2.5.el9 appstream 523 k 2026-02-12T12:27:07.148 INFO:teuthology.orchestra.run.trial063.stdout: libatomic x86_64 11.5.0-14.el9 baseos 23 k 2026-02-12T12:27:07.149 INFO:teuthology.orchestra.run.trial063.stdout: libcephsqlite x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 153 k 2026-02-12T12:27:07.149 INFO:teuthology.orchestra.run.trial063.stdout: libubsan x86_64 11.5.0-14.el9 appstream 176 k 2026-02-12T12:27:07.149 INFO:teuthology.orchestra.run.trial063.stdout: libxslt x86_64 1.1.34-12.el9 appstream 233 k 2026-02-12T12:27:07.149 INFO:teuthology.orchestra.run.trial063.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2026-02-12T12:27:07.149 INFO:teuthology.orchestra.run.trial063.stdout: protobuf x86_64 3.14.0-17.el9 appstream 1.0 M 2026-02-12T12:27:07.149 INFO:teuthology.orchestra.run.trial063.stdout: socat x86_64 1.7.4.1-8.el9 appstream 303 k 2026-02-12T12:27:07.149 INFO:teuthology.orchestra.run.trial063.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2026-02-12T12:27:07.149 INFO:teuthology.orchestra.run.trial063.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2026-02-12T12:27:07.149 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:27:07.149 INFO:teuthology.orchestra.run.trial063.stdout:Transaction Summary 2026-02-12T12:27:07.149 INFO:teuthology.orchestra.run.trial063.stdout:================================================================================ 2026-02-12T12:27:07.150 INFO:teuthology.orchestra.run.trial063.stdout:Install 12 Packages 2026-02-12T12:27:07.150 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:27:07.150 INFO:teuthology.orchestra.run.trial063.stdout:Total download size: 131 M 2026-02-12T12:27:07.150 INFO:teuthology.orchestra.run.trial063.stdout:Installed size: 977 M 2026-02-12T12:27:07.151 INFO:teuthology.orchestra.run.trial063.stdout:Downloading Packages: 2026-02-12T12:27:08.154 INFO:teuthology.orchestra.run.trial063.stdout:(1/12): libcephsqlite-20.3.0-5145.g5fdb7d8f.el9 197 kB/s | 153 kB 00:00 2026-02-12T12:27:17.386 INFO:teuthology.orchestra.run.trial063.stdout:[MIRROR] hwloc-libs-2.4.1-6.el9.x86_64.rpm: Curl error (6): Couldn't resolve host name for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/hwloc-libs-2.4.1-6.el9.x86_64.rpm [Could not resolve host: mirror.siena.edu] 2026-02-12T12:27:18.164 INFO:teuthology.orchestra.run.trial063.stdout:[MIRROR] libatomic-11.5.0-14.el9.x86_64.rpm: Curl error (6): Couldn't resolve host name for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/libatomic-11.5.0-14.el9.x86_64.rpm [Could not resolve host: mirror.siena.edu] 2026-02-12T12:27:22.603 INFO:teuthology.orchestra.run.trial063.stdout:(2/12): libatomic-11.5.0-14.el9.x86_64.rpm 1.6 kB/s | 23 kB 00:14 2026-02-12T12:27:22.927 INFO:teuthology.orchestra.run.trial063.stdout:(3/12): hwloc-libs-2.4.1-6.el9.x86_64.rpm 138 kB/s | 2.1 MB 00:15 2026-02-12T12:27:32.614 INFO:teuthology.orchestra.run.trial063.stdout:[MIRROR] lksctp-tools-1.0.19-2.el9.x86_64.rpm: Curl error (6): Couldn't resolve host name for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/lksctp-tools-1.0.19-2.el9.x86_64.rpm [Could not resolve host: mirror.siena.edu] 2026-02-12T12:27:32.896 INFO:teuthology.orchestra.run.trial063.stdout:(4/12): lksctp-tools-1.0.19-2.el9.x86_64.rpm 9.1 kB/s | 94 kB 00:10 2026-02-12T12:27:33.452 INFO:teuthology.orchestra.run.trial063.stdout:(5/12): libubsan-11.5.0-14.el9.x86_64.rpm 317 kB/s | 176 kB 00:00 2026-02-12T12:27:33.680 INFO:teuthology.orchestra.run.trial063.stdout:(6/12): libasan8-15.1.1-2.5.el9.x86_64.rpm 49 kB/s | 523 kB 00:10 2026-02-12T12:27:33.776 INFO:teuthology.orchestra.run.trial063.stdout:(7/12): libxslt-1.1.34-12.el9.x86_64.rpm 719 kB/s | 233 kB 00:00 2026-02-12T12:27:34.175 INFO:teuthology.orchestra.run.trial063.stdout:(8/12): socat-1.7.4.1-8.el9.x86_64.rpm 760 kB/s | 303 kB 00:00 2026-02-12T12:27:34.272 INFO:teuthology.orchestra.run.trial063.stdout:(9/12): xmlstarlet-1.6.1-20.el9.x86_64.rpm 657 kB/s | 64 kB 00:00 2026-02-12T12:27:34.468 INFO:teuthology.orchestra.run.trial063.stdout:(10/12): yaml-cpp-0.6.3-5.el9.x86_64.rpm 622 kB/s | 122 kB 00:00 2026-02-12T12:27:34.853 INFO:teuthology.orchestra.run.trial063.stdout:(11/12): protobuf-3.14.0-17.el9.x86_64.rpm 877 kB/s | 1.0 MB 00:01 2026-02-12T12:28:49.886 INFO:teuthology.orchestra.run.trial063.stdout:(12/12): ceph-test-20.3.0-5145.g5fdb7d8f.el9.x8 1.2 MB/s | 126 MB 01:42 2026-02-12T12:28:49.889 INFO:teuthology.orchestra.run.trial063.stdout:-------------------------------------------------------------------------------- 2026-02-12T12:28:49.889 INFO:teuthology.orchestra.run.trial063.stdout:Total 1.3 MB/s | 131 MB 01:42 2026-02-12T12:28:49.959 INFO:teuthology.orchestra.run.trial063.stdout:Running transaction check 2026-02-12T12:28:49.971 INFO:teuthology.orchestra.run.trial063.stdout:Transaction check succeeded. 2026-02-12T12:28:49.971 INFO:teuthology.orchestra.run.trial063.stdout:Running transaction test 2026-02-12T12:28:50.400 INFO:teuthology.orchestra.run.trial063.stdout:Transaction test succeeded. 2026-02-12T12:28:50.400 INFO:teuthology.orchestra.run.trial063.stdout:Running transaction 2026-02-12T12:28:50.864 INFO:teuthology.orchestra.run.trial063.stdout: Preparing : 1/1 2026-02-12T12:28:50.873 INFO:teuthology.orchestra.run.trial063.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/12 2026-02-12T12:28:50.882 INFO:teuthology.orchestra.run.trial063.stdout: Installing : socat-1.7.4.1-8.el9.x86_64 2/12 2026-02-12T12:28:50.887 INFO:teuthology.orchestra.run.trial063.stdout: Installing : protobuf-3.14.0-17.el9.x86_64 3/12 2026-02-12T12:28:50.891 INFO:teuthology.orchestra.run.trial063.stdout: Installing : libxslt-1.1.34-12.el9.x86_64 4/12 2026-02-12T12:28:50.893 INFO:teuthology.orchestra.run.trial063.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 5/12 2026-02-12T12:28:50.898 INFO:teuthology.orchestra.run.trial063.stdout: Installing : libubsan-11.5.0-14.el9.x86_64 6/12 2026-02-12T12:28:50.898 INFO:teuthology.orchestra.run.trial063.stdout: Installing : libasan8-15.1.1-2.5.el9.x86_64 7/12 2026-02-12T12:28:50.915 INFO:teuthology.orchestra.run.trial063.stdout: Running scriptlet: libasan8-15.1.1-2.5.el9.x86_64 7/12 2026-02-12T12:28:50.919 INFO:teuthology.orchestra.run.trial063.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 8/12 2026-02-12T12:28:50.929 INFO:teuthology.orchestra.run.trial063.stdout: Installing : libatomic-11.5.0-14.el9.x86_64 9/12 2026-02-12T12:28:50.938 INFO:teuthology.orchestra.run.trial063.stdout: Installing : hwloc-libs-2.4.1-6.el9.x86_64 10/12 2026-02-12T12:28:50.939 INFO:teuthology.orchestra.run.trial063.stdout: Installing : libcephsqlite-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 11/12 2026-02-12T12:28:58.052 INFO:teuthology.orchestra.run.trial063.stdout: Running scriptlet: libcephsqlite-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 11/12 2026-02-12T12:28:58.069 INFO:teuthology.orchestra.run.trial063.stdout: Installing : ceph-test-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 12/12 2026-02-12T12:28:58.834 INFO:teuthology.orchestra.run.trial063.stdout: Running scriptlet: ceph-test-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 12/12 2026-02-12T12:28:58.834 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : ceph-test-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 1/12 2026-02-12T12:28:58.834 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : libcephsqlite-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2/12 2026-02-12T12:28:58.835 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : hwloc-libs-2.4.1-6.el9.x86_64 3/12 2026-02-12T12:28:58.835 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : libatomic-11.5.0-14.el9.x86_64 4/12 2026-02-12T12:28:58.835 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 5/12 2026-02-12T12:28:58.835 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : libasan8-15.1.1-2.5.el9.x86_64 6/12 2026-02-12T12:28:58.835 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : libubsan-11.5.0-14.el9.x86_64 7/12 2026-02-12T12:28:58.835 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : libxslt-1.1.34-12.el9.x86_64 8/12 2026-02-12T12:28:58.835 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : protobuf-3.14.0-17.el9.x86_64 9/12 2026-02-12T12:28:58.835 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : socat-1.7.4.1-8.el9.x86_64 10/12 2026-02-12T12:28:58.835 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 11/12 2026-02-12T12:28:58.889 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 12/12 2026-02-12T12:28:58.889 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:28:58.890 INFO:teuthology.orchestra.run.trial063.stdout:Installed: 2026-02-12T12:28:58.890 INFO:teuthology.orchestra.run.trial063.stdout: ceph-test-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:28:58.890 INFO:teuthology.orchestra.run.trial063.stdout: hwloc-libs-2.4.1-6.el9.x86_64 2026-02-12T12:28:58.890 INFO:teuthology.orchestra.run.trial063.stdout: libasan8-15.1.1-2.5.el9.x86_64 2026-02-12T12:28:58.890 INFO:teuthology.orchestra.run.trial063.stdout: libatomic-11.5.0-14.el9.x86_64 2026-02-12T12:28:58.890 INFO:teuthology.orchestra.run.trial063.stdout: libcephsqlite-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:28:58.890 INFO:teuthology.orchestra.run.trial063.stdout: libubsan-11.5.0-14.el9.x86_64 2026-02-12T12:28:58.890 INFO:teuthology.orchestra.run.trial063.stdout: libxslt-1.1.34-12.el9.x86_64 2026-02-12T12:28:58.890 INFO:teuthology.orchestra.run.trial063.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2026-02-12T12:28:58.890 INFO:teuthology.orchestra.run.trial063.stdout: protobuf-3.14.0-17.el9.x86_64 2026-02-12T12:28:58.890 INFO:teuthology.orchestra.run.trial063.stdout: socat-1.7.4.1-8.el9.x86_64 2026-02-12T12:28:58.890 INFO:teuthology.orchestra.run.trial063.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2026-02-12T12:28:58.891 INFO:teuthology.orchestra.run.trial063.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2026-02-12T12:28:58.891 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:28:58.891 INFO:teuthology.orchestra.run.trial063.stdout:Complete! 2026-02-12T12:28:58.929 DEBUG:teuthology.orchestra.run.trial063:> sudo yum -y install ceph 2026-02-12T12:28:59.220 INFO:teuthology.orchestra.run.trial063.stdout:Last metadata expiration check: 0:02:33 ago on Thu 12 Feb 2026 12:26:26 PM UTC. 2026-02-12T12:28:59.306 INFO:teuthology.orchestra.run.trial063.stdout:Dependencies resolved. 2026-02-12T12:28:59.309 INFO:teuthology.orchestra.run.trial063.stdout:======================================================================================= 2026-02-12T12:28:59.309 INFO:teuthology.orchestra.run.trial063.stdout: Package Arch Version Repository Size 2026-02-12T12:28:59.310 INFO:teuthology.orchestra.run.trial063.stdout:======================================================================================= 2026-02-12T12:28:59.310 INFO:teuthology.orchestra.run.trial063.stdout:Installing: 2026-02-12T12:28:59.310 INFO:teuthology.orchestra.run.trial063.stdout: ceph x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 6.5 k 2026-02-12T12:28:59.310 INFO:teuthology.orchestra.run.trial063.stdout:Installing dependencies: 2026-02-12T12:28:59.310 INFO:teuthology.orchestra.run.trial063.stdout: abseil-cpp x86_64 20211102.0-4.el9 epel 551 k 2026-02-12T12:28:59.310 INFO:teuthology.orchestra.run.trial063.stdout: ceph-grafana-dashboards noarch 2:20.3.0-5145.g5fdb7d8f.el9 ceph-noarch 57 k 2026-02-12T12:28:59.310 INFO:teuthology.orchestra.run.trial063.stdout: ceph-mds x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 2.0 M 2026-02-12T12:28:59.310 INFO:teuthology.orchestra.run.trial063.stdout: ceph-mgr x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 937 k 2026-02-12T12:28:59.310 INFO:teuthology.orchestra.run.trial063.stdout: ceph-mgr-modules-core noarch 2:20.3.0-5145.g5fdb7d8f.el9 ceph-noarch 297 k 2026-02-12T12:28:59.310 INFO:teuthology.orchestra.run.trial063.stdout: ceph-mon x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 4.8 M 2026-02-12T12:28:59.311 INFO:teuthology.orchestra.run.trial063.stdout: ceph-osd x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 163 k 2026-02-12T12:28:59.311 INFO:teuthology.orchestra.run.trial063.stdout: ceph-osd-classic x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 15 M 2026-02-12T12:28:59.311 INFO:teuthology.orchestra.run.trial063.stdout: ceph-osd-crimson x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 87 M 2026-02-12T12:28:59.311 INFO:teuthology.orchestra.run.trial063.stdout: ceph-prometheus-alerts noarch 2:20.3.0-5145.g5fdb7d8f.el9 ceph-noarch 17 k 2026-02-12T12:28:59.311 INFO:teuthology.orchestra.run.trial063.stdout: cephadm noarch 2:20.3.0-5145.g5fdb7d8f.el9 ceph-noarch 1.0 M 2026-02-12T12:28:59.311 INFO:teuthology.orchestra.run.trial063.stdout: cryptsetup x86_64 2.8.1-2.el9 baseos 350 k 2026-02-12T12:28:59.311 INFO:teuthology.orchestra.run.trial063.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2026-02-12T12:28:59.311 INFO:teuthology.orchestra.run.trial063.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2026-02-12T12:28:59.311 INFO:teuthology.orchestra.run.trial063.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2026-02-12T12:28:59.311 INFO:teuthology.orchestra.run.trial063.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2026-02-12T12:28:59.311 INFO:teuthology.orchestra.run.trial063.stdout: ledmon-libs x86_64 1.1.0-3.el9 baseos 40 k 2026-02-12T12:28:59.311 INFO:teuthology.orchestra.run.trial063.stdout: libconfig x86_64 1.7.2-9.el9 baseos 72 k 2026-02-12T12:28:59.312 INFO:teuthology.orchestra.run.trial063.stdout: libgfortran x86_64 11.5.0-14.el9 baseos 794 k 2026-02-12T12:28:59.312 INFO:teuthology.orchestra.run.trial063.stdout: libquadmath x86_64 11.5.0-14.el9 baseos 184 k 2026-02-12T12:28:59.312 INFO:teuthology.orchestra.run.trial063.stdout: libstoragemgmt x86_64 1.10.1-1.el9 appstream 246 k 2026-02-12T12:28:59.312 INFO:teuthology.orchestra.run.trial063.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2026-02-12T12:28:59.312 INFO:teuthology.orchestra.run.trial063.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2026-02-12T12:28:59.312 INFO:teuthology.orchestra.run.trial063.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2026-02-12T12:28:59.312 INFO:teuthology.orchestra.run.trial063.stdout: openblas x86_64 0.3.29-1.el9 appstream 42 k 2026-02-12T12:28:59.312 INFO:teuthology.orchestra.run.trial063.stdout: openblas-openmp x86_64 0.3.29-1.el9 appstream 5.3 M 2026-02-12T12:28:59.312 INFO:teuthology.orchestra.run.trial063.stdout: protobuf-compiler x86_64 3.14.0-17.el9 crb 862 k 2026-02-12T12:28:59.312 INFO:teuthology.orchestra.run.trial063.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2026-02-12T12:28:59.312 INFO:teuthology.orchestra.run.trial063.stdout: python3-autocommand noarch 2.2.2-8.el9 epel 29 k 2026-02-12T12:28:59.312 INFO:teuthology.orchestra.run.trial063.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2026-02-12T12:28:59.313 INFO:teuthology.orchestra.run.trial063.stdout: python3-backports-tarfile noarch 1.2.0-1.el9 epel 60 k 2026-02-12T12:28:59.313 INFO:teuthology.orchestra.run.trial063.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2026-02-12T12:28:59.313 INFO:teuthology.orchestra.run.trial063.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2026-02-12T12:28:59.313 INFO:teuthology.orchestra.run.trial063.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2026-02-12T12:28:59.313 INFO:teuthology.orchestra.run.trial063.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2026-02-12T12:28:59.313 INFO:teuthology.orchestra.run.trial063.stdout: python3-cheroot noarch 10.0.1-4.el9 epel 173 k 2026-02-12T12:28:59.313 INFO:teuthology.orchestra.run.trial063.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2026-02-12T12:28:59.313 INFO:teuthology.orchestra.run.trial063.stdout: python3-cryptography x86_64 36.0.1-5.el9 baseos 1.2 M 2026-02-12T12:28:59.313 INFO:teuthology.orchestra.run.trial063.stdout: python3-devel x86_64 3.9.25-2.el9 appstream 244 k 2026-02-12T12:28:59.313 INFO:teuthology.orchestra.run.trial063.stdout: python3-google-auth noarch 1:2.45.0-1.el9 epel 254 k 2026-02-12T12:28:59.313 INFO:teuthology.orchestra.run.trial063.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2026-02-12T12:28:59.313 INFO:teuthology.orchestra.run.trial063.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2026-02-12T12:28:59.313 INFO:teuthology.orchestra.run.trial063.stdout: python3-isodate noarch 0.6.1-3.el9 epel 56 k 2026-02-12T12:28:59.314 INFO:teuthology.orchestra.run.trial063.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2026-02-12T12:28:59.314 INFO:teuthology.orchestra.run.trial063.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2026-02-12T12:28:59.314 INFO:teuthology.orchestra.run.trial063.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2026-02-12T12:28:59.314 INFO:teuthology.orchestra.run.trial063.stdout: python3-jaraco-context noarch 6.0.1-3.el9 epel 20 k 2026-02-12T12:28:59.314 INFO:teuthology.orchestra.run.trial063.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2026-02-12T12:28:59.314 INFO:teuthology.orchestra.run.trial063.stdout: python3-jaraco-text noarch 4.0.0-2.el9 epel 26 k 2026-02-12T12:28:59.314 INFO:teuthology.orchestra.run.trial063.stdout: python3-jinja2 noarch 2.11.3-8.el9 appstream 249 k 2026-02-12T12:28:59.314 INFO:teuthology.orchestra.run.trial063.stdout: python3-jmespath noarch 1.0.1-1.el9 appstream 48 k 2026-02-12T12:28:59.314 INFO:teuthology.orchestra.run.trial063.stdout: python3-kubernetes noarch 1:26.1.0-3.el9 epel 1.0 M 2026-02-12T12:28:59.314 INFO:teuthology.orchestra.run.trial063.stdout: python3-libstoragemgmt x86_64 1.10.1-1.el9 appstream 177 k 2026-02-12T12:28:59.314 INFO:teuthology.orchestra.run.trial063.stdout: python3-lxml x86_64 4.6.5-3.el9 appstream 1.2 M 2026-02-12T12:28:59.314 INFO:teuthology.orchestra.run.trial063.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2026-02-12T12:28:59.314 INFO:teuthology.orchestra.run.trial063.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2026-02-12T12:28:59.315 INFO:teuthology.orchestra.run.trial063.stdout: python3-msgpack x86_64 1.0.3-2.el9 epel 86 k 2026-02-12T12:28:59.315 INFO:teuthology.orchestra.run.trial063.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2026-02-12T12:28:59.315 INFO:teuthology.orchestra.run.trial063.stdout: python3-numpy x86_64 1:1.23.5-2.el9 appstream 6.1 M 2026-02-12T12:28:59.315 INFO:teuthology.orchestra.run.trial063.stdout: python3-numpy-f2py x86_64 1:1.23.5-2.el9 appstream 442 k 2026-02-12T12:28:59.315 INFO:teuthology.orchestra.run.trial063.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2026-02-12T12:28:59.315 INFO:teuthology.orchestra.run.trial063.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2026-02-12T12:28:59.315 INFO:teuthology.orchestra.run.trial063.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2026-02-12T12:28:59.315 INFO:teuthology.orchestra.run.trial063.stdout: python3-protobuf noarch 3.14.0-17.el9 appstream 267 k 2026-02-12T12:28:59.315 INFO:teuthology.orchestra.run.trial063.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2026-02-12T12:28:59.315 INFO:teuthology.orchestra.run.trial063.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2026-02-12T12:28:59.315 INFO:teuthology.orchestra.run.trial063.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2026-02-12T12:28:59.315 INFO:teuthology.orchestra.run.trial063.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2026-02-12T12:28:59.315 INFO:teuthology.orchestra.run.trial063.stdout: python3-pyparsing noarch 2.4.7-9.el9 baseos 150 k 2026-02-12T12:28:59.316 INFO:teuthology.orchestra.run.trial063.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2026-02-12T12:28:59.316 INFO:teuthology.orchestra.run.trial063.stdout: python3-requests noarch 2.25.1-10.el9 baseos 126 k 2026-02-12T12:28:59.316 INFO:teuthology.orchestra.run.trial063.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2026-02-12T12:28:59.316 INFO:teuthology.orchestra.run.trial063.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2026-02-12T12:28:59.316 INFO:teuthology.orchestra.run.trial063.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2026-02-12T12:28:59.316 INFO:teuthology.orchestra.run.trial063.stdout: python3-scipy x86_64 1.9.3-2.el9 appstream 19 M 2026-02-12T12:28:59.316 INFO:teuthology.orchestra.run.trial063.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2026-02-12T12:28:59.316 INFO:teuthology.orchestra.run.trial063.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2026-02-12T12:28:59.316 INFO:teuthology.orchestra.run.trial063.stdout: python3-typing-extensions noarch 4.15.0-1.el9 epel 86 k 2026-02-12T12:28:59.316 INFO:teuthology.orchestra.run.trial063.stdout: python3-urllib3 noarch 1.26.5-7.el9 baseos 218 k 2026-02-12T12:28:59.316 INFO:teuthology.orchestra.run.trial063.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2026-02-12T12:28:59.316 INFO:teuthology.orchestra.run.trial063.stdout: python3-xmlsec x86_64 1.3.13-1.el9 epel 48 k 2026-02-12T12:28:59.316 INFO:teuthology.orchestra.run.trial063.stdout: python3-xmltodict noarch 0.12.0-15.el9 epel 22 k 2026-02-12T12:28:59.317 INFO:teuthology.orchestra.run.trial063.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2026-02-12T12:28:59.317 INFO:teuthology.orchestra.run.trial063.stdout: unzip x86_64 6.0-59.el9 baseos 182 k 2026-02-12T12:28:59.317 INFO:teuthology.orchestra.run.trial063.stdout: xmlsec1 x86_64 1.2.29-13.el9 appstream 189 k 2026-02-12T12:28:59.317 INFO:teuthology.orchestra.run.trial063.stdout: xmlsec1-openssl x86_64 1.2.29-13.el9 appstream 90 k 2026-02-12T12:28:59.317 INFO:teuthology.orchestra.run.trial063.stdout: zip x86_64 3.0-35.el9 baseos 266 k 2026-02-12T12:28:59.317 INFO:teuthology.orchestra.run.trial063.stdout:Installing weak dependencies: 2026-02-12T12:28:59.317 INFO:teuthology.orchestra.run.trial063.stdout: ceph-mgr-cephadm noarch 2:20.3.0-5145.g5fdb7d8f.el9 ceph-noarch 183 k 2026-02-12T12:28:59.317 INFO:teuthology.orchestra.run.trial063.stdout: ceph-mgr-dashboard noarch 2:20.3.0-5145.g5fdb7d8f.el9 ceph-noarch 15 M 2026-02-12T12:28:59.317 INFO:teuthology.orchestra.run.trial063.stdout: ceph-mgr-diskprediction-local noarch 2:20.3.0-5145.g5fdb7d8f.el9 ceph-noarch 7.4 M 2026-02-12T12:28:59.317 INFO:teuthology.orchestra.run.trial063.stdout: ceph-mgr-k8sevents noarch 2:20.3.0-5145.g5fdb7d8f.el9 ceph-noarch 22 k 2026-02-12T12:28:59.317 INFO:teuthology.orchestra.run.trial063.stdout: ceph-mgr-rook noarch 2:20.3.0-5145.g5fdb7d8f.el9 ceph-noarch 50 k 2026-02-12T12:28:59.317 INFO:teuthology.orchestra.run.trial063.stdout: ceph-volume noarch 2:20.3.0-5145.g5fdb7d8f.el9 ceph-noarch 299 k 2026-02-12T12:28:59.318 INFO:teuthology.orchestra.run.trial063.stdout: python3-influxdb noarch 5.3.1-1.el9 epel 139 k 2026-02-12T12:28:59.318 INFO:teuthology.orchestra.run.trial063.stdout: python3-saml noarch 1.16.0-1.el9 epel 125 k 2026-02-12T12:28:59.318 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:28:59.318 INFO:teuthology.orchestra.run.trial063.stdout:Transaction Summary 2026-02-12T12:28:59.318 INFO:teuthology.orchestra.run.trial063.stdout:======================================================================================= 2026-02-12T12:28:59.318 INFO:teuthology.orchestra.run.trial063.stdout:Install 96 Packages 2026-02-12T12:28:59.318 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:28:59.319 INFO:teuthology.orchestra.run.trial063.stdout:Total download size: 189 M 2026-02-12T12:28:59.319 INFO:teuthology.orchestra.run.trial063.stdout:Installed size: 1.4 G 2026-02-12T12:28:59.319 INFO:teuthology.orchestra.run.trial063.stdout:Downloading Packages: 2026-02-12T12:29:00.050 INFO:teuthology.orchestra.run.trial063.stdout:(1/96): ceph-20.3.0-5145.g5fdb7d8f.el9.x86_64.r 43 kB/s | 6.5 kB 00:00 2026-02-12T12:29:00.317 INFO:teuthology.orchestra.run.trial063.stdout:(2/96): ceph-mds-20.3.0-5145.g5fdb7d8f.el9.x86_ 4.8 MB/s | 2.0 MB 00:00 2026-02-12T12:29:00.351 INFO:teuthology.orchestra.run.trial063.stdout:(3/96): ceph-osd-20.3.0-5145.g5fdb7d8f.el9.x86_ 4.7 MB/s | 163 kB 00:00 2026-02-12T12:29:00.360 INFO:teuthology.orchestra.run.trial063.stdout:(4/96): ceph-mgr-20.3.0-5145.g5fdb7d8f.el9.x86_ 2.0 MB/s | 937 kB 00:00 2026-02-12T12:29:01.053 INFO:teuthology.orchestra.run.trial063.stdout:(5/96): ceph-osd-classic-20.3.0-5145.g5fdb7d8f. 21 MB/s | 15 MB 00:00 2026-02-12T12:29:01.098 INFO:teuthology.orchestra.run.trial063.stdout:(6/96): ceph-grafana-dashboards-20.3.0-5145.g5f 1.3 MB/s | 57 kB 00:00 2026-02-12T12:29:01.143 INFO:teuthology.orchestra.run.trial063.stdout:(7/96): ceph-mgr-cephadm-20.3.0-5145.g5fdb7d8f. 3.9 MB/s | 183 kB 00:00 2026-02-12T12:29:01.852 INFO:teuthology.orchestra.run.trial063.stdout:(8/96): ceph-mgr-dashboard-20.3.0-5145.g5fdb7d8 21 MB/s | 15 MB 00:00 2026-02-12T12:29:01.858 INFO:teuthology.orchestra.run.trial063.stdout:(9/96): ceph-mon-20.3.0-5145.g5fdb7d8f.el9.x86_ 2.7 MB/s | 4.8 MB 00:01 2026-02-12T12:29:01.917 INFO:teuthology.orchestra.run.trial063.stdout:(10/96): ceph-mgr-k8sevents-20.3.0-5145.g5fdb7d 380 kB/s | 22 kB 00:00 2026-02-12T12:29:02.026 INFO:teuthology.orchestra.run.trial063.stdout:(11/96): ceph-mgr-modules-core-20.3.0-5145.g5fd 2.7 MB/s | 297 kB 00:00 2026-02-12T12:29:02.078 INFO:teuthology.orchestra.run.trial063.stdout:(12/96): ceph-mgr-rook-20.3.0-5145.g5fdb7d8f.el 967 kB/s | 50 kB 00:00 2026-02-12T12:29:02.121 INFO:teuthology.orchestra.run.trial063.stdout:(13/96): ceph-prometheus-alerts-20.3.0-5145.g5f 409 kB/s | 17 kB 00:00 2026-02-12T12:29:02.158 INFO:teuthology.orchestra.run.trial063.stdout:(14/96): ceph-mgr-diskprediction-local-20.3.0-5 24 MB/s | 7.4 MB 00:00 2026-02-12T12:29:02.209 INFO:teuthology.orchestra.run.trial063.stdout:(15/96): cephadm-20.3.0-5145.g5fdb7d8f.el9.noar 20 MB/s | 1.0 MB 00:00 2026-02-12T12:29:02.210 INFO:teuthology.orchestra.run.trial063.stdout:(16/96): ceph-volume-20.3.0-5145.g5fdb7d8f.el9. 3.3 MB/s | 299 kB 00:00 2026-02-12T12:29:02.275 INFO:teuthology.orchestra.run.trial063.stdout:(17/96): ledmon-libs-1.1.0-3.el9.x86_64.rpm 629 kB/s | 40 kB 00:00 2026-02-12T12:29:02.297 INFO:teuthology.orchestra.run.trial063.stdout:(18/96): libconfig-1.7.2-9.el9.x86_64.rpm 3.2 MB/s | 72 kB 00:00 2026-02-12T12:29:02.316 INFO:teuthology.orchestra.run.trial063.stdout:(19/96): cryptsetup-2.8.1-2.el9.x86_64.rpm 3.2 MB/s | 350 kB 00:00 2026-02-12T12:29:02.340 INFO:teuthology.orchestra.run.trial063.stdout:(20/96): libquadmath-11.5.0-14.el9.x86_64.rpm 7.6 MB/s | 184 kB 00:00 2026-02-12T12:29:02.364 INFO:teuthology.orchestra.run.trial063.stdout:(21/96): libgfortran-11.5.0-14.el9.x86_64.rpm 12 MB/s | 794 kB 00:00 2026-02-12T12:29:02.367 INFO:teuthology.orchestra.run.trial063.stdout:(22/96): python3-cffi-1.14.5-5.el9.x86_64.rpm 9.2 MB/s | 253 kB 00:00 2026-02-12T12:29:02.390 INFO:teuthology.orchestra.run.trial063.stdout:(23/96): python3-ply-3.11-14.el9.noarch.rpm 4.4 MB/s | 106 kB 00:00 2026-02-12T12:29:02.411 INFO:teuthology.orchestra.run.trial063.stdout:(24/96): python3-cryptography-36.0.1-5.el9.x86_ 26 MB/s | 1.2 MB 00:00 2026-02-12T12:29:02.414 INFO:teuthology.orchestra.run.trial063.stdout:(25/96): python3-pycparser-2.20-6.el9.noarch.rp 5.5 MB/s | 135 kB 00:00 2026-02-12T12:29:02.435 INFO:teuthology.orchestra.run.trial063.stdout:(26/96): python3-pyparsing-2.4.7-9.el9.noarch.r 6.1 MB/s | 150 kB 00:00 2026-02-12T12:29:02.438 INFO:teuthology.orchestra.run.trial063.stdout:(27/96): python3-requests-2.25.1-10.el9.noarch. 5.2 MB/s | 126 kB 00:00 2026-02-12T12:29:02.459 INFO:teuthology.orchestra.run.trial063.stdout:(28/96): python3-urllib3-1.26.5-7.el9.noarch.rp 8.8 MB/s | 218 kB 00:00 2026-02-12T12:29:02.462 INFO:teuthology.orchestra.run.trial063.stdout:(29/96): unzip-6.0-59.el9.x86_64.rpm 7.6 MB/s | 182 kB 00:00 2026-02-12T12:29:02.491 INFO:teuthology.orchestra.run.trial063.stdout:(30/96): zip-3.0-35.el9.x86_64.rpm 8.2 MB/s | 266 kB 00:00 2026-02-12T12:29:02.543 INFO:teuthology.orchestra.run.trial063.stdout:(31/96): flexiblas-3.0.4-8.el9.x86_64.rpm 404 kB/s | 33 kB 00:00 2026-02-12T12:29:02.575 INFO:teuthology.orchestra.run.trial063.stdout:(32/96): flexiblas-openblas-openmp-3.0.4-8.el9. 562 kB/s | 18 kB 00:00 2026-02-12T12:29:02.674 INFO:teuthology.orchestra.run.trial063.stdout:(33/96): libstoragemgmt-1.10.1-1.el9.x86_64.rpm 2.4 MB/s | 246 kB 00:00 2026-02-12T12:29:02.709 INFO:teuthology.orchestra.run.trial063.stdout:(34/96): lua-5.4.4-4.el9.x86_64.rpm 5.2 MB/s | 188 kB 00:00 2026-02-12T12:29:02.731 INFO:teuthology.orchestra.run.trial063.stdout:(35/96): flexiblas-netlib-3.0.4-8.el9.x86_64.rp 12 MB/s | 3.0 MB 00:00 2026-02-12T12:29:02.735 INFO:teuthology.orchestra.run.trial063.stdout:(36/96): openblas-0.3.29-1.el9.x86_64.rpm 1.6 MB/s | 42 kB 00:00 2026-02-12T12:29:02.799 INFO:teuthology.orchestra.run.trial063.stdout:(37/96): openblas-openmp-0.3.29-1.el9.x86_64.rp 78 MB/s | 5.3 MB 00:00 2026-02-12T12:29:02.827 INFO:teuthology.orchestra.run.trial063.stdout:(38/96): python3-devel-3.9.25-2.el9.x86_64.rpm 8.5 MB/s | 244 kB 00:00 2026-02-12T12:29:02.855 INFO:teuthology.orchestra.run.trial063.stdout:(39/96): python3-jinja2-2.11.3-8.el9.noarch.rpm 8.9 MB/s | 249 kB 00:00 2026-02-12T12:29:02.875 INFO:teuthology.orchestra.run.trial063.stdout:(40/96): python3-babel-2.9.1-2.el9.noarch.rpm 43 MB/s | 6.0 MB 00:00 2026-02-12T12:29:02.880 INFO:teuthology.orchestra.run.trial063.stdout:(41/96): python3-jmespath-1.0.1-1.el9.noarch.rp 1.8 MB/s | 48 kB 00:00 2026-02-12T12:29:02.902 INFO:teuthology.orchestra.run.trial063.stdout:(42/96): python3-libstoragemgmt-1.10.1-1.el9.x8 6.3 MB/s | 177 kB 00:00 2026-02-12T12:29:02.910 INFO:teuthology.orchestra.run.trial063.stdout:(43/96): python3-lxml-4.6.5-3.el9.x86_64.rpm 41 MB/s | 1.2 MB 00:00 2026-02-12T12:29:02.928 INFO:teuthology.orchestra.run.trial063.stdout:(44/96): python3-markupsafe-1.1.1-12.el9.x86_64 1.3 MB/s | 35 kB 00:00 2026-02-12T12:29:02.956 INFO:teuthology.orchestra.run.trial063.stdout:(45/96): python3-numpy-f2py-1.23.5-2.el9.x86_64 15 MB/s | 442 kB 00:00 2026-02-12T12:29:02.976 INFO:teuthology.orchestra.run.trial063.stdout:(46/96): python3-numpy-1.23.5-2.el9.x86_64.rpm 93 MB/s | 6.1 MB 00:00 2026-02-12T12:29:02.983 INFO:teuthology.orchestra.run.trial063.stdout:(47/96): python3-packaging-20.9-5.el9.noarch.rp 2.9 MB/s | 77 kB 00:00 2026-02-12T12:29:03.003 INFO:teuthology.orchestra.run.trial063.stdout:(48/96): python3-protobuf-3.14.0-17.el9.noarch. 9.4 MB/s | 267 kB 00:00 2026-02-12T12:29:03.009 INFO:teuthology.orchestra.run.trial063.stdout:(49/96): python3-pyasn1-0.4.8-6.el9.noarch.rpm 5.9 MB/s | 159 kB 00:00 2026-02-12T12:29:03.031 INFO:teuthology.orchestra.run.trial063.stdout:(50/96): python3-pyasn1-modules-0.4.8-6.el9.noa 9.9 MB/s | 279 kB 00:00 2026-02-12T12:29:03.035 INFO:teuthology.orchestra.run.trial063.stdout:(51/96): python3-requests-oauthlib-1.3.0-12.el9 2.0 MB/s | 54 kB 00:00 2026-02-12T12:29:03.061 INFO:teuthology.orchestra.run.trial063.stdout:(52/96): python3-toml-0.10.2-6.el9.noarch.rpm 1.6 MB/s | 42 kB 00:00 2026-02-12T12:29:03.087 INFO:teuthology.orchestra.run.trial063.stdout:(53/96): xmlsec1-1.2.29-13.el9.x86_64.rpm 7.2 MB/s | 189 kB 00:00 2026-02-12T12:29:03.113 INFO:teuthology.orchestra.run.trial063.stdout:(54/96): xmlsec1-openssl-1.2.29-13.el9.x86_64.r 3.4 MB/s | 90 kB 00:00 2026-02-12T12:29:03.221 INFO:teuthology.orchestra.run.trial063.stdout:(55/96): python3-scipy-1.9.3-2.el9.x86_64.rpm 102 MB/s | 19 MB 00:00 2026-02-12T12:29:03.221 INFO:teuthology.orchestra.run.trial063.stdout:(56/96): lua-devel-5.4.4-4.el9.x86_64.rpm 206 kB/s | 22 kB 00:00 2026-02-12T12:29:03.412 INFO:teuthology.orchestra.run.trial063.stdout:(57/96): protobuf-compiler-3.14.0-17.el9.x86_64 4.4 MB/s | 862 kB 00:00 2026-02-12T12:29:03.597 INFO:teuthology.orchestra.run.trial063.stdout:(58/96): grpc-data-1.46.7-10.el9.noarch.rpm 105 kB/s | 19 kB 00:00 2026-02-12T12:29:03.822 INFO:teuthology.orchestra.run.trial063.stdout:(59/96): luarocks-3.9.2-5.el9.noarch.rpm 672 kB/s | 151 kB 00:00 2026-02-12T12:29:03.844 INFO:teuthology.orchestra.run.trial063.stdout:(60/96): abseil-cpp-20211102.0-4.el9.x86_64.rpm 886 kB/s | 551 kB 00:00 2026-02-12T12:29:03.929 INFO:teuthology.orchestra.run.trial063.stdout:(61/96): python3-autocommand-2.2.2-8.el9.noarch 347 kB/s | 29 kB 00:00 2026-02-12T12:29:04.016 INFO:teuthology.orchestra.run.trial063.stdout:(62/96): python3-backports-tarfile-1.2.0-1.el9. 691 kB/s | 60 kB 00:00 2026-02-12T12:29:04.020 INFO:teuthology.orchestra.run.trial063.stdout:(63/96): python3-asyncssh-2.13.2-5.el9.noarch.r 2.7 MB/s | 548 kB 00:00 2026-02-12T12:29:04.086 INFO:teuthology.orchestra.run.trial063.stdout:(64/96): python3-cachetools-4.2.4-1.el9.noarch. 489 kB/s | 32 kB 00:00 2026-02-12T12:29:04.100 INFO:teuthology.orchestra.run.trial063.stdout:(65/96): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 517 kB/s | 43 kB 00:00 2026-02-12T12:29:04.151 INFO:teuthology.orchestra.run.trial063.stdout:(66/96): python3-certifi-2023.05.07-4.el9.noarc 216 kB/s | 14 kB 00:00 2026-02-12T12:29:04.197 INFO:teuthology.orchestra.run.trial063.stdout:(67/96): python3-cheroot-10.0.1-4.el9.noarch.rp 1.7 MB/s | 173 kB 00:00 2026-02-12T12:29:04.239 INFO:teuthology.orchestra.run.trial063.stdout:(68/96): python3-cherrypy-18.6.1-2.el9.noarch.r 4.0 MB/s | 358 kB 00:00 2026-02-12T12:29:04.303 INFO:teuthology.orchestra.run.trial063.stdout:(69/96): python3-google-auth-2.45.0-1.el9.noarc 2.4 MB/s | 254 kB 00:00 2026-02-12T12:29:04.398 INFO:teuthology.orchestra.run.trial063.stdout:(70/96): python3-grpcio-tools-1.46.7-10.el9.x86 1.5 MB/s | 144 kB 00:00 2026-02-12T12:29:04.430 INFO:teuthology.orchestra.run.trial063.stdout:(71/96): python3-grpcio-1.46.7-10.el9.x86_64.rp 11 MB/s | 2.0 MB 00:00 2026-02-12T12:29:04.493 INFO:teuthology.orchestra.run.trial063.stdout:(72/96): python3-isodate-0.6.1-3.el9.noarch.rpm 891 kB/s | 56 kB 00:00 2026-02-12T12:29:04.497 INFO:teuthology.orchestra.run.trial063.stdout:(73/96): python3-influxdb-5.3.1-1.el9.noarch.rp 1.4 MB/s | 139 kB 00:00 2026-02-12T12:29:04.555 INFO:teuthology.orchestra.run.trial063.stdout:(74/96): python3-jaraco-8.2.1-3.el9.noarch.rpm 172 kB/s | 11 kB 00:00 2026-02-12T12:29:04.595 INFO:teuthology.orchestra.run.trial063.stdout:(75/96): python3-jaraco-classes-3.2.1-5.el9.noa 181 kB/s | 18 kB 00:00 2026-02-12T12:29:04.618 INFO:teuthology.orchestra.run.trial063.stdout:(76/96): python3-jaraco-collections-3.0.0-8.el9 371 kB/s | 23 kB 00:00 2026-02-12T12:29:04.680 INFO:teuthology.orchestra.run.trial063.stdout:(77/96): python3-jaraco-functools-3.5.0-2.el9.n 313 kB/s | 19 kB 00:00 2026-02-12T12:29:04.683 INFO:teuthology.orchestra.run.trial063.stdout:(78/96): python3-jaraco-context-6.0.1-3.el9.noa 223 kB/s | 20 kB 00:00 2026-02-12T12:29:04.742 INFO:teuthology.orchestra.run.trial063.stdout:(79/96): python3-jaraco-text-4.0.0-2.el9.noarch 422 kB/s | 26 kB 00:00 2026-02-12T12:29:04.806 INFO:teuthology.orchestra.run.trial063.stdout:(80/96): python3-more-itertools-8.12.0-2.el9.no 1.2 MB/s | 79 kB 00:00 2026-02-12T12:29:04.870 INFO:teuthology.orchestra.run.trial063.stdout:(81/96): python3-msgpack-1.0.3-2.el9.x86_64.rpm 1.3 MB/s | 86 kB 00:00 2026-02-12T12:29:04.893 INFO:teuthology.orchestra.run.trial063.stdout:(82/96): python3-kubernetes-26.1.0-3.el9.noarch 4.9 MB/s | 1.0 MB 00:00 2026-02-12T12:29:04.933 INFO:teuthology.orchestra.run.trial063.stdout:(83/96): python3-natsort-7.1.1-5.el9.noarch.rpm 914 kB/s | 58 kB 00:00 2026-02-12T12:29:04.974 INFO:teuthology.orchestra.run.trial063.stdout:(84/96): python3-portend-3.1.0-2.el9.noarch.rpm 203 kB/s | 16 kB 00:00 2026-02-12T12:29:04.996 INFO:teuthology.orchestra.run.trial063.stdout:(85/96): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.4 MB/s | 90 kB 00:00 2026-02-12T12:29:05.056 INFO:teuthology.orchestra.run.trial063.stdout:(86/96): python3-repoze-lru-0.7-16.el9.noarch.r 378 kB/s | 31 kB 00:00 2026-02-12T12:29:05.061 INFO:teuthology.orchestra.run.trial063.stdout:(87/96): python3-routes-2.5.1-5.el9.noarch.rpm 2.8 MB/s | 188 kB 00:00 2026-02-12T12:29:05.126 INFO:teuthology.orchestra.run.trial063.stdout:(88/96): python3-saml-1.16.0-1.el9.noarch.rpm 1.9 MB/s | 125 kB 00:00 2026-02-12T12:29:05.138 INFO:teuthology.orchestra.run.trial063.stdout:(89/96): python3-rsa-4.9-2.el9.noarch.rpm 720 kB/s | 59 kB 00:00 2026-02-12T12:29:05.189 INFO:teuthology.orchestra.run.trial063.stdout:(90/96): python3-tempora-5.0.0-2.el9.noarch.rpm 570 kB/s | 36 kB 00:00 2026-02-12T12:29:05.221 INFO:teuthology.orchestra.run.trial063.stdout:(91/96): python3-typing-extensions-4.15.0-1.el9 1.0 MB/s | 86 kB 00:00 2026-02-12T12:29:05.253 INFO:teuthology.orchestra.run.trial063.stdout:(92/96): python3-websocket-client-1.2.3-2.el9.n 1.4 MB/s | 90 kB 00:00 2026-02-12T12:29:05.303 INFO:teuthology.orchestra.run.trial063.stdout:(93/96): python3-xmlsec-1.3.13-1.el9.x86_64.rpm 592 kB/s | 48 kB 00:00 2026-02-12T12:29:05.315 INFO:teuthology.orchestra.run.trial063.stdout:(94/96): python3-xmltodict-0.12.0-15.el9.noarch 356 kB/s | 22 kB 00:00 2026-02-12T12:29:05.385 INFO:teuthology.orchestra.run.trial063.stdout:(95/96): python3-zc-lockfile-2.0-10.el9.noarch. 244 kB/s | 20 kB 00:00 2026-02-12T12:29:07.311 INFO:teuthology.orchestra.run.trial063.stdout:(96/96): ceph-osd-crimson-20.3.0-5145.g5fdb7d8f 12 MB/s | 87 MB 00:06 2026-02-12T12:29:07.312 INFO:teuthology.orchestra.run.trial063.stdout:-------------------------------------------------------------------------------- 2026-02-12T12:29:07.312 INFO:teuthology.orchestra.run.trial063.stdout:Total 24 MB/s | 189 MB 00:08 2026-02-12T12:29:07.719 INFO:teuthology.orchestra.run.trial063.stdout:Running transaction check 2026-02-12T12:29:07.758 INFO:teuthology.orchestra.run.trial063.stdout:Transaction check succeeded. 2026-02-12T12:29:07.759 INFO:teuthology.orchestra.run.trial063.stdout:Running transaction test 2026-02-12T12:29:08.449 INFO:teuthology.orchestra.run.trial063.stdout:Transaction test succeeded. 2026-02-12T12:29:08.449 INFO:teuthology.orchestra.run.trial063.stdout:Running transaction 2026-02-12T12:29:09.170 INFO:teuthology.orchestra.run.trial063.stdout: Preparing : 1/1 2026-02-12T12:29:09.172 INFO:teuthology.orchestra.run.trial063.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/96 2026-02-12T12:29:09.176 INFO:teuthology.orchestra.run.trial063.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/96 2026-02-12T12:29:09.197 INFO:teuthology.orchestra.run.trial063.stdout: Installing : xmlsec1-1.2.29-13.el9.x86_64 3/96 2026-02-12T12:29:09.232 INFO:teuthology.orchestra.run.trial063.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 4/96 2026-02-12T12:29:09.238 INFO:teuthology.orchestra.run.trial063.stdout: Installing : python3-lxml-4.6.5-3.el9.x86_64 5/96 2026-02-12T12:29:09.241 INFO:teuthology.orchestra.run.trial063.stdout: Installing : lua-5.4.4-4.el9.x86_64 6/96 2026-02-12T12:29:09.246 INFO:teuthology.orchestra.run.trial063.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 7/96 2026-02-12T12:29:09.269 INFO:teuthology.orchestra.run.trial063.stdout: Installing : unzip-6.0-59.el9.x86_64 8/96 2026-02-12T12:29:09.282 INFO:teuthology.orchestra.run.trial063.stdout: Installing : python3-urllib3-1.26.5-7.el9.noarch 9/96 2026-02-12T12:29:09.285 INFO:teuthology.orchestra.run.trial063.stdout: Installing : python3-requests-2.25.1-10.el9.noarch 10/96 2026-02-12T12:29:09.291 INFO:teuthology.orchestra.run.trial063.stdout: Installing : libquadmath-11.5.0-14.el9.x86_64 11/96 2026-02-12T12:29:09.292 INFO:teuthology.orchestra.run.trial063.stdout: Installing : libgfortran-11.5.0-14.el9.x86_64 12/96 2026-02-12T12:29:09.304 INFO:teuthology.orchestra.run.trial063.stdout: Installing : ledmon-libs-1.1.0-3.el9.x86_64 13/96 2026-02-12T12:29:09.310 INFO:teuthology.orchestra.run.trial063.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 14/96 2026-02-12T12:29:09.334 INFO:teuthology.orchestra.run.trial063.stdout: Installing : zip-3.0-35.el9.x86_64 15/96 2026-02-12T12:29:09.337 INFO:teuthology.orchestra.run.trial063.stdout: Installing : luarocks-3.9.2-5.el9.noarch 16/96 2026-02-12T12:29:09.387 INFO:teuthology.orchestra.run.trial063.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 17/96 2026-02-12T12:29:09.400 INFO:teuthology.orchestra.run.trial063.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 18/96 2026-02-12T12:29:09.402 INFO:teuthology.orchestra.run.trial063.stdout: Installing : python3-rsa-4.9-2.el9.noarch 19/96 2026-02-12T12:29:09.407 INFO:teuthology.orchestra.run.trial063.stdout: Installing : xmlsec1-openssl-1.2.29-13.el9.x86_64 20/96 2026-02-12T12:29:09.412 INFO:teuthology.orchestra.run.trial063.stdout: Installing : python3-xmlsec-1.3.13-1.el9.x86_64 21/96 2026-02-12T12:29:09.416 INFO:teuthology.orchestra.run.trial063.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 22/96 2026-02-12T12:29:09.420 INFO:teuthology.orchestra.run.trial063.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 23/96 2026-02-12T12:29:09.434 INFO:teuthology.orchestra.run.trial063.stdout: Installing : python3-xmltodict-0.12.0-15.el9.noarch 24/96 2026-02-12T12:29:09.439 INFO:teuthology.orchestra.run.trial063.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 25/96 2026-02-12T12:29:09.444 INFO:teuthology.orchestra.run.trial063.stdout: Installing : python3-typing-extensions-4.15.0-1.el9.noarch 26/96 2026-02-12T12:29:09.455 INFO:teuthology.orchestra.run.trial063.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 27/96 2026-02-12T12:29:09.464 INFO:teuthology.orchestra.run.trial063.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 28/96 2026-02-12T12:29:09.471 INFO:teuthology.orchestra.run.trial063.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 29/96 2026-02-12T12:29:09.490 INFO:teuthology.orchestra.run.trial063.stdout: Installing : python3-msgpack-1.0.3-2.el9.x86_64 30/96 2026-02-12T12:29:09.501 INFO:teuthology.orchestra.run.trial063.stdout: Installing : python3-influxdb-5.3.1-1.el9.noarch 31/96 2026-02-12T12:29:09.517 INFO:teuthology.orchestra.run.trial063.stdout: Installing : python3-isodate-0.6.1-3.el9.noarch 32/96 2026-02-12T12:29:09.521 INFO:teuthology.orchestra.run.trial063.stdout: Installing : python3-saml-1.16.0-1.el9.noarch 33/96 2026-02-12T12:29:09.529 INFO:teuthology.orchestra.run.trial063.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 34/96 2026-02-12T12:29:09.569 INFO:teuthology.orchestra.run.trial063.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 35/96 2026-02-12T12:29:09.892 INFO:teuthology.orchestra.run.trial063.stdout: Installing : python3-google-auth-1:2.45.0-1.el9.noarch 36/96 2026-02-12T12:29:09.902 INFO:teuthology.orchestra.run.trial063.stdout: Installing : python3-kubernetes-1:26.1.0-3.el9.noarch 37/96 2026-02-12T12:29:09.905 INFO:teuthology.orchestra.run.trial063.stdout: Installing : python3-backports-tarfile-1.2.0-1.el9.noarch 38/96 2026-02-12T12:29:09.911 INFO:teuthology.orchestra.run.trial063.stdout: Installing : python3-jaraco-context-6.0.1-3.el9.noarch 39/96 2026-02-12T12:29:09.913 INFO:teuthology.orchestra.run.trial063.stdout: Installing : python3-autocommand-2.2.2-8.el9.noarch 40/96 2026-02-12T12:29:09.940 INFO:teuthology.orchestra.run.trial063.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 41/96 2026-02-12T12:29:09.984 INFO:teuthology.orchestra.run.trial063.stdout: Installing : abseil-cpp-20211102.0-4.el9.x86_64 42/96 2026-02-12T12:29:09.992 INFO:teuthology.orchestra.run.trial063.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 43/96 2026-02-12T12:29:09.998 INFO:teuthology.orchestra.run.trial063.stdout: Installing : protobuf-compiler-3.14.0-17.el9.x86_64 44/96 2026-02-12T12:29:10.001 INFO:teuthology.orchestra.run.trial063.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 45/96 2026-02-12T12:29:10.007 INFO:teuthology.orchestra.run.trial063.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 46/96 2026-02-12T12:29:10.011 INFO:teuthology.orchestra.run.trial063.stdout: Installing : python3-jaraco-text-4.0.0-2.el9.noarch 47/96 2026-02-12T12:29:10.017 INFO:teuthology.orchestra.run.trial063.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 48/96 2026-02-12T12:29:10.021 INFO:teuthology.orchestra.run.trial063.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 49/96 2026-02-12T12:29:10.049 INFO:teuthology.orchestra.run.trial063.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 50/96 2026-02-12T12:29:10.059 INFO:teuthology.orchestra.run.trial063.stdout: Installing : python3-protobuf-3.14.0-17.el9.noarch 51/96 2026-02-12T12:29:10.064 INFO:teuthology.orchestra.run.trial063.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 52/96 2026-02-12T12:29:10.071 INFO:teuthology.orchestra.run.trial063.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 53/96 2026-02-12T12:29:10.106 INFO:teuthology.orchestra.run.trial063.stdout: Installing : python3-jmespath-1.0.1-1.el9.noarch 54/96 2026-02-12T12:29:10.329 INFO:teuthology.orchestra.run.trial063.stdout: Installing : python3-devel-3.9.25-2.el9.x86_64 55/96 2026-02-12T12:29:10.352 INFO:teuthology.orchestra.run.trial063.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 56/96 2026-02-12T12:29:10.356 INFO:teuthology.orchestra.run.trial063.stdout: Installing : python3-jinja2-2.11.3-8.el9.noarch 57/96 2026-02-12T12:29:10.414 INFO:teuthology.orchestra.run.trial063.stdout: Installing : openblas-0.3.29-1.el9.x86_64 58/96 2026-02-12T12:29:10.416 INFO:teuthology.orchestra.run.trial063.stdout: Installing : openblas-openmp-0.3.29-1.el9.x86_64 59/96 2026-02-12T12:29:10.438 INFO:teuthology.orchestra.run.trial063.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 60/96 2026-02-12T12:29:10.762 INFO:teuthology.orchestra.run.trial063.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 61/96 2026-02-12T12:29:10.835 INFO:teuthology.orchestra.run.trial063.stdout: Installing : python3-numpy-1:1.23.5-2.el9.x86_64 62/96 2026-02-12T12:29:11.505 INFO:teuthology.orchestra.run.trial063.stdout: Installing : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 63/96 2026-02-12T12:29:11.522 INFO:teuthology.orchestra.run.trial063.stdout: Installing : python3-scipy-1.9.3-2.el9.x86_64 64/96 2026-02-12T12:29:11.532 INFO:teuthology.orchestra.run.trial063.stdout: Installing : python3-pyparsing-2.4.7-9.el9.noarch 65/96 2026-02-12T12:29:11.538 INFO:teuthology.orchestra.run.trial063.stdout: Installing : python3-packaging-20.9-5.el9.noarch 66/96 2026-02-12T12:29:11.552 INFO:teuthology.orchestra.run.trial063.stdout: Installing : python3-ply-3.11-14.el9.noarch 67/96 2026-02-12T12:29:11.567 INFO:teuthology.orchestra.run.trial063.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 68/96 2026-02-12T12:29:11.642 INFO:teuthology.orchestra.run.trial063.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 69/96 2026-02-12T12:29:11.651 INFO:teuthology.orchestra.run.trial063.stdout: Installing : python3-cryptography-36.0.1-5.el9.x86_64 70/96 2026-02-12T12:29:11.675 INFO:teuthology.orchestra.run.trial063.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 71/96 2026-02-12T12:29:11.706 INFO:teuthology.orchestra.run.trial063.stdout: Installing : python3-cheroot-10.0.1-4.el9.noarch 72/96 2026-02-12T12:29:11.758 INFO:teuthology.orchestra.run.trial063.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 73/96 2026-02-12T12:29:11.764 INFO:teuthology.orchestra.run.trial063.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 74/96 2026-02-12T12:29:11.767 INFO:teuthology.orchestra.run.trial063.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 75/96 2026-02-12T12:29:11.769 INFO:teuthology.orchestra.run.trial063.stdout: Installing : libconfig-1.7.2-9.el9.x86_64 76/96 2026-02-12T12:29:11.787 INFO:teuthology.orchestra.run.trial063.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-12T12:29:11.787 INFO:teuthology.orchestra.run.trial063.stdout:Creating group 'libstoragemgmt' with GID 991. 2026-02-12T12:29:11.787 INFO:teuthology.orchestra.run.trial063.stdout:Creating user 'libstoragemgmt' (daemon account for libstoragemgmt) with UID 991 and GID 991. 2026-02-12T12:29:11.787 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:29:11.796 INFO:teuthology.orchestra.run.trial063.stdout: Installing : libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-12T12:29:11.821 INFO:teuthology.orchestra.run.trial063.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-12T12:29:11.822 INFO:teuthology.orchestra.run.trial063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/libstoragemgmt.service → /usr/lib/systemd/system/libstoragemgmt.service. 2026-02-12T12:29:11.822 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:29:11.859 INFO:teuthology.orchestra.run.trial063.stdout: Installing : python3-libstoragemgmt-1.10.1-1.el9.x86_64 78/96 2026-02-12T12:29:12.448 INFO:teuthology.orchestra.run.trial063.stdout: Installing : cryptsetup-2.8.1-2.el9.x86_64 79/96 2026-02-12T12:29:17.786 INFO:teuthology.orchestra.run.trial063.stdout: Installing : ceph-osd-classic-2:20.3.0-5145.g5fdb7d8f.el9.x86_6 80/96 2026-02-12T12:29:17.850 INFO:teuthology.orchestra.run.trial063.stdout: Installing : ceph-osd-crimson-2:20.3.0-5145.g5fdb7d8f.el9.x86_6 81/96 2026-02-12T12:29:17.852 INFO:teuthology.orchestra.run.trial063.stdout: Installing : ceph-volume-2:20.3.0-5145.g5fdb7d8f.el9.noarch 82/96 2026-02-12T12:29:17.867 INFO:teuthology.orchestra.run.trial063.stdout: Running scriptlet: ceph-volume-2:20.3.0-5145.g5fdb7d8f.el9.noarch 82/96 2026-02-12T12:29:17.867 INFO:teuthology.orchestra.run.trial063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-12T12:29:17.867 INFO:teuthology.orchestra.run.trial063.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2026-02-12T12:29:17.867 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:29:17.878 INFO:teuthology.orchestra.run.trial063.stdout: Installing : ceph-osd-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 83/96 2026-02-12T12:29:17.901 INFO:teuthology.orchestra.run.trial063.stdout: Running scriptlet: ceph-osd-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 83/96 2026-02-12T12:29:17.901 INFO:teuthology.orchestra.run.trial063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-12T12:29:17.901 INFO:teuthology.orchestra.run.trial063.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2026-02-12T12:29:17.901 INFO:teuthology.orchestra.run.trial063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-12T12:29:17.902 INFO:teuthology.orchestra.run.trial063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-12T12:29:17.902 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:29:17.961 INFO:teuthology.orchestra.run.trial063.stdout: Running scriptlet: cephadm-2:20.3.0-5145.g5fdb7d8f.el9.noarch 84/96 2026-02-12T12:29:17.963 INFO:teuthology.orchestra.run.trial063.stdout: Installing : cephadm-2:20.3.0-5145.g5fdb7d8f.el9.noarch 84/96 2026-02-12T12:29:17.972 INFO:teuthology.orchestra.run.trial063.stdout: Installing : ceph-prometheus-alerts-2:20.3.0-5145.g5fdb7d8f.el9 85/96 2026-02-12T12:29:17.995 INFO:teuthology.orchestra.run.trial063.stdout: Installing : ceph-grafana-dashboards-2:20.3.0-5145.g5fdb7d8f.el 86/96 2026-02-12T12:29:17.997 INFO:teuthology.orchestra.run.trial063.stdout: Installing : ceph-mgr-cephadm-2:20.3.0-5145.g5fdb7d8f.el9.noarc 87/96 2026-02-12T12:29:19.568 INFO:teuthology.orchestra.run.trial063.stdout: Running scriptlet: ceph-mgr-cephadm-2:20.3.0-5145.g5fdb7d8f.el9.noarc 87/96 2026-02-12T12:29:19.575 INFO:teuthology.orchestra.run.trial063.stdout: Installing : ceph-mgr-dashboard-2:20.3.0-5145.g5fdb7d8f.el9.noa 88/96 2026-02-12T12:29:20.029 INFO:teuthology.orchestra.run.trial063.stdout: Running scriptlet: ceph-mgr-dashboard-2:20.3.0-5145.g5fdb7d8f.el9.noa 88/96 2026-02-12T12:29:20.030 INFO:teuthology.orchestra.run.trial063.stdout: Installing : ceph-mgr-diskprediction-local-2:20.3.0-5145.g5fdb7 89/96 2026-02-12T12:29:20.043 INFO:teuthology.orchestra.run.trial063.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:20.3.0-5145.g5fdb7 89/96 2026-02-12T12:29:20.044 INFO:teuthology.orchestra.run.trial063.stdout: Installing : ceph-mgr-k8sevents-2:20.3.0-5145.g5fdb7d8f.el9.noa 90/96 2026-02-12T12:29:20.105 INFO:teuthology.orchestra.run.trial063.stdout: Running scriptlet: ceph-mgr-k8sevents-2:20.3.0-5145.g5fdb7d8f.el9.noa 90/96 2026-02-12T12:29:20.146 INFO:teuthology.orchestra.run.trial063.stdout: Installing : ceph-mgr-modules-core-2:20.3.0-5145.g5fdb7d8f.el9. 91/96 2026-02-12T12:29:20.147 INFO:teuthology.orchestra.run.trial063.stdout: Installing : ceph-mgr-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 92/96 2026-02-12T12:29:20.167 INFO:teuthology.orchestra.run.trial063.stdout: Running scriptlet: ceph-mgr-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 92/96 2026-02-12T12:29:20.168 INFO:teuthology.orchestra.run.trial063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-12T12:29:20.168 INFO:teuthology.orchestra.run.trial063.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2026-02-12T12:29:20.168 INFO:teuthology.orchestra.run.trial063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-12T12:29:20.168 INFO:teuthology.orchestra.run.trial063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-12T12:29:20.168 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:29:20.180 INFO:teuthology.orchestra.run.trial063.stdout: Installing : ceph-mgr-rook-2:20.3.0-5145.g5fdb7d8f.el9.noarch 93/96 2026-02-12T12:29:20.394 INFO:teuthology.orchestra.run.trial063.stdout: Running scriptlet: ceph-mgr-rook-2:20.3.0-5145.g5fdb7d8f.el9.noarch 93/96 2026-02-12T12:29:20.395 INFO:teuthology.orchestra.run.trial063.stdout: Installing : ceph-mon-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 94/96 2026-02-12T12:29:20.416 INFO:teuthology.orchestra.run.trial063.stdout: Running scriptlet: ceph-mon-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 94/96 2026-02-12T12:29:20.417 INFO:teuthology.orchestra.run.trial063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-12T12:29:20.417 INFO:teuthology.orchestra.run.trial063.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2026-02-12T12:29:20.417 INFO:teuthology.orchestra.run.trial063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-12T12:29:20.417 INFO:teuthology.orchestra.run.trial063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-12T12:29:20.417 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:29:20.500 INFO:teuthology.orchestra.run.trial063.stdout: Installing : ceph-mds-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 95/96 2026-02-12T12:29:20.520 INFO:teuthology.orchestra.run.trial063.stdout: Running scriptlet: ceph-mds-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 95/96 2026-02-12T12:29:20.520 INFO:teuthology.orchestra.run.trial063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-12T12:29:20.520 INFO:teuthology.orchestra.run.trial063.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2026-02-12T12:29:20.520 INFO:teuthology.orchestra.run.trial063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-12T12:29:20.520 INFO:teuthology.orchestra.run.trial063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-12T12:29:20.520 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:29:20.521 INFO:teuthology.orchestra.run.trial063.stdout: Installing : ceph-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 96/96 2026-02-12T12:29:20.528 INFO:teuthology.orchestra.run.trial063.stdout: Running scriptlet: ceph-osd-classic-2:20.3.0-5145.g5fdb7d8f.el9.x86_6 96/96 2026-02-12T12:29:20.552 INFO:teuthology.orchestra.run.trial063.stdout: Running scriptlet: ceph-osd-crimson-2:20.3.0-5145.g5fdb7d8f.el9.x86_6 96/96 2026-02-12T12:29:21.862 INFO:teuthology.orchestra.run.trial063.stdout: Running scriptlet: ceph-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 96/96 2026-02-12T12:29:21.862 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : ceph-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 1/96 2026-02-12T12:29:21.862 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : ceph-mds-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2/96 2026-02-12T12:29:21.862 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : ceph-mgr-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 3/96 2026-02-12T12:29:21.862 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : ceph-mon-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 4/96 2026-02-12T12:29:21.863 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : ceph-osd-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 5/96 2026-02-12T12:29:21.863 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : ceph-osd-classic-2:20.3.0-5145.g5fdb7d8f.el9.x86_6 6/96 2026-02-12T12:29:21.863 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : ceph-osd-crimson-2:20.3.0-5145.g5fdb7d8f.el9.x86_6 7/96 2026-02-12T12:29:21.863 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : ceph-grafana-dashboards-2:20.3.0-5145.g5fdb7d8f.el 8/96 2026-02-12T12:29:21.863 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : ceph-mgr-cephadm-2:20.3.0-5145.g5fdb7d8f.el9.noarc 9/96 2026-02-12T12:29:21.863 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : ceph-mgr-dashboard-2:20.3.0-5145.g5fdb7d8f.el9.noa 10/96 2026-02-12T12:29:21.863 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : ceph-mgr-diskprediction-local-2:20.3.0-5145.g5fdb7 11/96 2026-02-12T12:29:21.863 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : ceph-mgr-k8sevents-2:20.3.0-5145.g5fdb7d8f.el9.noa 12/96 2026-02-12T12:29:21.863 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : ceph-mgr-modules-core-2:20.3.0-5145.g5fdb7d8f.el9. 13/96 2026-02-12T12:29:21.863 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : ceph-mgr-rook-2:20.3.0-5145.g5fdb7d8f.el9.noarch 14/96 2026-02-12T12:29:21.863 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : ceph-prometheus-alerts-2:20.3.0-5145.g5fdb7d8f.el9 15/96 2026-02-12T12:29:21.863 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : ceph-volume-2:20.3.0-5145.g5fdb7d8f.el9.noarch 16/96 2026-02-12T12:29:21.864 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : cephadm-2:20.3.0-5145.g5fdb7d8f.el9.noarch 17/96 2026-02-12T12:29:21.864 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : cryptsetup-2.8.1-2.el9.x86_64 18/96 2026-02-12T12:29:21.864 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : ledmon-libs-1.1.0-3.el9.x86_64 19/96 2026-02-12T12:29:21.864 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : libconfig-1.7.2-9.el9.x86_64 20/96 2026-02-12T12:29:21.864 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : libgfortran-11.5.0-14.el9.x86_64 21/96 2026-02-12T12:29:21.864 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : libquadmath-11.5.0-14.el9.x86_64 22/96 2026-02-12T12:29:21.864 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 23/96 2026-02-12T12:29:21.864 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : python3-cryptography-36.0.1-5.el9.x86_64 24/96 2026-02-12T12:29:21.864 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : python3-ply-3.11-14.el9.noarch 25/96 2026-02-12T12:29:21.864 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 26/96 2026-02-12T12:29:21.864 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : python3-pyparsing-2.4.7-9.el9.noarch 27/96 2026-02-12T12:29:21.864 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : python3-requests-2.25.1-10.el9.noarch 28/96 2026-02-12T12:29:21.864 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : python3-urllib3-1.26.5-7.el9.noarch 29/96 2026-02-12T12:29:21.865 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : unzip-6.0-59.el9.x86_64 30/96 2026-02-12T12:29:21.865 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : zip-3.0-35.el9.x86_64 31/96 2026-02-12T12:29:21.865 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 32/96 2026-02-12T12:29:21.865 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 33/96 2026-02-12T12:29:21.865 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 34/96 2026-02-12T12:29:21.865 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : libstoragemgmt-1.10.1-1.el9.x86_64 35/96 2026-02-12T12:29:21.865 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : lua-5.4.4-4.el9.x86_64 36/96 2026-02-12T12:29:21.865 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : openblas-0.3.29-1.el9.x86_64 37/96 2026-02-12T12:29:21.865 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : openblas-openmp-0.3.29-1.el9.x86_64 38/96 2026-02-12T12:29:21.865 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 39/96 2026-02-12T12:29:21.865 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : python3-devel-3.9.25-2.el9.x86_64 40/96 2026-02-12T12:29:21.865 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : python3-jinja2-2.11.3-8.el9.noarch 41/96 2026-02-12T12:29:21.866 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : python3-jmespath-1.0.1-1.el9.noarch 42/96 2026-02-12T12:29:21.866 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : python3-libstoragemgmt-1.10.1-1.el9.x86_64 43/96 2026-02-12T12:29:21.866 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : python3-lxml-4.6.5-3.el9.x86_64 44/96 2026-02-12T12:29:21.866 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 45/96 2026-02-12T12:29:21.866 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : python3-numpy-1:1.23.5-2.el9.x86_64 46/96 2026-02-12T12:29:21.866 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 47/96 2026-02-12T12:29:21.866 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 48/96 2026-02-12T12:29:21.866 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : python3-protobuf-3.14.0-17.el9.noarch 49/96 2026-02-12T12:29:21.866 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 50/96 2026-02-12T12:29:21.866 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 51/96 2026-02-12T12:29:21.866 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 52/96 2026-02-12T12:29:21.866 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : python3-scipy-1.9.3-2.el9.x86_64 53/96 2026-02-12T12:29:21.866 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 54/96 2026-02-12T12:29:21.867 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : xmlsec1-1.2.29-13.el9.x86_64 55/96 2026-02-12T12:29:21.867 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : xmlsec1-openssl-1.2.29-13.el9.x86_64 56/96 2026-02-12T12:29:21.867 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 57/96 2026-02-12T12:29:21.867 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : protobuf-compiler-3.14.0-17.el9.x86_64 58/96 2026-02-12T12:29:21.867 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : abseil-cpp-20211102.0-4.el9.x86_64 59/96 2026-02-12T12:29:21.867 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 60/96 2026-02-12T12:29:21.867 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 61/96 2026-02-12T12:29:21.867 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 62/96 2026-02-12T12:29:21.867 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : python3-autocommand-2.2.2-8.el9.noarch 63/96 2026-02-12T12:29:21.867 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : python3-backports-tarfile-1.2.0-1.el9.noarch 64/96 2026-02-12T12:29:21.867 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 65/96 2026-02-12T12:29:21.867 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 66/96 2026-02-12T12:29:21.867 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 67/96 2026-02-12T12:29:21.868 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : python3-cheroot-10.0.1-4.el9.noarch 68/96 2026-02-12T12:29:21.868 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 69/96 2026-02-12T12:29:21.868 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : python3-google-auth-1:2.45.0-1.el9.noarch 70/96 2026-02-12T12:29:21.868 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 71/96 2026-02-12T12:29:21.868 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 72/96 2026-02-12T12:29:21.868 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : python3-influxdb-5.3.1-1.el9.noarch 73/96 2026-02-12T12:29:21.868 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : python3-isodate-0.6.1-3.el9.noarch 74/96 2026-02-12T12:29:21.868 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 75/96 2026-02-12T12:29:21.868 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 76/96 2026-02-12T12:29:21.868 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 77/96 2026-02-12T12:29:21.868 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : python3-jaraco-context-6.0.1-3.el9.noarch 78/96 2026-02-12T12:29:21.868 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 79/96 2026-02-12T12:29:21.868 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : python3-jaraco-text-4.0.0-2.el9.noarch 80/96 2026-02-12T12:29:21.868 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : python3-kubernetes-1:26.1.0-3.el9.noarch 81/96 2026-02-12T12:29:21.869 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 82/96 2026-02-12T12:29:21.869 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : python3-msgpack-1.0.3-2.el9.x86_64 83/96 2026-02-12T12:29:21.869 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 84/96 2026-02-12T12:29:21.869 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 85/96 2026-02-12T12:29:21.869 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/96 2026-02-12T12:29:21.869 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 87/96 2026-02-12T12:29:21.869 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 88/96 2026-02-12T12:29:21.869 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 89/96 2026-02-12T12:29:21.869 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : python3-saml-1.16.0-1.el9.noarch 90/96 2026-02-12T12:29:21.869 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 91/96 2026-02-12T12:29:21.869 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : python3-typing-extensions-4.15.0-1.el9.noarch 92/96 2026-02-12T12:29:21.869 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 93/96 2026-02-12T12:29:21.869 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : python3-xmlsec-1.3.13-1.el9.x86_64 94/96 2026-02-12T12:29:21.870 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : python3-xmltodict-0.12.0-15.el9.noarch 95/96 2026-02-12T12:29:21.925 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 96/96 2026-02-12T12:29:21.926 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:29:21.926 INFO:teuthology.orchestra.run.trial063.stdout:Installed: 2026-02-12T12:29:21.926 INFO:teuthology.orchestra.run.trial063.stdout: abseil-cpp-20211102.0-4.el9.x86_64 2026-02-12T12:29:21.926 INFO:teuthology.orchestra.run.trial063.stdout: ceph-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:29:21.926 INFO:teuthology.orchestra.run.trial063.stdout: ceph-grafana-dashboards-2:20.3.0-5145.g5fdb7d8f.el9.noarch 2026-02-12T12:29:21.926 INFO:teuthology.orchestra.run.trial063.stdout: ceph-mds-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:29:21.926 INFO:teuthology.orchestra.run.trial063.stdout: ceph-mgr-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:29:21.926 INFO:teuthology.orchestra.run.trial063.stdout: ceph-mgr-cephadm-2:20.3.0-5145.g5fdb7d8f.el9.noarch 2026-02-12T12:29:21.926 INFO:teuthology.orchestra.run.trial063.stdout: ceph-mgr-dashboard-2:20.3.0-5145.g5fdb7d8f.el9.noarch 2026-02-12T12:29:21.926 INFO:teuthology.orchestra.run.trial063.stdout: ceph-mgr-diskprediction-local-2:20.3.0-5145.g5fdb7d8f.el9.noarch 2026-02-12T12:29:21.927 INFO:teuthology.orchestra.run.trial063.stdout: ceph-mgr-k8sevents-2:20.3.0-5145.g5fdb7d8f.el9.noarch 2026-02-12T12:29:21.927 INFO:teuthology.orchestra.run.trial063.stdout: ceph-mgr-modules-core-2:20.3.0-5145.g5fdb7d8f.el9.noarch 2026-02-12T12:29:21.927 INFO:teuthology.orchestra.run.trial063.stdout: ceph-mgr-rook-2:20.3.0-5145.g5fdb7d8f.el9.noarch 2026-02-12T12:29:21.927 INFO:teuthology.orchestra.run.trial063.stdout: ceph-mon-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:29:21.927 INFO:teuthology.orchestra.run.trial063.stdout: ceph-osd-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:29:21.927 INFO:teuthology.orchestra.run.trial063.stdout: ceph-osd-classic-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:29:21.927 INFO:teuthology.orchestra.run.trial063.stdout: ceph-osd-crimson-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:29:21.927 INFO:teuthology.orchestra.run.trial063.stdout: ceph-prometheus-alerts-2:20.3.0-5145.g5fdb7d8f.el9.noarch 2026-02-12T12:29:21.927 INFO:teuthology.orchestra.run.trial063.stdout: ceph-volume-2:20.3.0-5145.g5fdb7d8f.el9.noarch 2026-02-12T12:29:21.927 INFO:teuthology.orchestra.run.trial063.stdout: cephadm-2:20.3.0-5145.g5fdb7d8f.el9.noarch 2026-02-12T12:29:21.927 INFO:teuthology.orchestra.run.trial063.stdout: cryptsetup-2.8.1-2.el9.x86_64 2026-02-12T12:29:21.927 INFO:teuthology.orchestra.run.trial063.stdout: flexiblas-3.0.4-8.el9.x86_64 2026-02-12T12:29:21.927 INFO:teuthology.orchestra.run.trial063.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2026-02-12T12:29:21.928 INFO:teuthology.orchestra.run.trial063.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2026-02-12T12:29:21.928 INFO:teuthology.orchestra.run.trial063.stdout: grpc-data-1.46.7-10.el9.noarch 2026-02-12T12:29:21.928 INFO:teuthology.orchestra.run.trial063.stdout: ledmon-libs-1.1.0-3.el9.x86_64 2026-02-12T12:29:21.928 INFO:teuthology.orchestra.run.trial063.stdout: libconfig-1.7.2-9.el9.x86_64 2026-02-12T12:29:21.928 INFO:teuthology.orchestra.run.trial063.stdout: libgfortran-11.5.0-14.el9.x86_64 2026-02-12T12:29:21.928 INFO:teuthology.orchestra.run.trial063.stdout: libquadmath-11.5.0-14.el9.x86_64 2026-02-12T12:29:21.928 INFO:teuthology.orchestra.run.trial063.stdout: libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-12T12:29:21.928 INFO:teuthology.orchestra.run.trial063.stdout: lua-5.4.4-4.el9.x86_64 2026-02-12T12:29:21.928 INFO:teuthology.orchestra.run.trial063.stdout: lua-devel-5.4.4-4.el9.x86_64 2026-02-12T12:29:21.928 INFO:teuthology.orchestra.run.trial063.stdout: luarocks-3.9.2-5.el9.noarch 2026-02-12T12:29:21.928 INFO:teuthology.orchestra.run.trial063.stdout: openblas-0.3.29-1.el9.x86_64 2026-02-12T12:29:21.928 INFO:teuthology.orchestra.run.trial063.stdout: openblas-openmp-0.3.29-1.el9.x86_64 2026-02-12T12:29:21.928 INFO:teuthology.orchestra.run.trial063.stdout: protobuf-compiler-3.14.0-17.el9.x86_64 2026-02-12T12:29:21.929 INFO:teuthology.orchestra.run.trial063.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2026-02-12T12:29:21.929 INFO:teuthology.orchestra.run.trial063.stdout: python3-autocommand-2.2.2-8.el9.noarch 2026-02-12T12:29:21.929 INFO:teuthology.orchestra.run.trial063.stdout: python3-babel-2.9.1-2.el9.noarch 2026-02-12T12:29:21.929 INFO:teuthology.orchestra.run.trial063.stdout: python3-backports-tarfile-1.2.0-1.el9.noarch 2026-02-12T12:29:21.929 INFO:teuthology.orchestra.run.trial063.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2026-02-12T12:29:21.929 INFO:teuthology.orchestra.run.trial063.stdout: python3-cachetools-4.2.4-1.el9.noarch 2026-02-12T12:29:21.929 INFO:teuthology.orchestra.run.trial063.stdout: python3-certifi-2023.05.07-4.el9.noarch 2026-02-12T12:29:21.929 INFO:teuthology.orchestra.run.trial063.stdout: python3-cffi-1.14.5-5.el9.x86_64 2026-02-12T12:29:21.929 INFO:teuthology.orchestra.run.trial063.stdout: python3-cheroot-10.0.1-4.el9.noarch 2026-02-12T12:29:21.929 INFO:teuthology.orchestra.run.trial063.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2026-02-12T12:29:21.929 INFO:teuthology.orchestra.run.trial063.stdout: python3-cryptography-36.0.1-5.el9.x86_64 2026-02-12T12:29:21.929 INFO:teuthology.orchestra.run.trial063.stdout: python3-devel-3.9.25-2.el9.x86_64 2026-02-12T12:29:21.930 INFO:teuthology.orchestra.run.trial063.stdout: python3-google-auth-1:2.45.0-1.el9.noarch 2026-02-12T12:29:21.930 INFO:teuthology.orchestra.run.trial063.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2026-02-12T12:29:21.930 INFO:teuthology.orchestra.run.trial063.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2026-02-12T12:29:21.930 INFO:teuthology.orchestra.run.trial063.stdout: python3-influxdb-5.3.1-1.el9.noarch 2026-02-12T12:29:21.930 INFO:teuthology.orchestra.run.trial063.stdout: python3-isodate-0.6.1-3.el9.noarch 2026-02-12T12:29:21.930 INFO:teuthology.orchestra.run.trial063.stdout: python3-jaraco-8.2.1-3.el9.noarch 2026-02-12T12:29:21.930 INFO:teuthology.orchestra.run.trial063.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2026-02-12T12:29:21.930 INFO:teuthology.orchestra.run.trial063.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2026-02-12T12:29:21.930 INFO:teuthology.orchestra.run.trial063.stdout: python3-jaraco-context-6.0.1-3.el9.noarch 2026-02-12T12:29:21.930 INFO:teuthology.orchestra.run.trial063.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2026-02-12T12:29:21.930 INFO:teuthology.orchestra.run.trial063.stdout: python3-jaraco-text-4.0.0-2.el9.noarch 2026-02-12T12:29:21.930 INFO:teuthology.orchestra.run.trial063.stdout: python3-jinja2-2.11.3-8.el9.noarch 2026-02-12T12:29:21.930 INFO:teuthology.orchestra.run.trial063.stdout: python3-jmespath-1.0.1-1.el9.noarch 2026-02-12T12:29:21.931 INFO:teuthology.orchestra.run.trial063.stdout: python3-kubernetes-1:26.1.0-3.el9.noarch 2026-02-12T12:29:21.931 INFO:teuthology.orchestra.run.trial063.stdout: python3-libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-12T12:29:21.931 INFO:teuthology.orchestra.run.trial063.stdout: python3-lxml-4.6.5-3.el9.x86_64 2026-02-12T12:29:21.931 INFO:teuthology.orchestra.run.trial063.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2026-02-12T12:29:21.931 INFO:teuthology.orchestra.run.trial063.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2026-02-12T12:29:21.931 INFO:teuthology.orchestra.run.trial063.stdout: python3-msgpack-1.0.3-2.el9.x86_64 2026-02-12T12:29:21.931 INFO:teuthology.orchestra.run.trial063.stdout: python3-natsort-7.1.1-5.el9.noarch 2026-02-12T12:29:21.931 INFO:teuthology.orchestra.run.trial063.stdout: python3-numpy-1:1.23.5-2.el9.x86_64 2026-02-12T12:29:21.931 INFO:teuthology.orchestra.run.trial063.stdout: python3-numpy-f2py-1:1.23.5-2.el9.x86_64 2026-02-12T12:29:21.931 INFO:teuthology.orchestra.run.trial063.stdout: python3-packaging-20.9-5.el9.noarch 2026-02-12T12:29:21.931 INFO:teuthology.orchestra.run.trial063.stdout: python3-ply-3.11-14.el9.noarch 2026-02-12T12:29:21.931 INFO:teuthology.orchestra.run.trial063.stdout: python3-portend-3.1.0-2.el9.noarch 2026-02-12T12:29:21.931 INFO:teuthology.orchestra.run.trial063.stdout: python3-protobuf-3.14.0-17.el9.noarch 2026-02-12T12:29:21.932 INFO:teuthology.orchestra.run.trial063.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2026-02-12T12:29:21.932 INFO:teuthology.orchestra.run.trial063.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2026-02-12T12:29:21.932 INFO:teuthology.orchestra.run.trial063.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2026-02-12T12:29:21.932 INFO:teuthology.orchestra.run.trial063.stdout: python3-pycparser-2.20-6.el9.noarch 2026-02-12T12:29:21.932 INFO:teuthology.orchestra.run.trial063.stdout: python3-pyparsing-2.4.7-9.el9.noarch 2026-02-12T12:29:21.932 INFO:teuthology.orchestra.run.trial063.stdout: python3-repoze-lru-0.7-16.el9.noarch 2026-02-12T12:29:21.932 INFO:teuthology.orchestra.run.trial063.stdout: python3-requests-2.25.1-10.el9.noarch 2026-02-12T12:29:21.932 INFO:teuthology.orchestra.run.trial063.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2026-02-12T12:29:21.932 INFO:teuthology.orchestra.run.trial063.stdout: python3-routes-2.5.1-5.el9.noarch 2026-02-12T12:29:21.932 INFO:teuthology.orchestra.run.trial063.stdout: python3-rsa-4.9-2.el9.noarch 2026-02-12T12:29:21.932 INFO:teuthology.orchestra.run.trial063.stdout: python3-saml-1.16.0-1.el9.noarch 2026-02-12T12:29:21.932 INFO:teuthology.orchestra.run.trial063.stdout: python3-scipy-1.9.3-2.el9.x86_64 2026-02-12T12:29:21.932 INFO:teuthology.orchestra.run.trial063.stdout: python3-tempora-5.0.0-2.el9.noarch 2026-02-12T12:29:21.932 INFO:teuthology.orchestra.run.trial063.stdout: python3-toml-0.10.2-6.el9.noarch 2026-02-12T12:29:21.933 INFO:teuthology.orchestra.run.trial063.stdout: python3-typing-extensions-4.15.0-1.el9.noarch 2026-02-12T12:29:21.933 INFO:teuthology.orchestra.run.trial063.stdout: python3-urllib3-1.26.5-7.el9.noarch 2026-02-12T12:29:21.933 INFO:teuthology.orchestra.run.trial063.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2026-02-12T12:29:21.933 INFO:teuthology.orchestra.run.trial063.stdout: python3-xmlsec-1.3.13-1.el9.x86_64 2026-02-12T12:29:21.933 INFO:teuthology.orchestra.run.trial063.stdout: python3-xmltodict-0.12.0-15.el9.noarch 2026-02-12T12:29:21.933 INFO:teuthology.orchestra.run.trial063.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2026-02-12T12:29:21.933 INFO:teuthology.orchestra.run.trial063.stdout: unzip-6.0-59.el9.x86_64 2026-02-12T12:29:21.933 INFO:teuthology.orchestra.run.trial063.stdout: xmlsec1-1.2.29-13.el9.x86_64 2026-02-12T12:29:21.933 INFO:teuthology.orchestra.run.trial063.stdout: xmlsec1-openssl-1.2.29-13.el9.x86_64 2026-02-12T12:29:21.933 INFO:teuthology.orchestra.run.trial063.stdout: zip-3.0-35.el9.x86_64 2026-02-12T12:29:21.933 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:29:21.933 INFO:teuthology.orchestra.run.trial063.stdout:Complete! 2026-02-12T12:29:21.989 DEBUG:teuthology.orchestra.run.trial063:> sudo yum -y install ceph-base 2026-02-12T12:29:22.298 INFO:teuthology.orchestra.run.trial063.stdout:Last metadata expiration check: 0:02:56 ago on Thu 12 Feb 2026 12:26:26 PM UTC. 2026-02-12T12:29:22.364 INFO:teuthology.orchestra.run.trial063.stdout:Package ceph-base-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:29:22.383 INFO:teuthology.orchestra.run.trial063.stdout:Dependencies resolved. 2026-02-12T12:29:22.383 INFO:teuthology.orchestra.run.trial063.stdout:Nothing to do. 2026-02-12T12:29:22.384 INFO:teuthology.orchestra.run.trial063.stdout:Complete! 2026-02-12T12:29:22.405 DEBUG:teuthology.orchestra.run.trial063:> sudo yum -y install cephadm 2026-02-12T12:29:22.703 INFO:teuthology.orchestra.run.trial063.stdout:Last metadata expiration check: 0:02:56 ago on Thu 12 Feb 2026 12:26:26 PM UTC. 2026-02-12T12:29:22.769 INFO:teuthology.orchestra.run.trial063.stdout:Package cephadm-2:20.3.0-5145.g5fdb7d8f.el9.noarch is already installed. 2026-02-12T12:29:22.788 INFO:teuthology.orchestra.run.trial063.stdout:Dependencies resolved. 2026-02-12T12:29:22.788 INFO:teuthology.orchestra.run.trial063.stdout:Nothing to do. 2026-02-12T12:29:22.788 INFO:teuthology.orchestra.run.trial063.stdout:Complete! 2026-02-12T12:29:22.809 DEBUG:teuthology.orchestra.run.trial063:> sudo yum -y install ceph-immutable-object-cache 2026-02-12T12:29:23.107 INFO:teuthology.orchestra.run.trial063.stdout:Last metadata expiration check: 0:02:57 ago on Thu 12 Feb 2026 12:26:26 PM UTC. 2026-02-12T12:29:23.190 INFO:teuthology.orchestra.run.trial063.stdout:Dependencies resolved. 2026-02-12T12:29:23.191 INFO:teuthology.orchestra.run.trial063.stdout:================================================================================ 2026-02-12T12:29:23.191 INFO:teuthology.orchestra.run.trial063.stdout: Package Arch Version Repo Size 2026-02-12T12:29:23.191 INFO:teuthology.orchestra.run.trial063.stdout:================================================================================ 2026-02-12T12:29:23.191 INFO:teuthology.orchestra.run.trial063.stdout:Installing: 2026-02-12T12:29:23.191 INFO:teuthology.orchestra.run.trial063.stdout: ceph-immutable-object-cache x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 149 k 2026-02-12T12:29:23.191 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:29:23.191 INFO:teuthology.orchestra.run.trial063.stdout:Transaction Summary 2026-02-12T12:29:23.192 INFO:teuthology.orchestra.run.trial063.stdout:================================================================================ 2026-02-12T12:29:23.192 INFO:teuthology.orchestra.run.trial063.stdout:Install 1 Package 2026-02-12T12:29:23.192 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:29:23.192 INFO:teuthology.orchestra.run.trial063.stdout:Total download size: 149 k 2026-02-12T12:29:23.192 INFO:teuthology.orchestra.run.trial063.stdout:Installed size: 455 k 2026-02-12T12:29:23.192 INFO:teuthology.orchestra.run.trial063.stdout:Downloading Packages: 2026-02-12T12:29:23.276 INFO:teuthology.orchestra.run.trial063.stdout:ceph-immutable-object-cache-20.3.0-5145.g5fdb7d 1.7 MB/s | 149 kB 00:00 2026-02-12T12:29:23.276 INFO:teuthology.orchestra.run.trial063.stdout:-------------------------------------------------------------------------------- 2026-02-12T12:29:23.276 INFO:teuthology.orchestra.run.trial063.stdout:Total 1.7 MB/s | 149 kB 00:00 2026-02-12T12:29:23.276 INFO:teuthology.orchestra.run.trial063.stdout:Running transaction check 2026-02-12T12:29:23.281 INFO:teuthology.orchestra.run.trial063.stdout:Transaction check succeeded. 2026-02-12T12:29:23.281 INFO:teuthology.orchestra.run.trial063.stdout:Running transaction test 2026-02-12T12:29:23.313 INFO:teuthology.orchestra.run.trial063.stdout:Transaction test succeeded. 2026-02-12T12:29:23.313 INFO:teuthology.orchestra.run.trial063.stdout:Running transaction 2026-02-12T12:29:23.368 INFO:teuthology.orchestra.run.trial063.stdout: Preparing : 1/1 2026-02-12T12:29:23.369 INFO:teuthology.orchestra.run.trial063.stdout: Installing : ceph-immutable-object-cache-2:20.3.0-5145.g5fdb7d8f. 1/1 2026-02-12T12:29:23.392 INFO:teuthology.orchestra.run.trial063.stdout: Running scriptlet: ceph-immutable-object-cache-2:20.3.0-5145.g5fdb7d8f. 1/1 2026-02-12T12:29:23.392 INFO:teuthology.orchestra.run.trial063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-12T12:29:23.392 INFO:teuthology.orchestra.run.trial063.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2026-02-12T12:29:23.392 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:29:23.708 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : ceph-immutable-object-cache-2:20.3.0-5145.g5fdb7d8f. 1/1 2026-02-12T12:29:23.708 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:29:23.709 INFO:teuthology.orchestra.run.trial063.stdout:Installed: 2026-02-12T12:29:23.709 INFO:teuthology.orchestra.run.trial063.stdout: ceph-immutable-object-cache-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:29:23.709 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:29:23.709 INFO:teuthology.orchestra.run.trial063.stdout:Complete! 2026-02-12T12:29:23.736 DEBUG:teuthology.orchestra.run.trial063:> sudo yum -y install ceph-mgr 2026-02-12T12:29:24.039 INFO:teuthology.orchestra.run.trial063.stdout:Last metadata expiration check: 0:02:58 ago on Thu 12 Feb 2026 12:26:26 PM UTC. 2026-02-12T12:29:24.104 INFO:teuthology.orchestra.run.trial063.stdout:Package ceph-mgr-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:29:24.122 INFO:teuthology.orchestra.run.trial063.stdout:Dependencies resolved. 2026-02-12T12:29:24.123 INFO:teuthology.orchestra.run.trial063.stdout:Nothing to do. 2026-02-12T12:29:24.123 INFO:teuthology.orchestra.run.trial063.stdout:Complete! 2026-02-12T12:29:24.145 DEBUG:teuthology.orchestra.run.trial063:> sudo yum -y install ceph-mgr-dashboard 2026-02-12T12:29:24.444 INFO:teuthology.orchestra.run.trial063.stdout:Last metadata expiration check: 0:02:58 ago on Thu 12 Feb 2026 12:26:26 PM UTC. 2026-02-12T12:29:24.509 INFO:teuthology.orchestra.run.trial063.stdout:Package ceph-mgr-dashboard-2:20.3.0-5145.g5fdb7d8f.el9.noarch is already installed. 2026-02-12T12:29:24.527 INFO:teuthology.orchestra.run.trial063.stdout:Dependencies resolved. 2026-02-12T12:29:24.528 INFO:teuthology.orchestra.run.trial063.stdout:Nothing to do. 2026-02-12T12:29:24.528 INFO:teuthology.orchestra.run.trial063.stdout:Complete! 2026-02-12T12:29:24.549 DEBUG:teuthology.orchestra.run.trial063:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-12T12:29:24.849 INFO:teuthology.orchestra.run.trial063.stdout:Last metadata expiration check: 0:02:58 ago on Thu 12 Feb 2026 12:26:26 PM UTC. 2026-02-12T12:29:24.913 INFO:teuthology.orchestra.run.trial063.stdout:Package ceph-mgr-diskprediction-local-2:20.3.0-5145.g5fdb7d8f.el9.noarch is already installed. 2026-02-12T12:29:24.932 INFO:teuthology.orchestra.run.trial063.stdout:Dependencies resolved. 2026-02-12T12:29:24.933 INFO:teuthology.orchestra.run.trial063.stdout:Nothing to do. 2026-02-12T12:29:24.933 INFO:teuthology.orchestra.run.trial063.stdout:Complete! 2026-02-12T12:29:24.954 DEBUG:teuthology.orchestra.run.trial063:> sudo yum -y install ceph-mgr-rook 2026-02-12T12:29:25.253 INFO:teuthology.orchestra.run.trial063.stdout:Last metadata expiration check: 0:02:59 ago on Thu 12 Feb 2026 12:26:26 PM UTC. 2026-02-12T12:29:25.318 INFO:teuthology.orchestra.run.trial063.stdout:Package ceph-mgr-rook-2:20.3.0-5145.g5fdb7d8f.el9.noarch is already installed. 2026-02-12T12:29:25.336 INFO:teuthology.orchestra.run.trial063.stdout:Dependencies resolved. 2026-02-12T12:29:25.337 INFO:teuthology.orchestra.run.trial063.stdout:Nothing to do. 2026-02-12T12:29:25.337 INFO:teuthology.orchestra.run.trial063.stdout:Complete! 2026-02-12T12:29:25.358 DEBUG:teuthology.orchestra.run.trial063:> sudo yum -y install ceph-mgr-cephadm 2026-02-12T12:29:25.662 INFO:teuthology.orchestra.run.trial063.stdout:Last metadata expiration check: 0:02:59 ago on Thu 12 Feb 2026 12:26:26 PM UTC. 2026-02-12T12:29:25.728 INFO:teuthology.orchestra.run.trial063.stdout:Package ceph-mgr-cephadm-2:20.3.0-5145.g5fdb7d8f.el9.noarch is already installed. 2026-02-12T12:29:25.747 INFO:teuthology.orchestra.run.trial063.stdout:Dependencies resolved. 2026-02-12T12:29:25.747 INFO:teuthology.orchestra.run.trial063.stdout:Nothing to do. 2026-02-12T12:29:25.748 INFO:teuthology.orchestra.run.trial063.stdout:Complete! 2026-02-12T12:29:25.768 DEBUG:teuthology.orchestra.run.trial063:> sudo yum -y install ceph-osd 2026-02-12T12:29:26.073 INFO:teuthology.orchestra.run.trial063.stdout:Last metadata expiration check: 0:03:00 ago on Thu 12 Feb 2026 12:26:26 PM UTC. 2026-02-12T12:29:26.139 INFO:teuthology.orchestra.run.trial063.stdout:Package ceph-osd-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:29:26.158 INFO:teuthology.orchestra.run.trial063.stdout:Dependencies resolved. 2026-02-12T12:29:26.158 INFO:teuthology.orchestra.run.trial063.stdout:Nothing to do. 2026-02-12T12:29:26.158 INFO:teuthology.orchestra.run.trial063.stdout:Complete! 2026-02-12T12:29:26.181 DEBUG:teuthology.orchestra.run.trial063:> sudo yum -y install ceph-osd-classic 2026-02-12T12:29:26.485 INFO:teuthology.orchestra.run.trial063.stdout:Last metadata expiration check: 0:03:00 ago on Thu 12 Feb 2026 12:26:26 PM UTC. 2026-02-12T12:29:26.551 INFO:teuthology.orchestra.run.trial063.stdout:Package ceph-osd-classic-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:29:26.570 INFO:teuthology.orchestra.run.trial063.stdout:Dependencies resolved. 2026-02-12T12:29:26.570 INFO:teuthology.orchestra.run.trial063.stdout:Nothing to do. 2026-02-12T12:29:26.570 INFO:teuthology.orchestra.run.trial063.stdout:Complete! 2026-02-12T12:29:26.591 DEBUG:teuthology.orchestra.run.trial063:> sudo yum -y install ceph-fuse 2026-02-12T12:29:26.891 INFO:teuthology.orchestra.run.trial063.stdout:Last metadata expiration check: 0:03:00 ago on Thu 12 Feb 2026 12:26:26 PM UTC. 2026-02-12T12:29:26.976 INFO:teuthology.orchestra.run.trial063.stdout:Dependencies resolved. 2026-02-12T12:29:26.977 INFO:teuthology.orchestra.run.trial063.stdout:================================================================================ 2026-02-12T12:29:26.977 INFO:teuthology.orchestra.run.trial063.stdout: Package Arch Version Repo Size 2026-02-12T12:29:26.977 INFO:teuthology.orchestra.run.trial063.stdout:================================================================================ 2026-02-12T12:29:26.977 INFO:teuthology.orchestra.run.trial063.stdout:Installing: 2026-02-12T12:29:26.977 INFO:teuthology.orchestra.run.trial063.stdout: ceph-fuse x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 914 k 2026-02-12T12:29:26.977 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:29:26.978 INFO:teuthology.orchestra.run.trial063.stdout:Transaction Summary 2026-02-12T12:29:26.978 INFO:teuthology.orchestra.run.trial063.stdout:================================================================================ 2026-02-12T12:29:26.978 INFO:teuthology.orchestra.run.trial063.stdout:Install 1 Package 2026-02-12T12:29:26.978 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:29:26.978 INFO:teuthology.orchestra.run.trial063.stdout:Total download size: 914 k 2026-02-12T12:29:26.978 INFO:teuthology.orchestra.run.trial063.stdout:Installed size: 2.9 M 2026-02-12T12:29:26.978 INFO:teuthology.orchestra.run.trial063.stdout:Downloading Packages: 2026-02-12T12:29:27.507 INFO:teuthology.orchestra.run.trial063.stdout:ceph-fuse-20.3.0-5145.g5fdb7d8f.el9.x86_64.rpm 1.7 MB/s | 914 kB 00:00 2026-02-12T12:29:27.507 INFO:teuthology.orchestra.run.trial063.stdout:-------------------------------------------------------------------------------- 2026-02-12T12:29:27.507 INFO:teuthology.orchestra.run.trial063.stdout:Total 1.7 MB/s | 914 kB 00:00 2026-02-12T12:29:27.507 INFO:teuthology.orchestra.run.trial063.stdout:Running transaction check 2026-02-12T12:29:27.512 INFO:teuthology.orchestra.run.trial063.stdout:Transaction check succeeded. 2026-02-12T12:29:27.512 INFO:teuthology.orchestra.run.trial063.stdout:Running transaction test 2026-02-12T12:29:27.547 INFO:teuthology.orchestra.run.trial063.stdout:Transaction test succeeded. 2026-02-12T12:29:27.547 INFO:teuthology.orchestra.run.trial063.stdout:Running transaction 2026-02-12T12:29:27.637 INFO:teuthology.orchestra.run.trial063.stdout: Preparing : 1/1 2026-02-12T12:29:27.654 INFO:teuthology.orchestra.run.trial063.stdout: Installing : ceph-fuse-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 1/1 2026-02-12T12:29:27.894 INFO:teuthology.orchestra.run.trial063.stdout: Running scriptlet: ceph-fuse-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 1/1 2026-02-12T12:29:27.951 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : ceph-fuse-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 1/1 2026-02-12T12:29:27.951 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:29:27.951 INFO:teuthology.orchestra.run.trial063.stdout:Installed: 2026-02-12T12:29:27.951 INFO:teuthology.orchestra.run.trial063.stdout: ceph-fuse-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:29:27.952 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:29:27.952 INFO:teuthology.orchestra.run.trial063.stdout:Complete! 2026-02-12T12:29:27.979 DEBUG:teuthology.orchestra.run.trial063:> sudo yum -y install ceph-volume 2026-02-12T12:29:28.285 INFO:teuthology.orchestra.run.trial063.stdout:Last metadata expiration check: 0:03:02 ago on Thu 12 Feb 2026 12:26:26 PM UTC. 2026-02-12T12:29:28.349 INFO:teuthology.orchestra.run.trial063.stdout:Package ceph-volume-2:20.3.0-5145.g5fdb7d8f.el9.noarch is already installed. 2026-02-12T12:29:28.367 INFO:teuthology.orchestra.run.trial063.stdout:Dependencies resolved. 2026-02-12T12:29:28.367 INFO:teuthology.orchestra.run.trial063.stdout:Nothing to do. 2026-02-12T12:29:28.367 INFO:teuthology.orchestra.run.trial063.stdout:Complete! 2026-02-12T12:29:28.389 DEBUG:teuthology.orchestra.run.trial063:> sudo yum -y install librados-devel 2026-02-12T12:29:28.690 INFO:teuthology.orchestra.run.trial063.stdout:Last metadata expiration check: 0:03:02 ago on Thu 12 Feb 2026 12:26:26 PM UTC. 2026-02-12T12:29:28.773 INFO:teuthology.orchestra.run.trial063.stdout:Dependencies resolved. 2026-02-12T12:29:28.774 INFO:teuthology.orchestra.run.trial063.stdout:================================================================================ 2026-02-12T12:29:28.774 INFO:teuthology.orchestra.run.trial063.stdout: Package Arch Version Repo Size 2026-02-12T12:29:28.774 INFO:teuthology.orchestra.run.trial063.stdout:================================================================================ 2026-02-12T12:29:28.774 INFO:teuthology.orchestra.run.trial063.stdout:Installing: 2026-02-12T12:29:28.774 INFO:teuthology.orchestra.run.trial063.stdout: librados-devel x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 124 k 2026-02-12T12:29:28.775 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:29:28.775 INFO:teuthology.orchestra.run.trial063.stdout:Transaction Summary 2026-02-12T12:29:28.775 INFO:teuthology.orchestra.run.trial063.stdout:================================================================================ 2026-02-12T12:29:28.775 INFO:teuthology.orchestra.run.trial063.stdout:Install 1 Package 2026-02-12T12:29:28.775 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:29:28.775 INFO:teuthology.orchestra.run.trial063.stdout:Total download size: 124 k 2026-02-12T12:29:28.775 INFO:teuthology.orchestra.run.trial063.stdout:Installed size: 434 k 2026-02-12T12:29:28.775 INFO:teuthology.orchestra.run.trial063.stdout:Downloading Packages: 2026-02-12T12:29:29.154 INFO:teuthology.orchestra.run.trial063.stdout:librados-devel-20.3.0-5145.g5fdb7d8f.el9.x86_64 325 kB/s | 124 kB 00:00 2026-02-12T12:29:29.155 INFO:teuthology.orchestra.run.trial063.stdout:-------------------------------------------------------------------------------- 2026-02-12T12:29:29.155 INFO:teuthology.orchestra.run.trial063.stdout:Total 324 kB/s | 124 kB 00:00 2026-02-12T12:29:29.155 INFO:teuthology.orchestra.run.trial063.stdout:Running transaction check 2026-02-12T12:29:29.158 INFO:teuthology.orchestra.run.trial063.stdout:Transaction check succeeded. 2026-02-12T12:29:29.158 INFO:teuthology.orchestra.run.trial063.stdout:Running transaction test 2026-02-12T12:29:29.191 INFO:teuthology.orchestra.run.trial063.stdout:Transaction test succeeded. 2026-02-12T12:29:29.191 INFO:teuthology.orchestra.run.trial063.stdout:Running transaction 2026-02-12T12:29:29.246 INFO:teuthology.orchestra.run.trial063.stdout: Preparing : 1/1 2026-02-12T12:29:29.262 INFO:teuthology.orchestra.run.trial063.stdout: Installing : librados-devel-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 1/1 2026-02-12T12:29:29.341 INFO:teuthology.orchestra.run.trial063.stdout: Running scriptlet: librados-devel-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 1/1 2026-02-12T12:29:29.398 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : librados-devel-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 1/1 2026-02-12T12:29:29.399 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:29:29.399 INFO:teuthology.orchestra.run.trial063.stdout:Installed: 2026-02-12T12:29:29.399 INFO:teuthology.orchestra.run.trial063.stdout: librados-devel-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:29:29.399 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:29:29.399 INFO:teuthology.orchestra.run.trial063.stdout:Complete! 2026-02-12T12:29:29.426 DEBUG:teuthology.orchestra.run.trial063:> sudo yum -y install libcephfs2 2026-02-12T12:29:29.733 INFO:teuthology.orchestra.run.trial063.stdout:Last metadata expiration check: 0:03:03 ago on Thu 12 Feb 2026 12:26:26 PM UTC. 2026-02-12T12:29:29.798 INFO:teuthology.orchestra.run.trial063.stdout:Package libcephfs2-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:29:29.815 INFO:teuthology.orchestra.run.trial063.stdout:Dependencies resolved. 2026-02-12T12:29:29.816 INFO:teuthology.orchestra.run.trial063.stdout:Nothing to do. 2026-02-12T12:29:29.816 INFO:teuthology.orchestra.run.trial063.stdout:Complete! 2026-02-12T12:29:29.837 DEBUG:teuthology.orchestra.run.trial063:> sudo yum -y install libcephfs-devel 2026-02-12T12:29:30.139 INFO:teuthology.orchestra.run.trial063.stdout:Last metadata expiration check: 0:03:04 ago on Thu 12 Feb 2026 12:26:26 PM UTC. 2026-02-12T12:29:30.222 INFO:teuthology.orchestra.run.trial063.stdout:Dependencies resolved. 2026-02-12T12:29:30.222 INFO:teuthology.orchestra.run.trial063.stdout:================================================================================ 2026-02-12T12:29:30.222 INFO:teuthology.orchestra.run.trial063.stdout: Package Arch Version Repo Size 2026-02-12T12:29:30.222 INFO:teuthology.orchestra.run.trial063.stdout:================================================================================ 2026-02-12T12:29:30.222 INFO:teuthology.orchestra.run.trial063.stdout:Installing: 2026-02-12T12:29:30.222 INFO:teuthology.orchestra.run.trial063.stdout: libcephfs-devel x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 37 k 2026-02-12T12:29:30.222 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:29:30.222 INFO:teuthology.orchestra.run.trial063.stdout:Transaction Summary 2026-02-12T12:29:30.222 INFO:teuthology.orchestra.run.trial063.stdout:================================================================================ 2026-02-12T12:29:30.222 INFO:teuthology.orchestra.run.trial063.stdout:Install 1 Package 2026-02-12T12:29:30.222 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:29:30.222 INFO:teuthology.orchestra.run.trial063.stdout:Total download size: 37 k 2026-02-12T12:29:30.223 INFO:teuthology.orchestra.run.trial063.stdout:Installed size: 171 k 2026-02-12T12:29:30.223 INFO:teuthology.orchestra.run.trial063.stdout:Downloading Packages: 2026-02-12T12:29:30.462 INFO:teuthology.orchestra.run.trial063.stdout:libcephfs-devel-20.3.0-5145.g5fdb7d8f.el9.x86_6 156 kB/s | 37 kB 00:00 2026-02-12T12:29:30.463 INFO:teuthology.orchestra.run.trial063.stdout:-------------------------------------------------------------------------------- 2026-02-12T12:29:30.463 INFO:teuthology.orchestra.run.trial063.stdout:Total 155 kB/s | 37 kB 00:00 2026-02-12T12:29:30.463 INFO:teuthology.orchestra.run.trial063.stdout:Running transaction check 2026-02-12T12:29:30.465 INFO:teuthology.orchestra.run.trial063.stdout:Transaction check succeeded. 2026-02-12T12:29:30.465 INFO:teuthology.orchestra.run.trial063.stdout:Running transaction test 2026-02-12T12:29:30.468 INFO:teuthology.orchestra.run.trial063.stdout:Transaction test succeeded. 2026-02-12T12:29:30.469 INFO:teuthology.orchestra.run.trial063.stdout:Running transaction 2026-02-12T12:29:30.491 INFO:teuthology.orchestra.run.trial063.stdout: Preparing : 1/1 2026-02-12T12:29:30.507 INFO:teuthology.orchestra.run.trial063.stdout: Installing : libcephfs-devel-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 1/1 2026-02-12T12:29:30.562 INFO:teuthology.orchestra.run.trial063.stdout: Running scriptlet: libcephfs-devel-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 1/1 2026-02-12T12:29:30.618 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : libcephfs-devel-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 1/1 2026-02-12T12:29:30.619 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:29:30.619 INFO:teuthology.orchestra.run.trial063.stdout:Installed: 2026-02-12T12:29:30.619 INFO:teuthology.orchestra.run.trial063.stdout: libcephfs-devel-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:29:30.619 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:29:30.619 INFO:teuthology.orchestra.run.trial063.stdout:Complete! 2026-02-12T12:29:30.647 DEBUG:teuthology.orchestra.run.trial063:> sudo yum -y install librados2 2026-02-12T12:29:30.947 INFO:teuthology.orchestra.run.trial063.stdout:Last metadata expiration check: 0:03:04 ago on Thu 12 Feb 2026 12:26:26 PM UTC. 2026-02-12T12:29:31.012 INFO:teuthology.orchestra.run.trial063.stdout:Package librados2-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:29:31.029 INFO:teuthology.orchestra.run.trial063.stdout:Dependencies resolved. 2026-02-12T12:29:31.029 INFO:teuthology.orchestra.run.trial063.stdout:Nothing to do. 2026-02-12T12:29:31.029 INFO:teuthology.orchestra.run.trial063.stdout:Complete! 2026-02-12T12:29:31.051 DEBUG:teuthology.orchestra.run.trial063:> sudo yum -y install librbd1 2026-02-12T12:29:31.350 INFO:teuthology.orchestra.run.trial063.stdout:Last metadata expiration check: 0:03:05 ago on Thu 12 Feb 2026 12:26:26 PM UTC. 2026-02-12T12:29:31.414 INFO:teuthology.orchestra.run.trial063.stdout:Package librbd1-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:29:31.431 INFO:teuthology.orchestra.run.trial063.stdout:Dependencies resolved. 2026-02-12T12:29:31.432 INFO:teuthology.orchestra.run.trial063.stdout:Nothing to do. 2026-02-12T12:29:31.432 INFO:teuthology.orchestra.run.trial063.stdout:Complete! 2026-02-12T12:29:31.453 DEBUG:teuthology.orchestra.run.trial063:> sudo yum -y install python3-rados 2026-02-12T12:29:31.760 INFO:teuthology.orchestra.run.trial063.stdout:Last metadata expiration check: 0:03:05 ago on Thu 12 Feb 2026 12:26:26 PM UTC. 2026-02-12T12:29:31.825 INFO:teuthology.orchestra.run.trial063.stdout:Package python3-rados-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:29:31.842 INFO:teuthology.orchestra.run.trial063.stdout:Dependencies resolved. 2026-02-12T12:29:31.843 INFO:teuthology.orchestra.run.trial063.stdout:Nothing to do. 2026-02-12T12:29:31.843 INFO:teuthology.orchestra.run.trial063.stdout:Complete! 2026-02-12T12:29:31.864 DEBUG:teuthology.orchestra.run.trial063:> sudo yum -y install python3-rgw 2026-02-12T12:29:32.173 INFO:teuthology.orchestra.run.trial063.stdout:Last metadata expiration check: 0:03:06 ago on Thu 12 Feb 2026 12:26:26 PM UTC. 2026-02-12T12:29:32.237 INFO:teuthology.orchestra.run.trial063.stdout:Package python3-rgw-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:29:32.255 INFO:teuthology.orchestra.run.trial063.stdout:Dependencies resolved. 2026-02-12T12:29:32.255 INFO:teuthology.orchestra.run.trial063.stdout:Nothing to do. 2026-02-12T12:29:32.256 INFO:teuthology.orchestra.run.trial063.stdout:Complete! 2026-02-12T12:29:32.278 DEBUG:teuthology.orchestra.run.trial063:> sudo yum -y install python3-cephfs 2026-02-12T12:29:32.578 INFO:teuthology.orchestra.run.trial063.stdout:Last metadata expiration check: 0:03:06 ago on Thu 12 Feb 2026 12:26:26 PM UTC. 2026-02-12T12:29:32.643 INFO:teuthology.orchestra.run.trial063.stdout:Package python3-cephfs-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:29:32.661 INFO:teuthology.orchestra.run.trial063.stdout:Dependencies resolved. 2026-02-12T12:29:32.661 INFO:teuthology.orchestra.run.trial063.stdout:Nothing to do. 2026-02-12T12:29:32.661 INFO:teuthology.orchestra.run.trial063.stdout:Complete! 2026-02-12T12:29:32.682 DEBUG:teuthology.orchestra.run.trial063:> sudo yum -y install python3-rbd 2026-02-12T12:29:32.981 INFO:teuthology.orchestra.run.trial063.stdout:Last metadata expiration check: 0:03:06 ago on Thu 12 Feb 2026 12:26:26 PM UTC. 2026-02-12T12:29:33.046 INFO:teuthology.orchestra.run.trial063.stdout:Package python3-rbd-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:29:33.064 INFO:teuthology.orchestra.run.trial063.stdout:Dependencies resolved. 2026-02-12T12:29:33.064 INFO:teuthology.orchestra.run.trial063.stdout:Nothing to do. 2026-02-12T12:29:33.064 INFO:teuthology.orchestra.run.trial063.stdout:Complete! 2026-02-12T12:29:33.086 DEBUG:teuthology.orchestra.run.trial063:> sudo yum -y install rbd-fuse 2026-02-12T12:29:33.383 INFO:teuthology.orchestra.run.trial063.stdout:Last metadata expiration check: 0:03:07 ago on Thu 12 Feb 2026 12:26:26 PM UTC. 2026-02-12T12:29:33.467 INFO:teuthology.orchestra.run.trial063.stdout:Dependencies resolved. 2026-02-12T12:29:33.467 INFO:teuthology.orchestra.run.trial063.stdout:================================================================================ 2026-02-12T12:29:33.468 INFO:teuthology.orchestra.run.trial063.stdout: Package Architecture Version Repository Size 2026-02-12T12:29:33.468 INFO:teuthology.orchestra.run.trial063.stdout:================================================================================ 2026-02-12T12:29:33.468 INFO:teuthology.orchestra.run.trial063.stdout:Installing: 2026-02-12T12:29:33.468 INFO:teuthology.orchestra.run.trial063.stdout: rbd-fuse x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 87 k 2026-02-12T12:29:33.468 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:29:33.468 INFO:teuthology.orchestra.run.trial063.stdout:Transaction Summary 2026-02-12T12:29:33.468 INFO:teuthology.orchestra.run.trial063.stdout:================================================================================ 2026-02-12T12:29:33.468 INFO:teuthology.orchestra.run.trial063.stdout:Install 1 Package 2026-02-12T12:29:33.468 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:29:33.469 INFO:teuthology.orchestra.run.trial063.stdout:Total download size: 87 k 2026-02-12T12:29:33.469 INFO:teuthology.orchestra.run.trial063.stdout:Installed size: 236 k 2026-02-12T12:29:33.469 INFO:teuthology.orchestra.run.trial063.stdout:Downloading Packages: 2026-02-12T12:29:33.885 INFO:teuthology.orchestra.run.trial063.stdout:rbd-fuse-20.3.0-5145.g5fdb7d8f.el9.x86_64.rpm 207 kB/s | 87 kB 00:00 2026-02-12T12:29:33.886 INFO:teuthology.orchestra.run.trial063.stdout:-------------------------------------------------------------------------------- 2026-02-12T12:29:33.886 INFO:teuthology.orchestra.run.trial063.stdout:Total 207 kB/s | 87 kB 00:00 2026-02-12T12:29:33.886 INFO:teuthology.orchestra.run.trial063.stdout:Running transaction check 2026-02-12T12:29:33.890 INFO:teuthology.orchestra.run.trial063.stdout:Transaction check succeeded. 2026-02-12T12:29:33.890 INFO:teuthology.orchestra.run.trial063.stdout:Running transaction test 2026-02-12T12:29:33.922 INFO:teuthology.orchestra.run.trial063.stdout:Transaction test succeeded. 2026-02-12T12:29:33.922 INFO:teuthology.orchestra.run.trial063.stdout:Running transaction 2026-02-12T12:29:33.973 INFO:teuthology.orchestra.run.trial063.stdout: Preparing : 1/1 2026-02-12T12:29:33.990 INFO:teuthology.orchestra.run.trial063.stdout: Installing : rbd-fuse-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 1/1 2026-02-12T12:29:34.071 INFO:teuthology.orchestra.run.trial063.stdout: Running scriptlet: rbd-fuse-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 1/1 2026-02-12T12:29:34.129 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : rbd-fuse-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 1/1 2026-02-12T12:29:34.129 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:29:34.129 INFO:teuthology.orchestra.run.trial063.stdout:Installed: 2026-02-12T12:29:34.130 INFO:teuthology.orchestra.run.trial063.stdout: rbd-fuse-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:29:34.130 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:29:34.130 INFO:teuthology.orchestra.run.trial063.stdout:Complete! 2026-02-12T12:29:34.159 DEBUG:teuthology.orchestra.run.trial063:> sudo yum -y install rbd-mirror 2026-02-12T12:29:34.462 INFO:teuthology.orchestra.run.trial063.stdout:Last metadata expiration check: 0:03:08 ago on Thu 12 Feb 2026 12:26:26 PM UTC. 2026-02-12T12:29:34.545 INFO:teuthology.orchestra.run.trial063.stdout:Dependencies resolved. 2026-02-12T12:29:34.546 INFO:teuthology.orchestra.run.trial063.stdout:================================================================================ 2026-02-12T12:29:34.546 INFO:teuthology.orchestra.run.trial063.stdout: Package Arch Version Repo Size 2026-02-12T12:29:34.546 INFO:teuthology.orchestra.run.trial063.stdout:================================================================================ 2026-02-12T12:29:34.547 INFO:teuthology.orchestra.run.trial063.stdout:Installing: 2026-02-12T12:29:34.547 INFO:teuthology.orchestra.run.trial063.stdout: rbd-mirror x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 2.6 M 2026-02-12T12:29:34.547 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:29:34.547 INFO:teuthology.orchestra.run.trial063.stdout:Transaction Summary 2026-02-12T12:29:34.547 INFO:teuthology.orchestra.run.trial063.stdout:================================================================================ 2026-02-12T12:29:34.547 INFO:teuthology.orchestra.run.trial063.stdout:Install 1 Package 2026-02-12T12:29:34.547 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:29:34.547 INFO:teuthology.orchestra.run.trial063.stdout:Total download size: 2.6 M 2026-02-12T12:29:34.547 INFO:teuthology.orchestra.run.trial063.stdout:Installed size: 11 M 2026-02-12T12:29:34.547 INFO:teuthology.orchestra.run.trial063.stdout:Downloading Packages: 2026-02-12T12:29:35.332 INFO:teuthology.orchestra.run.trial063.stdout:rbd-mirror-20.3.0-5145.g5fdb7d8f.el9.x86_64.rpm 3.3 MB/s | 2.6 MB 00:00 2026-02-12T12:29:35.332 INFO:teuthology.orchestra.run.trial063.stdout:-------------------------------------------------------------------------------- 2026-02-12T12:29:35.332 INFO:teuthology.orchestra.run.trial063.stdout:Total 3.3 MB/s | 2.6 MB 00:00 2026-02-12T12:29:35.332 INFO:teuthology.orchestra.run.trial063.stdout:Running transaction check 2026-02-12T12:29:35.337 INFO:teuthology.orchestra.run.trial063.stdout:Transaction check succeeded. 2026-02-12T12:29:35.337 INFO:teuthology.orchestra.run.trial063.stdout:Running transaction test 2026-02-12T12:29:35.376 INFO:teuthology.orchestra.run.trial063.stdout:Transaction test succeeded. 2026-02-12T12:29:35.376 INFO:teuthology.orchestra.run.trial063.stdout:Running transaction 2026-02-12T12:29:35.539 INFO:teuthology.orchestra.run.trial063.stdout: Preparing : 1/1 2026-02-12T12:29:35.541 INFO:teuthology.orchestra.run.trial063.stdout: Installing : rbd-mirror-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 1/1 2026-02-12T12:29:35.565 INFO:teuthology.orchestra.run.trial063.stdout: Running scriptlet: rbd-mirror-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 1/1 2026-02-12T12:29:35.565 INFO:teuthology.orchestra.run.trial063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-12T12:29:35.565 INFO:teuthology.orchestra.run.trial063.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2026-02-12T12:29:35.565 INFO:teuthology.orchestra.run.trial063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-12T12:29:35.565 INFO:teuthology.orchestra.run.trial063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-12T12:29:35.565 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:29:35.884 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : rbd-mirror-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 1/1 2026-02-12T12:29:35.885 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:29:35.885 INFO:teuthology.orchestra.run.trial063.stdout:Installed: 2026-02-12T12:29:35.885 INFO:teuthology.orchestra.run.trial063.stdout: rbd-mirror-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:29:35.885 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:29:35.885 INFO:teuthology.orchestra.run.trial063.stdout:Complete! 2026-02-12T12:29:35.913 DEBUG:teuthology.orchestra.run.trial063:> sudo yum -y install rbd-nbd 2026-02-12T12:29:36.215 INFO:teuthology.orchestra.run.trial063.stdout:Last metadata expiration check: 0:03:10 ago on Thu 12 Feb 2026 12:26:26 PM UTC. 2026-02-12T12:29:36.297 INFO:teuthology.orchestra.run.trial063.stdout:Dependencies resolved. 2026-02-12T12:29:36.298 INFO:teuthology.orchestra.run.trial063.stdout:================================================================================ 2026-02-12T12:29:36.298 INFO:teuthology.orchestra.run.trial063.stdout: Package Architecture Version Repository Size 2026-02-12T12:29:36.298 INFO:teuthology.orchestra.run.trial063.stdout:================================================================================ 2026-02-12T12:29:36.299 INFO:teuthology.orchestra.run.trial063.stdout:Installing: 2026-02-12T12:29:36.299 INFO:teuthology.orchestra.run.trial063.stdout: rbd-nbd x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 167 k 2026-02-12T12:29:36.299 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:29:36.299 INFO:teuthology.orchestra.run.trial063.stdout:Transaction Summary 2026-02-12T12:29:36.299 INFO:teuthology.orchestra.run.trial063.stdout:================================================================================ 2026-02-12T12:29:36.299 INFO:teuthology.orchestra.run.trial063.stdout:Install 1 Package 2026-02-12T12:29:36.299 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:29:36.299 INFO:teuthology.orchestra.run.trial063.stdout:Total download size: 167 k 2026-02-12T12:29:36.299 INFO:teuthology.orchestra.run.trial063.stdout:Installed size: 496 k 2026-02-12T12:29:36.299 INFO:teuthology.orchestra.run.trial063.stdout:Downloading Packages: 2026-02-12T12:29:36.629 INFO:teuthology.orchestra.run.trial063.stdout:rbd-nbd-20.3.0-5145.g5fdb7d8f.el9.x86_64.rpm 504 kB/s | 167 kB 00:00 2026-02-12T12:29:36.629 INFO:teuthology.orchestra.run.trial063.stdout:-------------------------------------------------------------------------------- 2026-02-12T12:29:36.630 INFO:teuthology.orchestra.run.trial063.stdout:Total 503 kB/s | 167 kB 00:00 2026-02-12T12:29:36.630 INFO:teuthology.orchestra.run.trial063.stdout:Running transaction check 2026-02-12T12:29:36.634 INFO:teuthology.orchestra.run.trial063.stdout:Transaction check succeeded. 2026-02-12T12:29:36.634 INFO:teuthology.orchestra.run.trial063.stdout:Running transaction test 2026-02-12T12:29:36.666 INFO:teuthology.orchestra.run.trial063.stdout:Transaction test succeeded. 2026-02-12T12:29:36.666 INFO:teuthology.orchestra.run.trial063.stdout:Running transaction 2026-02-12T12:29:36.723 INFO:teuthology.orchestra.run.trial063.stdout: Preparing : 1/1 2026-02-12T12:29:36.739 INFO:teuthology.orchestra.run.trial063.stdout: Installing : rbd-nbd-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 1/1 2026-02-12T12:29:36.820 INFO:teuthology.orchestra.run.trial063.stdout: Running scriptlet: rbd-nbd-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 1/1 2026-02-12T12:29:36.877 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : rbd-nbd-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 1/1 2026-02-12T12:29:36.878 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:29:36.878 INFO:teuthology.orchestra.run.trial063.stdout:Installed: 2026-02-12T12:29:36.878 INFO:teuthology.orchestra.run.trial063.stdout: rbd-nbd-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:29:36.878 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:29:36.878 INFO:teuthology.orchestra.run.trial063.stdout:Complete! 2026-02-12T12:29:36.906 DEBUG:teuthology.orchestra.run.trial063:> sudo yum -y install ceph-osd-crimson 2026-02-12T12:29:37.217 INFO:teuthology.orchestra.run.trial063.stdout:Last metadata expiration check: 0:03:11 ago on Thu 12 Feb 2026 12:26:26 PM UTC. 2026-02-12T12:29:37.281 INFO:teuthology.orchestra.run.trial063.stdout:Package ceph-osd-crimson-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:29:37.299 INFO:teuthology.orchestra.run.trial063.stdout:Dependencies resolved. 2026-02-12T12:29:37.300 INFO:teuthology.orchestra.run.trial063.stdout:Nothing to do. 2026-02-12T12:29:37.300 INFO:teuthology.orchestra.run.trial063.stdout:Complete! 2026-02-12T12:29:37.321 DEBUG:teuthology.orchestra.run.trial063:> sudo yum -y install ceph-osd-crimson-debuginfo 2026-02-12T12:29:37.625 INFO:teuthology.orchestra.run.trial063.stdout:Last metadata expiration check: 0:03:11 ago on Thu 12 Feb 2026 12:26:26 PM UTC. 2026-02-12T12:29:37.708 INFO:teuthology.orchestra.run.trial063.stdout:Dependencies resolved. 2026-02-12T12:29:37.709 INFO:teuthology.orchestra.run.trial063.stdout:================================================================================ 2026-02-12T12:29:37.709 INFO:teuthology.orchestra.run.trial063.stdout: Package Arch Version Repo Size 2026-02-12T12:29:37.709 INFO:teuthology.orchestra.run.trial063.stdout:================================================================================ 2026-02-12T12:29:37.709 INFO:teuthology.orchestra.run.trial063.stdout:Installing: 2026-02-12T12:29:37.710 INFO:teuthology.orchestra.run.trial063.stdout: ceph-osd-crimson-debuginfo x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 429 M 2026-02-12T12:29:37.710 INFO:teuthology.orchestra.run.trial063.stdout:Installing weak dependencies: 2026-02-12T12:29:37.710 INFO:teuthology.orchestra.run.trial063.stdout: ceph-debugsource x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 12 M 2026-02-12T12:29:37.710 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:29:37.710 INFO:teuthology.orchestra.run.trial063.stdout:Transaction Summary 2026-02-12T12:29:37.710 INFO:teuthology.orchestra.run.trial063.stdout:================================================================================ 2026-02-12T12:29:37.710 INFO:teuthology.orchestra.run.trial063.stdout:Install 2 Packages 2026-02-12T12:29:37.710 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:29:37.710 INFO:teuthology.orchestra.run.trial063.stdout:Total download size: 441 M 2026-02-12T12:29:37.710 INFO:teuthology.orchestra.run.trial063.stdout:Installed size: 2.7 G 2026-02-12T12:29:37.710 INFO:teuthology.orchestra.run.trial063.stdout:Downloading Packages: 2026-02-12T12:29:40.484 INFO:teuthology.orchestra.run.trial063.stdout:(1/2): ceph-debugsource-20.3.0-5145.g5fdb7d8f.e 4.3 MB/s | 12 MB 00:02 2026-02-12T12:30:20.428 INFO:teuthology.orchestra.run.trial063.stdout:(2/2): ceph-osd-crimson-debuginfo-20.3.0-5145.g 10 MB/s | 429 MB 00:42 2026-02-12T12:30:20.429 INFO:teuthology.orchestra.run.trial063.stdout:-------------------------------------------------------------------------------- 2026-02-12T12:30:20.429 INFO:teuthology.orchestra.run.trial063.stdout:Total 10 MB/s | 441 MB 00:42 2026-02-12T12:30:20.429 INFO:teuthology.orchestra.run.trial063.stdout:Running transaction check 2026-02-12T12:30:20.434 INFO:teuthology.orchestra.run.trial063.stdout:Transaction check succeeded. 2026-02-12T12:30:20.434 INFO:teuthology.orchestra.run.trial063.stdout:Running transaction test 2026-02-12T12:30:21.797 INFO:teuthology.orchestra.run.trial063.stdout:Transaction test succeeded. 2026-02-12T12:30:21.797 INFO:teuthology.orchestra.run.trial063.stdout:Running transaction 2026-02-12T12:30:25.377 INFO:teuthology.orchestra.run.trial063.stdout: Preparing : 1/1 2026-02-12T12:30:47.518 INFO:teuthology.orchestra.run.trial063.stdout: Installing : ceph-debugsource-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 1/2 2026-02-12T12:30:47.536 INFO:teuthology.orchestra.run.trial063.stdout: Installing : ceph-osd-crimson-debuginfo-2:20.3.0-5145.g5fdb7d8f.e 2/2 2026-02-12T12:30:49.555 INFO:teuthology.orchestra.run.trial063.stdout: Running scriptlet: ceph-osd-crimson-debuginfo-2:20.3.0-5145.g5fdb7d8f.e 2/2 2026-02-12T12:30:49.555 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : ceph-debugsource-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 1/2 2026-02-12T12:30:49.613 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : ceph-osd-crimson-debuginfo-2:20.3.0-5145.g5fdb7d8f.e 2/2 2026-02-12T12:30:49.614 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:30:49.614 INFO:teuthology.orchestra.run.trial063.stdout:Installed: 2026-02-12T12:30:49.614 INFO:teuthology.orchestra.run.trial063.stdout: ceph-debugsource-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:30:49.614 INFO:teuthology.orchestra.run.trial063.stdout: ceph-osd-crimson-debuginfo-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:30:49.614 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:30:49.614 INFO:teuthology.orchestra.run.trial063.stdout:Complete! 2026-02-12T12:30:49.688 DEBUG:teuthology.parallel:result is None 2026-02-12T12:30:49.689 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0 2026-02-12T12:30:49.850 DEBUG:teuthology.orchestra.run.trial063:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-12T12:30:49.871 INFO:teuthology.orchestra.run.trial063.stdout:20.3.0-5145.g5fdb7d8f.el9 2026-02-12T12:30:49.872 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5145.g5fdb7d8f.el9 2026-02-12T12:30:49.872 INFO:teuthology.task.install:The correct ceph version 20.3.0-5145.g5fdb7d8f is installed. 2026-02-12T12:30:49.874 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2026-02-12T12:30:49.874 DEBUG:teuthology.orchestra.run.trial063:> set -ex 2026-02-12T12:30:49.875 DEBUG:teuthology.orchestra.run.trial063:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-12T12:30:49.938 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2026-02-12T12:30:49.938 DEBUG:teuthology.orchestra.run.trial063:> set -ex 2026-02-12T12:30:49.938 DEBUG:teuthology.orchestra.run.trial063:> sudo dd of=/usr/bin/daemon-helper 2026-02-12T12:30:50.002 DEBUG:teuthology.orchestra.run.trial063:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-12T12:30:50.066 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2026-02-12T12:30:50.067 DEBUG:teuthology.orchestra.run.trial063:> set -ex 2026-02-12T12:30:50.067 DEBUG:teuthology.orchestra.run.trial063:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-12T12:30:50.132 DEBUG:teuthology.orchestra.run.trial063:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-12T12:30:50.196 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2026-02-12T12:30:50.196 DEBUG:teuthology.orchestra.run.trial063:> set -ex 2026-02-12T12:30:50.197 DEBUG:teuthology.orchestra.run.trial063:> sudo dd of=/usr/bin/stdin-killer 2026-02-12T12:30:50.259 DEBUG:teuthology.orchestra.run.trial063:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-12T12:30:50.323 INFO:teuthology.run_tasks:Running task pexec... 2026-02-12T12:30:50.341 INFO:teuthology.task.pexec:Executing custom commands... 2026-02-12T12:30:50.341 DEBUG:teuthology.orchestra.run.trial063:> TESTDIR=/home/ubuntu/cephtest bash -s 2026-02-12T12:30:50.365 INFO:teuthology.task.pexec:Running commands on host ubuntu@trial063.front.sepia.ceph.com 2026-02-12T12:30:50.365 INFO:teuthology.task.pexec:sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson 2026-02-12T12:30:50.416 DEBUG:teuthology.parallel:result is None 2026-02-12T12:30:50.416 INFO:teuthology.run_tasks:Running task install... 2026-02-12T12:30:50.430 DEBUG:teuthology.task.install:project ceph 2026-02-12T12:30:50.430 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'debug', 'sha1': '5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0'}} 2026-02-12T12:30:50.430 DEBUG:teuthology.task.install:config {'flavor': 'debug', 'sha1': '5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0'} 2026-02-12T12:30:50.430 INFO:teuthology.task.install:Using flavor: debug 2026-02-12T12:30:50.438 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-osd', 'ceph-osd-classic', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-osd', 'ceph-osd-classic', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-12T12:30:50.439 INFO:teuthology.task.install:extra packages: [] 2026-02-12T12:30:50.439 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'debug', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0', 'tag': None, 'wait_for_package': False} 2026-02-12T12:30:50.439 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0 2026-02-12T12:30:50.596 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-moagrawa-seastore-io-uring-again-crimson-only/5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0/centos/9/flavors/debug/ 2026-02-12T12:30:50.597 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5145.g5fdb7d8f 2026-02-12T12:30:50.726 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-moagrawa-seastore-io-uring-again-crimson-only/5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0/centos/9/flavors/debug/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-moagrawa-seastore-io-uring-again-crimson-only/5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0/centos/9/flavors/debug/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-moagrawa-seastore-io-uring-again-crimson-only/5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0/centos/9/flavors/debug/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-12T12:30:50.726 DEBUG:teuthology.orchestra.run.trial063:> set -ex 2026-02-12T12:30:50.726 DEBUG:teuthology.orchestra.run.trial063:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-12T12:30:50.750 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2026-02-12T12:30:50.750 DEBUG:teuthology.orchestra.run.trial063:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-12T12:30:50.815 DEBUG:teuthology.orchestra.run.trial063:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-12T12:30:50.880 DEBUG:teuthology.orchestra.run.trial063:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-12T12:30:50.936 INFO:teuthology.orchestra.run.trial063.stdout:check_obsoletes = 1 2026-02-12T12:30:50.946 DEBUG:teuthology.orchestra.run.trial063:> sudo yum clean all 2026-02-12T12:30:51.105 INFO:teuthology.orchestra.run.trial063.stdout:56 files removed 2026-02-12T12:30:51.122 DEBUG:teuthology.orchestra.run.trial063:> sudo yum -y install ceph-radosgw 2026-02-12T12:30:52.224 INFO:teuthology.orchestra.run.trial063.stdout:ceph packages for x86_64 99 kB/s | 94 kB 00:00 2026-02-12T12:30:52.366 INFO:teuthology.orchestra.run.trial063.stdout:ceph noarch packages 209 kB/s | 24 kB 00:00 2026-02-12T12:30:52.493 INFO:teuthology.orchestra.run.trial063.stdout:ceph source packages 20 kB/s | 2.1 kB 00:00 2026-02-12T12:30:53.117 INFO:teuthology.orchestra.run.trial063.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.9 MB 00:00 2026-02-12T12:30:56.549 INFO:teuthology.orchestra.run.trial063.stdout:CentOS Stream 9 - AppStream 9.2 MB/s | 27 MB 00:02 2026-02-12T12:31:03.764 INFO:teuthology.orchestra.run.trial063.stdout:CentOS Stream 9 - CRB 1.6 MB/s | 7.8 MB 00:04 2026-02-12T12:31:05.559 INFO:teuthology.orchestra.run.trial063.stdout:CentOS Stream 9 - Extras packages 19 kB/s | 20 kB 00:01 2026-02-12T12:31:06.151 INFO:teuthology.orchestra.run.trial063.stdout:Extra Packages for Enterprise Linux 39 MB/s | 20 MB 00:00 2026-02-12T12:31:09.421 INFO:teuthology.orchestra.run.trial063.stdout:lab-extras 2.7 MB/s | 50 kB 00:00 2026-02-12T12:31:10.678 INFO:teuthology.orchestra.run.trial063.stdout:Package ceph-radosgw-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:31:10.696 INFO:teuthology.orchestra.run.trial063.stdout:Dependencies resolved. 2026-02-12T12:31:10.696 INFO:teuthology.orchestra.run.trial063.stdout:Nothing to do. 2026-02-12T12:31:10.697 INFO:teuthology.orchestra.run.trial063.stdout:Complete! 2026-02-12T12:31:10.721 DEBUG:teuthology.orchestra.run.trial063:> sudo yum -y install ceph-test 2026-02-12T12:31:11.022 INFO:teuthology.orchestra.run.trial063.stdout:Last metadata expiration check: 0:00:02 ago on Thu 12 Feb 2026 12:31:09 PM UTC. 2026-02-12T12:31:11.086 INFO:teuthology.orchestra.run.trial063.stdout:Package ceph-test-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:31:11.103 INFO:teuthology.orchestra.run.trial063.stdout:Dependencies resolved. 2026-02-12T12:31:11.104 INFO:teuthology.orchestra.run.trial063.stdout:Nothing to do. 2026-02-12T12:31:11.104 INFO:teuthology.orchestra.run.trial063.stdout:Complete! 2026-02-12T12:31:11.125 DEBUG:teuthology.orchestra.run.trial063:> sudo yum -y install ceph 2026-02-12T12:31:11.429 INFO:teuthology.orchestra.run.trial063.stdout:Last metadata expiration check: 0:00:02 ago on Thu 12 Feb 2026 12:31:09 PM UTC. 2026-02-12T12:31:11.493 INFO:teuthology.orchestra.run.trial063.stdout:Package ceph-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:31:11.510 INFO:teuthology.orchestra.run.trial063.stdout:Dependencies resolved. 2026-02-12T12:31:11.511 INFO:teuthology.orchestra.run.trial063.stdout:Nothing to do. 2026-02-12T12:31:11.511 INFO:teuthology.orchestra.run.trial063.stdout:Complete! 2026-02-12T12:31:11.532 DEBUG:teuthology.orchestra.run.trial063:> sudo yum -y install ceph-base 2026-02-12T12:31:11.833 INFO:teuthology.orchestra.run.trial063.stdout:Last metadata expiration check: 0:00:02 ago on Thu 12 Feb 2026 12:31:09 PM UTC. 2026-02-12T12:31:11.898 INFO:teuthology.orchestra.run.trial063.stdout:Package ceph-base-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:31:11.915 INFO:teuthology.orchestra.run.trial063.stdout:Dependencies resolved. 2026-02-12T12:31:11.916 INFO:teuthology.orchestra.run.trial063.stdout:Nothing to do. 2026-02-12T12:31:11.916 INFO:teuthology.orchestra.run.trial063.stdout:Complete! 2026-02-12T12:31:11.938 DEBUG:teuthology.orchestra.run.trial063:> sudo yum -y install cephadm 2026-02-12T12:31:12.235 INFO:teuthology.orchestra.run.trial063.stdout:Last metadata expiration check: 0:00:03 ago on Thu 12 Feb 2026 12:31:09 PM UTC. 2026-02-12T12:31:12.299 INFO:teuthology.orchestra.run.trial063.stdout:Package cephadm-2:20.3.0-5145.g5fdb7d8f.el9.noarch is already installed. 2026-02-12T12:31:12.316 INFO:teuthology.orchestra.run.trial063.stdout:Dependencies resolved. 2026-02-12T12:31:12.317 INFO:teuthology.orchestra.run.trial063.stdout:Nothing to do. 2026-02-12T12:31:12.317 INFO:teuthology.orchestra.run.trial063.stdout:Complete! 2026-02-12T12:31:12.339 DEBUG:teuthology.orchestra.run.trial063:> sudo yum -y install ceph-immutable-object-cache 2026-02-12T12:31:12.641 INFO:teuthology.orchestra.run.trial063.stdout:Last metadata expiration check: 0:00:03 ago on Thu 12 Feb 2026 12:31:09 PM UTC. 2026-02-12T12:31:12.707 INFO:teuthology.orchestra.run.trial063.stdout:Package ceph-immutable-object-cache-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:31:12.725 INFO:teuthology.orchestra.run.trial063.stdout:Dependencies resolved. 2026-02-12T12:31:12.725 INFO:teuthology.orchestra.run.trial063.stdout:Nothing to do. 2026-02-12T12:31:12.726 INFO:teuthology.orchestra.run.trial063.stdout:Complete! 2026-02-12T12:31:12.747 DEBUG:teuthology.orchestra.run.trial063:> sudo yum -y install ceph-mgr 2026-02-12T12:31:13.045 INFO:teuthology.orchestra.run.trial063.stdout:Last metadata expiration check: 0:00:04 ago on Thu 12 Feb 2026 12:31:09 PM UTC. 2026-02-12T12:31:13.109 INFO:teuthology.orchestra.run.trial063.stdout:Package ceph-mgr-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:31:13.127 INFO:teuthology.orchestra.run.trial063.stdout:Dependencies resolved. 2026-02-12T12:31:13.127 INFO:teuthology.orchestra.run.trial063.stdout:Nothing to do. 2026-02-12T12:31:13.127 INFO:teuthology.orchestra.run.trial063.stdout:Complete! 2026-02-12T12:31:13.149 DEBUG:teuthology.orchestra.run.trial063:> sudo yum -y install ceph-mgr-dashboard 2026-02-12T12:31:13.451 INFO:teuthology.orchestra.run.trial063.stdout:Last metadata expiration check: 0:00:04 ago on Thu 12 Feb 2026 12:31:09 PM UTC. 2026-02-12T12:31:13.515 INFO:teuthology.orchestra.run.trial063.stdout:Package ceph-mgr-dashboard-2:20.3.0-5145.g5fdb7d8f.el9.noarch is already installed. 2026-02-12T12:31:13.533 INFO:teuthology.orchestra.run.trial063.stdout:Dependencies resolved. 2026-02-12T12:31:13.533 INFO:teuthology.orchestra.run.trial063.stdout:Nothing to do. 2026-02-12T12:31:13.533 INFO:teuthology.orchestra.run.trial063.stdout:Complete! 2026-02-12T12:31:13.555 DEBUG:teuthology.orchestra.run.trial063:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-12T12:31:13.861 INFO:teuthology.orchestra.run.trial063.stdout:Last metadata expiration check: 0:00:04 ago on Thu 12 Feb 2026 12:31:09 PM UTC. 2026-02-12T12:31:13.926 INFO:teuthology.orchestra.run.trial063.stdout:Package ceph-mgr-diskprediction-local-2:20.3.0-5145.g5fdb7d8f.el9.noarch is already installed. 2026-02-12T12:31:13.944 INFO:teuthology.orchestra.run.trial063.stdout:Dependencies resolved. 2026-02-12T12:31:13.944 INFO:teuthology.orchestra.run.trial063.stdout:Nothing to do. 2026-02-12T12:31:13.944 INFO:teuthology.orchestra.run.trial063.stdout:Complete! 2026-02-12T12:31:13.966 DEBUG:teuthology.orchestra.run.trial063:> sudo yum -y install ceph-mgr-rook 2026-02-12T12:31:14.269 INFO:teuthology.orchestra.run.trial063.stdout:Last metadata expiration check: 0:00:05 ago on Thu 12 Feb 2026 12:31:09 PM UTC. 2026-02-12T12:31:14.334 INFO:teuthology.orchestra.run.trial063.stdout:Package ceph-mgr-rook-2:20.3.0-5145.g5fdb7d8f.el9.noarch is already installed. 2026-02-12T12:31:14.351 INFO:teuthology.orchestra.run.trial063.stdout:Dependencies resolved. 2026-02-12T12:31:14.352 INFO:teuthology.orchestra.run.trial063.stdout:Nothing to do. 2026-02-12T12:31:14.352 INFO:teuthology.orchestra.run.trial063.stdout:Complete! 2026-02-12T12:31:14.374 DEBUG:teuthology.orchestra.run.trial063:> sudo yum -y install ceph-mgr-cephadm 2026-02-12T12:31:14.678 INFO:teuthology.orchestra.run.trial063.stdout:Last metadata expiration check: 0:00:05 ago on Thu 12 Feb 2026 12:31:09 PM UTC. 2026-02-12T12:31:14.743 INFO:teuthology.orchestra.run.trial063.stdout:Package ceph-mgr-cephadm-2:20.3.0-5145.g5fdb7d8f.el9.noarch is already installed. 2026-02-12T12:31:14.761 INFO:teuthology.orchestra.run.trial063.stdout:Dependencies resolved. 2026-02-12T12:31:14.761 INFO:teuthology.orchestra.run.trial063.stdout:Nothing to do. 2026-02-12T12:31:14.761 INFO:teuthology.orchestra.run.trial063.stdout:Complete! 2026-02-12T12:31:14.784 DEBUG:teuthology.orchestra.run.trial063:> sudo yum -y install ceph-osd 2026-02-12T12:31:15.087 INFO:teuthology.orchestra.run.trial063.stdout:Last metadata expiration check: 0:00:06 ago on Thu 12 Feb 2026 12:31:09 PM UTC. 2026-02-12T12:31:15.152 INFO:teuthology.orchestra.run.trial063.stdout:Package ceph-osd-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:31:15.169 INFO:teuthology.orchestra.run.trial063.stdout:Dependencies resolved. 2026-02-12T12:31:15.170 INFO:teuthology.orchestra.run.trial063.stdout:Nothing to do. 2026-02-12T12:31:15.170 INFO:teuthology.orchestra.run.trial063.stdout:Complete! 2026-02-12T12:31:15.192 DEBUG:teuthology.orchestra.run.trial063:> sudo yum -y install ceph-osd-classic 2026-02-12T12:31:15.496 INFO:teuthology.orchestra.run.trial063.stdout:Last metadata expiration check: 0:00:06 ago on Thu 12 Feb 2026 12:31:09 PM UTC. 2026-02-12T12:31:15.561 INFO:teuthology.orchestra.run.trial063.stdout:Package ceph-osd-classic-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:31:15.579 INFO:teuthology.orchestra.run.trial063.stdout:Dependencies resolved. 2026-02-12T12:31:15.579 INFO:teuthology.orchestra.run.trial063.stdout:Nothing to do. 2026-02-12T12:31:15.580 INFO:teuthology.orchestra.run.trial063.stdout:Complete! 2026-02-12T12:31:15.602 DEBUG:teuthology.orchestra.run.trial063:> sudo yum -y install ceph-fuse 2026-02-12T12:31:15.897 INFO:teuthology.orchestra.run.trial063.stdout:Last metadata expiration check: 0:00:06 ago on Thu 12 Feb 2026 12:31:09 PM UTC. 2026-02-12T12:31:15.961 INFO:teuthology.orchestra.run.trial063.stdout:Package ceph-fuse-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:31:15.979 INFO:teuthology.orchestra.run.trial063.stdout:Dependencies resolved. 2026-02-12T12:31:15.979 INFO:teuthology.orchestra.run.trial063.stdout:Nothing to do. 2026-02-12T12:31:15.979 INFO:teuthology.orchestra.run.trial063.stdout:Complete! 2026-02-12T12:31:16.001 DEBUG:teuthology.orchestra.run.trial063:> sudo yum -y install ceph-volume 2026-02-12T12:31:16.298 INFO:teuthology.orchestra.run.trial063.stdout:Last metadata expiration check: 0:00:07 ago on Thu 12 Feb 2026 12:31:09 PM UTC. 2026-02-12T12:31:16.363 INFO:teuthology.orchestra.run.trial063.stdout:Package ceph-volume-2:20.3.0-5145.g5fdb7d8f.el9.noarch is already installed. 2026-02-12T12:31:16.380 INFO:teuthology.orchestra.run.trial063.stdout:Dependencies resolved. 2026-02-12T12:31:16.381 INFO:teuthology.orchestra.run.trial063.stdout:Nothing to do. 2026-02-12T12:31:16.381 INFO:teuthology.orchestra.run.trial063.stdout:Complete! 2026-02-12T12:31:16.403 DEBUG:teuthology.orchestra.run.trial063:> sudo yum -y install librados-devel 2026-02-12T12:31:16.703 INFO:teuthology.orchestra.run.trial063.stdout:Last metadata expiration check: 0:00:07 ago on Thu 12 Feb 2026 12:31:09 PM UTC. 2026-02-12T12:31:16.768 INFO:teuthology.orchestra.run.trial063.stdout:Package librados-devel-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:31:16.785 INFO:teuthology.orchestra.run.trial063.stdout:Dependencies resolved. 2026-02-12T12:31:16.786 INFO:teuthology.orchestra.run.trial063.stdout:Nothing to do. 2026-02-12T12:31:16.786 INFO:teuthology.orchestra.run.trial063.stdout:Complete! 2026-02-12T12:31:16.808 DEBUG:teuthology.orchestra.run.trial063:> sudo yum -y install libcephfs2 2026-02-12T12:31:17.107 INFO:teuthology.orchestra.run.trial063.stdout:Last metadata expiration check: 0:00:08 ago on Thu 12 Feb 2026 12:31:09 PM UTC. 2026-02-12T12:31:17.172 INFO:teuthology.orchestra.run.trial063.stdout:Package libcephfs2-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:31:17.190 INFO:teuthology.orchestra.run.trial063.stdout:Dependencies resolved. 2026-02-12T12:31:17.191 INFO:teuthology.orchestra.run.trial063.stdout:Nothing to do. 2026-02-12T12:31:17.191 INFO:teuthology.orchestra.run.trial063.stdout:Complete! 2026-02-12T12:31:17.213 DEBUG:teuthology.orchestra.run.trial063:> sudo yum -y install libcephfs-devel 2026-02-12T12:31:17.512 INFO:teuthology.orchestra.run.trial063.stdout:Last metadata expiration check: 0:00:08 ago on Thu 12 Feb 2026 12:31:09 PM UTC. 2026-02-12T12:31:17.577 INFO:teuthology.orchestra.run.trial063.stdout:Package libcephfs-devel-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:31:17.594 INFO:teuthology.orchestra.run.trial063.stdout:Dependencies resolved. 2026-02-12T12:31:17.595 INFO:teuthology.orchestra.run.trial063.stdout:Nothing to do. 2026-02-12T12:31:17.595 INFO:teuthology.orchestra.run.trial063.stdout:Complete! 2026-02-12T12:31:17.616 DEBUG:teuthology.orchestra.run.trial063:> sudo yum -y install librados2 2026-02-12T12:31:17.915 INFO:teuthology.orchestra.run.trial063.stdout:Last metadata expiration check: 0:00:08 ago on Thu 12 Feb 2026 12:31:09 PM UTC. 2026-02-12T12:31:17.979 INFO:teuthology.orchestra.run.trial063.stdout:Package librados2-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:31:17.997 INFO:teuthology.orchestra.run.trial063.stdout:Dependencies resolved. 2026-02-12T12:31:17.997 INFO:teuthology.orchestra.run.trial063.stdout:Nothing to do. 2026-02-12T12:31:17.998 INFO:teuthology.orchestra.run.trial063.stdout:Complete! 2026-02-12T12:31:18.020 DEBUG:teuthology.orchestra.run.trial063:> sudo yum -y install librbd1 2026-02-12T12:31:18.317 INFO:teuthology.orchestra.run.trial063.stdout:Last metadata expiration check: 0:00:09 ago on Thu 12 Feb 2026 12:31:09 PM UTC. 2026-02-12T12:31:18.381 INFO:teuthology.orchestra.run.trial063.stdout:Package librbd1-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:31:18.398 INFO:teuthology.orchestra.run.trial063.stdout:Dependencies resolved. 2026-02-12T12:31:18.399 INFO:teuthology.orchestra.run.trial063.stdout:Nothing to do. 2026-02-12T12:31:18.399 INFO:teuthology.orchestra.run.trial063.stdout:Complete! 2026-02-12T12:31:18.421 DEBUG:teuthology.orchestra.run.trial063:> sudo yum -y install python3-rados 2026-02-12T12:31:18.725 INFO:teuthology.orchestra.run.trial063.stdout:Last metadata expiration check: 0:00:09 ago on Thu 12 Feb 2026 12:31:09 PM UTC. 2026-02-12T12:31:18.789 INFO:teuthology.orchestra.run.trial063.stdout:Package python3-rados-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:31:18.807 INFO:teuthology.orchestra.run.trial063.stdout:Dependencies resolved. 2026-02-12T12:31:18.808 INFO:teuthology.orchestra.run.trial063.stdout:Nothing to do. 2026-02-12T12:31:18.808 INFO:teuthology.orchestra.run.trial063.stdout:Complete! 2026-02-12T12:31:18.830 DEBUG:teuthology.orchestra.run.trial063:> sudo yum -y install python3-rgw 2026-02-12T12:31:19.130 INFO:teuthology.orchestra.run.trial063.stdout:Last metadata expiration check: 0:00:10 ago on Thu 12 Feb 2026 12:31:09 PM UTC. 2026-02-12T12:31:19.195 INFO:teuthology.orchestra.run.trial063.stdout:Package python3-rgw-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:31:19.212 INFO:teuthology.orchestra.run.trial063.stdout:Dependencies resolved. 2026-02-12T12:31:19.213 INFO:teuthology.orchestra.run.trial063.stdout:Nothing to do. 2026-02-12T12:31:19.213 INFO:teuthology.orchestra.run.trial063.stdout:Complete! 2026-02-12T12:31:19.235 DEBUG:teuthology.orchestra.run.trial063:> sudo yum -y install python3-cephfs 2026-02-12T12:31:19.535 INFO:teuthology.orchestra.run.trial063.stdout:Last metadata expiration check: 0:00:10 ago on Thu 12 Feb 2026 12:31:09 PM UTC. 2026-02-12T12:31:19.600 INFO:teuthology.orchestra.run.trial063.stdout:Package python3-cephfs-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:31:19.619 INFO:teuthology.orchestra.run.trial063.stdout:Dependencies resolved. 2026-02-12T12:31:19.620 INFO:teuthology.orchestra.run.trial063.stdout:Nothing to do. 2026-02-12T12:31:19.620 INFO:teuthology.orchestra.run.trial063.stdout:Complete! 2026-02-12T12:31:19.641 DEBUG:teuthology.orchestra.run.trial063:> sudo yum -y install python3-rbd 2026-02-12T12:31:19.941 INFO:teuthology.orchestra.run.trial063.stdout:Last metadata expiration check: 0:00:10 ago on Thu 12 Feb 2026 12:31:09 PM UTC. 2026-02-12T12:31:20.006 INFO:teuthology.orchestra.run.trial063.stdout:Package python3-rbd-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:31:20.023 INFO:teuthology.orchestra.run.trial063.stdout:Dependencies resolved. 2026-02-12T12:31:20.024 INFO:teuthology.orchestra.run.trial063.stdout:Nothing to do. 2026-02-12T12:31:20.024 INFO:teuthology.orchestra.run.trial063.stdout:Complete! 2026-02-12T12:31:20.046 DEBUG:teuthology.orchestra.run.trial063:> sudo yum -y install rbd-fuse 2026-02-12T12:31:20.348 INFO:teuthology.orchestra.run.trial063.stdout:Last metadata expiration check: 0:00:11 ago on Thu 12 Feb 2026 12:31:09 PM UTC. 2026-02-12T12:31:20.412 INFO:teuthology.orchestra.run.trial063.stdout:Package rbd-fuse-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:31:20.430 INFO:teuthology.orchestra.run.trial063.stdout:Dependencies resolved. 2026-02-12T12:31:20.430 INFO:teuthology.orchestra.run.trial063.stdout:Nothing to do. 2026-02-12T12:31:20.431 INFO:teuthology.orchestra.run.trial063.stdout:Complete! 2026-02-12T12:31:20.452 DEBUG:teuthology.orchestra.run.trial063:> sudo yum -y install rbd-mirror 2026-02-12T12:31:20.761 INFO:teuthology.orchestra.run.trial063.stdout:Last metadata expiration check: 0:00:11 ago on Thu 12 Feb 2026 12:31:09 PM UTC. 2026-02-12T12:31:20.825 INFO:teuthology.orchestra.run.trial063.stdout:Package rbd-mirror-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:31:20.842 INFO:teuthology.orchestra.run.trial063.stdout:Dependencies resolved. 2026-02-12T12:31:20.843 INFO:teuthology.orchestra.run.trial063.stdout:Nothing to do. 2026-02-12T12:31:20.843 INFO:teuthology.orchestra.run.trial063.stdout:Complete! 2026-02-12T12:31:20.864 DEBUG:teuthology.orchestra.run.trial063:> sudo yum -y install rbd-nbd 2026-02-12T12:31:21.166 INFO:teuthology.orchestra.run.trial063.stdout:Last metadata expiration check: 0:00:12 ago on Thu 12 Feb 2026 12:31:09 PM UTC. 2026-02-12T12:31:21.231 INFO:teuthology.orchestra.run.trial063.stdout:Package rbd-nbd-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:31:21.248 INFO:teuthology.orchestra.run.trial063.stdout:Dependencies resolved. 2026-02-12T12:31:21.249 INFO:teuthology.orchestra.run.trial063.stdout:Nothing to do. 2026-02-12T12:31:21.249 INFO:teuthology.orchestra.run.trial063.stdout:Complete! 2026-02-12T12:31:21.274 DEBUG:teuthology.parallel:result is None 2026-02-12T12:31:21.275 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0 2026-02-12T12:31:21.436 DEBUG:teuthology.orchestra.run.trial063:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-12T12:31:21.457 INFO:teuthology.orchestra.run.trial063.stdout:20.3.0-5145.g5fdb7d8f.el9 2026-02-12T12:31:21.458 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5145.g5fdb7d8f.el9 2026-02-12T12:31:21.458 INFO:teuthology.task.install:The correct ceph version 20.3.0-5145.g5fdb7d8f is installed. 2026-02-12T12:31:21.460 INFO:teuthology.task.install.util:Utilities already shipped, skip it... 2026-02-12T12:31:21.461 INFO:teuthology.run_tasks:Running task ceph... 2026-02-12T12:31:21.566 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2026-02-12T12:31:21.566 DEBUG:teuthology.orchestra.run.trial063:> sudo chmod 777 /var/log/ceph 2026-02-12T12:31:21.591 INFO:tasks.ceph:Disabling ceph logrotate... 2026-02-12T12:31:21.591 DEBUG:teuthology.orchestra.run.trial063:> sudo rm -f -- /etc/logrotate.d/ceph 2026-02-12T12:31:21.653 INFO:tasks.ceph:Creating extra log directories... 2026-02-12T12:31:21.654 DEBUG:teuthology.orchestra.run.trial063:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2026-02-12T12:31:21.728 INFO:tasks.ceph:Creating ceph cluster ceph... 2026-02-12T12:31:21.729 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore main device type': 'RANDOM_BLOCK_SSD', 'seastore max concurrent transactions': 8}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2026-02-12T12:31:21.729 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/mnt/teuthology/moagrawa-2026-02-10_06:35:43-crimson-rados-main-distro-debug-trial/42515', 'branch': 'main', 'description': 'crimson-rados/perf/{0-crimson_install clusters/{crimson-fixed-1} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/rbm$/{crimson_seastore_rbm} perf_ignorelist settings/optimized workloads/sample_radosbench}', 'email': None, 'first_in_suite': False, 'flavor': 'debug', 'job_id': '42515', 'kernel': {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'trial', 'name': 'moagrawa-2026-02-10_06:35:43-crimson-rados-main-distro-debug-trial', 'no_nested_subset': False, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd': {'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore main device type': 'RANDOM_BLOCK_SSD', 'seastore max concurrent transactions': 8}}, 'crimson_compat': True, 'flavor': 'debug', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH'], 'sha1': '5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'global': {'osd crush chooseleaf type': 0, 'osd pool default pg num': 128, 'osd pool default pgp num': 128, 'osd pool default size': 2}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'debug', 'sha1': '5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0'}}, 'workunit': {'branch': 'seastore_io_uring', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0'}}, 'owner': 'scheduled_moagrawa@soko04.front.sepia.ceph.com', 'priority': 1000, 'repo': 'https://github.com/ceph/ceph.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 2425, 'sha1': '5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'seastore_io_uring', 'suite_path': '/home/teuthworker/src/github.com_mohit84_ceph_5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/mohit84/ceph.git', 'suite_sha1': '5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0', 'targets': {'trial063.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHAvkLfvTmZ12iefE8b9mikT5HhlKlAe8pMet91IgN5SYYpWssChVxyc7J5+xOcjW+ECQpbWmh3P1eoeCRBpbYM='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'flavor': 'debug', 'sha1': '5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0'}}, {'pexec': {'all': ['sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson']}}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore main device type': 'RANDOM_BLOCK_SSD', 'seastore max concurrent transactions': 8}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}}, 'wait-for-scrub': False, 'crimson_compat': True, 'flavor': 'debug', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH'], 'sha1': '5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0', 'cluster': 'ceph'}}, {'ssh_keys': None}, {'cbt': {'benchmarks': {'radosbench': {'concurrent_ops': 4, 'concurrent_procs': 2, 'op_size': [4096], 'pool_monitoring_list': ['collectl'], 'pool_profile': 'replicated', 'run_monitoring_list': ['collectl'], 'time': 60, 'write_only': True}}, 'cluster': {'iterations': 1, 'osds_per_node': 3, 'pool_profiles': {'replicated': {'pg_size': 256, 'pgp_size': 256, 'replication': 'replicated'}}, 'user': 'ubuntu'}, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'c433f1062990a0488dc29a553589bc609a460691', 'timestamp': '2026-02-10_06:35:43', 'tube': 'trial', 'user': 'moagrawa', 'verbose': False, 'worker_log': '/home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.2161924'} 2026-02-12T12:31:21.730 DEBUG:teuthology.orchestra.run.trial063:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2026-02-12T12:31:21.776 DEBUG:teuthology.orchestra.run.trial063:> sudo install -d -m0777 -- /var/run/ceph 2026-02-12T12:31:21.840 DEBUG:teuthology.orchestra.run.trial063:> set -ex 2026-02-12T12:31:21.840 DEBUG:teuthology.orchestra.run.trial063:> dd if=/scratch_devs of=/dev/stdout 2026-02-12T12:31:21.897 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2026-02-12T12:31:21.898 DEBUG:teuthology.orchestra.run.trial063:> stat /dev/vg_nvme/lv_1 2026-02-12T12:31:21.953 INFO:teuthology.orchestra.run.trial063.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2026-02-12T12:31:21.953 INFO:teuthology.orchestra.run.trial063.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-12T12:31:21.953 INFO:teuthology.orchestra.run.trial063.stdout:Device: 6h/6d Inode: 1312 Links: 1 2026-02-12T12:31:21.953 INFO:teuthology.orchestra.run.trial063.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-12T12:31:21.953 INFO:teuthology.orchestra.run.trial063.stdout:Context: system_u:object_r:device_t:s0 2026-02-12T12:31:21.953 INFO:teuthology.orchestra.run.trial063.stdout:Access: 2026-02-12 12:29:35.710587700 +0000 2026-02-12T12:31:21.953 INFO:teuthology.orchestra.run.trial063.stdout:Modify: 2026-02-12 12:27:00.024123976 +0000 2026-02-12T12:31:21.953 INFO:teuthology.orchestra.run.trial063.stdout:Change: 2026-02-12 12:27:00.024123976 +0000 2026-02-12T12:31:21.954 INFO:teuthology.orchestra.run.trial063.stdout: Birth: 2026-02-12 12:27:00.024123976 +0000 2026-02-12T12:31:21.954 DEBUG:teuthology.orchestra.run.trial063:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2026-02-12T12:31:22.015 INFO:teuthology.orchestra.run.trial063.stderr:1+0 records in 2026-02-12T12:31:22.015 INFO:teuthology.orchestra.run.trial063.stderr:1+0 records out 2026-02-12T12:31:22.016 INFO:teuthology.orchestra.run.trial063.stderr:512 bytes copied, 0.000171032 s, 3.0 MB/s 2026-02-12T12:31:22.017 DEBUG:teuthology.orchestra.run.trial063:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2026-02-12T12:31:22.074 DEBUG:teuthology.orchestra.run.trial063:> stat /dev/vg_nvme/lv_2 2026-02-12T12:31:22.131 INFO:teuthology.orchestra.run.trial063.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2026-02-12T12:31:22.131 INFO:teuthology.orchestra.run.trial063.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-12T12:31:22.131 INFO:teuthology.orchestra.run.trial063.stdout:Device: 6h/6d Inode: 1307 Links: 1 2026-02-12T12:31:22.132 INFO:teuthology.orchestra.run.trial063.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-12T12:31:22.132 INFO:teuthology.orchestra.run.trial063.stdout:Context: system_u:object_r:device_t:s0 2026-02-12T12:31:22.132 INFO:teuthology.orchestra.run.trial063.stdout:Access: 2026-02-12 12:29:35.710587700 +0000 2026-02-12T12:31:22.132 INFO:teuthology.orchestra.run.trial063.stdout:Modify: 2026-02-12 12:27:00.024123976 +0000 2026-02-12T12:31:22.132 INFO:teuthology.orchestra.run.trial063.stdout:Change: 2026-02-12 12:27:00.024123976 +0000 2026-02-12T12:31:22.132 INFO:teuthology.orchestra.run.trial063.stdout: Birth: 2026-02-12 12:27:00.024123976 +0000 2026-02-12T12:31:22.132 DEBUG:teuthology.orchestra.run.trial063:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2026-02-12T12:31:22.194 INFO:teuthology.orchestra.run.trial063.stderr:1+0 records in 2026-02-12T12:31:22.194 INFO:teuthology.orchestra.run.trial063.stderr:1+0 records out 2026-02-12T12:31:22.194 INFO:teuthology.orchestra.run.trial063.stderr:512 bytes copied, 0.000182143 s, 2.8 MB/s 2026-02-12T12:31:22.195 DEBUG:teuthology.orchestra.run.trial063:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2026-02-12T12:31:22.252 DEBUG:teuthology.orchestra.run.trial063:> stat /dev/vg_nvme/lv_3 2026-02-12T12:31:22.308 INFO:teuthology.orchestra.run.trial063.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2026-02-12T12:31:22.308 INFO:teuthology.orchestra.run.trial063.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-12T12:31:22.308 INFO:teuthology.orchestra.run.trial063.stdout:Device: 6h/6d Inode: 1304 Links: 1 2026-02-12T12:31:22.309 INFO:teuthology.orchestra.run.trial063.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-12T12:31:22.309 INFO:teuthology.orchestra.run.trial063.stdout:Context: system_u:object_r:device_t:s0 2026-02-12T12:31:22.309 INFO:teuthology.orchestra.run.trial063.stdout:Access: 2026-02-12 12:29:35.710587700 +0000 2026-02-12T12:31:22.309 INFO:teuthology.orchestra.run.trial063.stdout:Modify: 2026-02-12 12:27:00.024123976 +0000 2026-02-12T12:31:22.309 INFO:teuthology.orchestra.run.trial063.stdout:Change: 2026-02-12 12:27:00.024123976 +0000 2026-02-12T12:31:22.309 INFO:teuthology.orchestra.run.trial063.stdout: Birth: 2026-02-12 12:27:00.024123976 +0000 2026-02-12T12:31:22.309 DEBUG:teuthology.orchestra.run.trial063:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2026-02-12T12:31:22.371 INFO:teuthology.orchestra.run.trial063.stderr:1+0 records in 2026-02-12T12:31:22.371 INFO:teuthology.orchestra.run.trial063.stderr:1+0 records out 2026-02-12T12:31:22.371 INFO:teuthology.orchestra.run.trial063.stderr:512 bytes copied, 0.00013283 s, 3.9 MB/s 2026-02-12T12:31:22.372 DEBUG:teuthology.orchestra.run.trial063:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2026-02-12T12:31:22.431 DEBUG:teuthology.orchestra.run.trial063:> stat /dev/vg_nvme/lv_4 2026-02-12T12:31:22.489 INFO:teuthology.orchestra.run.trial063.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2026-02-12T12:31:22.490 INFO:teuthology.orchestra.run.trial063.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-12T12:31:22.490 INFO:teuthology.orchestra.run.trial063.stdout:Device: 6h/6d Inode: 1293 Links: 1 2026-02-12T12:31:22.490 INFO:teuthology.orchestra.run.trial063.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-12T12:31:22.490 INFO:teuthology.orchestra.run.trial063.stdout:Context: system_u:object_r:device_t:s0 2026-02-12T12:31:22.490 INFO:teuthology.orchestra.run.trial063.stdout:Access: 2026-02-12 12:29:35.710587700 +0000 2026-02-12T12:31:22.490 INFO:teuthology.orchestra.run.trial063.stdout:Modify: 2026-02-12 12:27:00.024123976 +0000 2026-02-12T12:31:22.490 INFO:teuthology.orchestra.run.trial063.stdout:Change: 2026-02-12 12:27:00.024123976 +0000 2026-02-12T12:31:22.490 INFO:teuthology.orchestra.run.trial063.stdout: Birth: 2026-02-12 12:27:00.024123976 +0000 2026-02-12T12:31:22.491 DEBUG:teuthology.orchestra.run.trial063:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2026-02-12T12:31:22.551 INFO:teuthology.orchestra.run.trial063.stderr:1+0 records in 2026-02-12T12:31:22.552 INFO:teuthology.orchestra.run.trial063.stderr:1+0 records out 2026-02-12T12:31:22.552 INFO:teuthology.orchestra.run.trial063.stderr:512 bytes copied, 0.000134863 s, 3.8 MB/s 2026-02-12T12:31:22.553 DEBUG:teuthology.orchestra.run.trial063:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2026-02-12T12:31:22.610 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-12T12:31:22.611 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@trial063.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2026-02-12T12:31:22.611 INFO:tasks.ceph:Generating config... 2026-02-12T12:31:22.613 INFO:tasks.ceph:[osd] debug monc = 20 2026-02-12T12:31:22.613 INFO:tasks.ceph:[osd] crimson cpu num = 4 2026-02-12T12:31:22.613 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2026-02-12T12:31:22.613 INFO:tasks.ceph:[osd] debug filestore = 0/0 2026-02-12T12:31:22.614 INFO:tasks.ceph:[osd] debug journal = 0/0 2026-02-12T12:31:22.614 INFO:tasks.ceph:[osd] debug ms = 0/0 2026-02-12T12:31:22.614 INFO:tasks.ceph:[osd] debug osd = 0/0 2026-02-12T12:31:22.614 INFO:tasks.ceph:[osd] debug seastore = 20 2026-02-12T12:31:22.614 INFO:tasks.ceph:[osd] debug seastore backref = 20 2026-02-12T12:31:22.614 INFO:tasks.ceph:[osd] debug seastore cache = 20 2026-02-12T12:31:22.614 INFO:tasks.ceph:[osd] debug seastore cleaner = 20 2026-02-12T12:31:22.614 INFO:tasks.ceph:[osd] debug seastore device = 20 2026-02-12T12:31:22.614 INFO:tasks.ceph:[osd] debug seastore epm = 20 2026-02-12T12:31:22.614 INFO:tasks.ceph:[osd] debug seastore fixedkv tree = 20 2026-02-12T12:31:22.614 INFO:tasks.ceph:[osd] debug seastore journal = 20 2026-02-12T12:31:22.614 INFO:tasks.ceph:[osd] debug seastore lba = 20 2026-02-12T12:31:22.615 INFO:tasks.ceph:[osd] debug seastore odata = 20 2026-02-12T12:31:22.615 INFO:tasks.ceph:[osd] debug seastore omap = 20 2026-02-12T12:31:22.615 INFO:tasks.ceph:[osd] debug seastore onode = 20 2026-02-12T12:31:22.615 INFO:tasks.ceph:[osd] debug seastore t = 20 2026-02-12T12:31:22.615 INFO:tasks.ceph:[osd] debug seastore tm = 20 2026-02-12T12:31:22.615 INFO:tasks.ceph:[osd] osd objectstore = seastore 2026-02-12T12:31:22.615 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2026-02-12T12:31:22.615 INFO:tasks.ceph:[osd] seastore cachepin size pershard = 64M 2026-02-12T12:31:22.615 INFO:tasks.ceph:[osd] seastore main device type = RANDOM_BLOCK_SSD 2026-02-12T12:31:22.615 INFO:tasks.ceph:[osd] seastore max concurrent transactions = 8 2026-02-12T12:31:22.615 INFO:tasks.ceph:[global] auth client required = none 2026-02-12T12:31:22.615 INFO:tasks.ceph:[global] auth cluster required = none 2026-02-12T12:31:22.615 INFO:tasks.ceph:[global] auth service required = none 2026-02-12T12:31:22.616 INFO:tasks.ceph:[global] auth supported = none 2026-02-12T12:31:22.616 INFO:tasks.ceph:[global] debug asok = 0/0 2026-02-12T12:31:22.616 INFO:tasks.ceph:[global] debug auth = 0/0 2026-02-12T12:31:22.616 INFO:tasks.ceph:[global] debug bdev = 0/0 2026-02-12T12:31:22.616 INFO:tasks.ceph:[global] debug bluefs = 0/0 2026-02-12T12:31:22.616 INFO:tasks.ceph:[global] debug bluestore = 0/0 2026-02-12T12:31:22.616 INFO:tasks.ceph:[global] debug buffer = 0/0 2026-02-12T12:31:22.616 INFO:tasks.ceph:[global] debug civetweb = 0/0 2026-02-12T12:31:22.616 INFO:tasks.ceph:[global] debug client = 0/0 2026-02-12T12:31:22.616 INFO:tasks.ceph:[global] debug compressor = 0/0 2026-02-12T12:31:22.616 INFO:tasks.ceph:[global] debug context = 0/0 2026-02-12T12:31:22.616 INFO:tasks.ceph:[global] debug crush = 0/0 2026-02-12T12:31:22.617 INFO:tasks.ceph:[global] debug crypto = 0/0 2026-02-12T12:31:22.617 INFO:tasks.ceph:[global] debug dpdk = 0/0 2026-02-12T12:31:22.617 INFO:tasks.ceph:[global] debug eventtrace = 0/0 2026-02-12T12:31:22.617 INFO:tasks.ceph:[global] debug filer = 0/0 2026-02-12T12:31:22.617 INFO:tasks.ceph:[global] debug filestore = 0/0 2026-02-12T12:31:22.617 INFO:tasks.ceph:[global] debug finisher = 0/0 2026-02-12T12:31:22.617 INFO:tasks.ceph:[global] debug fuse = 0/0 2026-02-12T12:31:22.617 INFO:tasks.ceph:[global] debug heartbeatmap = 0/0 2026-02-12T12:31:22.617 INFO:tasks.ceph:[global] debug javaclient = 0/0 2026-02-12T12:31:22.617 INFO:tasks.ceph:[global] debug journal = 0/0 2026-02-12T12:31:22.617 INFO:tasks.ceph:[global] debug journaler = 0/0 2026-02-12T12:31:22.617 INFO:tasks.ceph:[global] debug kstore = 0/0 2026-02-12T12:31:22.617 INFO:tasks.ceph:[global] debug leveldb = 0/0 2026-02-12T12:31:22.618 INFO:tasks.ceph:[global] debug lockdep = 0/0 2026-02-12T12:31:22.618 INFO:tasks.ceph:[global] debug mds = 0/0 2026-02-12T12:31:22.618 INFO:tasks.ceph:[global] debug mds balancer = 0/0 2026-02-12T12:31:22.618 INFO:tasks.ceph:[global] debug mds locker = 0/0 2026-02-12T12:31:22.618 INFO:tasks.ceph:[global] debug mds log = 0/0 2026-02-12T12:31:22.618 INFO:tasks.ceph:[global] debug mds log expire = 0/0 2026-02-12T12:31:22.618 INFO:tasks.ceph:[global] debug mds migrator = 0/0 2026-02-12T12:31:22.618 INFO:tasks.ceph:[global] debug memdb = 0/0 2026-02-12T12:31:22.618 INFO:tasks.ceph:[global] debug mgr = 0/0 2026-02-12T12:31:22.618 INFO:tasks.ceph:[global] debug mgrc = 0/0 2026-02-12T12:31:22.618 INFO:tasks.ceph:[global] debug mon = 0/0 2026-02-12T12:31:22.618 INFO:tasks.ceph:[global] debug monc = 0/0 2026-02-12T12:31:22.618 INFO:tasks.ceph:[global] debug ms = 0/0 2026-02-12T12:31:22.619 INFO:tasks.ceph:[global] debug objclass = 0/0 2026-02-12T12:31:22.619 INFO:tasks.ceph:[global] debug objectcacher = 0/0 2026-02-12T12:31:22.619 INFO:tasks.ceph:[global] debug objecter = 0/0 2026-02-12T12:31:22.619 INFO:tasks.ceph:[global] debug optracker = 0/0 2026-02-12T12:31:22.619 INFO:tasks.ceph:[global] debug osd = 0/0 2026-02-12T12:31:22.619 INFO:tasks.ceph:[global] debug paxos = 0/0 2026-02-12T12:31:22.619 INFO:tasks.ceph:[global] debug perfcounter = 0/0 2026-02-12T12:31:22.619 INFO:tasks.ceph:[global] debug rados = 0/0 2026-02-12T12:31:22.619 INFO:tasks.ceph:[global] debug rbd = 0/0 2026-02-12T12:31:22.619 INFO:tasks.ceph:[global] debug rbd mirror = 0/0 2026-02-12T12:31:22.619 INFO:tasks.ceph:[global] debug rbd replay = 0/0 2026-02-12T12:31:22.619 INFO:tasks.ceph:[global] debug refs = 0/0 2026-02-12T12:31:22.619 INFO:tasks.ceph:[global] debug rgw = 0/0 2026-02-12T12:31:22.619 INFO:tasks.ceph:[global] debug rgw lifecycle = 0/0 2026-02-12T12:31:22.620 INFO:tasks.ceph:[global] debug rgw notification = 0/0 2026-02-12T12:31:22.620 INFO:tasks.ceph:[global] debug rgw sync = 0/0 2026-02-12T12:31:22.620 INFO:tasks.ceph:[global] debug rocksdb = 0/0 2026-02-12T12:31:22.620 INFO:tasks.ceph:[global] debug striper = 0/0 2026-02-12T12:31:22.620 INFO:tasks.ceph:[global] debug throttle = 0/0 2026-02-12T12:31:22.620 INFO:tasks.ceph:[global] debug timer = 0/0 2026-02-12T12:31:22.620 INFO:tasks.ceph:[global] debug tp = 0/0 2026-02-12T12:31:22.620 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2026-02-12T12:31:22.620 INFO:tasks.ceph:[global] ms client mode = crc 2026-02-12T12:31:22.620 INFO:tasks.ceph:[global] ms cluster mode = crc 2026-02-12T12:31:22.620 INFO:tasks.ceph:[global] ms mon client mode = crc 2026-02-12T12:31:22.620 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2026-02-12T12:31:22.620 INFO:tasks.ceph:[global] ms mon service mode = crc 2026-02-12T12:31:22.621 INFO:tasks.ceph:[global] ms service mode = crc 2026-02-12T12:31:22.621 INFO:tasks.ceph:[mgr] debug mgr = 20 2026-02-12T12:31:22.621 INFO:tasks.ceph:[mgr] debug ms = 1 2026-02-12T12:31:22.621 INFO:tasks.ceph:[mon] debug mon = 0/0 2026-02-12T12:31:22.621 INFO:tasks.ceph:[mon] debug ms = 0/0 2026-02-12T12:31:22.621 INFO:tasks.ceph:[mon] debug paxos = 0/0 2026-02-12T12:31:22.621 INFO:tasks.ceph:[mon] osd pool default crimson = True 2026-02-12T12:31:22.621 INFO:tasks.ceph:Setting up mon.a... 2026-02-12T12:31:22.621 DEBUG:teuthology.orchestra.run.trial063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2026-02-12T12:31:22.691 INFO:teuthology.orchestra.run.trial063.stdout:creating /etc/ceph/ceph.keyring 2026-02-12T12:31:22.694 DEBUG:teuthology.orchestra.run.trial063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2026-02-12T12:31:22.778 DEBUG:teuthology.orchestra.run.trial063:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-12T12:31:22.843 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '10.20.193.63')] 2026-02-12T12:31:22.843 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'none', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '10.20.193.63', 'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': 'true', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore main device type': 'RANDOM_BLOCK_SSD', 'seastore max concurrent transactions': 8}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': '0/0', 'debug mon': '0/0', 'debug paxos': '0/0', 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}} 2026-02-12T12:31:22.845 DEBUG:teuthology.orchestra.run.trial063:> set -ex 2026-02-12T12:31:22.845 DEBUG:teuthology.orchestra.run.trial063:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2026-02-12T12:31:22.899 DEBUG:teuthology.orchestra.run.trial063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 10.20.193.63 --print /home/ubuntu/cephtest/ceph.monmap 2026-02-12T12:31:22.979 INFO:teuthology.orchestra.run.trial063.stderr:2026-02-12T12:31:22.978+0000 7f5f7f282f40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:31:22.979 INFO:teuthology.orchestra.run.trial063.stderr:2026-02-12T12:31:22.978+0000 7f5f7f282f40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:31:22.980 INFO:teuthology.orchestra.run.trial063.stderr:2026-02-12T12:31:22.978+0000 7f5f7f282f40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:31:22.981 INFO:teuthology.orchestra.run.trial063.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2026-02-12T12:31:22.981 INFO:teuthology.orchestra.run.trial063.stdout:monmaptool: generated fsid 0ceebbf4-3330-468c-a6ec-0a57b2aaee66 2026-02-12T12:31:22.981 INFO:teuthology.orchestra.run.trial063.stdout:setting min_mon_release = tentacle 2026-02-12T12:31:22.981 INFO:teuthology.orchestra.run.trial063.stdout:epoch 0 2026-02-12T12:31:22.981 INFO:teuthology.orchestra.run.trial063.stdout:fsid 0ceebbf4-3330-468c-a6ec-0a57b2aaee66 2026-02-12T12:31:22.981 INFO:teuthology.orchestra.run.trial063.stdout:last_changed 2026-02-12T12:31:22.978773+0000 2026-02-12T12:31:22.981 INFO:teuthology.orchestra.run.trial063.stdout:created 2026-02-12T12:31:22.978773+0000 2026-02-12T12:31:22.982 INFO:teuthology.orchestra.run.trial063.stdout:min_mon_release 20 (tentacle) 2026-02-12T12:31:22.982 INFO:teuthology.orchestra.run.trial063.stdout:election_strategy: 1 2026-02-12T12:31:22.982 INFO:teuthology.orchestra.run.trial063.stdout:0: [v2:10.20.193.63:3300/0,v1:10.20.193.63:6789/0] mon.a 2026-02-12T12:31:22.982 INFO:teuthology.orchestra.run.trial063.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2026-02-12T12:31:22.983 DEBUG:teuthology.orchestra.run.trial063:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2026-02-12T12:31:23.040 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 0ceebbf4-3330-468c-a6ec-0a57b2aaee66... 2026-02-12T12:31:23.042 DEBUG:teuthology.orchestra.run.trial063:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2026-02-12T12:31:23.119 INFO:teuthology.orchestra.run.trial063.stdout:[global] 2026-02-12T12:31:23.119 INFO:teuthology.orchestra.run.trial063.stdout: chdir = "" 2026-02-12T12:31:23.120 INFO:teuthology.orchestra.run.trial063.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2026-02-12T12:31:23.120 INFO:teuthology.orchestra.run.trial063.stdout: auth supported = none 2026-02-12T12:31:23.120 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:31:23.120 INFO:teuthology.orchestra.run.trial063.stdout: filestore xattr use omap = true 2026-02-12T12:31:23.120 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:31:23.120 INFO:teuthology.orchestra.run.trial063.stdout: mon clock drift allowed = 1.000 2026-02-12T12:31:23.120 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:31:23.120 INFO:teuthology.orchestra.run.trial063.stdout: osd crush chooseleaf type = 0 2026-02-12T12:31:23.120 INFO:teuthology.orchestra.run.trial063.stdout: auth debug = true 2026-02-12T12:31:23.120 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:31:23.120 INFO:teuthology.orchestra.run.trial063.stdout: ms die on old message = true 2026-02-12T12:31:23.121 INFO:teuthology.orchestra.run.trial063.stdout: ms die on bug = true 2026-02-12T12:31:23.121 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:31:23.121 INFO:teuthology.orchestra.run.trial063.stdout: mon max pg per osd = 10000 # >= luminous 2026-02-12T12:31:23.121 INFO:teuthology.orchestra.run.trial063.stdout: mon pg warn max object skew = 0 2026-02-12T12:31:23.121 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:31:23.121 INFO:teuthology.orchestra.run.trial063.stdout: # disable pg_autoscaler by default for new pools 2026-02-12T12:31:23.121 INFO:teuthology.orchestra.run.trial063.stdout: osd_pool_default_pg_autoscale_mode = off 2026-02-12T12:31:23.121 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:31:23.121 INFO:teuthology.orchestra.run.trial063.stdout: osd pool default size = 2 2026-02-12T12:31:23.121 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:31:23.121 INFO:teuthology.orchestra.run.trial063.stdout: mon osd allow primary affinity = true 2026-02-12T12:31:23.122 INFO:teuthology.orchestra.run.trial063.stdout: mon osd allow pg remap = true 2026-02-12T12:31:23.122 INFO:teuthology.orchestra.run.trial063.stdout: mon warn on legacy crush tunables = false 2026-02-12T12:31:23.122 INFO:teuthology.orchestra.run.trial063.stdout: mon warn on crush straw calc version zero = false 2026-02-12T12:31:23.122 INFO:teuthology.orchestra.run.trial063.stdout: mon warn on no sortbitwise = false 2026-02-12T12:31:23.122 INFO:teuthology.orchestra.run.trial063.stdout: mon warn on osd down out interval zero = false 2026-02-12T12:31:23.122 INFO:teuthology.orchestra.run.trial063.stdout: mon warn on too few osds = false 2026-02-12T12:31:23.122 INFO:teuthology.orchestra.run.trial063.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2026-02-12T12:31:23.122 INFO:teuthology.orchestra.run.trial063.stdout: mon_warn_on_pool_no_redundancy = false 2026-02-12T12:31:23.122 INFO:teuthology.orchestra.run.trial063.stdout: mon_allow_pool_size_one = true 2026-02-12T12:31:23.122 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:31:23.122 INFO:teuthology.orchestra.run.trial063.stdout: osd pool default erasure code profile = plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2026-02-12T12:31:23.122 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:31:23.122 INFO:teuthology.orchestra.run.trial063.stdout: osd default data pool replay window = 5 2026-02-12T12:31:23.123 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:31:23.123 INFO:teuthology.orchestra.run.trial063.stdout: mon allow pool delete = true 2026-02-12T12:31:23.123 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:31:23.123 INFO:teuthology.orchestra.run.trial063.stdout: mon cluster log file level = debug 2026-02-12T12:31:23.123 INFO:teuthology.orchestra.run.trial063.stdout: debug asserts on shutdown = true 2026-02-12T12:31:23.123 INFO:teuthology.orchestra.run.trial063.stdout: mon health detail to clog = false 2026-02-12T12:31:23.123 INFO:teuthology.orchestra.run.trial063.stdout: mon host = 10.20.193.63 2026-02-12T12:31:23.123 INFO:teuthology.orchestra.run.trial063.stdout: auth client required = none 2026-02-12T12:31:23.123 INFO:teuthology.orchestra.run.trial063.stdout: auth cluster required = none 2026-02-12T12:31:23.123 INFO:teuthology.orchestra.run.trial063.stdout: auth service required = none 2026-02-12T12:31:23.123 INFO:teuthology.orchestra.run.trial063.stdout: debug asok = 0/0 2026-02-12T12:31:23.123 INFO:teuthology.orchestra.run.trial063.stdout: debug auth = 0/0 2026-02-12T12:31:23.123 INFO:teuthology.orchestra.run.trial063.stdout: debug bdev = 0/0 2026-02-12T12:31:23.124 INFO:teuthology.orchestra.run.trial063.stdout: debug bluefs = 0/0 2026-02-12T12:31:23.124 INFO:teuthology.orchestra.run.trial063.stdout: debug bluestore = 0/0 2026-02-12T12:31:23.124 INFO:teuthology.orchestra.run.trial063.stdout: debug buffer = 0/0 2026-02-12T12:31:23.124 INFO:teuthology.orchestra.run.trial063.stdout: debug civetweb = 0/0 2026-02-12T12:31:23.124 INFO:teuthology.orchestra.run.trial063.stdout: debug client = 0/0 2026-02-12T12:31:23.124 INFO:teuthology.orchestra.run.trial063.stdout: debug compressor = 0/0 2026-02-12T12:31:23.124 INFO:teuthology.orchestra.run.trial063.stdout: debug context = 0/0 2026-02-12T12:31:23.124 INFO:teuthology.orchestra.run.trial063.stdout: debug crush = 0/0 2026-02-12T12:31:23.124 INFO:teuthology.orchestra.run.trial063.stdout: debug crypto = 0/0 2026-02-12T12:31:23.124 INFO:teuthology.orchestra.run.trial063.stdout: debug dpdk = 0/0 2026-02-12T12:31:23.124 INFO:teuthology.orchestra.run.trial063.stdout: debug eventtrace = 0/0 2026-02-12T12:31:23.124 INFO:teuthology.orchestra.run.trial063.stdout: debug filer = 0/0 2026-02-12T12:31:23.124 INFO:teuthology.orchestra.run.trial063.stdout: debug filestore = 0/0 2026-02-12T12:31:23.125 INFO:teuthology.orchestra.run.trial063.stdout: debug finisher = 0/0 2026-02-12T12:31:23.125 INFO:teuthology.orchestra.run.trial063.stdout: debug fuse = 0/0 2026-02-12T12:31:23.125 INFO:teuthology.orchestra.run.trial063.stdout: debug heartbeatmap = 0/0 2026-02-12T12:31:23.125 INFO:teuthology.orchestra.run.trial063.stdout: debug javaclient = 0/0 2026-02-12T12:31:23.125 INFO:teuthology.orchestra.run.trial063.stdout: debug journal = 0/0 2026-02-12T12:31:23.125 INFO:teuthology.orchestra.run.trial063.stdout: debug journaler = 0/0 2026-02-12T12:31:23.125 INFO:teuthology.orchestra.run.trial063.stdout: debug kstore = 0/0 2026-02-12T12:31:23.125 INFO:teuthology.orchestra.run.trial063.stdout: debug leveldb = 0/0 2026-02-12T12:31:23.125 INFO:teuthology.orchestra.run.trial063.stdout: debug lockdep = 0/0 2026-02-12T12:31:23.125 INFO:teuthology.orchestra.run.trial063.stdout: debug mds = 0/0 2026-02-12T12:31:23.125 INFO:teuthology.orchestra.run.trial063.stdout: debug mds balancer = 0/0 2026-02-12T12:31:23.125 INFO:teuthology.orchestra.run.trial063.stdout: debug mds locker = 0/0 2026-02-12T12:31:23.125 INFO:teuthology.orchestra.run.trial063.stdout: debug mds log = 0/0 2026-02-12T12:31:23.126 INFO:teuthology.orchestra.run.trial063.stdout: debug mds log expire = 0/0 2026-02-12T12:31:23.126 INFO:teuthology.orchestra.run.trial063.stdout: debug mds migrator = 0/0 2026-02-12T12:31:23.126 INFO:teuthology.orchestra.run.trial063.stdout: debug memdb = 0/0 2026-02-12T12:31:23.126 INFO:teuthology.orchestra.run.trial063.stdout: debug mgr = 0/0 2026-02-12T12:31:23.126 INFO:teuthology.orchestra.run.trial063.stdout: debug mgrc = 0/0 2026-02-12T12:31:23.126 INFO:teuthology.orchestra.run.trial063.stdout: debug mon = 0/0 2026-02-12T12:31:23.126 INFO:teuthology.orchestra.run.trial063.stdout: debug monc = 0/0 2026-02-12T12:31:23.126 INFO:teuthology.orchestra.run.trial063.stdout: debug ms = 0/0 2026-02-12T12:31:23.126 INFO:teuthology.orchestra.run.trial063.stdout: debug objclass = 0/0 2026-02-12T12:31:23.126 INFO:teuthology.orchestra.run.trial063.stdout: debug objectcacher = 0/0 2026-02-12T12:31:23.126 INFO:teuthology.orchestra.run.trial063.stdout: debug objecter = 0/0 2026-02-12T12:31:23.126 INFO:teuthology.orchestra.run.trial063.stdout: debug optracker = 0/0 2026-02-12T12:31:23.126 INFO:teuthology.orchestra.run.trial063.stdout: debug osd = 0/0 2026-02-12T12:31:23.127 INFO:teuthology.orchestra.run.trial063.stdout: debug paxos = 0/0 2026-02-12T12:31:23.127 INFO:teuthology.orchestra.run.trial063.stdout: debug perfcounter = 0/0 2026-02-12T12:31:23.127 INFO:teuthology.orchestra.run.trial063.stdout: debug rados = 0/0 2026-02-12T12:31:23.127 INFO:teuthology.orchestra.run.trial063.stdout: debug rbd = 0/0 2026-02-12T12:31:23.127 INFO:teuthology.orchestra.run.trial063.stdout: debug rbd mirror = 0/0 2026-02-12T12:31:23.127 INFO:teuthology.orchestra.run.trial063.stdout: debug rbd replay = 0/0 2026-02-12T12:31:23.127 INFO:teuthology.orchestra.run.trial063.stdout: debug refs = 0/0 2026-02-12T12:31:23.127 INFO:teuthology.orchestra.run.trial063.stdout: debug rgw = 0/0 2026-02-12T12:31:23.127 INFO:teuthology.orchestra.run.trial063.stdout: debug rgw lifecycle = 0/0 2026-02-12T12:31:23.127 INFO:teuthology.orchestra.run.trial063.stdout: debug rgw notification = 0/0 2026-02-12T12:31:23.127 INFO:teuthology.orchestra.run.trial063.stdout: debug rgw sync = 0/0 2026-02-12T12:31:23.127 INFO:teuthology.orchestra.run.trial063.stdout: debug rocksdb = 0/0 2026-02-12T12:31:23.127 INFO:teuthology.orchestra.run.trial063.stdout: debug striper = 0/0 2026-02-12T12:31:23.128 INFO:teuthology.orchestra.run.trial063.stdout: debug throttle = 0/0 2026-02-12T12:31:23.128 INFO:teuthology.orchestra.run.trial063.stdout: debug timer = 0/0 2026-02-12T12:31:23.128 INFO:teuthology.orchestra.run.trial063.stdout: debug tp = 0/0 2026-02-12T12:31:23.128 INFO:teuthology.orchestra.run.trial063.stdout: enable experimental unrecoverable data corrupting features = crimson 2026-02-12T12:31:23.128 INFO:teuthology.orchestra.run.trial063.stdout: ms client mode = crc 2026-02-12T12:31:23.128 INFO:teuthology.orchestra.run.trial063.stdout: ms cluster mode = crc 2026-02-12T12:31:23.128 INFO:teuthology.orchestra.run.trial063.stdout: ms mon client mode = crc 2026-02-12T12:31:23.128 INFO:teuthology.orchestra.run.trial063.stdout: ms mon cluster mode = crc 2026-02-12T12:31:23.128 INFO:teuthology.orchestra.run.trial063.stdout: ms mon service mode = crc 2026-02-12T12:31:23.128 INFO:teuthology.orchestra.run.trial063.stdout: ms service mode = crc 2026-02-12T12:31:23.128 INFO:teuthology.orchestra.run.trial063.stdout: fsid = 0ceebbf4-3330-468c-a6ec-0a57b2aaee66 2026-02-12T12:31:23.128 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:31:23.128 INFO:teuthology.orchestra.run.trial063.stdout:[osd] 2026-02-12T12:31:23.129 INFO:teuthology.orchestra.run.trial063.stdout: osd journal size = 100 2026-02-12T12:31:23.129 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:31:23.129 INFO:teuthology.orchestra.run.trial063.stdout: osd scrub load threshold = 5.0 2026-02-12T12:31:23.129 INFO:teuthology.orchestra.run.trial063.stdout: osd scrub max interval = 600 2026-02-12T12:31:23.129 INFO:teuthology.orchestra.run.trial063.stdout: osd mclock profile = high_recovery_ops 2026-02-12T12:31:23.129 INFO:teuthology.orchestra.run.trial063.stdout: osd mclock skip benchmark = true 2026-02-12T12:31:23.129 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:31:23.129 INFO:teuthology.orchestra.run.trial063.stdout: osd recover clone overlap = true 2026-02-12T12:31:23.129 INFO:teuthology.orchestra.run.trial063.stdout: osd recovery max chunk = 1048576 2026-02-12T12:31:23.129 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:31:23.129 INFO:teuthology.orchestra.run.trial063.stdout: osd debug shutdown = true 2026-02-12T12:31:23.129 INFO:teuthology.orchestra.run.trial063.stdout: osd debug op order = true 2026-02-12T12:31:23.130 INFO:teuthology.orchestra.run.trial063.stdout: osd debug verify stray on activate = true 2026-02-12T12:31:23.130 INFO:teuthology.orchestra.run.trial063.stdout: osd debug trim objects = true 2026-02-12T12:31:23.130 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:31:23.130 INFO:teuthology.orchestra.run.trial063.stdout: osd open classes on start = true 2026-02-12T12:31:23.130 INFO:teuthology.orchestra.run.trial063.stdout: osd debug pg log writeout = true 2026-02-12T12:31:23.130 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:31:23.130 INFO:teuthology.orchestra.run.trial063.stdout: osd deep scrub update digest min age = 30 2026-02-12T12:31:23.130 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:31:23.130 INFO:teuthology.orchestra.run.trial063.stdout: osd map max advance = 10 2026-02-12T12:31:23.130 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:31:23.131 INFO:teuthology.orchestra.run.trial063.stdout: journal zero on create = true 2026-02-12T12:31:23.131 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:31:23.131 INFO:teuthology.orchestra.run.trial063.stdout: filestore ondisk finisher threads = 3 2026-02-12T12:31:23.131 INFO:teuthology.orchestra.run.trial063.stdout: filestore apply finisher threads = 3 2026-02-12T12:31:23.131 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:31:23.131 INFO:teuthology.orchestra.run.trial063.stdout: bdev debug aio = true 2026-02-12T12:31:23.131 INFO:teuthology.orchestra.run.trial063.stdout: osd debug misdirected ops = true 2026-02-12T12:31:23.131 INFO:teuthology.orchestra.run.trial063.stdout: debug monc = 20 2026-02-12T12:31:23.131 INFO:teuthology.orchestra.run.trial063.stdout: crimson cpu num = 4 2026-02-12T12:31:23.131 INFO:teuthology.orchestra.run.trial063.stdout: crimson osd obc lru size = 10 2026-02-12T12:31:23.131 INFO:teuthology.orchestra.run.trial063.stdout: debug filestore = 0/0 2026-02-12T12:31:23.131 INFO:teuthology.orchestra.run.trial063.stdout: debug journal = 0/0 2026-02-12T12:31:23.131 INFO:teuthology.orchestra.run.trial063.stdout: debug ms = 0/0 2026-02-12T12:31:23.132 INFO:teuthology.orchestra.run.trial063.stdout: debug osd = 0/0 2026-02-12T12:31:23.132 INFO:teuthology.orchestra.run.trial063.stdout: debug seastore = 20 2026-02-12T12:31:23.132 INFO:teuthology.orchestra.run.trial063.stdout: debug seastore backref = 20 2026-02-12T12:31:23.132 INFO:teuthology.orchestra.run.trial063.stdout: debug seastore cache = 20 2026-02-12T12:31:23.132 INFO:teuthology.orchestra.run.trial063.stdout: debug seastore cleaner = 20 2026-02-12T12:31:23.132 INFO:teuthology.orchestra.run.trial063.stdout: debug seastore device = 20 2026-02-12T12:31:23.132 INFO:teuthology.orchestra.run.trial063.stdout: debug seastore epm = 20 2026-02-12T12:31:23.132 INFO:teuthology.orchestra.run.trial063.stdout: debug seastore fixedkv tree = 20 2026-02-12T12:31:23.132 INFO:teuthology.orchestra.run.trial063.stdout: debug seastore journal = 20 2026-02-12T12:31:23.132 INFO:teuthology.orchestra.run.trial063.stdout: debug seastore lba = 20 2026-02-12T12:31:23.132 INFO:teuthology.orchestra.run.trial063.stdout: debug seastore odata = 20 2026-02-12T12:31:23.132 INFO:teuthology.orchestra.run.trial063.stdout: debug seastore omap = 20 2026-02-12T12:31:23.132 INFO:teuthology.orchestra.run.trial063.stdout: debug seastore onode = 20 2026-02-12T12:31:23.132 INFO:teuthology.orchestra.run.trial063.stdout: debug seastore t = 20 2026-02-12T12:31:23.133 INFO:teuthology.orchestra.run.trial063.stdout: debug seastore tm = 20 2026-02-12T12:31:23.133 INFO:teuthology.orchestra.run.trial063.stdout: osd objectstore = seastore 2026-02-12T12:31:23.133 INFO:teuthology.orchestra.run.trial063.stdout: osd shutdown pgref assert = True 2026-02-12T12:31:23.133 INFO:teuthology.orchestra.run.trial063.stdout: seastore cachepin size pershard = 64M 2026-02-12T12:31:23.133 INFO:teuthology.orchestra.run.trial063.stdout: seastore main device type = RANDOM_BLOCK_SSD 2026-02-12T12:31:23.133 INFO:teuthology.orchestra.run.trial063.stdout: seastore max concurrent transactions = 8 2026-02-12T12:31:23.133 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:31:23.133 INFO:teuthology.orchestra.run.trial063.stdout:[mgr] 2026-02-12T12:31:23.133 INFO:teuthology.orchestra.run.trial063.stdout: debug ms = 1 2026-02-12T12:31:23.133 INFO:teuthology.orchestra.run.trial063.stdout: debug mgr = 20 2026-02-12T12:31:23.133 INFO:teuthology.orchestra.run.trial063.stdout: debug mon = 20 2026-02-12T12:31:23.133 INFO:teuthology.orchestra.run.trial063.stdout: debug auth = 20 2026-02-12T12:31:23.133 INFO:teuthology.orchestra.run.trial063.stdout: mon reweight min pgs per osd = 4 2026-02-12T12:31:23.134 INFO:teuthology.orchestra.run.trial063.stdout: mon reweight min bytes per osd = 10 2026-02-12T12:31:23.134 INFO:teuthology.orchestra.run.trial063.stdout: mgr/telemetry/nag = false 2026-02-12T12:31:23.134 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:31:23.134 INFO:teuthology.orchestra.run.trial063.stdout:[mon] 2026-02-12T12:31:23.134 INFO:teuthology.orchestra.run.trial063.stdout: debug ms = 0/0 2026-02-12T12:31:23.134 INFO:teuthology.orchestra.run.trial063.stdout: debug mon = 0/0 2026-02-12T12:31:23.134 INFO:teuthology.orchestra.run.trial063.stdout: debug paxos = 0/0 2026-02-12T12:31:23.134 INFO:teuthology.orchestra.run.trial063.stdout: debug auth = 20 2026-02-12T12:31:23.134 INFO:teuthology.orchestra.run.trial063.stdout: mon data avail warn = 5 2026-02-12T12:31:23.134 INFO:teuthology.orchestra.run.trial063.stdout: mon mgr mkfs grace = 240 2026-02-12T12:31:23.134 INFO:teuthology.orchestra.run.trial063.stdout: mon reweight min pgs per osd = 4 2026-02-12T12:31:23.134 INFO:teuthology.orchestra.run.trial063.stdout: mon osd reporter subtree level = osd 2026-02-12T12:31:23.135 INFO:teuthology.orchestra.run.trial063.stdout: mon osd prime pg temp = true 2026-02-12T12:31:23.135 INFO:teuthology.orchestra.run.trial063.stdout: mon reweight min bytes per osd = 10 2026-02-12T12:31:23.135 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:31:23.135 INFO:teuthology.orchestra.run.trial063.stdout: # rotate auth tickets quickly to exercise renewal paths 2026-02-12T12:31:23.135 INFO:teuthology.orchestra.run.trial063.stdout: auth mon ticket ttl = 660 # 11m 2026-02-12T12:31:23.135 INFO:teuthology.orchestra.run.trial063.stdout: auth service ticket ttl = 240 # 4m 2026-02-12T12:31:23.135 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:31:23.135 INFO:teuthology.orchestra.run.trial063.stdout: # don't complain about insecure global_id in the test suite 2026-02-12T12:31:23.135 INFO:teuthology.orchestra.run.trial063.stdout: mon_warn_on_insecure_global_id_reclaim = false 2026-02-12T12:31:23.135 INFO:teuthology.orchestra.run.trial063.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2026-02-12T12:31:23.135 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:31:23.135 INFO:teuthology.orchestra.run.trial063.stdout: # 1m isn't quite enough 2026-02-12T12:31:23.135 INFO:teuthology.orchestra.run.trial063.stdout: mon_down_mkfs_grace = 2m 2026-02-12T12:31:23.136 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:31:23.136 INFO:teuthology.orchestra.run.trial063.stdout: mon_warn_on_filestore_osds = false 2026-02-12T12:31:23.136 INFO:teuthology.orchestra.run.trial063.stdout: osd pool default crimson = True 2026-02-12T12:31:23.136 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:31:23.136 INFO:teuthology.orchestra.run.trial063.stdout:[client] 2026-02-12T12:31:23.136 INFO:teuthology.orchestra.run.trial063.stdout: rgw cache enabled = true 2026-02-12T12:31:23.136 INFO:teuthology.orchestra.run.trial063.stdout: rgw enable ops log = true 2026-02-12T12:31:23.136 INFO:teuthology.orchestra.run.trial063.stdout: rgw enable usage log = true 2026-02-12T12:31:23.136 INFO:teuthology.orchestra.run.trial063.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2026-02-12T12:31:23.136 INFO:teuthology.orchestra.run.trial063.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2026-02-12T12:31:23.136 INFO:teuthology.orchestra.run.trial063.stdout:[mon.a] 2026-02-12T12:31:23.138 INFO:tasks.ceph:Creating admin key on mon.a... 2026-02-12T12:31:23.138 DEBUG:teuthology.orchestra.run.trial063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2026-02-12T12:31:23.229 INFO:tasks.ceph:Copying monmap to all nodes... 2026-02-12T12:31:23.229 DEBUG:teuthology.orchestra.run.trial063:> set -ex 2026-02-12T12:31:23.229 DEBUG:teuthology.orchestra.run.trial063:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2026-02-12T12:31:23.245 DEBUG:teuthology.orchestra.run.trial063:> set -ex 2026-02-12T12:31:23.246 DEBUG:teuthology.orchestra.run.trial063:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2026-02-12T12:31:23.301 INFO:tasks.ceph:Sending monmap to node ubuntu@trial063.front.sepia.ceph.com 2026-02-12T12:31:23.302 DEBUG:teuthology.orchestra.run.trial063:> set -ex 2026-02-12T12:31:23.302 DEBUG:teuthology.orchestra.run.trial063:> sudo dd of=/etc/ceph/ceph.keyring 2026-02-12T12:31:23.302 DEBUG:teuthology.orchestra.run.trial063:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-12T12:31:23.374 DEBUG:teuthology.orchestra.run.trial063:> set -ex 2026-02-12T12:31:23.374 DEBUG:teuthology.orchestra.run.trial063:> dd of=/home/ubuntu/cephtest/ceph.monmap 2026-02-12T12:31:23.430 INFO:tasks.ceph:Setting up mon nodes... 2026-02-12T12:31:23.431 INFO:tasks.ceph:Setting up mgr nodes... 2026-02-12T12:31:23.431 DEBUG:teuthology.orchestra.run.trial063:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2026-02-12T12:31:23.521 INFO:teuthology.orchestra.run.trial063.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2026-02-12T12:31:23.524 INFO:tasks.ceph:Setting up mds nodes... 2026-02-12T12:31:23.524 INFO:tasks.ceph_client:Setting up client nodes... 2026-02-12T12:31:23.524 DEBUG:teuthology.orchestra.run.trial063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2026-02-12T12:31:23.566 INFO:teuthology.orchestra.run.trial063.stdout:creating /etc/ceph/ceph.client.0.keyring 2026-02-12T12:31:23.577 INFO:tasks.ceph:Running mkfs on osd nodes... 2026-02-12T12:31:23.578 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@trial063.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2026-02-12T12:31:23.578 DEBUG:teuthology.orchestra.run.trial063:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2026-02-12T12:31:23.642 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-12T12:31:23.642 INFO:tasks.ceph:role: osd.0 2026-02-12T12:31:23.642 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@trial063.front.sepia.ceph.com 2026-02-12T12:31:23.642 DEBUG:teuthology.orchestra.run.trial063:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2026-02-12T12:31:23.709 INFO:teuthology.orchestra.run.trial063.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=45875200 blks 2026-02-12T12:31:23.709 INFO:teuthology.orchestra.run.trial063.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-12T12:31:23.709 INFO:teuthology.orchestra.run.trial063.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-12T12:31:23.709 INFO:teuthology.orchestra.run.trial063.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-12T12:31:23.710 INFO:teuthology.orchestra.run.trial063.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-12T12:31:23.710 INFO:teuthology.orchestra.run.trial063.stdout: = sunit=0 swidth=0 blks 2026-02-12T12:31:23.710 INFO:teuthology.orchestra.run.trial063.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-12T12:31:23.710 INFO:teuthology.orchestra.run.trial063.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-12T12:31:23.710 INFO:teuthology.orchestra.run.trial063.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-12T12:31:23.710 INFO:teuthology.orchestra.run.trial063.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-12T12:31:24.278 INFO:teuthology.orchestra.run.trial063.stdout:Discarding blocks...Done. 2026-02-12T12:31:24.280 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@trial063.front.sepia.ceph.com -o noatime 2026-02-12T12:31:24.280 DEBUG:teuthology.orchestra.run.trial063:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2026-02-12T12:31:24.349 DEBUG:teuthology.orchestra.run.trial063:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2026-02-12T12:31:24.416 DEBUG:teuthology.orchestra.run.trial063:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2026-02-12T12:31:24.482 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-12T12:31:24.482 INFO:tasks.ceph:role: osd.1 2026-02-12T12:31:24.482 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@trial063.front.sepia.ceph.com 2026-02-12T12:31:24.483 DEBUG:teuthology.orchestra.run.trial063:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2026-02-12T12:31:24.547 INFO:teuthology.orchestra.run.trial063.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=45875200 blks 2026-02-12T12:31:24.547 INFO:teuthology.orchestra.run.trial063.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-12T12:31:24.548 INFO:teuthology.orchestra.run.trial063.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-12T12:31:24.548 INFO:teuthology.orchestra.run.trial063.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-12T12:31:24.548 INFO:teuthology.orchestra.run.trial063.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-12T12:31:24.548 INFO:teuthology.orchestra.run.trial063.stdout: = sunit=0 swidth=0 blks 2026-02-12T12:31:24.548 INFO:teuthology.orchestra.run.trial063.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-12T12:31:24.548 INFO:teuthology.orchestra.run.trial063.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-12T12:31:24.548 INFO:teuthology.orchestra.run.trial063.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-12T12:31:24.548 INFO:teuthology.orchestra.run.trial063.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-12T12:31:25.104 INFO:teuthology.orchestra.run.trial063.stdout:Discarding blocks...Done. 2026-02-12T12:31:25.106 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@trial063.front.sepia.ceph.com -o noatime 2026-02-12T12:31:25.106 DEBUG:teuthology.orchestra.run.trial063:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2026-02-12T12:31:25.174 DEBUG:teuthology.orchestra.run.trial063:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2026-02-12T12:31:25.241 DEBUG:teuthology.orchestra.run.trial063:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2026-02-12T12:31:25.305 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-12T12:31:25.305 INFO:tasks.ceph:role: osd.2 2026-02-12T12:31:25.305 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@trial063.front.sepia.ceph.com 2026-02-12T12:31:25.305 DEBUG:teuthology.orchestra.run.trial063:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2026-02-12T12:31:25.372 INFO:teuthology.orchestra.run.trial063.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=45875200 blks 2026-02-12T12:31:25.372 INFO:teuthology.orchestra.run.trial063.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-12T12:31:25.372 INFO:teuthology.orchestra.run.trial063.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-12T12:31:25.372 INFO:teuthology.orchestra.run.trial063.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-12T12:31:25.372 INFO:teuthology.orchestra.run.trial063.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-12T12:31:25.372 INFO:teuthology.orchestra.run.trial063.stdout: = sunit=0 swidth=0 blks 2026-02-12T12:31:25.372 INFO:teuthology.orchestra.run.trial063.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-12T12:31:25.373 INFO:teuthology.orchestra.run.trial063.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-12T12:31:25.373 INFO:teuthology.orchestra.run.trial063.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-12T12:31:25.373 INFO:teuthology.orchestra.run.trial063.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-12T12:31:25.981 INFO:teuthology.orchestra.run.trial063.stdout:Discarding blocks...Done. 2026-02-12T12:31:25.983 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@trial063.front.sepia.ceph.com -o noatime 2026-02-12T12:31:25.984 DEBUG:teuthology.orchestra.run.trial063:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2026-02-12T12:31:26.051 DEBUG:teuthology.orchestra.run.trial063:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2026-02-12T12:31:26.118 DEBUG:teuthology.orchestra.run.trial063:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-12T12:31:26.268 INFO:teuthology.orchestra.run.trial063.stderr:INFO  osd - OSD::main: parsing early config 2026-02-12T12:31:26.320 INFO:teuthology.orchestra.run.trial063.stderr:INFO 2026-02-12 12:31:26,319 seastar - Reactor backend: linux-aio 2026-02-12T12:31:26.337 INFO:teuthology.orchestra.run.trial063.stderr:INFO 2026-02-12 12:31:26,336 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-12T12:31:26.338 INFO:teuthology.orchestra.run.trial063.stderr:INFO 2026-02-12 12:31:26,336 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-12T12:31:26.340 INFO:teuthology.orchestra.run.trial063.stderr:==20796==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-12T12:31:26.399 INFO:teuthology.orchestra.run.trial063.stderr:DEBUG 2026-02-12 12:31:26,398 [shard 0:main] seastar - reactor::drain 2026-02-12T12:31:28.508 INFO:teuthology.orchestra.run.trial063.stderr:----------------------------------------------------- 2026-02-12T12:31:28.508 INFO:teuthology.orchestra.run.trial063.stderr:Suppressions used: 2026-02-12T12:31:28.508 INFO:teuthology.orchestra.run.trial063.stderr: count bytes template 2026-02-12T12:31:28.508 INFO:teuthology.orchestra.run.trial063.stderr: 1 8 MallocExtension::Register 2026-02-12T12:31:28.509 INFO:teuthology.orchestra.run.trial063.stderr:----------------------------------------------------- 2026-02-12T12:31:28.509 INFO:teuthology.orchestra.run.trial063.stderr: 2026-02-12T12:31:28.527 INFO:teuthology.orchestra.run.trial063.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-12T12:31:28.527 INFO:teuthology.orchestra.run.trial063.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-12T12:31:28.575 INFO:teuthology.orchestra.run.trial063.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-12T12:31:28.575 INFO:teuthology.orchestra.run.trial063.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-12T12:31:28.582 INFO:teuthology.orchestra.run.trial063.stderr:INFO 2026-02-12 12:31:28,581 seastar - Reactor backend: linux-aio 2026-02-12T12:31:28.590 INFO:teuthology.orchestra.run.trial063.stderr:INFO 2026-02-12 12:31:28,589 [shard 3:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-12T12:31:28.590 INFO:teuthology.orchestra.run.trial063.stderr:INFO 2026-02-12 12:31:28,589 [shard 3:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-12T12:31:28.594 INFO:teuthology.orchestra.run.trial063.stderr:==20794==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-12T12:31:28.594 INFO:teuthology.orchestra.run.trial063.stderr:INFO 2026-02-12 12:31:28,593 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-12T12:31:28.736 INFO:teuthology.orchestra.run.trial063.stderr:Reactor stalled for 32 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e4796c 0x9e47c55 0x9ea85f3 0x9ea8de6 0x9ea9054 0x9ea919c 0x9ea965b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xee77f 0x9d39c22 0x9d39cee 0x9d61e0d 0x9d66fb7 0x9d27585 0x9d26585 0x7bb28d7 0x7bbc3a0 0x7a3ca55 0x7a7cdbd 0x7113237 0x6f948ba 0x754bf4a 0x754c09e 0x9f024bb 0x9e56790 0x9f2ff2b 0x9f9870e 0x9f99f2c 0x9c17b31 0x9c1836c 0xed0853 /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb47b04 2026-02-12T12:31:28.736 INFO:teuthology.orchestra.run.trial063.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e4796c 0x9e47c55 0x9ea85f3 0x9ea8de6 0x9ea9054 0x9ea919c 0x9ea965b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc597 /lib64/libasan.so.8+0x8eea /lib64/libasan.so.8+0xedc65 0xe1e15b 0x9bc3ff7 0x9d3b018 0x9d3b111 0x7bb2246 0x7bbc3a0 0x7a3ca55 0x7a7cdbd 0x7113237 0x6f93ace 0x6f978cf 0x6f9d479 0x74115d6 0x7433c3e 0x7283c37 0x7283d2f 0x7354fe2 0x735509b 0x7355252 0x7433cbf 0x7283dac 0x7283e31 0x72afa7b 0x741c466 0x741c4eb 0x741c5c9 0x729b554 0x73a9088 0x74287fc 0x7428877 0x74288f2 0x74ec4a2 0x75da55c 0x9f024bb 0x9e56790 0x9f2ff2b 0x9f9870e 0x9fb319e 0x9fb381a 0x26ed857 0xa5c1720 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T12:31:28.736 INFO:teuthology.orchestra.run.trial063.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff917280f7 0xffffffff913f7fa6 0xffffffff913fb960 0xffffffff913fccc1 0xffffffff91424923 0xffffffff91424ae5 0xffffffff913c9858 0xffffffff913d3b7b 0xffffffff913d3fdb 0xffffffff910910d2 0xffffffff91d4d451 0xffffffff91e00ba2 2026-02-12T12:31:28.736 INFO:teuthology.orchestra.run.trial063.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e4796c 0x9e47c55 0x9ea85f3 0x9ea8de6 0x9ea9054 0x9ea919c 0x9ea965b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1297e4 /lib64/libasan.so.8+0xc6631 /lib64/libasan.so.8+0x151b7 0x316c8b5 0x9c70901 0x9c70c76 0x9c71725 0x9c71c23 0x9c72253 0x9c723bd 0x9d5272d 0x9d65530 0x9d27585 0x9d26585 0x7bb28d7 0x7bbc3a0 0x7a3ca55 0x7a7cdbd 0x7113237 0x6f93ace 0x6f978cf 0x6f9d479 0x74115d6 0x7433c3e 0x7283c37 0x7283d2f 0x7354fe2 0x735509b 0x7355252 0x7433cbf 0x7283dac 0x7283e31 0x72afa7b 0x741c466 0x741c4eb 0x741c5c9 0x729b554 0x73a9088 0x74287fc 0x7428877 0x74288f2 0x74ec4a2 0x75da55c 0x9f024bb 0x9e56790 0x9f2ff2b 0x9f9870e 0x9fb319e 0x9fb381a 0x26ed857 0xa5c1720 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T12:31:28.736 INFO:teuthology.orchestra.run.trial063.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e4796c 0x9e47c55 0x9ea85f3 0x9ea8de6 0x9ea9054 0x9ea919c 0x9ea965b /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x191c1b /lib64/libasan.so.8+0xc748 /lib64/libasan.so.8+0x91c1 /lib64/libasan.so.8+0xee7fc 0x9b44d0b 0x9bc24a6 0x9bc25bc 0x9bc2698 0x9bc2d54 0x9bc29cd 0x9bc2ebb 0x9bc32bf 0x9bc357c 0x9d50e47 0x9d51025 0x9d51281 0x9d51372 0x9d51f12 0x9d52516 0x9d528a9 0x9d65530 0x9d27585 0x9d26585 0x7bb28d7 0x7bbc3a0 0x7a3ca55 0x7a7cdbd 0x7113237 0x6f93ace 0x6f978cf 0x6f9d479 0x74115d6 0x7433c3e 0x7283c37 0x7283d2f 0x7354fe2 0x735509b 0x7355252 0x7433cbf 0x7283dac 0x7283e31 0x72afa7b 0x741c466 0x741c4eb 0x741c5c9 0x729b554 0x73a9088 0x74287fc 0x7428877 0x74288f2 0x74ec4a2 0x75da55c 0x9f024bb 0x9e56790 0x9f2ff2b 0x9f9870e 0x9fb319e 0x9fb381a 0x26ed857 0xa5c1720 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T12:31:28.736 INFO:teuthology.orchestra.run.trial063.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff9170bc89 0xffffffff913d0dc7 0xffffffff910910a4 0xffffffff91d4d451 0xffffffff91e00ba2 2026-02-12T12:31:28.881 INFO:teuthology.orchestra.run.trial063.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e4796c 0x9e47c55 0x9ea85f3 0x9ea8de6 0x9ea9054 0x9ea919c 0x9ea965b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1006c8 /lib64/libasan.so.8+0xc6619 /lib64/libasan.so.8+0x151b7 0x112781c 0xf2fda3 0x9c70450 0x9c7074d 0x9c709c0 0x9c70c76 0x9c71238 0x9c796df 0x9c79dbf 0x9d6290c 0x9d66fb7 0x9d27585 0x9d26585 0x7bb28d7 0x7bbc3a0 0x7a3ca55 0x7a7cdbd 0x7113237 0x6f3791e 0x6f39c86 0x747c276 0x729b1dd 0x729b2df 0x729b3d7 0x73541ae 0x7354267 0x735441a 0x7354552 0x73545cd 0x7354652 0x73547ad 0x741bee3 0x747c2ff 0x7282672 0x729b554 0x73a9088 0x74287fc 0x7428877 0x74288f2 0x74ec4a2 0x75da55c 0x9f024bb 0x9e56790 0x9f2ff2b 0x9f9870e 0x9fb319e 0x9fb381a 0x26ed857 0xa5c1720 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T12:31:28.881 INFO:teuthology.orchestra.run.trial063.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e4796c 0x9e47c55 0x9ea85f3 0x9ea8de6 0x9ea9054 0x9ea919c 0x9ea965b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x15896 0x1092cf7 0xf2fdae 0x9c70450 0x9c7074d 0x9c709c0 0x9c70c76 0x9c71238 0x9c796df 0x9c79dbf 0x9d6290c 0x9d66fb7 0x9d27585 0x9d26585 0x7bb28d7 0x7bbc3a0 0x7a3ca55 0x7a7cdbd 0x7113237 0x6f3791e 0x6f39c86 0x747c276 0x729b1dd 0x729b2df 0x729b3d7 0x73541ae 0x7354267 0x735441a 0x7354552 0x73545cd 0x7354652 0x73547ad 0x741bee3 0x747c2ff 0x7282672 0x729b554 0x73a9088 0x74287fc 0x7428877 0x74288f2 0x74ec4a2 0x75da55c 0x9f024bb 0x9e56790 0x9f2ff2b 0x9f9870e 0x9fb319e 0x9fb381a 0x26ed857 0xa5c1720 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T12:31:28.881 INFO:teuthology.orchestra.run.trial063.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e4796c 0x9e47c55 0x9ea85f3 0x9ea8de6 0x9ea9054 0x9ea919c 0x9ea965b /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x8cc0c /lib64/libasan.so.8+0x1006bc /lib64/libasan.so.8+0xc6619 /lib64/libasan.so.8+0x157e7 0x1092cf7 0xf2fdae 0x9c70450 0x9c7074d 0x9c709c0 0x9c70c76 0x9c71725 0x9c71c23 0x9c72253 0x9c723bd 0x9d5272d 0x9d65530 0x9d27585 0x9d26585 0x7bb28d7 0x7bbc3a0 0x7a3ca55 0x7a7cdbd 0x7113237 0x6f3791e 0x6f39c86 0x747c276 0x729b1dd 0x729b2df 0x729b3d7 0x73541ae 0x7354267 0x735441a 0x7354552 0x73545cd 0x7354652 0x73547ad 0x741bee3 0x747c2ff 0x7282672 0x729b554 0x73a9088 0x74287fc 0x7428877 0x74288f2 0x74ec4a2 0x75da55c 0x9f024bb 0x9e56790 0x9f2ff2b 0x9f9870e 0x9fb319e 0x9fb381a 0x26ed857 0xa5c1720 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T12:31:31.076 INFO:teuthology.orchestra.run.trial063.stderr:----------------------------------------------------- 2026-02-12T12:31:31.077 INFO:teuthology.orchestra.run.trial063.stderr:Suppressions used: 2026-02-12T12:31:31.077 INFO:teuthology.orchestra.run.trial063.stderr: count bytes template 2026-02-12T12:31:31.077 INFO:teuthology.orchestra.run.trial063.stderr: 1 8 MallocExtension::Register 2026-02-12T12:31:31.077 INFO:teuthology.orchestra.run.trial063.stderr:----------------------------------------------------- 2026-02-12T12:31:31.077 INFO:teuthology.orchestra.run.trial063.stderr: 2026-02-12T12:31:31.107 DEBUG:teuthology.orchestra.run.trial063:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2026-02-12T12:31:31.132 DEBUG:teuthology.orchestra.run.trial063:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-12T12:31:31.283 INFO:teuthology.orchestra.run.trial063.stderr:INFO  osd - OSD::main: parsing early config 2026-02-12T12:31:31.334 INFO:teuthology.orchestra.run.trial063.stderr:INFO 2026-02-12 12:31:31,333 seastar - Reactor backend: linux-aio 2026-02-12T12:31:31.352 INFO:teuthology.orchestra.run.trial063.stderr:INFO 2026-02-12 12:31:31,351 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-12T12:31:31.352 INFO:teuthology.orchestra.run.trial063.stderr:INFO 2026-02-12 12:31:31,351 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-12T12:31:31.354 INFO:teuthology.orchestra.run.trial063.stderr:==20864==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-12T12:31:31.413 INFO:teuthology.orchestra.run.trial063.stderr:DEBUG 2026-02-12 12:31:31,412 [shard 0:main] seastar - reactor::drain 2026-02-12T12:31:33.471 INFO:teuthology.orchestra.run.trial063.stderr:----------------------------------------------------- 2026-02-12T12:31:33.472 INFO:teuthology.orchestra.run.trial063.stderr:Suppressions used: 2026-02-12T12:31:33.472 INFO:teuthology.orchestra.run.trial063.stderr: count bytes template 2026-02-12T12:31:33.472 INFO:teuthology.orchestra.run.trial063.stderr: 1 8 MallocExtension::Register 2026-02-12T12:31:33.472 INFO:teuthology.orchestra.run.trial063.stderr:----------------------------------------------------- 2026-02-12T12:31:33.472 INFO:teuthology.orchestra.run.trial063.stderr: 2026-02-12T12:31:33.491 INFO:teuthology.orchestra.run.trial063.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-12T12:31:33.491 INFO:teuthology.orchestra.run.trial063.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-12T12:31:33.537 INFO:teuthology.orchestra.run.trial063.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-12T12:31:33.538 INFO:teuthology.orchestra.run.trial063.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-12T12:31:33.545 INFO:teuthology.orchestra.run.trial063.stderr:INFO 2026-02-12 12:31:33,544 seastar - Reactor backend: linux-aio 2026-02-12T12:31:33.562 INFO:teuthology.orchestra.run.trial063.stderr:INFO 2026-02-12 12:31:33,561 [shard 2:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-12T12:31:33.562 INFO:teuthology.orchestra.run.trial063.stderr:INFO 2026-02-12 12:31:33,561 [shard 2:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-12T12:31:33.565 INFO:teuthology.orchestra.run.trial063.stderr:==20862==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-12T12:31:33.566 INFO:teuthology.orchestra.run.trial063.stderr:INFO 2026-02-12 12:31:33,565 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-12T12:31:33.705 INFO:teuthology.orchestra.run.trial063.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e4796c 0x9e47c55 0x9ea85f3 0x9ea8de6 0x9ea9054 0x9ea919c 0x9ea965b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc661c /lib64/libasan.so.8+0x151b7 0x9c702ec 0x9c7074d 0x9c709c0 0x9c726e7 0x9d560be 0x9d57d10 0x9d5836a 0x9d588b0 0x9d62b3e 0x9d66fb7 0x9d27585 0x9d26585 0x7bb28d7 0x7bbc3a0 0x7a3ca55 0x7a7cdbd 0x7113237 0x6f93ace 0x6f978cf 0x6f9d479 0x74115d6 0x7433c3e 0x7283c37 0x7283d2f 0x7354fe2 0x735509b 0x7355252 0x7433cbf 0x7283dac 0x7283e31 0x72afa7b 0x741c466 0x741c4eb 0x741c5c9 0x729b554 0x73a9088 0x74287fc 0x7428877 0x74288f2 0x74ec4a2 0x75da55c 0x9f024bb 0x9e56790 0x9f2ff2b 0x9f9870e 0x9fb319e 0x9fb381a 0x26ed857 0xa5c1720 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T12:31:33.705 INFO:teuthology.orchestra.run.trial063.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e4796c 0x9e47c55 0x9ea85f3 0x9ea8de6 0x9ea9054 0x9ea919c 0x9ea965b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1006bc /lib64/libasan.so.8+0xc6619 /lib64/libasan.so.8+0x157e7 0x1092cf7 0xf2fdae 0x9d5308d 0x9d5367a 0x9d539fd 0x9d63b54 0x9d66fb7 0x9d27585 0x9d26585 0x7bb28d7 0x7bbc3a0 0x7a3ca55 0x7a7cdbd 0x7113237 0x6f93ace 0x6f978cf 0x6f9d479 0x74115d6 0x7433c3e 0x7283c37 0x7283d2f 0x7354fe2 0x735509b 0x7355252 0x7433cbf 0x7283dac 0x7283e31 0x72afa7b 0x741c466 0x741c4eb 0x741c5c9 0x729b554 0x73a9088 0x74287fc 0x7428877 0x74288f2 0x74ec4a2 0x75da55c 0x9f024bb 0x9e56790 0x9f2ff2b 0x9f9870e 0x9fb319e 0x9fb381a 0x26ed857 0xa5c1720 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T12:31:33.705 INFO:teuthology.orchestra.run.trial063.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e4796c 0x9e47c55 0x9ea85f3 0x9ea8de6 0x9ea9054 0x9ea919c 0x9ea965b /lib64/libc.so.6+0x3fc2f 0xe075e0 0x11277c5 0xf2fda3 0x9c70450 0x9c7074d 0x9c709c0 0x9c70c76 0x9c71725 0x9c71c23 0x9c72253 0x9c723bd 0x9d5272d 0x9d65530 0x9d27585 0x9d26585 0x7bb28d7 0x7bbc3a0 0x7a3ca55 0x7a7cdbd 0x7113237 0x6f93ace 0x6f978cf 0x6f9d479 0x74115d6 0x7433c3e 0x7283c37 0x7283d2f 0x7354fe2 0x735509b 0x7355252 0x7433cbf 0x7283dac 0x7283e31 0x72afa7b 0x741c466 0x741c4eb 0x741c5c9 0x729b554 0x73a9088 0x74287fc 0x7428877 0x74288f2 0x74ec4a2 0x75da55c 0x9f024bb 0x9e56790 0x9f2ff2b 0x9f9870e 0x9fb319e 0x9fb381a 0x26ed857 0xa5c1720 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T12:31:33.852 INFO:teuthology.orchestra.run.trial063.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e4796c 0x9e47c55 0x9ea85f3 0x9ea8de6 0x9ea9054 0x9ea919c 0x9ea965b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x129858 /lib64/libasan.so.8+0xc6631 /lib64/libasan.so.8+0x151b7 0x9c702ec 0x9c7074d 0x9c709c0 0x9c719c0 0x9c71f2e 0x9c72253 0x9c723bd 0x9d5272d 0x9d65530 0x9d27585 0x9d26585 0x7bb28d7 0x7bbc3a0 0x7a3ca55 0x7a7cdbd 0x7113237 0x6f3791e 0x6f39c86 0x747c276 0x729b1dd 0x729b2df 0x729b3d7 0x73541ae 0x7354267 0x735441a 0x7354552 0x73545cd 0x7354652 0x73547ad 0x741bee3 0x747c2ff 0x7282672 0x729b554 0x73a9088 0x74287fc 0x7428877 0x74288f2 0x74ec4a2 0x75da55c 0x9f024bb 0x9e56790 0x9f2ff2b 0x9f9870e 0x9fb319e 0x9fb381a 0x26ed857 0xa5c1720 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T12:31:33.852 INFO:teuthology.orchestra.run.trial063.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e4796c 0x9e47c55 0x9ea85f3 0x9ea8de6 0x9ea9054 0x9ea919c 0x9ea965b /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x191c1b /lib64/libasan.so.8+0xc748 /lib64/libasan.so.8+0x91c1 /lib64/libasan.so.8+0xee7fc 0x9b44d0b 0x9bc24a6 0x9bc25bc 0x9bc2698 0x9bc2d54 0x9bc29cd 0x9bc2ebb 0x9bc32bf 0x9bc357c 0x9bc43c4 0x9d3b018 0x9d3b111 0x7bb2246 0x7bbc3a0 0x7a3ca55 0x7a7cdbd 0x7113237 0x6f3791e 0x6f39c86 0x747c276 0x729b1dd 0x729b2df 0x729b3d7 0x73541ae 0x7354267 0x735441a 0x7354552 0x73545cd 0x7354652 0x73547ad 0x741bee3 0x747c2ff 0x7282672 0x729b554 0x73a9088 0x74287fc 0x7428877 0x74288f2 0x74ec4a2 0x75da55c 0x9f024bb 0x9e56790 0x9f2ff2b 0x9f9870e 0x9fb319e 0x9fb381a 0x26ed857 0xa5c1720 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T12:31:33.852 INFO:teuthology.orchestra.run.trial063.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff9145e108 0xffffffff913e9091 0xffffffff913cb9cd 0xffffffff913d3b7b 0xffffffff913d3fdb 0xffffffff910910d2 0xffffffff91d4d451 0xffffffff91e00ba2 2026-02-12T12:31:33.853 INFO:teuthology.orchestra.run.trial063.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e4796c 0x9e47c55 0x9ea85f3 0x9ea8de6 0x9ea9054 0x9ea919c 0x9ea965b /lib64/libc.so.6+0x3fc2f 0xcb11fb 0x1092d87 0xf2fdae 0x9c70450 0x9c7074d 0x9c709c0 0x9c70c76 0x9c71238 0x9c796df 0x9c79dbf 0x9d6290c 0x9d66fb7 0x9d27585 0x9d26585 0x7bb28d7 0x7bbc3a0 0x7a3ca55 0x7a7cdbd 0x7113237 0x6f3791e 0x6f39c86 0x747c276 0x729b1dd 0x729b2df 0x729b3d7 0x73541ae 0x7354267 0x735441a 0x7354552 0x73545cd 0x7354652 0x73547ad 0x741bee3 0x747c2ff 0x7282672 0x729b554 0x73a9088 0x74287fc 0x7428877 0x74288f2 0x74ec4a2 0x75da55c 0x9f024bb 0x9e56790 0x9f2ff2b 0x9f9870e 0x9fb319e 0x9fb381a 0x26ed857 0xa5c1720 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T12:31:35.967 INFO:teuthology.orchestra.run.trial063.stderr:----------------------------------------------------- 2026-02-12T12:31:35.967 INFO:teuthology.orchestra.run.trial063.stderr:Suppressions used: 2026-02-12T12:31:35.967 INFO:teuthology.orchestra.run.trial063.stderr: count bytes template 2026-02-12T12:31:35.967 INFO:teuthology.orchestra.run.trial063.stderr: 1 8 MallocExtension::Register 2026-02-12T12:31:35.968 INFO:teuthology.orchestra.run.trial063.stderr:----------------------------------------------------- 2026-02-12T12:31:35.968 INFO:teuthology.orchestra.run.trial063.stderr: 2026-02-12T12:31:35.997 DEBUG:teuthology.orchestra.run.trial063:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2026-02-12T12:31:36.022 DEBUG:teuthology.orchestra.run.trial063:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-12T12:31:36.172 INFO:teuthology.orchestra.run.trial063.stderr:INFO  osd - OSD::main: parsing early config 2026-02-12T12:31:36.232 INFO:teuthology.orchestra.run.trial063.stderr:INFO 2026-02-12 12:31:36,231 seastar - Reactor backend: linux-aio 2026-02-12T12:31:36.244 INFO:teuthology.orchestra.run.trial063.stderr:INFO 2026-02-12 12:31:36,243 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-12T12:31:36.245 INFO:teuthology.orchestra.run.trial063.stderr:INFO 2026-02-12 12:31:36,244 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-12T12:31:36.247 INFO:teuthology.orchestra.run.trial063.stderr:==20930==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-12T12:31:36.306 INFO:teuthology.orchestra.run.trial063.stderr:DEBUG 2026-02-12 12:31:36,305 [shard 0:main] seastar - reactor::drain 2026-02-12T12:31:38.404 INFO:teuthology.orchestra.run.trial063.stderr:----------------------------------------------------- 2026-02-12T12:31:38.405 INFO:teuthology.orchestra.run.trial063.stderr:Suppressions used: 2026-02-12T12:31:38.405 INFO:teuthology.orchestra.run.trial063.stderr: count bytes template 2026-02-12T12:31:38.405 INFO:teuthology.orchestra.run.trial063.stderr: 1 8 MallocExtension::Register 2026-02-12T12:31:38.405 INFO:teuthology.orchestra.run.trial063.stderr:----------------------------------------------------- 2026-02-12T12:31:38.405 INFO:teuthology.orchestra.run.trial063.stderr: 2026-02-12T12:31:38.424 INFO:teuthology.orchestra.run.trial063.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-12T12:31:38.424 INFO:teuthology.orchestra.run.trial063.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-12T12:31:38.471 INFO:teuthology.orchestra.run.trial063.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-12T12:31:38.472 INFO:teuthology.orchestra.run.trial063.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-12T12:31:38.478 INFO:teuthology.orchestra.run.trial063.stderr:INFO 2026-02-12 12:31:38,478 seastar - Reactor backend: linux-aio 2026-02-12T12:31:38.488 INFO:teuthology.orchestra.run.trial063.stderr:INFO 2026-02-12 12:31:38,487 [shard 2:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-12T12:31:38.489 INFO:teuthology.orchestra.run.trial063.stderr:INFO 2026-02-12 12:31:38,487 [shard 2:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-12T12:31:38.492 INFO:teuthology.orchestra.run.trial063.stderr:==20928==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-12T12:31:38.492 INFO:teuthology.orchestra.run.trial063.stderr:INFO 2026-02-12 12:31:38,491 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-12T12:31:38.631 INFO:teuthology.orchestra.run.trial063.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e4796c 0x9e47c55 0x9ea85f3 0x9ea8de6 0x9ea9054 0x9ea919c 0x9ea965b /lib64/libc.so.6+0x3fc2f 0xcb8a54 0x11277b3 0xf2fda3 0x9c70283 0x9c7074d 0x9c709c0 0x9c726e7 0x9c72c34 0x9c72e33 0x9d58155 0x9d588b0 0x9d62b3e 0x9d66fb7 0x9d27585 0x9d26585 0x7bb28d7 0x7bbc3a0 0x7a3ca55 0x7a7cdbd 0x7113237 0x6f93ace 0x6f978cf 0x6f9d479 0x74115d6 0x7433c3e 0x7283c37 0x7283d2f 0x7354fe2 0x735509b 0x7355252 0x7433cbf 0x7283dac 0x7283e31 0x72afa7b 0x741c466 0x741c4eb 0x741c5c9 0x729b554 0x73a9088 0x74287fc 0x7428877 0x74288f2 0x74ec4a2 0x75da55c 0x9f024bb 0x9e56790 0x9f2ff2b 0x9f9870e 0x9fb319e 0x9fb381a 0x26ed857 0xa5c1720 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T12:31:38.632 INFO:teuthology.orchestra.run.trial063.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e4796c 0x9e47c55 0x9ea85f3 0x9ea8de6 0x9ea9054 0x9ea919c 0x9ea965b /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x8cc25 /lib64/libasan.so.8+0x1006bc /lib64/libasan.so.8+0xc6619 /lib64/libasan.so.8+0x157e7 0x1092cf7 0xf2fdae 0x9c70283 0x9c7074d 0x9c709c0 0x9c726e7 0x9d560be 0x9d57d10 0x9d5836a 0x9d588b0 0x9d62b3e 0x9d66fb7 0x9d27585 0x9d26585 0x7bb28d7 0x7bbc3a0 0x7a3ca55 0x7a7cdbd 0x7113237 0x6f93ace 0x6f978cf 0x6f9d479 0x74115d6 0x7433c3e 0x7283c37 0x7283d2f 0x7354fe2 0x735509b 0x7355252 0x7433cbf 0x7283dac 0x7283e31 0x72afa7b 0x741c466 0x741c4eb 0x741c5c9 0x729b554 0x73a9088 0x74287fc 0x7428877 0x74288f2 0x74ec4a2 0x75da55c 0x9f024bb 0x9e56790 0x9f2ff2b 0x9f9870e 0x9fb319e 0x9fb381a 0x26ed857 0xa5c1720 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T12:31:38.632 INFO:teuthology.orchestra.run.trial063.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e4796c 0x9e47c55 0x9ea85f3 0x9ea8de6 0x9ea9054 0x9ea919c 0x9ea965b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xb04ef /lib64/libasan.so.8+0xb1477 0xb6a317 0x1092c80 0xf2fdae 0x9c73f6e 0x9c748ad 0x9d5e7f5 0x9d68edb 0x78a0be9 0x7bb280e 0x7bbc3a0 0x7a3ca55 0x7a7cdbd 0x7113237 0x6f93ace 0x6f978cf 0x6f9d479 0x74115d6 0x7433c3e 0x7283c37 0x7283d2f 0x7354fe2 0x735509b 0x7355252 0x7433cbf 0x7283dac 0x7283e31 0x72afa7b 0x741c466 0x741c4eb 0x741c5c9 0x729b554 0x73a9088 0x74287fc 0x7428877 0x74288f2 0x74ec4a2 0x75da55c 0x9f024bb 0x9e56790 0x9f2ff2b 0x9f9870e 0x9fb319e 0x9fb381a 0x26ed857 0xa5c1720 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T12:31:38.780 INFO:teuthology.orchestra.run.trial063.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e4796c 0x9e47c55 0x9ea85f3 0x9ea8de6 0x9ea9054 0x9ea919c 0x9ea965b /lib64/libc.so.6+0x3fc2f 0x9c70b9c 0x9c71238 0x9c796df 0x9c79dbf 0x9d6290c 0x9d66fb7 0x9d27585 0x9d26585 0x7bb28d7 0x7bbc3a0 0x7a3ca55 0x7a7cdbd 0x7113237 0x6f3791e 0x6f39c86 0x747c276 0x729b1dd 0x729b2df 0x729b3d7 0x73541ae 0x7354267 0x735441a 0x7354552 0x73545cd 0x7354652 0x73547ad 0x741bee3 0x747c2ff 0x7282672 0x729b554 0x73a9088 0x74287fc 0x7428877 0x74288f2 0x74ec4a2 0x75da55c 0x9f024bb 0x9e56790 0x9f2ff2b 0x9f9870e 0x9fb319e 0x9fb381a 0x26ed857 0xa5c1720 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T12:31:38.780 INFO:teuthology.orchestra.run.trial063.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e4796c 0x9e47c55 0x9ea85f3 0x9ea8de6 0x9ea9054 0x9ea919c 0x9ea965b /lib64/libc.so.6+0x3fc2f 0x316c834 0x9c709ae 0x9c70c76 0x9c71725 0x9c71c23 0x9c72253 0x9c723bd 0x9d5272d 0x9d65530 0x9d27585 0x9d26585 0x7bb28d7 0x7bbc3a0 0x7a3ca55 0x7a7cdbd 0x7113237 0x6f3791e 0x6f39c86 0x747c276 0x729b1dd 0x729b2df 0x729b3d7 0x73541ae 0x7354267 0x735441a 0x7354552 0x73545cd 0x7354652 0x73547ad 0x741bee3 0x747c2ff 0x7282672 0x729b554 0x73a9088 0x74287fc 0x7428877 0x74288f2 0x74ec4a2 0x75da55c 0x9f024bb 0x9e56790 0x9f2ff2b 0x9f9870e 0x9fb319e 0x9fb381a 0x26ed857 0xa5c1720 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T12:31:38.780 INFO:teuthology.orchestra.run.trial063.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e4796c 0x9e47c55 0x9ea85f3 0x9ea8de6 0x9ea9054 0x9ea919c 0x9ea965b /lib64/libc.so.6+0x3fc2f 0x1092dc8 0xf2fdae 0x9c70450 0x9c7074d 0x9c709c0 0x9c70c76 0x9c71725 0x9c71c23 0x9c72253 0x9c723bd 0x9d5272d 0x9d65530 0x9d27585 0x9d26585 0x7bb28d7 0x7bbc3a0 0x7a3ca55 0x7a7cdbd 0x7113237 0x6f3791e 0x6f39c86 0x747c276 0x729b1dd 0x729b2df 0x729b3d7 0x73541ae 0x7354267 0x735441a 0x7354552 0x73545cd 0x7354652 0x73547ad 0x741bee3 0x747c2ff 0x7282672 0x729b554 0x73a9088 0x74287fc 0x7428877 0x74288f2 0x74ec4a2 0x75da55c 0x9f024bb 0x9e56790 0x9f2ff2b 0x9f9870e 0x9fb319e 0x9fb381a 0x26ed857 0xa5c1720 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T12:31:40.962 INFO:teuthology.orchestra.run.trial063.stderr:----------------------------------------------------- 2026-02-12T12:31:40.963 INFO:teuthology.orchestra.run.trial063.stderr:Suppressions used: 2026-02-12T12:31:40.963 INFO:teuthology.orchestra.run.trial063.stderr: count bytes template 2026-02-12T12:31:40.963 INFO:teuthology.orchestra.run.trial063.stderr: 1 8 MallocExtension::Register 2026-02-12T12:31:40.963 INFO:teuthology.orchestra.run.trial063.stderr:----------------------------------------------------- 2026-02-12T12:31:40.963 INFO:teuthology.orchestra.run.trial063.stderr: 2026-02-12T12:31:40.993 DEBUG:teuthology.orchestra.run.trial063:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2026-02-12T12:31:41.018 INFO:tasks.ceph:Reading keys from all nodes... 2026-02-12T12:31:41.019 DEBUG:teuthology.orchestra.run.trial063:> set -ex 2026-02-12T12:31:41.019 DEBUG:teuthology.orchestra.run.trial063:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2026-02-12T12:31:41.082 DEBUG:teuthology.orchestra.run.trial063:> set -ex 2026-02-12T12:31:41.083 DEBUG:teuthology.orchestra.run.trial063:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2026-02-12T12:31:41.146 DEBUG:teuthology.orchestra.run.trial063:> set -ex 2026-02-12T12:31:41.147 DEBUG:teuthology.orchestra.run.trial063:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2026-02-12T12:31:41.209 DEBUG:teuthology.orchestra.run.trial063:> set -ex 2026-02-12T12:31:41.209 DEBUG:teuthology.orchestra.run.trial063:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2026-02-12T12:31:41.273 DEBUG:teuthology.orchestra.run.trial063:> set -ex 2026-02-12T12:31:41.273 DEBUG:teuthology.orchestra.run.trial063:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2026-02-12T12:31:41.329 INFO:tasks.ceph:Adding keys to all mons... 2026-02-12T12:31:41.330 DEBUG:teuthology.orchestra.run.trial063:> sudo tee -a /etc/ceph/ceph.keyring 2026-02-12T12:31:41.392 INFO:teuthology.orchestra.run.trial063.stdout:[mgr.x] 2026-02-12T12:31:41.392 INFO:teuthology.orchestra.run.trial063.stdout: key = AQAbyI1pn18EHxAAQnyZ1itUnHyVOZlvb+Lzyw== 2026-02-12T12:31:41.392 INFO:teuthology.orchestra.run.trial063.stdout:[osd.0] 2026-02-12T12:31:41.392 INFO:teuthology.orchestra.run.trial063.stdout: key = AQAgyI1pavhFJxAAQEXUp78/V0kJAdlP8XaJfA== 2026-02-12T12:31:41.392 INFO:teuthology.orchestra.run.trial063.stdout:[osd.1] 2026-02-12T12:31:41.393 INFO:teuthology.orchestra.run.trial063.stdout: key = AQAlyI1pWaiEJRAAR3zxpY3X5OZRARkaBI8Omg== 2026-02-12T12:31:41.393 INFO:teuthology.orchestra.run.trial063.stdout:[osd.2] 2026-02-12T12:31:41.393 INFO:teuthology.orchestra.run.trial063.stdout: key = AQAqyI1pPOoiIRAAkWEm3ZEqXFO/F0i08qjxGA== 2026-02-12T12:31:41.393 INFO:teuthology.orchestra.run.trial063.stdout:[client.0] 2026-02-12T12:31:41.393 INFO:teuthology.orchestra.run.trial063.stdout: key = AQAbyI1prIGvIRAAmqioC3G2IDezDDJi52S9xQ== 2026-02-12T12:31:41.394 DEBUG:teuthology.orchestra.run.trial063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2026-02-12T12:31:41.482 DEBUG:teuthology.orchestra.run.trial063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-12T12:31:41.530 DEBUG:teuthology.orchestra.run.trial063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-12T12:31:41.578 DEBUG:teuthology.orchestra.run.trial063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-12T12:31:41.626 DEBUG:teuthology.orchestra.run.trial063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-12T12:31:41.676 INFO:tasks.ceph:Running mkfs on mon nodes... 2026-02-12T12:31:41.676 DEBUG:teuthology.orchestra.run.trial063:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2026-02-12T12:31:41.700 DEBUG:teuthology.orchestra.run.trial063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2026-02-12T12:31:41.786 INFO:teuthology.orchestra.run.trial063.stderr:2026-02-12T12:31:41.785+0000 7fab97906c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:31:41.786 INFO:teuthology.orchestra.run.trial063.stderr:2026-02-12T12:31:41.785+0000 7fab97906c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:31:41.793 DEBUG:teuthology.orchestra.run.trial063:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2026-02-12T12:31:41.818 DEBUG:teuthology.orchestra.run.trial063:> rm -- /home/ubuntu/cephtest/ceph.monmap 2026-02-12T12:31:41.874 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2026-02-12T12:31:41.875 INFO:tasks.ceph.mon.a:Restarting daemon 2026-02-12T12:31:41.875 DEBUG:teuthology.orchestra.run.trial063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2026-02-12T12:31:41.916 INFO:tasks.ceph.mon.a:Started 2026-02-12T12:31:41.916 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2026-02-12T12:31:41.917 INFO:tasks.ceph.mgr.x:Restarting daemon 2026-02-12T12:31:41.917 DEBUG:teuthology.orchestra.run.trial063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2026-02-12T12:31:41.919 INFO:tasks.ceph.mgr.x:Started 2026-02-12T12:31:41.919 DEBUG:tasks.ceph:set 0 configs 2026-02-12T12:31:41.920 DEBUG:teuthology.orchestra.run.trial063:> sudo ceph --cluster ceph config dump 2026-02-12T12:31:41.980 INFO:tasks.ceph.mon.a.trial063.stderr:2026-02-12T12:31:41.978+0000 7f8bc2e83c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:31:41.980 INFO:tasks.ceph.mon.a.trial063.stderr:2026-02-12T12:31:41.978+0000 7f8bc2e83c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:31:41.980 INFO:tasks.ceph.mon.a.trial063.stderr:2026-02-12T12:31:41.978+0000 7f8bc2e83c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:31:41.982 INFO:tasks.ceph.mgr.x.trial063.stderr:2026-02-12T12:31:41.981+0000 7f8c30031fc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:31:41.995 INFO:teuthology.orchestra.run.trial063.stderr:2026-02-12T12:31:41.992+0000 7fa1f850b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:31:41.996 INFO:teuthology.orchestra.run.trial063.stderr:2026-02-12T12:31:41.996+0000 7fa1f850b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:31:42.027 INFO:teuthology.orchestra.run.trial063.stdout:WHO MASK LEVEL OPTION VALUE RO 2026-02-12T12:31:42.036 INFO:tasks.ceph:Setting crush tunables to default 2026-02-12T12:31:42.036 DEBUG:teuthology.orchestra.run.trial063:> sudo ceph --cluster ceph osd crush tunables default 2026-02-12T12:31:42.110 INFO:teuthology.orchestra.run.trial063.stderr:2026-02-12T12:31:42.106+0000 7fc7405e4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:31:42.111 INFO:teuthology.orchestra.run.trial063.stderr:2026-02-12T12:31:42.110+0000 7fc7405e4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:31:42.145 INFO:teuthology.orchestra.run.trial063.stderr:adjusted tunables profile to default 2026-02-12T12:31:42.153 INFO:tasks.ceph:check_enable_crimson: True 2026-02-12T12:31:42.153 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2026-02-12T12:31:42.153 DEBUG:teuthology.orchestra.run.trial063:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2026-02-12T12:31:42.185 INFO:tasks.ceph.mgr.x.trial063.stderr:2026-02-12T12:31:42.184+0000 7f8c30031fc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:31:42.186 INFO:tasks.ceph.mgr.x.trial063.stderr:2026-02-12T12:31:42.185+0000 7f8c30031fc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:31:42.228 INFO:teuthology.orchestra.run.trial063.stderr:2026-02-12T12:31:42.224+0000 7fa26a05a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:31:42.229 INFO:teuthology.orchestra.run.trial063.stderr:2026-02-12T12:31:42.228+0000 7fa26a05a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:31:42.268 INFO:tasks.ceph.mon.a.trial063.stderr:2026-02-12T12:31:42.267+0000 7f8bbb1f1640 -1 WARNING: experimental feature 'crimson' is enabled 2026-02-12T12:31:42.268 INFO:tasks.ceph.mon.a.trial063.stderr:Please be aware that this feature is experimental, untested, 2026-02-12T12:31:42.269 INFO:tasks.ceph.mon.a.trial063.stderr:unsupported, and may result in data corruption, data loss, 2026-02-12T12:31:42.269 INFO:tasks.ceph.mon.a.trial063.stderr:and/or irreparable damage to your cluster. Do not use 2026-02-12T12:31:42.269 INFO:tasks.ceph.mon.a.trial063.stderr:feature with important data. 2026-02-12T12:31:42.269 INFO:tasks.ceph.mon.a.trial063.stderr: 2026-02-12T12:31:43.028 INFO:tasks.ceph.mgr.x.trial063.stderr:/usr/lib64/python3.9/site-packages/scipy/__init__.py:73: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2026-02-12T12:31:43.029 INFO:tasks.ceph.mgr.x.trial063.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2026-02-12T12:31:43.029 INFO:tasks.ceph.mgr.x.trial063.stderr: from numpy import show_config as show_numpy_config 2026-02-12T12:31:43.155 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2026-02-12T12:31:43.156 DEBUG:teuthology.orchestra.run.trial063:> set -ex 2026-02-12T12:31:43.156 DEBUG:teuthology.orchestra.run.trial063:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2026-02-12T12:31:43.219 DEBUG:teuthology.orchestra.run.trial063:> set -ex 2026-02-12T12:31:43.219 DEBUG:teuthology.orchestra.run.trial063:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2026-02-12T12:31:43.282 DEBUG:teuthology.orchestra.run.trial063:> set -ex 2026-02-12T12:31:43.282 DEBUG:teuthology.orchestra.run.trial063:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2026-02-12T12:31:43.346 DEBUG:teuthology.orchestra.run.trial063:> sudo ceph --cluster ceph osd new 327d06b7-8ca5-40a0-ad4f-755b0878e1ee 0 2026-02-12T12:31:43.458 INFO:teuthology.orchestra.run.trial063.stderr:2026-02-12T12:31:43.454+0000 7f056c58a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:31:43.459 INFO:teuthology.orchestra.run.trial063.stderr:2026-02-12T12:31:43.458+0000 7f056c58a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:31:43.498 INFO:teuthology.orchestra.run.trial063.stdout:0 2026-02-12T12:31:43.507 DEBUG:teuthology.orchestra.run.trial063:> sudo ceph --cluster ceph osd new dce03db3-52c2-40c3-b9ec-de5cc9227cde 1 2026-02-12T12:31:43.581 INFO:teuthology.orchestra.run.trial063.stderr:2026-02-12T12:31:43.577+0000 7fea91acb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:31:43.582 INFO:teuthology.orchestra.run.trial063.stderr:2026-02-12T12:31:43.581+0000 7fea91acb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:31:43.619 INFO:teuthology.orchestra.run.trial063.stdout:1 2026-02-12T12:31:43.627 DEBUG:teuthology.orchestra.run.trial063:> sudo ceph --cluster ceph osd new 538db773-9089-49ed-b1e1-8fba4eeff801 2 2026-02-12T12:31:43.702 INFO:teuthology.orchestra.run.trial063.stderr:2026-02-12T12:31:43.698+0000 7fd11310b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:31:43.703 INFO:teuthology.orchestra.run.trial063.stderr:2026-02-12T12:31:43.702+0000 7fd11310b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:31:43.742 INFO:teuthology.orchestra.run.trial063.stdout:2 2026-02-12T12:31:43.751 DEBUG:teuthology.orchestra.run.trial063:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2026-02-12T12:31:43.779 INFO:tasks.ceph.osd.0:Restarting daemon 2026-02-12T12:31:43.779 DEBUG:teuthology.orchestra.run.trial063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2026-02-12T12:31:43.821 INFO:tasks.ceph.osd.0:Started 2026-02-12T12:31:43.821 DEBUG:teuthology.orchestra.run.trial063:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2026-02-12T12:31:43.849 INFO:tasks.ceph.osd.1:Restarting daemon 2026-02-12T12:31:43.850 DEBUG:teuthology.orchestra.run.trial063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2026-02-12T12:31:43.892 INFO:tasks.ceph.osd.1:Started 2026-02-12T12:31:43.892 DEBUG:teuthology.orchestra.run.trial063:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2026-02-12T12:31:43.921 INFO:tasks.ceph.osd.2:Restarting daemon 2026-02-12T12:31:43.921 DEBUG:teuthology.orchestra.run.trial063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2026-02-12T12:31:43.963 INFO:tasks.ceph.osd.2:Started 2026-02-12T12:31:43.964 DEBUG:teuthology.orchestra.run.trial063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-12T12:31:44.041 INFO:teuthology.orchestra.run.trial063.stderr:2026-02-12T12:31:44.038+0000 7f76f7b0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:31:44.043 INFO:teuthology.orchestra.run.trial063.stderr:2026-02-12T12:31:44.042+0000 7f76f7b0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:31:44.075 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:31:44.075 INFO:teuthology.orchestra.run.trial063.stdout:{"epoch":6,"fsid":"0ceebbf4-3330-468c-a6ec-0a57b2aaee66","created":"2026-02-12T12:31:41.989046+0000","modified":"2026-02-12T12:31:43.740654+0000","last_up_change":"0.000000","last_in_change":"2026-02-12T12:31:43.740654+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"327d06b7-8ca5-40a0-ad4f-755b0878e1ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dce03db3-52c2-40c3-b9ec-de5cc9227cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"538db773-9089-49ed-b1e1-8fba4eeff801","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-12T12:31:44.084 INFO:tasks.ceph.ceph_manager.ceph:[] 2026-02-12T12:31:44.084 INFO:tasks.ceph:Waiting for OSDs to come up 2026-02-12T12:31:44.386 DEBUG:teuthology.orchestra.run.trial063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-12T12:31:44.457 INFO:teuthology.misc.health.trial063.stderr:2026-02-12T12:31:44.454+0000 7f21d3b5c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:31:44.458 INFO:teuthology.misc.health.trial063.stderr:2026-02-12T12:31:44.457+0000 7f21d3b5c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:31:44.491 INFO:teuthology.misc.health.trial063.stdout: 2026-02-12T12:31:44.491 INFO:teuthology.misc.health.trial063.stdout:{"epoch":6,"fsid":"0ceebbf4-3330-468c-a6ec-0a57b2aaee66","created":"2026-02-12T12:31:41.989046+0000","modified":"2026-02-12T12:31:43.740654+0000","last_up_change":"0.000000","last_in_change":"2026-02-12T12:31:43.740654+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"327d06b7-8ca5-40a0-ad4f-755b0878e1ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dce03db3-52c2-40c3-b9ec-de5cc9227cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"538db773-9089-49ed-b1e1-8fba4eeff801","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-12T12:31:44.499 DEBUG:teuthology.misc:0 of 3 OSDs are up 2026-02-12T12:31:45.432 INFO:tasks.ceph.mgr.x.trial063.stderr:2026-02-12T12:31:45.431+0000 7f8b4ab9b640 -1 mgr.server handle_report got status from non-daemon mon.a 2026-02-12T12:31:50.801 DEBUG:teuthology.orchestra.run.trial063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-12T12:31:50.870 INFO:teuthology.misc.health.trial063.stderr:2026-02-12T12:31:50.867+0000 7ff13eacb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:31:50.872 INFO:teuthology.misc.health.trial063.stderr:2026-02-12T12:31:50.871+0000 7ff13eacb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:31:50.983 INFO:teuthology.misc.health.trial063.stdout: 2026-02-12T12:31:50.983 INFO:teuthology.misc.health.trial063.stdout:{"epoch":8,"fsid":"0ceebbf4-3330-468c-a6ec-0a57b2aaee66","created":"2026-02-12T12:31:41.989046+0000","modified":"2026-02-12T12:31:50.913579+0000","last_up_change":"0.000000","last_in_change":"2026-02-12T12:31:43.740654+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"327d06b7-8ca5-40a0-ad4f-755b0878e1ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dce03db3-52c2-40c3-b9ec-de5cc9227cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"538db773-9089-49ed-b1e1-8fba4eeff801","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-12T12:31:50.992 DEBUG:teuthology.misc:0 of 3 OSDs are up 2026-02-12T12:31:57.293 DEBUG:teuthology.orchestra.run.trial063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-12T12:31:57.364 INFO:teuthology.misc.health.trial063.stderr:2026-02-12T12:31:57.360+0000 7fd10685a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:31:57.365 INFO:teuthology.misc.health.trial063.stderr:2026-02-12T12:31:57.363+0000 7fd10685a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:31:57.476 INFO:teuthology.misc.health.trial063.stdout: 2026-02-12T12:31:57.476 INFO:teuthology.misc.health.trial063.stdout:{"epoch":13,"fsid":"0ceebbf4-3330-468c-a6ec-0a57b2aaee66","created":"2026-02-12T12:31:41.989046+0000","modified":"2026-02-12T12:31:56.934814+0000","last_up_change":"2026-02-12T12:31:52.921461+0000","last_in_change":"2026-02-12T12:31:43.740654+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-12T12:31:54.442274+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"327d06b7-8ca5-40a0-ad4f-755b0878e1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.63:6806","nonce":2676210856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.63:6805","nonce":2676210856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.63:6813","nonce":2676210856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.63:6812","nonce":2676210856}]},"public_addr":"10.20.193.63:6806/2676210856","cluster_addr":"10.20.193.63:6805/2676210856","heartbeat_back_addr":"10.20.193.63:6813/2676210856","heartbeat_front_addr":"10.20.193.63:6812/2676210856","state":["exists","up"]},{"osd":1,"uuid":"dce03db3-52c2-40c3-b9ec-de5cc9227cde","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.63:6803","nonce":3405487706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.63:6804","nonce":3405487706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.63:6808","nonce":3405487706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.63:6807","nonce":3405487706}]},"public_addr":"10.20.193.63:6803/3405487706","cluster_addr":"10.20.193.63:6804/3405487706","heartbeat_back_addr":"10.20.193.63:6808/3405487706","heartbeat_front_addr":"10.20.193.63:6807/3405487706","state":["exists","up"]},{"osd":2,"uuid":"538db773-9089-49ed-b1e1-8fba4eeff801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.63:6809","nonce":770573261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.63:6810","nonce":770573261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.63:6811","nonce":770573261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.63:6802","nonce":770573261}]},"public_addr":"10.20.193.63:6809/770573261","cluster_addr":"10.20.193.63:6810/770573261","heartbeat_back_addr":"10.20.193.63:6811/770573261","heartbeat_front_addr":"10.20.193.63:6802/770573261","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-12T12:31:57.484 DEBUG:teuthology.misc:3 of 3 OSDs are up 2026-02-12T12:31:57.485 INFO:tasks.ceph:Creating RBD pool 2026-02-12T12:31:57.485 DEBUG:teuthology.orchestra.run.trial063:> sudo ceph --cluster ceph osd pool create rbd 8 2026-02-12T12:31:57.559 INFO:teuthology.orchestra.run.trial063.stderr:2026-02-12T12:31:57.553+0000 7f4349206640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:31:57.560 INFO:teuthology.orchestra.run.trial063.stderr:2026-02-12T12:31:57.558+0000 7f4349206640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:31:58.269 INFO:teuthology.orchestra.run.trial063.stderr:pool 'rbd' created 2026-02-12T12:31:58.285 DEBUG:teuthology.orchestra.run.trial063:> rbd --cluster ceph pool init rbd 2026-02-12T12:31:58.325 INFO:teuthology.orchestra.run.trial063.stderr:2026-02-12T12:31:58.323+0000 7f38a48ea240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:31:58.325 INFO:teuthology.orchestra.run.trial063.stderr:2026-02-12T12:31:58.323+0000 7f38a48ea240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:31:58.327 INFO:teuthology.orchestra.run.trial063.stderr:2026-02-12T12:31:58.325+0000 7f38a48ea240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:32:01.305 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2026-02-12T12:32:01.306 DEBUG:teuthology.orchestra.run.trial063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2026-02-12T12:32:01.306 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2026-02-12T12:32:01.424 INFO:teuthology.orchestra.run.trial063.stderr:2026-02-12T12:32:01.419+0000 7f03101e4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:32:01.425 INFO:teuthology.orchestra.run.trial063.stderr:2026-02-12T12:32:01.423+0000 7f03101e4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:32:01.535 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:32:01.545 INFO:teuthology.orchestra.run.trial063.stdout:[{"version":1,"timestamp":"0.000000","name":"","changes":[]}] 2026-02-12T12:32:01.546 INFO:tasks.ceph_manager:config epoch is 1 2026-02-12T12:32:01.546 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2026-02-12T12:32:01.546 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2026-02-12T12:32:01.546 DEBUG:teuthology.orchestra.run.trial063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2026-02-12T12:32:01.624 INFO:teuthology.orchestra.run.trial063.stderr:2026-02-12T12:32:01.620+0000 7f559698c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:32:01.625 INFO:teuthology.orchestra.run.trial063.stderr:2026-02-12T12:32:01.623+0000 7f559698c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:32:01.766 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:32:01.775 INFO:teuthology.orchestra.run.trial063.stdout:{"epoch":5,"flags":0,"active_gid":4103,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.63:6800","nonce":2838481324},{"type":"v1","addr":"10.20.193.63:6801","nonce":2838481324}]},"active_addr":"10.20.193.63:6801/2838481324","active_change":"2026-02-12T12:31:44.426206+0000","active_mgr_features":4544132024016699391,"available":true,"standbys":[],"modules":["iostat","nfs"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_pg_upmap_activity":{"name":"update_pg_upmap_activity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Updates pg_upmap activity stats to be used in `balancer status detail`","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often the agent on each host will gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"The first TCP port the agent will try to bind to. Up to 1000 subsequent ports will be attempted if this port cannot be bound","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this option can facilitate debugging daemons that encounter runtime problems.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"How frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"Fraction of total system memory to divide among autotuned daemons. Converged systems hosting both Ceph and compute should set a lower value; systems with more than 10 GiB of memory per OSD may benefit from a slightly larger value so that the abundance of RAM can be fully utilized","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command persistent logging","long_desc":"","tags":[],"see_also":[]},"certificate_automated_rotation_enabled":{"name":"certificate_automated_rotation_enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"This flag controls whether cephadm automatically rotates certificates upon expiration.","long_desc":"","tags":[],"see_also":[]},"certificate_check_debug_mode":{"name":"certificate_check_debug_mode","type":"bool","level":"dev","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"FOR TESTING ONLY: This flag forces the certificate check instead of waiting for certificate_check_period.","long_desc":"","tags":[],"see_also":[]},"certificate_check_period":{"name":"certificate_check_period","type":"int","level":"advanced","flags":0,"default_value":"1","min":"0","max":"30","enum_allowed":[],"desc":"Specifies how often (in days) the certificate should be checked for validity.","long_desc":"","tags":[],"see_also":[]},"certificate_duration_days":{"name":"certificate_duration_days","type":"int","level":"advanced","flags":0,"default_value":"1095","min":"90","max":"3650","enum_allowed":[],"desc":"Specifies the duration of self-signed certificates generated and signed by the cephadm root CA","long_desc":"","tags":[],"see_also":[]},"certificate_renewal_threshold_days":{"name":"certificate_renewal_threshold_days","type":"int","level":"advanced","flags":0,"default_value":"30","min":"10","max":"90","enum_allowed":[],"desc":"Specifies the lead time in days to initiate certificate renewal before expiration.","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently Podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.28.1","min":"","max":"","enum_allowed":[],"desc":"Alertmanager container image","long_desc":"","tags":[],"see_also":[]},"container_image_alloy":{"name":"container_image_alloy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/alloy:latest","min":"","max":"","enum_allowed":[],"desc":"Alloy container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"Elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:12.3.1","min":"","max":"","enum_allowed":[],"desc":"Grafana container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"Haproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.9.1","min":"","max":"","enum_allowed":[],"desc":"Node exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.6","min":"","max":"","enum_allowed":[],"desc":"Nvmeof container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"Oauth2 proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v3.6.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba metrics container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"Snmp gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"quay.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"Location of dashboards to include in Grafana","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"How frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether the hardware monitoring daemon be deployed on every host?","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at the debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"Max number of daemons per service per host: guard against mistakes and runaways","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"Max number of OSDs that will be drained simultaneously when OSDs are removed","long_desc":"","tags":[],"see_also":[]},"max_parallel_osd_upgrades":{"name":"max_parallel_osd_upgrades","type":"int","level":"advanced","flags":0,"default_value":"16","min":"","max":"","enum_allowed":[],"desc":"Maximum number of OSD daemons upgraded in parallel.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"Mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default IP address for RedFish API (OOB management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"Location of alerts to include in Prometheus Alertmanager","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry URL for login purposes. This is not the default registry.","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"Cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Customized SSH config file to distribute to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many sequential SSH connection liveness check failures before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often SSH connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"stray_daemon_check_interval":{"name":"stray_daemon_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"How frequently cephadm should check for the presence of stray daemons","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use the cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest to ensure that all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROM_ALERT_CREDENTIAL_CACHE_TTL":{"name":"PROM_ALERT_CREDENTIAL_CACHE_TTL","type":"int","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_HOSTNAME_PER_DAEMON":{"name":"RGW_HOSTNAME_PER_DAEMON","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crypto_caller":{"name":"crypto_caller","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":true,"error_string":"","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"healthcheck_history_max_entries":{"name":"healthcheck_history_max_entries","type":"int","level":"advanced","flags":1,"default_value":"1000","min":"","max":"","enum_allowed":[],"desc":"Maximum number of health check history entries to keep","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"prometheus_tls_secret_name":{"name":"prometheus_tls_secret_name","type":"str","level":"advanced","flags":0,"default_value":"rook-ceph-prometheus-server-tls","min":"","max":"","enum_allowed":[],"desc":"name of tls secret in k8s for prometheus","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_cloning":{"name":"pause_cloning","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_purging":{"name":"pause_purging","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous subvolume purge threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"tentacle":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"force_disabled_modules":{},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"10.20.193.63:0","nonce":2420713811}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"10.20.193.63:0","nonce":365717242}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"10.20.193.63:0","nonce":334031026}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"10.20.193.63:0","nonce":1216552194}]}]} 2026-02-12T12:32:01.779 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2026-02-12T12:32:01.780 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2026-02-12T12:32:01.780 DEBUG:teuthology.orchestra.run.trial063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-12T12:32:01.855 INFO:teuthology.orchestra.run.trial063.stderr:2026-02-12T12:32:01.852+0000 7f8b3f9f5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:32:01.857 INFO:teuthology.orchestra.run.trial063.stderr:2026-02-12T12:32:01.855+0000 7f8b3f9f5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:32:01.966 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:32:01.966 INFO:teuthology.orchestra.run.trial063.stdout:{"epoch":17,"fsid":"0ceebbf4-3330-468c-a6ec-0a57b2aaee66","created":"2026-02-12T12:31:41.989046+0000","modified":"2026-02-12T12:32:01.276682+0000","last_up_change":"2026-02-12T12:31:52.921461+0000","last_in_change":"2026-02-12T12:31:43.740654+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-12T12:31:54.442274+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-12T12:31:57.673572+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"327d06b7-8ca5-40a0-ad4f-755b0878e1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.63:6806","nonce":2676210856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.63:6805","nonce":2676210856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.63:6813","nonce":2676210856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.63:6812","nonce":2676210856}]},"public_addr":"10.20.193.63:6806/2676210856","cluster_addr":"10.20.193.63:6805/2676210856","heartbeat_back_addr":"10.20.193.63:6813/2676210856","heartbeat_front_addr":"10.20.193.63:6812/2676210856","state":["exists","up"]},{"osd":1,"uuid":"dce03db3-52c2-40c3-b9ec-de5cc9227cde","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.63:6803","nonce":3405487706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.63:6804","nonce":3405487706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.63:6808","nonce":3405487706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.63:6807","nonce":3405487706}]},"public_addr":"10.20.193.63:6803/3405487706","cluster_addr":"10.20.193.63:6804/3405487706","heartbeat_back_addr":"10.20.193.63:6808/3405487706","heartbeat_front_addr":"10.20.193.63:6807/3405487706","state":["exists","up"]},{"osd":2,"uuid":"538db773-9089-49ed-b1e1-8fba4eeff801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.63:6809","nonce":770573261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.63:6810","nonce":770573261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.63:6811","nonce":770573261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.63:6802","nonce":770573261}]},"public_addr":"10.20.193.63:6809/770573261","cluster_addr":"10.20.193.63:6810/770573261","heartbeat_back_addr":"10.20.193.63:6811/770573261","heartbeat_front_addr":"10.20.193.63:6802/770573261","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-12T12:32:01.977 INFO:tasks.ceph.ceph_manager.ceph:all up! 2026-02-12T12:32:01.977 DEBUG:teuthology.orchestra.run.trial063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-12T12:32:02.053 INFO:teuthology.orchestra.run.trial063.stderr:2026-02-12T12:32:02.049+0000 7f2cb5ecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:32:02.054 INFO:teuthology.orchestra.run.trial063.stderr:2026-02-12T12:32:02.052+0000 7f2cb5ecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:32:02.163 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:32:02.163 INFO:teuthology.orchestra.run.trial063.stdout:{"epoch":17,"fsid":"0ceebbf4-3330-468c-a6ec-0a57b2aaee66","created":"2026-02-12T12:31:41.989046+0000","modified":"2026-02-12T12:32:01.276682+0000","last_up_change":"2026-02-12T12:31:52.921461+0000","last_in_change":"2026-02-12T12:31:43.740654+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-12T12:31:54.442274+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-12T12:31:57.673572+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"327d06b7-8ca5-40a0-ad4f-755b0878e1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.63:6806","nonce":2676210856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.63:6805","nonce":2676210856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.63:6813","nonce":2676210856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.63:6812","nonce":2676210856}]},"public_addr":"10.20.193.63:6806/2676210856","cluster_addr":"10.20.193.63:6805/2676210856","heartbeat_back_addr":"10.20.193.63:6813/2676210856","heartbeat_front_addr":"10.20.193.63:6812/2676210856","state":["exists","up"]},{"osd":1,"uuid":"dce03db3-52c2-40c3-b9ec-de5cc9227cde","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.63:6803","nonce":3405487706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.63:6804","nonce":3405487706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.63:6808","nonce":3405487706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.63:6807","nonce":3405487706}]},"public_addr":"10.20.193.63:6803/3405487706","cluster_addr":"10.20.193.63:6804/3405487706","heartbeat_back_addr":"10.20.193.63:6808/3405487706","heartbeat_front_addr":"10.20.193.63:6807/3405487706","state":["exists","up"]},{"osd":2,"uuid":"538db773-9089-49ed-b1e1-8fba4eeff801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.63:6809","nonce":770573261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.63:6810","nonce":770573261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.63:6811","nonce":770573261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.63:6802","nonce":770573261}]},"public_addr":"10.20.193.63:6809/770573261","cluster_addr":"10.20.193.63:6810/770573261","heartbeat_back_addr":"10.20.193.63:6811/770573261","heartbeat_front_addr":"10.20.193.63:6802/770573261","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-12T12:32:02.173 DEBUG:teuthology.orchestra.run.trial063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-12T12:32:02.174 DEBUG:teuthology.orchestra.run.trial063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-12T12:32:02.174 DEBUG:teuthology.orchestra.run.trial063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-12T12:32:02.255 INFO:teuthology.orchestra.run.trial063.stderr:2026-02-12T12:32:02.250+0000 7f3468c1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:32:02.255 INFO:teuthology.orchestra.run.trial063.stderr:2026-02-12T12:32:02.250+0000 7fec91a1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:32:02.256 INFO:teuthology.orchestra.run.trial063.stderr:2026-02-12T12:32:02.254+0000 7f3468c1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:32:02.256 INFO:teuthology.orchestra.run.trial063.stderr:2026-02-12T12:32:02.254+0000 7fec91a1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:32:02.257 INFO:teuthology.orchestra.run.trial063.stderr:2026-02-12T12:32:02.252+0000 7f4d43f0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:32:02.258 INFO:teuthology.orchestra.run.trial063.stderr:2026-02-12T12:32:02.256+0000 7f4d43f0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:32:02.271 INFO:teuthology.orchestra.run.trial063.stdout:38654705674 2026-02-12T12:32:02.271 DEBUG:teuthology.orchestra.run.trial063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-12T12:32:02.272 INFO:teuthology.orchestra.run.trial063.stdout:42949672969 2026-02-12T12:32:02.273 DEBUG:teuthology.orchestra.run.trial063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-12T12:32:02.274 INFO:teuthology.orchestra.run.trial063.stdout:42949672969 2026-02-12T12:32:02.274 DEBUG:teuthology.orchestra.run.trial063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-12T12:32:02.392 INFO:teuthology.orchestra.run.trial063.stderr:2026-02-12T12:32:02.388+0000 7f6c35364640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:32:02.393 INFO:teuthology.orchestra.run.trial063.stderr:2026-02-12T12:32:02.388+0000 7f96b843d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:32:02.393 INFO:teuthology.orchestra.run.trial063.stderr:2026-02-12T12:32:02.388+0000 7f85803a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:32:02.394 INFO:teuthology.orchestra.run.trial063.stderr:2026-02-12T12:32:02.392+0000 7f96b843d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:32:02.394 INFO:teuthology.orchestra.run.trial063.stderr:2026-02-12T12:32:02.392+0000 7f6c35364640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:32:02.395 INFO:teuthology.orchestra.run.trial063.stderr:2026-02-12T12:32:02.392+0000 7f85803a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:32:02.506 INFO:teuthology.orchestra.run.trial063.stdout:42949672969 2026-02-12T12:32:02.507 INFO:teuthology.orchestra.run.trial063.stdout:38654705674 2026-02-12T12:32:02.513 INFO:teuthology.orchestra.run.trial063.stdout:42949672969 2026-02-12T12:32:02.519 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705674 got 38654705674 for osd.1 2026-02-12T12:32:02.519 DEBUG:teuthology.parallel:result is None 2026-02-12T12:32:02.520 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672969 got 42949672969 for osd.0 2026-02-12T12:32:02.520 DEBUG:teuthology.parallel:result is None 2026-02-12T12:32:02.525 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672969 got 42949672969 for osd.2 2026-02-12T12:32:02.525 DEBUG:teuthology.parallel:result is None 2026-02-12T12:32:02.525 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-12T12:32:02.525 DEBUG:teuthology.orchestra.run.trial063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T12:32:02.638 INFO:teuthology.orchestra.run.trial063.stderr:2026-02-12T12:32:02.634+0000 7fcaf0b93640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:32:02.639 INFO:teuthology.orchestra.run.trial063.stderr:2026-02-12T12:32:02.637+0000 7fcaf0b93640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:32:02.748 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:32:02.748 INFO:teuthology.orchestra.run.trial063.stderr:dumped all 2026-02-12T12:32:02.756 INFO:teuthology.orchestra.run.trial063.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2026-02-12T12:32:02.430076+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":157286352,"kb_used":62917052,"kb_used_data":62917052,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":94369300,"statfs":{"total":161061224448,"available":96634163200,"internally_reserved":0,"allocated":64427061248,"data_stored":64427061248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.493025"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-12T12:32:01.290639+0000","last_change":"2026-02-12T12:32:01.294283+0000","last_active":"2026-02-12T12:32:01.290639+0000","last_peered":"2026-02-12T12:32:01.290639+0000","last_clean":"2026-02-12T12:32:01.290639+0000","last_became_active":"2026-02-12T12:31:59.312499+0000","last_became_peered":"2026-02-12T12:31:59.312499+0000","last_unstale":"2026-02-12T12:32:01.290639+0000","last_undegraded":"2026-02-12T12:32:01.290639+0000","last_fullsized":"2026-02-12T12:32:01.290639+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T12:31:58.267408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T12:31:58.267408+0000","last_clean_scrub_stamp":"2026-02-12T12:31:58.267408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-12T12:32:01.292469+0000","last_change":"2026-02-12T12:32:01.294779+0000","last_active":"2026-02-12T12:32:01.292469+0000","last_peered":"2026-02-12T12:32:01.292469+0000","last_clean":"2026-02-12T12:32:01.292469+0000","last_became_active":"2026-02-12T12:31:59.306352+0000","last_became_peered":"2026-02-12T12:31:59.306352+0000","last_unstale":"2026-02-12T12:32:01.292469+0000","last_undegraded":"2026-02-12T12:32:01.292469+0000","last_fullsized":"2026-02-12T12:32:01.292469+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T12:31:58.267408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T12:31:58.267408+0000","last_clean_scrub_stamp":"2026-02-12T12:31:58.267408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-12T12:32:01.292444+0000","last_change":"2026-02-12T12:32:01.296639+0000","last_active":"2026-02-12T12:32:01.292444+0000","last_peered":"2026-02-12T12:32:01.292444+0000","last_clean":"2026-02-12T12:32:01.292444+0000","last_became_active":"2026-02-12T12:31:59.310601+0000","last_became_peered":"2026-02-12T12:31:59.310601+0000","last_unstale":"2026-02-12T12:32:01.292444+0000","last_undegraded":"2026-02-12T12:32:01.292444+0000","last_fullsized":"2026-02-12T12:32:01.292444+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T12:31:58.267408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T12:31:58.267408+0000","last_clean_scrub_stamp":"2026-02-12T12:31:58.267408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-12T12:32:01.292399+0000","last_change":"2026-02-12T12:32:01.294164+0000","last_active":"2026-02-12T12:32:01.292399+0000","last_peered":"2026-02-12T12:32:01.292399+0000","last_clean":"2026-02-12T12:32:01.292399+0000","last_became_active":"2026-02-12T12:31:59.314413+0000","last_became_peered":"2026-02-12T12:31:59.314413+0000","last_unstale":"2026-02-12T12:32:01.292399+0000","last_undegraded":"2026-02-12T12:32:01.292399+0000","last_fullsized":"2026-02-12T12:32:01.292399+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T12:31:58.267408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T12:31:58.267408+0000","last_clean_scrub_stamp":"2026-02-12T12:31:58.267408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'1","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-12T12:32:01.291133+0000","last_change":"2026-02-12T12:32:01.291133+0000","last_active":"2026-02-12T12:32:01.291133+0000","last_peered":"2026-02-12T12:32:01.291133+0000","last_clean":"2026-02-12T12:32:01.291133+0000","last_became_active":"2026-02-12T12:31:59.322763+0000","last_became_peered":"2026-02-12T12:31:59.322763+0000","last_unstale":"2026-02-12T12:32:01.291133+0000","last_undegraded":"2026-02-12T12:32:01.291133+0000","last_fullsized":"2026-02-12T12:32:01.291133+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T12:31:58.267408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T12:31:58.267408+0000","last_clean_scrub_stamp":"2026-02-12T12:31:58.267408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-12T12:32:01.290417+0000","last_change":"2026-02-12T12:32:01.292259+0000","last_active":"2026-02-12T12:32:01.290417+0000","last_peered":"2026-02-12T12:32:01.290417+0000","last_clean":"2026-02-12T12:32:01.290417+0000","last_became_active":"2026-02-12T12:31:59.316804+0000","last_became_peered":"2026-02-12T12:31:59.316804+0000","last_unstale":"2026-02-12T12:32:01.290417+0000","last_undegraded":"2026-02-12T12:32:01.290417+0000","last_fullsized":"2026-02-12T12:32:01.290417+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T12:31:58.267408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T12:31:58.267408+0000","last_clean_scrub_stamp":"2026-02-12T12:31:58.267408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-12T12:32:01.288269+0000","last_change":"2026-02-12T12:32:01.290595+0000","last_active":"2026-02-12T12:32:01.288269+0000","last_peered":"2026-02-12T12:32:01.288269+0000","last_clean":"2026-02-12T12:32:01.288269+0000","last_became_active":"2026-02-12T12:31:59.323426+0000","last_became_peered":"2026-02-12T12:31:59.323426+0000","last_unstale":"2026-02-12T12:32:01.288269+0000","last_undegraded":"2026-02-12T12:32:01.288269+0000","last_fullsized":"2026-02-12T12:32:01.288269+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T12:31:58.267408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T12:31:58.267408+0000","last_clean_scrub_stamp":"2026-02-12T12:31:58.267408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":16,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-12T12:32:01.292399+0000","last_change":"2026-02-12T12:32:01.296366+0000","last_active":"2026-02-12T12:32:01.292399+0000","last_peered":"2026-02-12T12:32:01.292399+0000","last_clean":"2026-02-12T12:32:01.292399+0000","last_became_active":"2026-02-12T12:31:59.304060+0000","last_became_peered":"2026-02-12T12:31:59.304060+0000","last_unstale":"2026-02-12T12:32:01.292399+0000","last_undegraded":"2026-02-12T12:32:01.292399+0000","last_fullsized":"2026-02-12T12:32:01.292399+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T12:31:58.267408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T12:31:58.267408+0000","last_clean_scrub_stamp":"2026-02-12T12:31:58.267408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":90,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-12T12:32:01.291586+0000","last_change":"2026-02-12T12:31:55.959508+0000","last_active":"2026-02-12T12:32:01.291586+0000","last_peered":"2026-02-12T12:32:01.291586+0000","last_clean":"2026-02-12T12:32:01.291586+0000","last_became_active":"2026-02-12T12:31:55.952326+0000","last_became_peered":"2026-02-12T12:31:55.952326+0000","last_unstale":"2026-02-12T12:32:01.291586+0000","last_undegraded":"2026-02-12T12:32:01.291586+0000","last_fullsized":"2026-02-12T12:32:01.291586+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T12:31:54.923087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T12:31:54.923087+0000","last_clean_scrub_stamp":"2026-02-12T12:31:54.923087+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":161061224448,"available":96634163200,"internally_reserved":0,"allocated":64427061248,"data_stored":64427061248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53687074816,"available":32211079168,"internally_reserved":0,"allocated":21475995648,"data_stored":21475995648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20971820,"kb_used_data":20971820,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456964,"statfs":{"total":53687074816,"available":32211931136,"internally_reserved":0,"allocated":21475143680,"data_stored":21475143680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972580,"kb_used_data":20972580,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456204,"statfs":{"total":53687074816,"available":32211152896,"internally_reserved":0,"allocated":21475921920,"data_stored":21475921920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972652,"kb_used_data":20972652,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456132,"statfs":{"total":53687074816,"available":32211079168,"internally_reserved":0,"allocated":21475995648,"data_stored":21475995648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53687074816,"available":32211079168,"internally_reserved":0,"allocated":21475995648,"data_stored":21475995648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53687074816,"available":32211152896,"internally_reserved":0,"allocated":21475921920,"data_stored":21475921920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53687074816,"available":32211079168,"internally_reserved":0,"allocated":21475995648,"data_stored":21475995648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53687074816,"available":32211931136,"internally_reserved":0,"allocated":21475143680,"data_stored":21475143680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T12:32:02.758 DEBUG:teuthology.orchestra.run.trial063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T12:32:02.835 INFO:teuthology.orchestra.run.trial063.stderr:2026-02-12T12:32:02.831+0000 7f6933611640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:32:02.836 INFO:teuthology.orchestra.run.trial063.stderr:2026-02-12T12:32:02.834+0000 7f6933611640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:32:02.945 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:32:02.945 INFO:teuthology.orchestra.run.trial063.stderr:dumped all 2026-02-12T12:32:02.955 INFO:teuthology.orchestra.run.trial063.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2026-02-12T12:32:02.430076+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":157286352,"kb_used":62917052,"kb_used_data":62917052,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":94369300,"statfs":{"total":161061224448,"available":96634163200,"internally_reserved":0,"allocated":64427061248,"data_stored":64427061248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.493025"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-12T12:32:01.290639+0000","last_change":"2026-02-12T12:32:01.294283+0000","last_active":"2026-02-12T12:32:01.290639+0000","last_peered":"2026-02-12T12:32:01.290639+0000","last_clean":"2026-02-12T12:32:01.290639+0000","last_became_active":"2026-02-12T12:31:59.312499+0000","last_became_peered":"2026-02-12T12:31:59.312499+0000","last_unstale":"2026-02-12T12:32:01.290639+0000","last_undegraded":"2026-02-12T12:32:01.290639+0000","last_fullsized":"2026-02-12T12:32:01.290639+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T12:31:58.267408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T12:31:58.267408+0000","last_clean_scrub_stamp":"2026-02-12T12:31:58.267408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-12T12:32:01.292469+0000","last_change":"2026-02-12T12:32:01.294779+0000","last_active":"2026-02-12T12:32:01.292469+0000","last_peered":"2026-02-12T12:32:01.292469+0000","last_clean":"2026-02-12T12:32:01.292469+0000","last_became_active":"2026-02-12T12:31:59.306352+0000","last_became_peered":"2026-02-12T12:31:59.306352+0000","last_unstale":"2026-02-12T12:32:01.292469+0000","last_undegraded":"2026-02-12T12:32:01.292469+0000","last_fullsized":"2026-02-12T12:32:01.292469+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T12:31:58.267408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T12:31:58.267408+0000","last_clean_scrub_stamp":"2026-02-12T12:31:58.267408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-12T12:32:01.292444+0000","last_change":"2026-02-12T12:32:01.296639+0000","last_active":"2026-02-12T12:32:01.292444+0000","last_peered":"2026-02-12T12:32:01.292444+0000","last_clean":"2026-02-12T12:32:01.292444+0000","last_became_active":"2026-02-12T12:31:59.310601+0000","last_became_peered":"2026-02-12T12:31:59.310601+0000","last_unstale":"2026-02-12T12:32:01.292444+0000","last_undegraded":"2026-02-12T12:32:01.292444+0000","last_fullsized":"2026-02-12T12:32:01.292444+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T12:31:58.267408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T12:31:58.267408+0000","last_clean_scrub_stamp":"2026-02-12T12:31:58.267408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-12T12:32:01.292399+0000","last_change":"2026-02-12T12:32:01.294164+0000","last_active":"2026-02-12T12:32:01.292399+0000","last_peered":"2026-02-12T12:32:01.292399+0000","last_clean":"2026-02-12T12:32:01.292399+0000","last_became_active":"2026-02-12T12:31:59.314413+0000","last_became_peered":"2026-02-12T12:31:59.314413+0000","last_unstale":"2026-02-12T12:32:01.292399+0000","last_undegraded":"2026-02-12T12:32:01.292399+0000","last_fullsized":"2026-02-12T12:32:01.292399+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T12:31:58.267408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T12:31:58.267408+0000","last_clean_scrub_stamp":"2026-02-12T12:31:58.267408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'1","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-12T12:32:01.291133+0000","last_change":"2026-02-12T12:32:01.291133+0000","last_active":"2026-02-12T12:32:01.291133+0000","last_peered":"2026-02-12T12:32:01.291133+0000","last_clean":"2026-02-12T12:32:01.291133+0000","last_became_active":"2026-02-12T12:31:59.322763+0000","last_became_peered":"2026-02-12T12:31:59.322763+0000","last_unstale":"2026-02-12T12:32:01.291133+0000","last_undegraded":"2026-02-12T12:32:01.291133+0000","last_fullsized":"2026-02-12T12:32:01.291133+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T12:31:58.267408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T12:31:58.267408+0000","last_clean_scrub_stamp":"2026-02-12T12:31:58.267408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-12T12:32:01.290417+0000","last_change":"2026-02-12T12:32:01.292259+0000","last_active":"2026-02-12T12:32:01.290417+0000","last_peered":"2026-02-12T12:32:01.290417+0000","last_clean":"2026-02-12T12:32:01.290417+0000","last_became_active":"2026-02-12T12:31:59.316804+0000","last_became_peered":"2026-02-12T12:31:59.316804+0000","last_unstale":"2026-02-12T12:32:01.290417+0000","last_undegraded":"2026-02-12T12:32:01.290417+0000","last_fullsized":"2026-02-12T12:32:01.290417+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T12:31:58.267408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T12:31:58.267408+0000","last_clean_scrub_stamp":"2026-02-12T12:31:58.267408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-12T12:32:01.288269+0000","last_change":"2026-02-12T12:32:01.290595+0000","last_active":"2026-02-12T12:32:01.288269+0000","last_peered":"2026-02-12T12:32:01.288269+0000","last_clean":"2026-02-12T12:32:01.288269+0000","last_became_active":"2026-02-12T12:31:59.323426+0000","last_became_peered":"2026-02-12T12:31:59.323426+0000","last_unstale":"2026-02-12T12:32:01.288269+0000","last_undegraded":"2026-02-12T12:32:01.288269+0000","last_fullsized":"2026-02-12T12:32:01.288269+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T12:31:58.267408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T12:31:58.267408+0000","last_clean_scrub_stamp":"2026-02-12T12:31:58.267408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":16,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-12T12:32:01.292399+0000","last_change":"2026-02-12T12:32:01.296366+0000","last_active":"2026-02-12T12:32:01.292399+0000","last_peered":"2026-02-12T12:32:01.292399+0000","last_clean":"2026-02-12T12:32:01.292399+0000","last_became_active":"2026-02-12T12:31:59.304060+0000","last_became_peered":"2026-02-12T12:31:59.304060+0000","last_unstale":"2026-02-12T12:32:01.292399+0000","last_undegraded":"2026-02-12T12:32:01.292399+0000","last_fullsized":"2026-02-12T12:32:01.292399+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T12:31:58.267408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T12:31:58.267408+0000","last_clean_scrub_stamp":"2026-02-12T12:31:58.267408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":90,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-12T12:32:01.291586+0000","last_change":"2026-02-12T12:31:55.959508+0000","last_active":"2026-02-12T12:32:01.291586+0000","last_peered":"2026-02-12T12:32:01.291586+0000","last_clean":"2026-02-12T12:32:01.291586+0000","last_became_active":"2026-02-12T12:31:55.952326+0000","last_became_peered":"2026-02-12T12:31:55.952326+0000","last_unstale":"2026-02-12T12:32:01.291586+0000","last_undegraded":"2026-02-12T12:32:01.291586+0000","last_fullsized":"2026-02-12T12:32:01.291586+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T12:31:54.923087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T12:31:54.923087+0000","last_clean_scrub_stamp":"2026-02-12T12:31:54.923087+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":161061224448,"available":96634163200,"internally_reserved":0,"allocated":64427061248,"data_stored":64427061248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53687074816,"available":32211079168,"internally_reserved":0,"allocated":21475995648,"data_stored":21475995648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20971820,"kb_used_data":20971820,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456964,"statfs":{"total":53687074816,"available":32211931136,"internally_reserved":0,"allocated":21475143680,"data_stored":21475143680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972580,"kb_used_data":20972580,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456204,"statfs":{"total":53687074816,"available":32211152896,"internally_reserved":0,"allocated":21475921920,"data_stored":21475921920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972652,"kb_used_data":20972652,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456132,"statfs":{"total":53687074816,"available":32211079168,"internally_reserved":0,"allocated":21475995648,"data_stored":21475995648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53687074816,"available":32211079168,"internally_reserved":0,"allocated":21475995648,"data_stored":21475995648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53687074816,"available":32211152896,"internally_reserved":0,"allocated":21475921920,"data_stored":21475921920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53687074816,"available":32211079168,"internally_reserved":0,"allocated":21475995648,"data_stored":21475995648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53687074816,"available":32211931136,"internally_reserved":0,"allocated":21475143680,"data_stored":21475143680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T12:32:02.956 INFO:tasks.ceph.ceph_manager.ceph:clean! 2026-02-12T12:32:02.957 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2026-02-12T12:32:02.957 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2026-02-12T12:32:02.957 DEBUG:teuthology.orchestra.run.trial063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-12T12:32:03.032 INFO:teuthology.orchestra.run.trial063.stderr:2026-02-12T12:32:03.028+0000 7f0ae9dec640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:32:03.033 INFO:teuthology.orchestra.run.trial063.stderr:2026-02-12T12:32:03.031+0000 7f0ae9dec640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:32:03.169 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:32:03.169 INFO:teuthology.orchestra.run.trial063.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2026-02-12T12:32:03.178 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2026-02-12T12:32:03.179 INFO:teuthology.run_tasks:Running task ssh_keys... 2026-02-12T12:32:03.275 DEBUG:teuthology.orchestra.run.trial063:> echo 'StrictHostKeyChecking no 2026-02-12T12:32:03.275 DEBUG:teuthology.orchestra.run.trial063:> ' > /home/ubuntu/.ssh/config && echo 'UserKnownHostsFile ' /dev/null >> /home/ubuntu/.ssh/config && chmod 600 /home/ubuntu/.ssh/config 2026-02-12T12:32:03.293 INFO:teuthology.task.ssh_keys:generated public key AAAAB3NzaC1yc2EAAAADAQABAAABAQDoK5RbBgfFObp9L2xp2L4MdSJopskBzgzdLR5ljf9vefdMlfCIc9On6dcFhCydW+x70gQvsuAD0qWmzKF8zkPETxzVpNaqsrDebjN0OQgM9oHO2VAxNZ5QPsOlUzwRQ0SMG/qbXirZeR7PDb+Fn2+0uA1Gslz7ipJppkA3wukyT+woVnm3iFfuSDzYwuB4BBrYGQoZ4AD7D3sDW5vIOJcdbnvUVV9u8Un8eVMDYQtdWsVp9M0PPht9yvzcRqAKTbsThD9RbJHB2wBZUycde2KxmKkoFv6Z3tS2cCzBpIzWsE4OVVw8tshJ0ZxWl/4Y+/WfIOzbjq+Kw0o5OQqDSvU/ 2026-02-12T12:32:03.293 INFO:teuthology.task.ssh_keys:pushing keys to trial063.front.sepia.ceph.com for ubuntu 2026-02-12T12:32:03.293 DEBUG:teuthology.orchestra.run.trial063:> rm -f -- /home/ubuntu/.ssh/id_rsa 2026-02-12T12:32:03.349 DEBUG:teuthology.orchestra.run.trial063:> set -ex 2026-02-12T12:32:03.349 DEBUG:teuthology.orchestra.run.trial063:> dd of=/home/ubuntu/.ssh/id_rsa 2026-02-12T12:32:03.349 DEBUG:teuthology.orchestra.run.trial063:> chmod 0500 /home/ubuntu/.ssh/id_rsa 2026-02-12T12:32:03.405 DEBUG:teuthology.orchestra.run.trial063:> rm -f -- /home/ubuntu/.ssh/id_rsa.pub 2026-02-12T12:32:03.461 DEBUG:teuthology.orchestra.run.trial063:> set -ex 2026-02-12T12:32:03.461 DEBUG:teuthology.orchestra.run.trial063:> dd of=/home/ubuntu/.ssh/id_rsa.pub 2026-02-12T12:32:03.517 DEBUG:teuthology.orchestra.run.trial063:> cp -v -a /home/ubuntu/.ssh/authorized_keys /home/ubuntu/.ssh/authorized_keys_2026-02-12_12:32:03:517732 2026-02-12T12:32:03.573 INFO:teuthology.orchestra.run.trial063.stdout:'/home/ubuntu/.ssh/authorized_keys' -> '/home/ubuntu/.ssh/authorized_keys_2026-02-12_12:32:03:517732' 2026-02-12T12:32:03.574 DEBUG:teuthology.orchestra.run.trial063:> set -ex 2026-02-12T12:32:03.574 DEBUG:teuthology.orchestra.run.trial063:> dd of=/home/ubuntu/.ssh/authorized_keys conv=notrunc oflag=append 2026-02-12T12:32:03.629 INFO:teuthology.run_tasks:Running task cbt... 2026-02-12T12:32:03.654 INFO:tasks.cbt:cbt configuration is {'cluster': {'user': 'ubuntu', 'head': 'ubuntu@trial063.front.sepia.ceph.com', 'osds': ['ubuntu@trial063.front.sepia.ceph.com'], 'mons': ['ubuntu@trial063.front.sepia.ceph.com'], 'clients': ['ubuntu@trial063.front.sepia.ceph.com'], 'rgws': {'ubuntu@trial063.front.sepia.ceph.com': None}, 'osds_per_node': 3, 'rebuild_every_test': False, 'use_existing': True, 'is_teuthology': True, 'iterations': 1, 'tmp_dir': '/tmp/cbt', 'pool_profiles': {'replicated': {'pg_size': 256, 'pgp_size': 256, 'replication': 'replicated'}}, 'pid_dir': '/var/run/ceph'}, 'benchmarks': {'radosbench': {'concurrent_ops': 4, 'concurrent_procs': 2, 'op_size': [4096], 'pool_monitoring_list': ['collectl'], 'pool_profile': 'replicated', 'run_monitoring_list': ['collectl'], 'time': 60, 'write_only': True}}, 'client_endpoints': None, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}} 2026-02-12T12:32:03.654 DEBUG:teuthology.orchestra.run.trial063:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/cbt 2026-02-12T12:32:03.690 DEBUG:teuthology.orchestra.run.trial063:> set -ex 2026-02-12T12:32:03.690 DEBUG:teuthology.orchestra.run.trial063:> dd of=/home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2026-02-12T12:32:03.743 DEBUG:teuthology.orchestra.run.trial063:> git clone --depth 1 -b master https://github.com/ceph/cbt.git /home/ubuntu/cephtest/cbt 2026-02-12T12:32:03.806 INFO:teuthology.orchestra.run.trial063.stderr:Cloning into '/home/ubuntu/cephtest/cbt'... 2026-02-12T12:32:04.059 INFO:tasks.cbt:Installing collectl 2026-02-12T12:32:04.059 DEBUG:teuthology.orchestra.run.trial063:> sudo mkdir collectl && cd collectl && sudo wget https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download -O collectl.tar.gz && sudo tar -xvf collectl.tar.gz && cd collectl-4.3.1 && sudo ./INSTALL 2026-02-12T12:32:04.134 INFO:teuthology.orchestra.run.trial063.stderr:--2026-02-12 12:32:04-- https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download 2026-02-12T12:32:04.186 INFO:teuthology.orchestra.run.trial063.stderr:Resolving sourceforge.net (sourceforge.net)... 104.18.12.149, 104.18.13.149, 2606:4700::6812:c95, ... 2026-02-12T12:32:04.189 INFO:teuthology.orchestra.run.trial063.stderr:Connecting to sourceforge.net (sourceforge.net)|104.18.12.149|:443... connected. 2026-02-12T12:32:04.407 INFO:teuthology.orchestra.run.trial063.stderr:HTTP request sent, awaiting response... 302 Found 2026-02-12T12:32:04.408 INFO:teuthology.orchestra.run.trial063.stderr:Location: https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABpjchE_0pdddENCY69q2GV61vA305S34NaLd5s0a3mHRDGm73QyOB5t25A9oZYd00QNVqR-GmIgElq_vswr0DbXoeY7w%3D%3D&use_mirror=pilotfiber&r= [following] 2026-02-12T12:32:04.408 INFO:teuthology.orchestra.run.trial063.stderr:--2026-02-12 12:32:04-- https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABpjchE_0pdddENCY69q2GV61vA305S34NaLd5s0a3mHRDGm73QyOB5t25A9oZYd00QNVqR-GmIgElq_vswr0DbXoeY7w%3D%3D&use_mirror=pilotfiber&r= 2026-02-12T12:32:04.442 INFO:teuthology.orchestra.run.trial063.stderr:Resolving downloads.sourceforge.net (downloads.sourceforge.net)... 104.18.12.149, 104.18.13.149, 2606:4700::6812:c95, ... 2026-02-12T12:32:04.445 INFO:teuthology.orchestra.run.trial063.stderr:Connecting to downloads.sourceforge.net (downloads.sourceforge.net)|104.18.12.149|:443... connected. 2026-02-12T12:32:04.625 INFO:teuthology.orchestra.run.trial063.stderr:HTTP request sent, awaiting response... 302 Found 2026-02-12T12:32:04.625 INFO:teuthology.orchestra.run.trial063.stderr:Location: https://pilotfiber.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?viasf=1 [following] 2026-02-12T12:32:04.626 INFO:teuthology.orchestra.run.trial063.stderr:--2026-02-12 12:32:04-- https://pilotfiber.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?viasf=1 2026-02-12T12:32:04.631 INFO:teuthology.orchestra.run.trial063.stderr:Resolving pilotfiber.dl.sourceforge.net (pilotfiber.dl.sourceforge.net)... 216.158.133.250 2026-02-12T12:32:04.634 INFO:teuthology.orchestra.run.trial063.stderr:Connecting to pilotfiber.dl.sourceforge.net (pilotfiber.dl.sourceforge.net)|216.158.133.250|:443... connected. 2026-02-12T12:32:04.718 INFO:teuthology.orchestra.run.trial063.stderr:HTTP request sent, awaiting response... 200 OK 2026-02-12T12:32:04.718 INFO:teuthology.orchestra.run.trial063.stderr:Length: 620693 (606K) [application/x-gzip] 2026-02-12T12:32:04.718 INFO:teuthology.orchestra.run.trial063.stderr:Saving to: ‘collectl.tar.gz’ 2026-02-12T12:32:04.718 INFO:teuthology.orchestra.run.trial063.stderr: 2026-02-12T12:32:04.721 INFO:teuthology.orchestra.run.trial063.stderr: 0K .......... .......... .......... .......... .......... 8% 14.9M 0s 2026-02-12T12:32:04.731 INFO:teuthology.orchestra.run.trial063.stderr: 50K .......... .......... .......... .......... .......... 16% 4.91M 0s 2026-02-12T12:32:04.734 INFO:teuthology.orchestra.run.trial063.stderr: 100K .......... .......... .......... .......... .......... 24% 15.3M 0s 2026-02-12T12:32:04.739 INFO:teuthology.orchestra.run.trial063.stderr: 150K .......... .......... .......... .......... .......... 32% 16.2M 0s 2026-02-12T12:32:04.739 INFO:teuthology.orchestra.run.trial063.stderr: 200K .......... .......... .......... .......... .......... 41% 247M 0s 2026-02-12T12:32:04.746 INFO:teuthology.orchestra.run.trial063.stderr: 250K .......... .......... .......... .......... .......... 49% 7.51M 0s 2026-02-12T12:32:04.746 INFO:teuthology.orchestra.run.trial063.stderr: 300K .......... .......... .......... .......... .......... 57% 226M 0s 2026-02-12T12:32:04.746 INFO:teuthology.orchestra.run.trial063.stderr: 350K .......... .......... .......... .......... .......... 65% 285M 0s 2026-02-12T12:32:04.752 INFO:teuthology.orchestra.run.trial063.stderr: 400K .......... .......... .......... .......... .......... 74% 13.8M 0s 2026-02-12T12:32:04.752 INFO:teuthology.orchestra.run.trial063.stderr: 450K .......... .......... .......... .......... .......... 82% 278M 0s 2026-02-12T12:32:04.752 INFO:teuthology.orchestra.run.trial063.stderr: 500K .......... .......... .......... .......... .......... 90% 335M 0s 2026-02-12T12:32:04.752 INFO:teuthology.orchestra.run.trial063.stderr: 550K .......... .......... .......... .......... .......... 98% 213M 0s 2026-02-12T12:32:04.752 INFO:teuthology.orchestra.run.trial063.stderr: 600K ...... 100% 11.4T=0.03s 2026-02-12T12:32:04.752 INFO:teuthology.orchestra.run.trial063.stderr: 2026-02-12T12:32:04.752 INFO:teuthology.orchestra.run.trial063.stderr:2026-02-12 12:32:04 (19.3 MB/s) - ‘collectl.tar.gz’ saved [620693/620693] 2026-02-12T12:32:04.753 INFO:teuthology.orchestra.run.trial063.stderr: 2026-02-12T12:32:04.769 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/ 2026-02-12T12:32:04.770 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/collectl.conf 2026-02-12T12:32:04.770 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/docs/ 2026-02-12T12:32:04.770 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/docs/SlowProc.html 2026-02-12T12:32:04.770 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/docs/BuddyInfo.html 2026-02-12T12:32:04.770 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/docs/WhySummary.html 2026-02-12T12:32:04.770 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/docs/Socket.html 2026-02-12T12:32:04.770 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/docs/colmux.html 2026-02-12T12:32:04.770 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/docs/Tutorial.html 2026-02-12T12:32:04.770 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/docs/Gexpr.html 2026-02-12T12:32:04.770 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/docs/Import.html 2026-02-12T12:32:04.770 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/docs/style.css 2026-02-12T12:32:04.771 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/docs/Ganglia3.jpg 2026-02-12T12:32:04.771 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/docs/Ganglia2.jpg 2026-02-12T12:32:04.771 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/docs/FileNaming.html 2026-02-12T12:32:04.771 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/docs/TheMath.html 2026-02-12T12:32:04.771 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/docs/Environmental.html 2026-02-12T12:32:04.771 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/docs/Process.html 2026-02-12T12:32:04.771 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/docs/OutputFiles.html 2026-02-12T12:32:04.771 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/docs/HiResTime.html 2026-02-12T12:32:04.771 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/docs/OutputFormats.html 2026-02-12T12:32:04.771 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/docs/slub.jpg 2026-02-12T12:32:04.771 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/docs/Lustre.html 2026-02-12T12:32:04.771 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/docs/Documentation.html 2026-02-12T12:32:04.771 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/docs/Inodes.html 2026-02-12T12:32:04.772 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/docs/Graphite.html 2026-02-12T12:32:04.772 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/docs/InputFiles.html 2026-02-12T12:32:04.772 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/docs/Export.html 2026-02-12T12:32:04.772 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/docs/Interrupts.html 2026-02-12T12:32:04.772 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/docs/Startup.html 2026-02-12T12:32:04.772 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/docs/Performance.html 2026-02-12T12:32:04.772 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/docs/Messages.html 2026-02-12T12:32:04.772 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/docs/Logging.html 2026-02-12T12:32:04.772 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/docs/Data-detail.html 2026-02-12T12:32:04.772 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/docs/CPUs.html 2026-02-12T12:32:04.772 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/docs/Matrix.html 2026-02-12T12:32:04.772 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/docs/Data-brief.html 2026-02-12T12:32:04.773 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/docs/SlabInfo.html 2026-02-12T12:32:04.773 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/docs/Misc.html 2026-02-12T12:32:04.773 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/docs/Examples.html 2026-02-12T12:32:04.773 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/docs/OpenStack.html 2026-02-12T12:32:04.773 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/docs/Plotfiles.html 2026-02-12T12:32:04.773 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/docs/ColmuxCPU.jpg 2026-02-12T12:32:04.773 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/docs/Tutorial-Lustre.html 2026-02-12T12:32:04.773 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/docs/Data-verbose.html 2026-02-12T12:32:04.773 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/docs/Data.html 2026-02-12T12:32:04.773 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/docs/Exceptions.html 2026-02-12T12:32:04.773 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/docs/RunningAsAService.html 2026-02-12T12:32:04.773 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/docs/FAQ-collectl.html 2026-02-12T12:32:04.773 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/docs/Ganglia.jpg 2026-02-12T12:32:04.773 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/docs/OperationalModes.html 2026-02-12T12:32:04.774 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/docs/Playback.html 2026-02-12T12:32:04.774 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/docs/Hello.html 2026-02-12T12:32:04.774 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/docs/Sockets.html 2026-02-12T12:32:04.774 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/docs/Network.html 2026-02-12T12:32:04.774 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/docs/Infiniband.html 2026-02-12T12:32:04.774 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/docs/Memory.html 2026-02-12T12:32:04.774 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/docs/ColmuxLustre.jpg 2026-02-12T12:32:04.774 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/docs/Disks.html 2026-02-12T12:32:04.774 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/docs/NfsInfo.html 2026-02-12T12:32:04.774 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/service/ 2026-02-12T12:32:04.774 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/service/collectl.service 2026-02-12T12:32:04.774 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/initd/ 2026-02-12T12:32:04.774 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/initd/collectl-suse 2026-02-12T12:32:04.775 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/initd/collectl-debian 2026-02-12T12:32:04.775 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/initd/collectl-generic 2026-02-12T12:32:04.775 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/initd/collectl 2026-02-12T12:32:04.775 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/misc.ph 2026-02-12T12:32:04.775 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/gexpr.ph 2026-02-12T12:32:04.775 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/statsd.ph 2026-02-12T12:32:04.775 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/envrules.std 2026-02-12T12:32:04.775 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/hello.ph 2026-02-12T12:32:04.775 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/vmsum.ph 2026-02-12T12:32:04.775 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/proctree.ph 2026-02-12T12:32:04.775 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/README-WINDOWS 2026-02-12T12:32:04.775 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/formatit.ph 2026-02-12T12:32:04.775 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/colmux 2026-02-12T12:32:04.776 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/INSTALL 2026-02-12T12:32:04.776 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/graphite.ph 2026-02-12T12:32:04.776 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/RELEASE-collectl 2026-02-12T12:32:04.776 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/vmstat.ph 2026-02-12T12:32:04.776 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/README 2026-02-12T12:32:04.776 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/man1/ 2026-02-12T12:32:04.776 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/man1/collectl.1 2026-02-12T12:32:04.776 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/man1/colmux.1 2026-02-12T12:32:04.776 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/lexpr.ph 2026-02-12T12:32:04.776 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/client.pl 2026-02-12T12:32:04.776 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/collectl 2026-02-12T12:32:04.776 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/COPYING 2026-02-12T12:32:04.776 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/vnet.ph 2026-02-12T12:32:04.777 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/GPL 2026-02-12T12:32:04.777 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/UNINSTALL 2026-02-12T12:32:04.777 INFO:teuthology.orchestra.run.trial063.stdout:collectl-4.3.1/ARTISTIC 2026-02-12T12:32:04.813 INFO:teuthology.orchestra.run.trial063.stderr:./INSTALL: line 90: chkconfig: command not found 2026-02-12T12:32:04.814 DEBUG:teuthology.orchestra.run.trial063:> sudo yum -y install librbd-devel pdsh pdsh-rcmd-ssh perf 2026-02-12T12:32:05.161 INFO:teuthology.orchestra.run.trial063.stdout:Last metadata expiration check: 0:00:56 ago on Thu 12 Feb 2026 12:31:09 PM UTC. 2026-02-12T12:32:05.248 INFO:teuthology.orchestra.run.trial063.stdout:Dependencies resolved. 2026-02-12T12:32:05.248 INFO:teuthology.orchestra.run.trial063.stdout:================================================================================ 2026-02-12T12:32:05.248 INFO:teuthology.orchestra.run.trial063.stdout: Package Arch Version Repository Size 2026-02-12T12:32:05.248 INFO:teuthology.orchestra.run.trial063.stdout:================================================================================ 2026-02-12T12:32:05.248 INFO:teuthology.orchestra.run.trial063.stdout:Installing: 2026-02-12T12:32:05.248 INFO:teuthology.orchestra.run.trial063.stdout: librbd-devel x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 24 k 2026-02-12T12:32:05.249 INFO:teuthology.orchestra.run.trial063.stdout: pdsh x86_64 2.34-7.el9 epel 118 k 2026-02-12T12:32:05.249 INFO:teuthology.orchestra.run.trial063.stdout: pdsh-rcmd-ssh x86_64 2.34-7.el9 epel 13 k 2026-02-12T12:32:05.249 INFO:teuthology.orchestra.run.trial063.stdout: perf x86_64 5.14.0-677.el9 appstream 3.2 M 2026-02-12T12:32:05.249 INFO:teuthology.orchestra.run.trial063.stdout:Installing dependencies: 2026-02-12T12:32:05.249 INFO:teuthology.orchestra.run.trial063.stdout: bzip2 x86_64 1.0.8-10.el9 baseos 56 k 2026-02-12T12:32:05.249 INFO:teuthology.orchestra.run.trial063.stdout: libradospp-devel x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 31 k 2026-02-12T12:32:05.249 INFO:teuthology.orchestra.run.trial063.stdout: libtraceevent x86_64 1.8.4-2.el9 baseos 290 k 2026-02-12T12:32:05.249 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:32:05.249 INFO:teuthology.orchestra.run.trial063.stdout:Transaction Summary 2026-02-12T12:32:05.249 INFO:teuthology.orchestra.run.trial063.stdout:================================================================================ 2026-02-12T12:32:05.249 INFO:teuthology.orchestra.run.trial063.stdout:Install 7 Packages 2026-02-12T12:32:05.249 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:32:05.250 INFO:teuthology.orchestra.run.trial063.stdout:Total download size: 3.7 M 2026-02-12T12:32:05.250 INFO:teuthology.orchestra.run.trial063.stdout:Installed size: 14 M 2026-02-12T12:32:05.250 INFO:teuthology.orchestra.run.trial063.stdout:Downloading Packages: 2026-02-12T12:32:05.584 INFO:teuthology.orchestra.run.trial063.stdout:(1/7): libradospp-devel-20.3.0-5145.g5fdb7d8f.e 448 kB/s | 31 kB 00:00 2026-02-12T12:32:05.586 INFO:teuthology.orchestra.run.trial063.stdout:(2/7): librbd-devel-20.3.0-5145.g5fdb7d8f.el9.x 332 kB/s | 24 kB 00:00 2026-02-12T12:32:05.617 INFO:teuthology.orchestra.run.trial063.stdout:(3/7): bzip2-1.0.8-10.el9.x86_64.rpm 541 kB/s | 56 kB 00:00 2026-02-12T12:32:05.664 INFO:teuthology.orchestra.run.trial063.stdout:(4/7): pdsh-2.34-7.el9.x86_64.rpm 2.5 MB/s | 118 kB 00:00 2026-02-12T12:32:05.676 INFO:teuthology.orchestra.run.trial063.stdout:(5/7): pdsh-rcmd-ssh-2.34-7.el9.x86_64.rpm 1.1 MB/s | 13 kB 00:00 2026-02-12T12:32:05.741 INFO:teuthology.orchestra.run.trial063.stdout:(6/7): libtraceevent-1.8.4-2.el9.x86_64.rpm 1.8 MB/s | 290 kB 00:00 2026-02-12T12:32:09.683 INFO:teuthology.orchestra.run.trial063.stdout:(7/7): perf-5.14.0-677.el9.x86_64.rpm 789 kB/s | 3.2 MB 00:04 2026-02-12T12:32:09.685 INFO:teuthology.orchestra.run.trial063.stdout:-------------------------------------------------------------------------------- 2026-02-12T12:32:09.685 INFO:teuthology.orchestra.run.trial063.stdout:Total 848 kB/s | 3.7 MB 00:04 2026-02-12T12:32:09.716 INFO:teuthology.orchestra.run.trial063.stdout:Running transaction check 2026-02-12T12:32:09.726 INFO:teuthology.orchestra.run.trial063.stdout:Transaction check succeeded. 2026-02-12T12:32:09.726 INFO:teuthology.orchestra.run.trial063.stdout:Running transaction test 2026-02-12T12:32:09.785 INFO:teuthology.orchestra.run.trial063.stdout:Transaction test succeeded. 2026-02-12T12:32:09.785 INFO:teuthology.orchestra.run.trial063.stdout:Running transaction 2026-02-12T12:32:09.860 INFO:teuthology.orchestra.run.trial063.stdout: Preparing : 1/1 2026-02-12T12:32:09.866 INFO:teuthology.orchestra.run.trial063.stdout: Installing : pdsh-rcmd-ssh-2.34-7.el9.x86_64 1/7 2026-02-12T12:32:09.907 INFO:teuthology.orchestra.run.trial063.stdout: Installing : pdsh-2.34-7.el9.x86_64 2/7 2026-02-12T12:32:09.913 INFO:teuthology.orchestra.run.trial063.stdout: Installing : libtraceevent-1.8.4-2.el9.x86_64 3/7 2026-02-12T12:32:09.917 INFO:teuthology.orchestra.run.trial063.stdout: Installing : bzip2-1.0.8-10.el9.x86_64 4/7 2026-02-12T12:32:09.919 INFO:teuthology.orchestra.run.trial063.stdout: Installing : libradospp-devel-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 5/7 2026-02-12T12:32:10.080 INFO:teuthology.orchestra.run.trial063.stdout: Installing : librbd-devel-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 6/7 2026-02-12T12:32:10.101 INFO:teuthology.orchestra.run.trial063.stdout: Installing : perf-5.14.0-677.el9.x86_64 7/7 2026-02-12T12:32:10.218 INFO:teuthology.orchestra.run.trial063.stdout: Running scriptlet: perf-5.14.0-677.el9.x86_64 7/7 2026-02-12T12:32:10.218 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : libradospp-devel-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 1/7 2026-02-12T12:32:10.218 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : librbd-devel-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2/7 2026-02-12T12:32:10.218 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : bzip2-1.0.8-10.el9.x86_64 3/7 2026-02-12T12:32:10.218 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : libtraceevent-1.8.4-2.el9.x86_64 4/7 2026-02-12T12:32:10.218 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : perf-5.14.0-677.el9.x86_64 5/7 2026-02-12T12:32:10.218 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : pdsh-2.34-7.el9.x86_64 6/7 2026-02-12T12:32:10.278 INFO:teuthology.orchestra.run.trial063.stdout: Verifying : pdsh-rcmd-ssh-2.34-7.el9.x86_64 7/7 2026-02-12T12:32:10.278 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:32:10.279 INFO:teuthology.orchestra.run.trial063.stdout:Installed: 2026-02-12T12:32:10.279 INFO:teuthology.orchestra.run.trial063.stdout: bzip2-1.0.8-10.el9.x86_64 2026-02-12T12:32:10.279 INFO:teuthology.orchestra.run.trial063.stdout: libradospp-devel-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:32:10.279 INFO:teuthology.orchestra.run.trial063.stdout: librbd-devel-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:32:10.279 INFO:teuthology.orchestra.run.trial063.stdout: libtraceevent-1.8.4-2.el9.x86_64 2026-02-12T12:32:10.279 INFO:teuthology.orchestra.run.trial063.stdout: pdsh-2.34-7.el9.x86_64 2026-02-12T12:32:10.279 INFO:teuthology.orchestra.run.trial063.stdout: pdsh-rcmd-ssh-2.34-7.el9.x86_64 2026-02-12T12:32:10.279 INFO:teuthology.orchestra.run.trial063.stdout: perf-5.14.0-677.el9.x86_64 2026-02-12T12:32:10.279 INFO:teuthology.orchestra.run.trial063.stdout: 2026-02-12T12:32:10.279 INFO:teuthology.orchestra.run.trial063.stdout:Complete! 2026-02-12T12:32:10.310 DEBUG:teuthology.orchestra.run.trial063:> python3 -m pip install -r /home/ubuntu/cephtest/cbt/requirements.txt 2026-02-12T12:32:10.452 INFO:teuthology.orchestra.run.trial063.stdout:Defaulting to user installation because normal site-packages is not writeable 2026-02-12T12:32:10.464 INFO:teuthology.orchestra.run.trial063.stdout:Requirement already satisfied: pyyaml in /usr/lib64/python3.9/site-packages (from -r /home/ubuntu/cephtest/cbt/requirements.txt (line 1)) (5.4.1) 2026-02-12T12:32:10.464 INFO:teuthology.orchestra.run.trial063.stdout:Requirement already satisfied: lxml in /usr/lib64/python3.9/site-packages (from -r /home/ubuntu/cephtest/cbt/requirements.txt (line 2)) (4.6.5) 2026-02-12T12:32:10.786 INFO:teuthology.orchestra.run.trial063.stdout:Collecting matplotlib 2026-02-12T12:32:10.805 INFO:teuthology.orchestra.run.trial063.stdout: Downloading matplotlib-3.9.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (8.3 MB) 2026-02-12T12:32:10.868 INFO:teuthology.orchestra.run.trial063.stdout:Requirement already satisfied: python-dateutil>=2.7 in /usr/lib/python3.9/site-packages (from matplotlib->-r /home/ubuntu/cephtest/cbt/requirements.txt (line 3)) (2.9.0.post0) 2026-02-12T12:32:11.245 INFO:teuthology.orchestra.run.trial063.stdout:Collecting pillow>=8 2026-02-12T12:32:11.249 INFO:teuthology.orchestra.run.trial063.stdout: Downloading pillow-11.3.0-cp39-cp39-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (6.6 MB) 2026-02-12T12:32:11.289 INFO:teuthology.orchestra.run.trial063.stdout:Collecting cycler>=0.10 2026-02-12T12:32:11.292 INFO:teuthology.orchestra.run.trial063.stdout: Downloading cycler-0.12.1-py3-none-any.whl (8.3 kB) 2026-02-12T12:32:11.293 INFO:teuthology.orchestra.run.trial063.stdout:Requirement already satisfied: packaging>=20.0 in /usr/lib/python3.9/site-packages (from matplotlib->-r /home/ubuntu/cephtest/cbt/requirements.txt (line 3)) (20.9) 2026-02-12T12:32:11.326 INFO:teuthology.orchestra.run.trial063.stdout:Collecting importlib-resources>=3.2.0 2026-02-12T12:32:11.330 INFO:teuthology.orchestra.run.trial063.stdout: Downloading importlib_resources-6.5.2-py3-none-any.whl (37 kB) 2026-02-12T12:32:11.457 INFO:teuthology.orchestra.run.trial063.stdout:Collecting kiwisolver>=1.3.1 2026-02-12T12:32:11.463 INFO:teuthology.orchestra.run.trial063.stdout: Downloading kiwisolver-1.4.7-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.6 MB) 2026-02-12T12:32:11.471 INFO:teuthology.orchestra.run.trial063.stdout:Requirement already satisfied: numpy>=1.23 in /usr/lib64/python3.9/site-packages (from matplotlib->-r /home/ubuntu/cephtest/cbt/requirements.txt (line 3)) (1.23.5) 2026-02-12T12:32:11.472 INFO:teuthology.orchestra.run.trial063.stdout:Requirement already satisfied: pyparsing>=2.3.1 in /usr/lib/python3.9/site-packages (from matplotlib->-r /home/ubuntu/cephtest/cbt/requirements.txt (line 3)) (2.4.7) 2026-02-12T12:32:11.607 INFO:teuthology.orchestra.run.trial063.stdout:Collecting contourpy>=1.0.1 2026-02-12T12:32:11.611 INFO:teuthology.orchestra.run.trial063.stdout: Downloading contourpy-1.3.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (321 kB) 2026-02-12T12:32:11.944 INFO:teuthology.orchestra.run.trial063.stdout:Collecting fonttools>=4.22.0 2026-02-12T12:32:11.949 INFO:teuthology.orchestra.run.trial063.stdout: Downloading fonttools-4.60.2-cp39-cp39-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (4.8 MB) 2026-02-12T12:32:12.026 INFO:teuthology.orchestra.run.trial063.stdout:Collecting zipp>=3.1.0 2026-02-12T12:32:12.030 INFO:teuthology.orchestra.run.trial063.stdout: Downloading zipp-3.23.0-py3-none-any.whl (10 kB) 2026-02-12T12:32:12.041 INFO:teuthology.orchestra.run.trial063.stdout:Requirement already satisfied: six>=1.5 in /usr/lib/python3.9/site-packages (from python-dateutil>=2.7->matplotlib->-r /home/ubuntu/cephtest/cbt/requirements.txt (line 3)) (1.15.0) 2026-02-12T12:32:12.172 INFO:teuthology.orchestra.run.trial063.stdout:Installing collected packages: zipp, pillow, kiwisolver, importlib-resources, fonttools, cycler, contourpy, matplotlib 2026-02-12T12:32:13.220 INFO:teuthology.orchestra.run.trial063.stdout:Successfully installed contourpy-1.3.0 cycler-0.12.1 fonttools-4.60.2 importlib-resources-6.5.2 kiwisolver-1.4.7 matplotlib-3.9.4 pillow-11.3.0 zipp-3.23.0 2026-02-12T12:32:13.331 INFO:tasks.cbt:benchmark: radosbench 2026-02-12T12:32:13.332 DEBUG:teuthology.orchestra.run.trial063:> sudo /sbin/sysctl -q -w kernel.perf_event_paranoid=0 2026-02-12T12:32:13.358 DEBUG:teuthology.orchestra.run.trial063:> /home/ubuntu/cephtest/cbt/cbt.py -a /home/ubuntu/cephtest/archive/cbt /home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2026-02-12T12:32:13.470 INFO:teuthology.orchestra.run.trial063.stderr:12:32:13 - DEBUG - cbt - Settings.cluster: 2026-02-12T12:32:13.470 INFO:teuthology.orchestra.run.trial063.stderr: {'archive_dir': '/home/ubuntu/cephtest/archive/cbt', 2026-02-12T12:32:13.470 INFO:teuthology.orchestra.run.trial063.stderr: 'clients': ['ubuntu@trial063.front.sepia.ceph.com'], 2026-02-12T12:32:13.470 INFO:teuthology.orchestra.run.trial063.stderr: 'conf_file': '/etc/ceph/ceph.conf', 2026-02-12T12:32:13.470 INFO:teuthology.orchestra.run.trial063.stderr: 'head': 'ubuntu@trial063.front.sepia.ceph.com', 2026-02-12T12:32:13.471 INFO:teuthology.orchestra.run.trial063.stderr: 'is_teuthology': True, 2026-02-12T12:32:13.471 INFO:teuthology.orchestra.run.trial063.stderr: 'iterations': 1, 2026-02-12T12:32:13.471 INFO:teuthology.orchestra.run.trial063.stderr: 'mons': ['ubuntu@trial063.front.sepia.ceph.com'], 2026-02-12T12:32:13.471 INFO:teuthology.orchestra.run.trial063.stderr: 'osds': ['ubuntu@trial063.front.sepia.ceph.com'], 2026-02-12T12:32:13.471 INFO:teuthology.orchestra.run.trial063.stderr: 'osds_per_node': 3, 2026-02-12T12:32:13.471 INFO:teuthology.orchestra.run.trial063.stderr: 'pid_dir': '/var/run/ceph', 2026-02-12T12:32:13.471 INFO:teuthology.orchestra.run.trial063.stderr: 'pool_profiles': {'replicated': {'pg_size': 256, 2026-02-12T12:32:13.471 INFO:teuthology.orchestra.run.trial063.stderr: 'pgp_size': 256, 2026-02-12T12:32:13.471 INFO:teuthology.orchestra.run.trial063.stderr: 'replication': 'replicated'}}, 2026-02-12T12:32:13.471 INFO:teuthology.orchestra.run.trial063.stderr: 'rebuild_every_test': False, 2026-02-12T12:32:13.471 INFO:teuthology.orchestra.run.trial063.stderr: 'rgws': {'ubuntu@trial063.front.sepia.ceph.com': None}, 2026-02-12T12:32:13.471 INFO:teuthology.orchestra.run.trial063.stderr: 'tmp_dir': '/tmp/cbt', 2026-02-12T12:32:13.472 INFO:teuthology.orchestra.run.trial063.stderr: 'use_existing': True, 2026-02-12T12:32:13.472 INFO:teuthology.orchestra.run.trial063.stderr: 'user': 'ubuntu'} 2026-02-12T12:32:13.472 INFO:teuthology.orchestra.run.trial063.stderr:12:32:13 - INFO - cbt - Results dir: /home/ubuntu/cephtest/archive/cbt/results/00000000/id-8b8185cb 2026-02-12T12:32:13.472 INFO:teuthology.orchestra.run.trial063.stderr:12:32:13 - INFO - cbt - Pausing for 60s for idle monitoring. 2026-02-12T12:32:13.472 INFO:teuthology.orchestra.run.trial063.stderr:12:32:13 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial063.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/idle_monitoring/collectl 2026-02-12T12:32:13.634 INFO:teuthology.orchestra.run.trial063.stderr:12:32:13 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial063.front.sepia.ceph.com collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/idle_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2026-02-12T12:32:13.636 INFO:teuthology.orchestra.run.trial063.stderr:12:32:13 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial063.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/idle_monitoring/perf 2026-02-12T12:32:13.796 INFO:teuthology.orchestra.run.trial063.stderr:12:32:13 - DEBUG - cbt - PerfMonitoring: remote_node 2026-02-12T12:32:13.796 INFO:teuthology.orchestra.run.trial063.stderr:12:32:13 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial063.front.sepia.ceph.com for pid in `cat /var/run/ceph/crimson-osd.*.pid`; do perf stat -p ${pid} -o /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/idle_monitoring/perf/perf_stat.${pid} & done 2026-02-12T12:33:13.858 INFO:teuthology.orchestra.run.trial063.stderr:12:33:13 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial063.front.sepia.ceph.com pkill -SIGINT -f collectl 2026-02-12T12:33:14.035 INFO:teuthology.orchestra.run.trial063.stderr:12:33:14 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial063.front.sepia.ceph.com sudo pkill -SIGINT -f perf\ 2026-02-12T12:33:14.221 INFO:teuthology.orchestra.run.trial063.stderr:12:33:14 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial063.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/idle_monitoring/perf/perf.data 2026-02-12T12:33:14.223 INFO:teuthology.orchestra.run.trial063.stderr:12:33:14 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial063.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/idle_monitoring/perf/perf_stat.* 2026-02-12T12:33:14.225 INFO:teuthology.orchestra.run.trial063.stderr:12:33:14 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial063.front.sepia.ceph.com sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/* 2026-02-12T12:33:14.392 INFO:teuthology.orchestra.run.trial063.stderr:12:33:14 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=rpdcp -f 10 -R ssh -w ubuntu@trial063.front.sepia.ceph.com -r /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/* /home/ubuntu/cephtest/archive/cbt/results/00000000/id-8b8185cb 2026-02-12T12:33:14.552 INFO:teuthology.orchestra.run.trial063.stderr:12:33:14 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial063.front.sepia.ceph.com sudo killall -9 rados 2026-02-12T12:33:14.730 INFO:teuthology.orchestra.run.trial063.stderr:12:33:14 - INFO - cbt - Results dir: /home/ubuntu/cephtest/archive/cbt/results/00000000/id-8b8185cb 2026-02-12T12:33:14.730 INFO:teuthology.orchestra.run.trial063.stderr:12:33:14 - INFO - cbt - Skipping existing test in /home/ubuntu/cephtest/archive/cbt/results/00000000/id-8b8185cb. 2026-02-12T12:33:14.730 INFO:teuthology.orchestra.run.trial063.stderr:12:33:14 - INFO - cbt - Running benchmark /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004 2026-02-12T12:33:14.730 INFO:teuthology.orchestra.run.trial063.stderr:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-8b8185cb 2026-02-12T12:33:14.730 INFO:teuthology.orchestra.run.trial063.stderr:{'concurrent_ops': 4, 'concurrent_procs': 2, 'pool_profile': 'replicated', 'time': 60, 'write_only': True, 'op_size': 4096, 'pool_monitoring_list': 'collectl', 'run_monitoring_list': 'collectl', 'benchmark': 'radosbench', 'iteration': 0} == iteration 0 == 2026-02-12T12:33:14.730 INFO:teuthology.orchestra.run.trial063.stderr:12:33:14 - INFO - cbt - Setting OSD Read Ahead to: 0 2026-02-12T12:33:14.731 INFO:teuthology.orchestra.run.trial063.stderr:12:33:14 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial063.front.sepia.ceph.com find /dev/disk/by-partlabel/osd-device-*data -exec readlink {} \; | cut -d"/" -f 3 | sed "s/[0-9]$//" | xargs -I{} sudo sh -c "echo 0 > /sys/block/'{}'/queue/read_ahead_kb" 2026-02-12T12:33:14.731 INFO:teuthology.orchestra.run.trial063.stderr:12:33:14 - DEBUG - cbt - Cleaning existing temporary run directory: /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004 2026-02-12T12:33:14.732 INFO:teuthology.orchestra.run.trial063.stderr:12:33:14 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial063.front.sepia.ceph.com sudo rm -rf /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004 2026-02-12T12:33:14.900 INFO:teuthology.orchestra.run.trial063.stderr:12:33:14 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial063.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/pool_monitoring/collectl 2026-02-12T12:33:15.057 INFO:teuthology.orchestra.run.trial063.stderr:12:33:15 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial063.front.sepia.ceph.com collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/pool_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2026-02-12T12:33:15.059 INFO:teuthology.orchestra.run.trial063.stderr:12:33:15 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial063.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/pool_monitoring/perf 2026-02-12T12:33:15.217 INFO:teuthology.orchestra.run.trial063.stderr:12:33:15 - DEBUG - cbt - PerfMonitoring: remote_node 2026-02-12T12:33:15.217 INFO:teuthology.orchestra.run.trial063.stderr:12:33:15 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial063.front.sepia.ceph.com for pid in `cat /var/run/ceph/crimson-osd.*.pid`; do perf stat -p ${pid} -o /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/pool_monitoring/perf/perf_stat.${pid} & done 2026-02-12T12:33:15.218 INFO:teuthology.orchestra.run.trial063.stderr:12:33:15 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial063.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool delete rados-bench-cbt rados-bench-cbt --yes-i-really-really-mean-it 2026-02-12T12:33:15.561 INFO:teuthology.orchestra.run.trial063.stderr:12:33:15 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial063.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool create rados-bench-cbt 256 256 2026-02-12T12:33:16.541 INFO:teuthology.orchestra.run.trial063.stderr:12:33:16 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial063.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool application enable rados-bench-cbt radosbench 2026-02-12T12:33:17.542 INFO:teuthology.orchestra.run.trial063.stderr:12:33:17 - INFO - cbt - Checking Health after pool creation. 2026-02-12T12:33:22.547 INFO:teuthology.orchestra.run.trial063.stderr:12:33:22 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial063.front.sepia.ceph.com /usr/bin/ceph -c /etc/ceph/ceph.conf health 2026-02-12T12:33:22.899 INFO:teuthology.orchestra.run.trial063.stderr:12:33:22 - INFO - cbt - Final Pool Health Check. 2026-02-12T12:33:27.904 INFO:teuthology.orchestra.run.trial063.stderr:12:33:27 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial063.front.sepia.ceph.com /usr/bin/ceph -c /etc/ceph/ceph.conf health 2026-02-12T12:33:28.255 INFO:teuthology.orchestra.run.trial063.stderr:12:33:28 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial063.front.sepia.ceph.com pkill -SIGINT -f collectl 2026-02-12T12:33:28.425 INFO:teuthology.orchestra.run.trial063.stderr:12:33:28 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial063.front.sepia.ceph.com sudo pkill -SIGINT -f perf\ 2026-02-12T12:33:28.606 INFO:teuthology.orchestra.run.trial063.stderr:12:33:28 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial063.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/pool_monitoring/perf/perf.data 2026-02-12T12:33:28.607 INFO:teuthology.orchestra.run.trial063.stderr:12:33:28 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial063.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/pool_monitoring/perf/perf_stat.* 2026-02-12T12:33:28.609 INFO:teuthology.orchestra.run.trial063.stderr:12:33:28 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial063.front.sepia.ceph.com sync 2026-02-12T12:33:28.826 INFO:teuthology.orchestra.run.trial063.stderr:12:33:28 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial063.front.sepia.ceph.com echo 3 | sudo tee /proc/sys/vm/drop_caches 2026-02-12T12:33:29.811 INFO:teuthology.orchestra.run.trial063.stderr:12:33:29 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial063.front.sepia.ceph.com /usr/bin/rados -c /etc/ceph/ceph.conf -v 2026-02-12T12:33:30.022 INFO:teuthology.orchestra.run.trial063.stderr:12:33:30 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial063.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/write 2026-02-12T12:33:30.178 INFO:teuthology.orchestra.run.trial063.stderr:12:33:30 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial063.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf daemon osd.0 config show > /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/write/ceph_settings.out 2026-02-12T12:33:30.461 INFO:teuthology.orchestra.run.trial063.stderr:12:33:30 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial063.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/write/collectl 2026-02-12T12:33:30.625 INFO:teuthology.orchestra.run.trial063.stderr:12:33:30 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial063.front.sepia.ceph.com collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/write/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2026-02-12T12:33:30.626 INFO:teuthology.orchestra.run.trial063.stderr:12:33:30 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial063.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/write/perf 2026-02-12T12:33:30.789 INFO:teuthology.orchestra.run.trial063.stderr:12:33:30 - DEBUG - cbt - PerfMonitoring: remote_node 2026-02-12T12:33:30.789 INFO:teuthology.orchestra.run.trial063.stderr:12:33:30 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial063.front.sepia.ceph.com for pid in `cat /var/run/ceph/crimson-osd.*.pid`; do perf stat -p ${pid} -o /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/write/perf/perf_stat.${pid} & done 2026-02-12T12:33:30.790 INFO:teuthology.orchestra.run.trial063.stderr:12:33:30 - INFO - cbt - Running radosbench write test. 2026-02-12T12:33:30.791 INFO:teuthology.orchestra.run.trial063.stderr:12:33:30 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial063.front.sepia.ceph.com /usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench -b 4096 60 write --concurrent-ios 4 --run-name `hostname -f`-0 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/write/objecter.0.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/write/output.0 2026-02-12T12:33:30.792 INFO:teuthology.orchestra.run.trial063.stderr:12:33:30 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial063.front.sepia.ceph.com /usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench -b 4096 60 write --concurrent-ios 4 --run-name `hostname -f`-1 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/write/objecter.1.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/write/output.1 2026-02-12T12:34:32.012 INFO:teuthology.orchestra.run.trial063.stderr:12:34:32 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial063.front.sepia.ceph.com pkill -SIGINT -f collectl 2026-02-12T12:34:32.198 INFO:teuthology.orchestra.run.trial063.stderr:12:34:32 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial063.front.sepia.ceph.com sudo pkill -SIGINT -f perf\ 2026-02-12T12:34:32.371 INFO:teuthology.orchestra.run.trial063.stderr:12:34:32 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial063.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/write/perf/perf.data 2026-02-12T12:34:32.373 INFO:teuthology.orchestra.run.trial063.stderr:12:34:32 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial063.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/write/perf/perf_stat.* 2026-02-12T12:34:32.374 INFO:teuthology.orchestra.run.trial063.stderr:12:34:32 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial063.front.sepia.ceph.com find /var/run/ceph/ceph-osd*.asok -maxdepth 1 -exec sudo /usr/bin/ceph --admin-daemon {} dump_historic_ops \; > /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/write/historic_ops.out 2026-02-12T12:34:32.731 INFO:teuthology.orchestra.run.trial063.stderr:12:34:32 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial063.front.sepia.ceph.com sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/write/* 2026-02-12T12:34:32.898 INFO:teuthology.orchestra.run.trial063.stderr:12:34:32 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=rpdcp -f 10 -R ssh -w ubuntu@trial063.front.sepia.ceph.com -r /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/write/* /home/ubuntu/cephtest/archive/cbt/results/00000000/id-8b8185cb/write 2026-02-12T12:34:33.062 INFO:teuthology.orchestra.run.trial063.stderr:12:34:33 - INFO - cbt - Convert results to json format. 2026-02-12T12:34:33.073 DEBUG:teuthology.run_tasks:Unwinding manager cbt 2026-02-12T12:34:33.088 DEBUG:teuthology.orchestra.run.trial063:> rm --one-file-system -rf -- /home/ubuntu/cephtest/cbt 2026-02-12T12:34:33.116 INFO:tasks.cbt_performance:Collecting CBT performance data config 2026-02-12T12:34:33.116 INFO:tasks.cbt_performance:reading cbt preferences 2026-02-12T12:34:33.116 DEBUG:teuthology.orchestra.run.trial063:> find /home/ubuntu/cephtest -name 'json_output.*' 2026-02-12T12:34:33.174 INFO:teuthology.orchestra.run.trial063.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-8b8185cb/write/json_output.0.trial063.front.sepia.ceph.com 2026-02-12T12:34:33.174 INFO:teuthology.orchestra.run.trial063.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-8b8185cb/write/json_output.1.trial063.front.sepia.ceph.com 2026-02-12T12:34:33.175 DEBUG:teuthology.orchestra.run.trial063:> set -ex 2026-02-12T12:34:33.175 DEBUG:teuthology.orchestra.run.trial063:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-8b8185cb/write/json_output.0.trial063.front.sepia.ceph.com of=/dev/stdout 2026-02-12T12:34:33.231 INFO:tasks.cbt_performance:reading total cpu cycles 2026-02-12T12:34:33.232 DEBUG:teuthology.orchestra.run.trial063:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-8b8185cb/write -name 'perf_stat.*' 2026-02-12T12:34:33.287 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2026-02-12T12:34:33.287 INFO:tasks.cbt_performance:total cpu cycles: 0 2026-02-12T12:34:33.288 DEBUG:teuthology.orchestra.run.trial063:> set -ex 2026-02-12T12:34:33.288 DEBUG:teuthology.orchestra.run.trial063:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-8b8185cb/write/json_output.1.trial063.front.sepia.ceph.com of=/dev/stdout 2026-02-12T12:34:33.343 INFO:tasks.cbt_performance:reading total cpu cycles 2026-02-12T12:34:33.344 DEBUG:teuthology.orchestra.run.trial063:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-8b8185cb/write -name 'perf_stat.*' 2026-02-12T12:34:33.399 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2026-02-12T12:34:33.399 INFO:tasks.cbt_performance:total cpu cycles: 0 2026-02-12T12:34:33.403 WARNING:tasks.cbt_performance:Could not send performance data to http://mira118.front.sepia.ceph.com:4000/cbt_performance: HTTPConnectionPool(host='mira118.front.sepia.ceph.com', port=4000): Max retries exceeded with url: /cbt_performance (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -2] Name or service not known')) 2026-02-12T12:34:33.404 WARNING:tasks.cbt_performance:Continuing without performance data collection 2026-02-12T12:34:33.404 DEBUG:teuthology.run_tasks:Unwinding manager ssh_keys 2026-02-12T12:34:33.422 INFO:teuthology.task.ssh_keys:Cleaning up SSH keys 2026-02-12T12:34:33.422 INFO:teuthology.task.ssh_keys:cleaning up keys added for testing 2026-02-12T12:34:33.422 INFO:teuthology.task.ssh_keys: cleaning up keys for user ubuntu on trial063.front.sepia.ceph.com 2026-02-12T12:34:33.422 DEBUG:teuthology.orchestra.run.trial063:> rm -- /home/ubuntu/.ssh/id_rsa 2026-02-12T12:34:33.455 DEBUG:teuthology.orchestra.run.trial063:> rm -- /home/ubuntu/.ssh/id_rsa.pub 2026-02-12T12:34:33.511 DEBUG:teuthology.orchestra.run.trial063:> stat -c '"%a"' /home/ubuntu/.ssh/authorized_keys 2026-02-12T12:34:33.567 INFO:teuthology.orchestra.run.trial063.stdout:"600" 2026-02-12T12:34:33.568 DEBUG:teuthology.orchestra.run.trial063:> mv -- /home/ubuntu/.ssh/authorized_keys_2026-02-12_12:32:03:517732 /home/ubuntu/.ssh/authorized_keys 2026-02-12T12:34:33.624 DEBUG:teuthology.orchestra.run.trial063:> chmod 600 /home/ubuntu/.ssh/authorized_keys 2026-02-12T12:34:33.681 DEBUG:teuthology.orchestra.run.trial063:> rm /home/ubuntu/.ssh/config 2026-02-12T12:34:33.738 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2026-02-12T12:34:33.753 DEBUG:teuthology.orchestra.run.trial063:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2026-02-12T12:34:33.852 INFO:teuthology.orchestra.run.trial063.stderr:2026-02-12T12:34:33.846+0000 7f927ae51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:34:33.853 INFO:teuthology.orchestra.run.trial063.stderr:2026-02-12T12:34:33.851+0000 7f927ae51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:34:33.975 INFO:teuthology.misc:Shutting down mds daemons... 2026-02-12T12:34:33.976 INFO:teuthology.misc:Shutting down osd daemons... 2026-02-12T12:34:33.976 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2026-02-12T12:34:33.976 INFO:teuthology.orchestra.run:waiting for 300 2026-02-12T12:34:34.056 INFO:tasks.ceph.osd.0:Stopped 2026-02-12T12:34:34.057 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2026-02-12T12:34:34.057 INFO:teuthology.orchestra.run:waiting for 300 2026-02-12T12:34:34.143 INFO:tasks.ceph.osd.1:Stopped 2026-02-12T12:34:34.143 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2026-02-12T12:34:34.144 INFO:teuthology.orchestra.run:waiting for 300 2026-02-12T12:34:34.240 INFO:tasks.ceph.osd.2:Stopped 2026-02-12T12:34:34.240 INFO:teuthology.misc:Shutting down mgr daemons... 2026-02-12T12:34:34.240 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2026-02-12T12:34:34.240 INFO:teuthology.orchestra.run:waiting for 300 2026-02-12T12:34:34.277 INFO:tasks.ceph.mgr.x:Stopped 2026-02-12T12:34:34.277 INFO:teuthology.misc:Shutting down mon daemons... 2026-02-12T12:34:34.277 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2026-02-12T12:34:34.277 INFO:teuthology.orchestra.run:waiting for 300 2026-02-12T12:34:34.327 INFO:tasks.ceph.mon.a:Stopped 2026-02-12T12:34:34.327 INFO:tasks.ceph:Checking cluster log for badness... 2026-02-12T12:34:34.327 DEBUG:teuthology.orchestra.run.trial063:> sudo grep -E '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | grep -E -v '\(MDS_ALL_DOWN\)' | grep -E -v '\(MDS_UP_LESS_THAN_MAX\)' | grep -E -v '\(POOL_APP_NOT_ENABLED\)' | grep -E -v '\(PG_' | grep -E -v '\(OSD_' | grep -E -v '\(OBJECT_' | grep -E -v 'overall HEALTH' | head -n 1 2026-02-12T12:34:34.393 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@trial063.front.sepia.ceph.com 2026-02-12T12:34:34.393 DEBUG:teuthology.orchestra.run.trial063:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2026-02-12T12:34:34.469 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@trial063.front.sepia.ceph.com 2026-02-12T12:34:34.470 DEBUG:teuthology.orchestra.run.trial063:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2026-02-12T12:34:34.538 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@trial063.front.sepia.ceph.com 2026-02-12T12:34:34.539 DEBUG:teuthology.orchestra.run.trial063:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2026-02-12T12:34:34.612 INFO:tasks.ceph:Archiving mon data... 2026-02-12T12:34:34.614 DEBUG:teuthology.misc:Transferring archived files from trial063:/var/lib/ceph/mon/ceph-a to /home/teuthworker/mnt/teuthology/moagrawa-2026-02-10_06:35:43-crimson-rados-main-distro-debug-trial/42515/data/mon.a.tgz 2026-02-12T12:34:34.614 DEBUG:teuthology.orchestra.run.trial063:> mktemp 2026-02-12T12:34:34.669 INFO:teuthology.orchestra.run.trial063.stdout:/tmp/tmp.wxvGkhCESn 2026-02-12T12:34:34.669 DEBUG:teuthology.orchestra.run.trial063:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.wxvGkhCESn 2026-02-12T12:34:34.758 DEBUG:teuthology.orchestra.run.trial063:> sudo chmod 0666 /tmp/tmp.wxvGkhCESn 2026-02-12T12:34:34.840 DEBUG:teuthology.orchestra.remote:trial063:/tmp/tmp.wxvGkhCESn is 195KB 2026-02-12T12:34:34.894 DEBUG:teuthology.orchestra.run.trial063:> rm -fr /tmp/tmp.wxvGkhCESn 2026-02-12T12:34:34.910 INFO:tasks.ceph:Cleaning ceph cluster... 2026-02-12T12:34:34.911 DEBUG:teuthology.orchestra.run.trial063:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2026-02-12T12:34:35.031 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2026-02-12T12:34:35.031 INFO:tasks.ceph:Archiving crash dumps... 2026-02-12T12:34:35.034 DEBUG:teuthology.misc:Transferring archived files from trial063:/var/lib/ceph/crash to /home/teuthworker/mnt/teuthology/moagrawa-2026-02-10_06:35:43-crimson-rados-main-distro-debug-trial/42515/remote/trial063/crash 2026-02-12T12:34:35.035 DEBUG:teuthology.orchestra.run.trial063:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2026-02-12T12:34:35.095 INFO:tasks.ceph:Compressing logs... 2026-02-12T12:34:35.095 DEBUG:teuthology.orchestra.run.trial063:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2026-02-12T12:34:35.159 INFO:teuthology.orchestra.run.trial063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2026-02-12T12:34:35.161 INFO:teuthology.orchestra.run.trial063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22222.log 2026-02-12T12:34:35.161 INFO:teuthology.orchestra.run.trial063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2026-02-12T12:34:35.162 INFO:teuthology.orchestra.run.trial063.stderr:/var/log/ceph/ceph-osd.2.log: /var/log/ceph/ceph-client.admin.22222.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2026-02-12T12:34:35.162 INFO:teuthology.orchestra.run.trial063.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.22222.log.gz 2026-02-12T12:34:35.162 INFO:teuthology.orchestra.run.trial063.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.20161.log 2026-02-12T12:34:35.162 INFO:teuthology.orchestra.run.trial063.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21595.log 2026-02-12T12:34:35.162 INFO:teuthology.orchestra.run.trial063.stderr:/var/log/ceph/ceph.tmp-client.admin.20161.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25883.log 2026-02-12T12:34:35.162 INFO:teuthology.orchestra.run.trial063.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2026-02-12T12:34:35.162 INFO:teuthology.orchestra.run.trial063.stderr: 67.7% -- replaced with /var/log/ceph/ceph.tmp-client.admin.20161.log.gz 2026-02-12T12:34:35.162 INFO:teuthology.orchestra.run.trial063.stderr:/var/log/ceph/ceph-client.admin.21595.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22385.log 2026-02-12T12:34:35.162 INFO:teuthology.orchestra.run.trial063.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.21595.log.gz 2026-02-12T12:34:35.162 INFO:teuthology.orchestra.run.trial063.stderr:/var/log/ceph/ceph-client.admin.25883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21826.log 2026-02-12T12:34:35.162 INFO:teuthology.orchestra.run.trial063.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.25883.log.gz 2026-02-12T12:34:35.163 INFO:teuthology.orchestra.run.trial063.stderr:/var/log/ceph/ceph-client.admin.22385.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22290.log 2026-02-12T12:34:35.163 INFO:teuthology.orchestra.run.trial063.stderr: 66.9% -- replaced with /var/log/ceph/ceph-client.admin.22385.log.gz 2026-02-12T12:34:35.172 INFO:teuthology.orchestra.run.trial063.stderr:/var/log/ceph/ceph-client.admin.21826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22512.log 2026-02-12T12:34:35.172 INFO:teuthology.orchestra.run.trial063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22045.log/var/log/ceph/ceph-client.admin.22290.log: 2026-02-12T12:34:35.172 INFO:teuthology.orchestra.run.trial063.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.21826.log.gz 2026-02-12T12:34:35.172 INFO:teuthology.orchestra.run.trial063.stderr:/var/log/ceph/ceph-client.admin.22512.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.22290.log.gz 2026-02-12T12:34:35.172 INFO:teuthology.orchestra.run.trial063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22860.log 2026-02-12T12:34:35.172 INFO:teuthology.orchestra.run.trial063.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.22512.log.gz 2026-02-12T12:34:35.172 INFO:teuthology.orchestra.run.trial063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22859.log 2026-02-12T12:34:35.173 INFO:teuthology.orchestra.run.trial063.stderr:/var/log/ceph/ceph-client.admin.22045.log: /var/log/ceph/ceph-client.admin.22860.log: 50.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.22045.log.gz -5 2026-02-12T12:34:35.173 INFO:teuthology.orchestra.run.trial063.stderr: --verbose -- /var/log/ceph/ceph-client.admin.22610.log 2026-02-12T12:34:35.173 INFO:teuthology.orchestra.run.trial063.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.22860.log.gz 2026-02-12T12:34:35.173 INFO:teuthology.orchestra.run.trial063.stderr:/var/log/ceph/ceph-client.admin.22859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21732.log 2026-02-12T12:34:35.173 INFO:teuthology.orchestra.run.trial063.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.22859.log.gz 2026-02-12T12:34:35.173 INFO:teuthology.orchestra.run.trial063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22360.log 2026-02-12T12:34:35.173 INFO:teuthology.orchestra.run.trial063.stderr:/var/log/ceph/ceph-client.admin.22610.log: /var/log/ceph/ceph-client.admin.21732.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21453.log 2026-02-12T12:34:35.173 INFO:teuthology.orchestra.run.trial063.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.22610.log.gz 2026-02-12T12:34:35.173 INFO:teuthology.orchestra.run.trial063.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.21732.log.gz 2026-02-12T12:34:35.173 INFO:teuthology.orchestra.run.trial063.stderr:/var/log/ceph/ceph-client.admin.22360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23051.log 2026-02-12T12:34:35.173 INFO:teuthology.orchestra.run.trial063.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.22360.log.gz 2026-02-12T12:34:35.173 INFO:teuthology.orchestra.run.trial063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25794.log 2026-02-12T12:34:35.174 INFO:teuthology.orchestra.run.trial063.stderr:/var/log/ceph/ceph-client.admin.21453.log: /var/log/ceph/ceph-client.admin.23051.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.21453.log.gz 2026-02-12T12:34:35.174 INFO:teuthology.orchestra.run.trial063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22463.log 2026-02-12T12:34:35.174 INFO:teuthology.orchestra.run.trial063.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.23051.log.gz 2026-02-12T12:34:35.174 INFO:teuthology.orchestra.run.trial063.stderr:/var/log/ceph/ceph-client.admin.25794.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26433.log 2026-02-12T12:34:35.174 INFO:teuthology.orchestra.run.trial063.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.25794.log.gz 2026-02-12T12:34:35.174 INFO:teuthology.orchestra.run.trial063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21779.log 2026-02-12T12:34:35.174 INFO:teuthology.orchestra.run.trial063.stderr:/var/log/ceph/ceph-client.admin.22463.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.22463.log.gz 2026-02-12T12:34:35.174 INFO:teuthology.orchestra.run.trial063.stderr:/var/log/ceph/ceph-client.admin.26433.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23002.log 2026-02-12T12:34:35.174 INFO:teuthology.orchestra.run.trial063.stderr: 66.1% -- replaced with /var/log/ceph/ceph-client.admin.26433.log.gz 2026-02-12T12:34:35.174 INFO:teuthology.orchestra.run.trial063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22953.log 2026-02-12T12:34:35.174 INFO:teuthology.orchestra.run.trial063.stderr:/var/log/ceph/ceph-client.admin.21779.log: /var/log/ceph/ceph-client.admin.23002.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.21779.log.gz 2026-02-12T12:34:35.174 INFO:teuthology.orchestra.run.trial063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2026-02-12T12:34:35.174 INFO:teuthology.orchestra.run.trial063.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.23002.log.gz 2026-02-12T12:34:35.175 INFO:teuthology.orchestra.run.trial063.stderr:/var/log/ceph/ceph-client.admin.22953.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25735.log 2026-02-12T12:34:35.175 INFO:teuthology.orchestra.run.trial063.stderr:/var/log/ceph/ceph.audit.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.22953.log.gz 2026-02-12T12:34:35.175 INFO:teuthology.orchestra.run.trial063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2026-02-12T12:34:35.175 INFO:teuthology.orchestra.run.trial063.stderr:/var/log/ceph/ceph-client.admin.25735.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25679.log 2026-02-12T12:34:35.175 INFO:teuthology.orchestra.run.trial063.stderr: 87.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2026-02-12T12:34:35.175 INFO:teuthology.orchestra.run.trial063.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.25735.log.gz 2026-02-12T12:34:35.175 INFO:teuthology.orchestra.run.trial063.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25829.log 2026-02-12T12:34:35.175 INFO:teuthology.orchestra.run.trial063.stderr:/var/log/ceph/ceph-client.admin.25679.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27051.log 2026-02-12T12:34:35.175 INFO:teuthology.orchestra.run.trial063.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.25679.log.gz 2026-02-12T12:34:35.175 INFO:teuthology.orchestra.run.trial063.stderr:/var/log/ceph/ceph-client.admin.25829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21548.log 2026-02-12T12:34:35.175 INFO:teuthology.orchestra.run.trial063.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.25829.log.gz 2026-02-12T12:34:35.175 INFO:teuthology.orchestra.run.trial063.stderr:/var/log/ceph/ceph-client.admin.27051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22712.log 2026-02-12T12:34:35.175 INFO:teuthology.orchestra.run.trial063.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.27051.log.gz 2026-02-12T12:34:35.176 INFO:teuthology.orchestra.run.trial063.stderr:/var/log/ceph/ceph-client.admin.21548.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22075.log 2026-02-12T12:34:35.176 INFO:teuthology.orchestra.run.trial063.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.21548.log.gz 2026-02-12T12:34:35.176 INFO:teuthology.orchestra.run.trial063.stderr:/var/log/ceph/ceph-client.admin.22712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26437.log 2026-02-12T12:34:35.176 INFO:teuthology.orchestra.run.trial063.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.22712.log.gz 2026-02-12T12:34:35.176 INFO:teuthology.orchestra.run.trial063.stderr:/var/log/ceph/ceph-client.admin.22075.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2026-02-12T12:34:35.176 INFO:teuthology.orchestra.run.trial063.stderr:/var/log/ceph/ceph-client.admin.26437.log: gzip 66.7% -- replaced with /var/log/ceph/ceph-client.admin.26437.log.gz -5 2026-02-12T12:34:35.176 INFO:teuthology.orchestra.run.trial063.stderr: --verbose -- /var/log/ceph/ceph-client.admin.22713.log 2026-02-12T12:34:35.176 INFO:teuthology.orchestra.run.trial063.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.22075.log.gz 2026-02-12T12:34:35.176 INFO:teuthology.orchestra.run.trial063.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2026-02-12T12:34:35.176 INFO:teuthology.orchestra.run.trial063.stderr:/var/log/ceph/ceph-client.admin.22713.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22711.log 2026-02-12T12:34:35.176 INFO:teuthology.orchestra.run.trial063.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.22713.log.gz 2026-02-12T12:34:35.176 INFO:teuthology.orchestra.run.trial063.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22858.log 2026-02-12T12:34:35.176 INFO:teuthology.orchestra.run.trial063.stderr: 84.9% -- replaced with /var/log/ceph/ceph.log.gz 2026-02-12T12:34:35.177 INFO:teuthology.orchestra.run.trial063.stderr:/var/log/ceph/ceph-client.admin.22711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22561.log 2026-02-12T12:34:35.177 INFO:teuthology.orchestra.run.trial063.stderr:/var/log/ceph/ceph-client.admin.22858.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.22711.log.gz 2026-02-12T12:34:35.177 INFO:teuthology.orchestra.run.trial063.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.22858.log.gz 2026-02-12T12:34:35.177 INFO:teuthology.orchestra.run.trial063.stderr:/var/log/ceph/ceph-client.admin.22561.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.22561.log.gz 2026-02-12T12:34:35.177 INFO:teuthology.orchestra.run.trial063.stderr: 92.0% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2026-02-12T12:34:39.068 INFO:teuthology.orchestra.run.trial063.stderr: 91.9% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2026-02-12T12:34:39.215 INFO:teuthology.orchestra.run.trial063.stderr: 91.9% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2026-02-12T12:34:39.404 INFO:teuthology.orchestra.run.trial063.stderr: 91.9% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2026-02-12T12:34:39.406 INFO:teuthology.orchestra.run.trial063.stderr: 2026-02-12T12:34:39.406 INFO:teuthology.orchestra.run.trial063.stderr:real 0m4.255s 2026-02-12T12:34:39.406 INFO:teuthology.orchestra.run.trial063.stderr:user 0m11.633s 2026-02-12T12:34:39.406 INFO:teuthology.orchestra.run.trial063.stderr:sys 0m0.594s 2026-02-12T12:34:39.406 INFO:tasks.ceph:Archiving logs... 2026-02-12T12:34:39.407 DEBUG:teuthology.misc:Transferring archived files from trial063:/var/log/ceph to /home/teuthworker/mnt/teuthology/moagrawa-2026-02-10_06:35:43-crimson-rados-main-distro-debug-trial/42515/remote/trial063/log 2026-02-12T12:34:39.408 DEBUG:teuthology.orchestra.run.trial063:> sudo tar c -f - -C /var/log/ceph -- . 2026-02-12T12:34:40.509 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-12T12:34:40.573 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-12T12:34:40.583 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2026-02-12T12:34:40.584 DEBUG:teuthology.orchestra.run.trial063:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-12T12:34:40.609 DEBUG:teuthology.run_tasks:Unwinding manager clock 2026-02-12T12:34:40.620 INFO:teuthology.task.clock:Checking final clock skew... 2026-02-12T12:34:40.620 DEBUG:teuthology.orchestra.run.trial063:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-12T12:34:40.664 INFO:teuthology.orchestra.run.trial063.stderr:bash: line 1: ntpq: command not found 2026-02-12T12:34:40.682 INFO:teuthology.orchestra.run.trial063.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-12T12:34:40.682 INFO:teuthology.orchestra.run.trial063.stdout:=============================================================================== 2026-02-12T12:34:40.682 INFO:teuthology.orchestra.run.trial063.stdout:^- ntp11.kernfusion.at 2 6 377 120 -2916us[-2872us] +/- 75ms 2026-02-12T12:34:40.682 INFO:teuthology.orchestra.run.trial063.stdout:^* 216-144-228-179-host.col> 2 6 377 57 +401us[ +453us] +/- 11ms 2026-02-12T12:34:40.682 INFO:teuthology.orchestra.run.trial063.stdout:^- 172-104-28-175.ip.linode> 2 6 377 56 -5796us[-5796us] +/- 57ms 2026-02-12T12:34:40.682 INFO:teuthology.orchestra.run.trial063.stdout:^+ 96-19-94-82.cpe.sparklig> 1 6 377 55 -301us[ -301us] +/- 32ms 2026-02-12T12:34:40.684 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2026-02-12T12:34:40.697 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2026-02-12T12:34:40.697 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2026-02-12T12:34:40.710 DEBUG:teuthology.orchestra.run.trial063:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2026-02-12T12:34:40.770 DEBUG:teuthology.orchestra.run.trial063:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-12T12:34:40.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-12T12:34:40.796 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2026-02-12T12:34:40.812 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2026-02-12T12:34:40.828 INFO:teuthology.task.internal:Duration was 581.616450 seconds 2026-02-12T12:34:40.828 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2026-02-12T12:34:40.845 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2026-02-12T12:34:40.845 DEBUG:teuthology.orchestra.run.trial063:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-12T12:34:40.882 INFO:teuthology.orchestra.run.trial063.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-12T12:34:41.195 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2026-02-12T12:34:41.195 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial063.front.sepia.ceph.com 2026-02-12T12:34:41.196 DEBUG:teuthology.orchestra.run.trial063:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-12T12:34:41.254 INFO:teuthology.task.internal.syslog:Gathering journactl... 2026-02-12T12:34:41.254 DEBUG:teuthology.orchestra.run.trial063:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-12T12:34:41.340 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2026-02-12T12:34:41.340 DEBUG:teuthology.orchestra.run.trial063:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-12T12:34:41.362 INFO:teuthology.orchestra.run.trial063.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-12T12:34:41.364 INFO:teuthology.orchestra.run.trial063.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-12T12:34:41.364 INFO:teuthology.orchestra.run.trial063.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-12T12:34:41.364 INFO:teuthology.orchestra.run.trial063.stderr: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-02-12T12:34:41.364 INFO:teuthology.orchestra.run.trial063.stderr:/home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-12T12:34:41.367 INFO:teuthology.orchestra.run.trial063.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: 88.1% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-12T12:34:41.369 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2026-02-12T12:34:41.386 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2026-02-12T12:34:41.386 DEBUG:teuthology.orchestra.run.trial063:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-12T12:34:41.433 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2026-02-12T12:34:41.451 DEBUG:teuthology.orchestra.run.trial063:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-12T12:34:41.498 INFO:teuthology.orchestra.run.trial063.stdout:kernel.core_pattern = core 2026-02-12T12:34:41.511 DEBUG:teuthology.orchestra.run.trial063:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-12T12:34:41.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-12T12:34:41.567 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2026-02-12T12:34:41.584 INFO:teuthology.task.internal:Transferring archived files... 2026-02-12T12:34:41.585 DEBUG:teuthology.misc:Transferring archived files from trial063:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/moagrawa-2026-02-10_06:35:43-crimson-rados-main-distro-debug-trial/42515/remote/trial063 2026-02-12T12:34:41.586 DEBUG:teuthology.orchestra.run.trial063:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-12T12:34:41.667 INFO:teuthology.task.internal:Removing archive directory... 2026-02-12T12:34:41.668 DEBUG:teuthology.orchestra.run.trial063:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-12T12:34:41.698 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2026-02-12T12:34:41.717 INFO:teuthology.task.internal:Not uploading archives. 2026-02-12T12:34:41.717 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2026-02-12T12:34:41.734 INFO:teuthology.task.internal:Tidying up after the test... 2026-02-12T12:34:41.734 DEBUG:teuthology.orchestra.run.trial063:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-12T12:34:41.756 INFO:teuthology.orchestra.run.trial063.stdout: 786525 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 12 12:34 /home/ubuntu/cephtest 2026-02-12T12:34:41.757 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2026-02-12T12:34:41.775 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2026-02-12T12:34:41.814 INFO:teuthology.run:Summary data: description: crimson-rados/perf/{0-crimson_install clusters/{crimson-fixed-1} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/rbm$/{crimson_seastore_rbm} perf_ignorelist settings/optimized workloads/sample_radosbench} duration: 581.6164498329163 flavor: debug owner: scheduled_moagrawa@soko04.front.sepia.ceph.com success: true 2026-02-12T12:34:41.814 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-12T12:34:41.930 INFO:teuthology.run:pass