2026-02-12T12:02:58.248 INFO:root:teuthology version: 0.0.1.dev29+gc433f1062 2026-02-12T12:02:58.266 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-12T12:02:58.346 INFO:teuthology.run:Config: archive_path: /home/teuthworker/mnt/teuthology/moagrawa-2026-02-10_06:35:43-crimson-rados-main-distro-debug-trial/42503 branch: main description: crimson-rados/perf/{0-crimson_install clusters/{crimson-fixed-1} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/rbm$/{crimson_seastore_rbm} perf_ignorelist settings/optimized workloads/fio_4M_rand_rw} email: null first_in_suite: false flavor: debug job_id: '42503' kernel: branch: distro kdb: 1 sha1: distro last_in_suite: false machine_type: trial name: moagrawa-2026-02-10_06:35:43-crimson-rados-main-distro-debug-trial no_nested_subset: false os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: global: auth client required: none auth cluster required: none auth service required: none auth supported: none debug asok: 0/0 debug auth: 0/0 debug bdev: 0/0 debug bluefs: 0/0 debug bluestore: 0/0 debug buffer: 0/0 debug civetweb: 0/0 debug client: 0/0 debug compressor: 0/0 debug context: 0/0 debug crush: 0/0 debug crypto: 0/0 debug dpdk: 0/0 debug eventtrace: 0/0 debug filer: 0/0 debug filestore: 0/0 debug finisher: 0/0 debug fuse: 0/0 debug heartbeatmap: 0/0 debug javaclient: 0/0 debug journal: 0/0 debug journaler: 0/0 debug kstore: 0/0 debug leveldb: 0/0 debug lockdep: 0/0 debug mds: 0/0 debug mds balancer: 0/0 debug mds locker: 0/0 debug mds log: 0/0 debug mds log expire: 0/0 debug mds migrator: 0/0 debug memdb: 0/0 debug mgr: 0/0 debug mgrc: 0/0 debug mon: 0/0 debug monc: 0/0 debug ms: 0/0 debug objclass: 0/0 debug objectcacher: 0/0 debug objecter: 0/0 debug optracker: 0/0 debug osd: 0/0 debug paxos: 0/0 debug perfcounter: 0/0 debug rados: 0/0 debug rbd: 0/0 debug rbd mirror: 0/0 debug rbd replay: 0/0 debug refs: 0/0 debug rgw: 0/0 debug rgw lifecycle: 0/0 debug rgw notification: 0/0 debug rgw sync: 0/0 debug rocksdb: 0/0 debug striper: 0/0 debug throttle: 0/0 debug timer: 0/0 debug tp: 0/0 enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 0/0 debug ms: 0/0 debug paxos: 0/0 osd pool default crimson: true osd: crimson cpu num: 4 crimson osd obc lru size: 10 debug filestore: 0/0 debug journal: 0/0 debug ms: 0/0 debug osd: 0/0 debug seastore: 20 debug seastore backref: 20 debug seastore cache: 20 debug seastore cleaner: 20 debug seastore device: 20 debug seastore epm: 20 debug seastore fixedkv tree: 20 debug seastore journal: 20 debug seastore lba: 20 debug seastore odata: 20 debug seastore omap: 20 debug seastore onode: 20 debug seastore t: 20 debug seastore tm: 20 osd objectstore: seastore osd shutdown pgref assert: true seastore cachepin size pershard: 64M seastore main device type: RANDOM_BLOCK_SSD seastore max concurrent transactions: 8 crimson_compat: true flavor: debug log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) - \(PG_ - \(OSD_ - \(OBJECT_ - overall HEALTH sha1: 5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log global: osd crush chooseleaf type: 0 osd pool default pg num: 128 osd pool default pgp num: 128 osd pool default size: 2 mon: {} install: ceph: flavor: debug sha1: 5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0 workunit: branch: seastore_io_uring env: CRIMSON_COMPAT: '1' sha1: 5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0 owner: scheduled_moagrawa@soko04.front.sepia.ceph.com priority: 1000 repo: https://github.com/ceph/ceph.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 2425 sha1: 5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0 sleep_before_teardown: 0 suite: crimson-rados suite_branch: seastore_io_uring suite_path: /home/teuthworker/src/github.com_mohit84_ceph_5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0/qa suite_relpath: qa suite_repo: https://github.com/mohit84/ceph.git suite_sha1: 5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0 targets: trial127.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGdObeFTFkrlBKhCkAGcyYnDwtG7jTAb01C177Aek7c7uMRXKelJQnjrbSpKUeabk2CzMIsbCcy8Niz0Wz8/ERI= tasks: - install: extra_packages: deb: - ceph-osd-crimson - ceph-osd-crimson-dbg rpm: - ceph-osd-crimson - ceph-osd-crimson-debuginfo - pexec: all: - sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson - install: null - ceph: conf: osd: debug monc: 20 wait-for-scrub: false - ssh_keys: null - cbt: benchmarks: librbdfio: iodepth: - 32 log_avg_msec: 100 mode: - randrw norandommap: true op_size: - 4194304 osd_ra: - 4096 pool_profile: rbd procs_per_volume: - 1 time: 60 vol_size: 4096 volumes_per_client: - 2 cluster: iterations: 1 osds_per_node: 3 pool_profiles: rbd: pg_size: 128 pgp_size: 128 replication: 3 user: ubuntu monitoring_profiles: perf: args: stat -p {pid} -o {perf_dir}/perf_stat.{pid} nodes: - osds perf_cmd: perf pid_glob: crimson-osd.*.pid teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: c433f1062990a0488dc29a553589bc609a460691 timestamp: 2026-02-10_06:35:43 tube: trial user: moagrawa verbose: false worker_log: /home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.2161924 2026-02-12T12:02:58.346 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_mohit84_ceph_5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0/qa; will attempt to use it 2026-02-12T12:02:58.347 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_mohit84_ceph_5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0/qa/tasks 2026-02-12T12:02:58.347 INFO:teuthology.run_tasks:Running task internal.check_packages... 2026-02-12T12:02:58.349 INFO:teuthology.task.internal:Checking packages... 2026-02-12T12:02:58.349 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'debug' and ceph hash '5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0' 2026-02-12T12:02:58.349 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2026-02-12T12:02:58.349 INFO:teuthology.packaging:ref: None 2026-02-12T12:02:58.349 INFO:teuthology.packaging:tag: None 2026-02-12T12:02:58.349 INFO:teuthology.packaging:branch: main 2026-02-12T12:02:58.349 INFO:teuthology.packaging:sha1: 5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0 2026-02-12T12:02:58.349 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&ref=main 2026-02-12T12:02:58.496 INFO:teuthology.task.internal:Found packages for ceph version 20.3.0-5273.g2bf58db4 2026-02-12T12:02:58.497 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2026-02-12T12:02:58.510 INFO:teuthology.task.internal:no buildpackages task found 2026-02-12T12:02:58.510 INFO:teuthology.run_tasks:Running task internal.save_config... 2026-02-12T12:02:58.523 INFO:teuthology.task.internal:Saving configuration 2026-02-12T12:02:58.532 INFO:teuthology.run_tasks:Running task internal.check_lock... 2026-02-12T12:02:58.544 INFO:teuthology.task.internal.check_lock:Checking locks... 2026-02-12T12:02:58.685 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial127.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/moagrawa-2026-02-10_06:35:43-crimson-rados-main-distro-debug-trial/42503', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-12 11:59:12.926459', 'locked_by': 'scheduled_moagrawa@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGdObeFTFkrlBKhCkAGcyYnDwtG7jTAb01C177Aek7c7uMRXKelJQnjrbSpKUeabk2CzMIsbCcy8Niz0Wz8/ERI='} 2026-02-12T12:02:58.685 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2026-02-12T12:02:58.702 INFO:teuthology.task.internal:roles: ubuntu@trial127.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2026-02-12T12:02:58.702 INFO:teuthology.run_tasks:Running task console_log... 2026-02-12T12:02:58.807 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x763da0ed9da0>, signals=[15]) 2026-02-12T12:02:58.807 INFO:teuthology.run_tasks:Running task internal.connect... 2026-02-12T12:02:58.822 INFO:teuthology.task.internal:Opening connections... 2026-02-12T12:02:58.822 DEBUG:teuthology.task.internal:connecting to ubuntu@trial127.front.sepia.ceph.com 2026-02-12T12:02:58.823 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial127.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-12T12:02:58.887 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2026-02-12T12:02:58.899 DEBUG:teuthology.orchestra.run.trial127:> uname -m 2026-02-12T12:02:58.958 INFO:teuthology.orchestra.run.trial127.stdout:x86_64 2026-02-12T12:02:58.959 DEBUG:teuthology.orchestra.run.trial127:> cat /etc/os-release 2026-02-12T12:02:59.014 INFO:teuthology.orchestra.run.trial127.stdout:NAME="CentOS Stream" 2026-02-12T12:02:59.014 INFO:teuthology.orchestra.run.trial127.stdout:VERSION="9" 2026-02-12T12:02:59.014 INFO:teuthology.orchestra.run.trial127.stdout:ID="centos" 2026-02-12T12:02:59.014 INFO:teuthology.orchestra.run.trial127.stdout:ID_LIKE="rhel fedora" 2026-02-12T12:02:59.014 INFO:teuthology.orchestra.run.trial127.stdout:VERSION_ID="9" 2026-02-12T12:02:59.014 INFO:teuthology.orchestra.run.trial127.stdout:PLATFORM_ID="platform:el9" 2026-02-12T12:02:59.014 INFO:teuthology.orchestra.run.trial127.stdout:PRETTY_NAME="CentOS Stream 9" 2026-02-12T12:02:59.014 INFO:teuthology.orchestra.run.trial127.stdout:ANSI_COLOR="0;31" 2026-02-12T12:02:59.015 INFO:teuthology.orchestra.run.trial127.stdout:LOGO="fedora-logo-icon" 2026-02-12T12:02:59.015 INFO:teuthology.orchestra.run.trial127.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2026-02-12T12:02:59.015 INFO:teuthology.orchestra.run.trial127.stdout:HOME_URL="https://centos.org/" 2026-02-12T12:02:59.015 INFO:teuthology.orchestra.run.trial127.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2026-02-12T12:02:59.015 INFO:teuthology.orchestra.run.trial127.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2026-02-12T12:02:59.015 INFO:teuthology.orchestra.run.trial127.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2026-02-12T12:02:59.016 INFO:teuthology.lock.ops:Updating trial127.front.sepia.ceph.com on lock server 2026-02-12T12:02:59.102 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2026-02-12T12:02:59.114 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2026-02-12T12:02:59.125 INFO:teuthology.task.internal:Checking for old test directory... 2026-02-12T12:02:59.125 DEBUG:teuthology.orchestra.run.trial127:> test '!' -e /home/ubuntu/cephtest 2026-02-12T12:02:59.140 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2026-02-12T12:02:59.151 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2026-02-12T12:02:59.152 DEBUG:teuthology.orchestra.run.trial127:> test -z $(ls -A /var/lib/ceph) 2026-02-12T12:02:59.197 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2026-02-12T12:02:59.302 INFO:teuthology.run_tasks:Running task kernel... 2026-02-12T12:02:59.328 INFO:teuthology.task.kernel:normalize config orig: {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'} 2026-02-12T12:02:59.328 INFO:teuthology.task.kernel:config {'mon.a': {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'}}, timeout 300 2026-02-12T12:02:59.328 DEBUG:teuthology.orchestra.run.trial127:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-12T12:02:59.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-12T12:02:59.344 DEBUG:teuthology.orchestra.run.trial127:> uname -r 2026-02-12T12:02:59.399 INFO:teuthology.orchestra.run.trial127.stdout:5.14.0-645.el9.x86_64 2026-02-12T12:02:59.399 INFO:teuthology.task.kernel:Running kernel on trial127: 5.14.0-645.el9.x86_64 2026-02-12T12:02:59.399 DEBUG:teuthology.orchestra.run.trial127:> sudo yum install -y kernel 2026-02-12T12:03:00.870 INFO:teuthology.orchestra.run.trial127.stdout:CentOS Stream 9 - BaseOS 8.0 MB/s | 8.9 MB 00:01 2026-02-12T12:03:04.986 INFO:teuthology.orchestra.run.trial127.stdout:CentOS Stream 9 - AppStream 7.5 MB/s | 27 MB 00:03 2026-02-12T12:03:15.746 INFO:teuthology.orchestra.run.trial127.stdout:CentOS Stream 9 - CRB 947 kB/s | 7.8 MB 00:08 2026-02-12T12:03:16.629 INFO:teuthology.orchestra.run.trial127.stdout:CentOS Stream 9 - Extras packages 116 kB/s | 20 kB 00:00 2026-02-12T12:03:17.098 INFO:teuthology.orchestra.run.trial127.stdout:Extra Packages for Enterprise Linux 51 MB/s | 20 MB 00:00 2026-02-12T12:03:20.316 INFO:teuthology.orchestra.run.trial127.stdout:lab-extras 2.5 MB/s | 50 kB 00:00 2026-02-12T12:03:21.380 INFO:teuthology.orchestra.run.trial127.stdout:Package kernel-5.14.0-645.el9.x86_64 is already installed. 2026-02-12T12:03:21.399 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-12T12:03:21.401 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-12T12:03:21.401 INFO:teuthology.orchestra.run.trial127.stdout: Package Architecture Version Repository Size 2026-02-12T12:03:21.401 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-12T12:03:21.401 INFO:teuthology.orchestra.run.trial127.stdout:Installing: 2026-02-12T12:03:21.401 INFO:teuthology.orchestra.run.trial127.stdout: kernel x86_64 5.14.0-677.el9 baseos 768 k 2026-02-12T12:03:21.401 INFO:teuthology.orchestra.run.trial127.stdout:Installing dependencies: 2026-02-12T12:03:21.401 INFO:teuthology.orchestra.run.trial127.stdout: kernel-core x86_64 5.14.0-677.el9 baseos 17 M 2026-02-12T12:03:21.401 INFO:teuthology.orchestra.run.trial127.stdout: kernel-modules x86_64 5.14.0-677.el9 baseos 40 M 2026-02-12T12:03:21.402 INFO:teuthology.orchestra.run.trial127.stdout: kernel-modules-core x86_64 5.14.0-677.el9 baseos 31 M 2026-02-12T12:03:21.402 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:03:21.402 INFO:teuthology.orchestra.run.trial127.stdout:Transaction Summary 2026-02-12T12:03:21.402 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-12T12:03:21.402 INFO:teuthology.orchestra.run.trial127.stdout:Install 4 Packages 2026-02-12T12:03:21.402 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:03:21.402 INFO:teuthology.orchestra.run.trial127.stdout:Total download size: 88 M 2026-02-12T12:03:21.402 INFO:teuthology.orchestra.run.trial127.stdout:Installed size: 135 M 2026-02-12T12:03:21.402 INFO:teuthology.orchestra.run.trial127.stdout:Downloading Packages: 2026-02-12T12:03:21.613 INFO:teuthology.orchestra.run.trial127.stdout:(1/4): kernel-5.14.0-677.el9.x86_64.rpm 5.9 MB/s | 768 kB 00:00 2026-02-12T12:03:21.760 INFO:teuthology.orchestra.run.trial127.stdout:(2/4): kernel-core-5.14.0-677.el9.x86_64.rpm 62 MB/s | 17 MB 00:00 2026-02-12T12:03:21.946 INFO:teuthology.orchestra.run.trial127.stdout:(3/4): kernel-modules-5.14.0-677.el9.x86_64.rpm 86 MB/s | 40 MB 00:00 2026-02-12T12:03:22.213 INFO:teuthology.orchestra.run.trial127.stdout:(4/4): kernel-modules-core-5.14.0-677.el9.x86_6 51 MB/s | 31 MB 00:00 2026-02-12T12:03:22.213 INFO:teuthology.orchestra.run.trial127.stdout:-------------------------------------------------------------------------------- 2026-02-12T12:03:22.213 INFO:teuthology.orchestra.run.trial127.stdout:Total 109 MB/s | 88 MB 00:00 2026-02-12T12:03:22.403 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction check 2026-02-12T12:03:22.600 INFO:teuthology.orchestra.run.trial127.stdout:Transaction check succeeded. 2026-02-12T12:03:22.600 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction test 2026-02-12T12:03:22.944 INFO:teuthology.orchestra.run.trial127.stdout:Transaction test succeeded. 2026-02-12T12:03:22.945 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction 2026-02-12T12:03:24.033 INFO:teuthology.orchestra.run.trial127.stdout: Preparing : 1/1 2026-02-12T12:03:24.316 INFO:teuthology.orchestra.run.trial127.stdout: Installing : kernel-modules-core-5.14.0-677.el9.x86_64 1/4 2026-02-12T12:03:24.335 INFO:teuthology.orchestra.run.trial127.stdout: Installing : kernel-core-5.14.0-677.el9.x86_64 2/4 2026-02-12T12:03:26.097 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: kernel-core-5.14.0-677.el9.x86_64 2/4 2026-02-12T12:03:26.126 INFO:teuthology.orchestra.run.trial127.stdout: Installing : kernel-modules-5.14.0-677.el9.x86_64 3/4 2026-02-12T12:03:29.149 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: kernel-modules-5.14.0-677.el9.x86_64 3/4 2026-02-12T12:03:29.155 INFO:teuthology.orchestra.run.trial127.stdout: Installing : kernel-5.14.0-677.el9.x86_64 4/4 2026-02-12T12:03:32.251 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: kernel-modules-core-5.14.0-677.el9.x86_64 4/4 2026-02-12T12:04:02.793 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: kernel-core-5.14.0-677.el9.x86_64 4/4 2026-02-12T12:04:02.814 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: kernel-modules-5.14.0-677.el9.x86_64 4/4 2026-02-12T12:04:02.877 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: kernel-5.14.0-677.el9.x86_64 4/4 2026-02-12T12:04:02.877 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : kernel-5.14.0-677.el9.x86_64 1/4 2026-02-12T12:04:02.877 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : kernel-core-5.14.0-677.el9.x86_64 2/4 2026-02-12T12:04:02.877 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : kernel-modules-5.14.0-677.el9.x86_64 3/4 2026-02-12T12:04:02.930 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : kernel-modules-core-5.14.0-677.el9.x86_64 4/4 2026-02-12T12:04:02.931 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:04:02.931 INFO:teuthology.orchestra.run.trial127.stdout:Installed: 2026-02-12T12:04:02.931 INFO:teuthology.orchestra.run.trial127.stdout: kernel-5.14.0-677.el9.x86_64 2026-02-12T12:04:02.931 INFO:teuthology.orchestra.run.trial127.stdout: kernel-core-5.14.0-677.el9.x86_64 2026-02-12T12:04:02.931 INFO:teuthology.orchestra.run.trial127.stdout: kernel-modules-5.14.0-677.el9.x86_64 2026-02-12T12:04:02.931 INFO:teuthology.orchestra.run.trial127.stdout: kernel-modules-core-5.14.0-677.el9.x86_64 2026-02-12T12:04:02.931 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:04:02.931 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-12T12:04:02.992 DEBUG:teuthology.orchestra.run.trial127:> rpm -q kernel | sort -rV | head -n 1 2026-02-12T12:04:03.017 INFO:teuthology.orchestra.run.trial127.stdout:kernel-5.14.0-677.el9.x86_64 2026-02-12T12:04:03.019 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-677.el9.x86_64 2026-02-12T12:04:03.019 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-645.el9.x86_64 Expected: 5.14.0-677.el9.x86_64 2026-02-12T12:04:03.019 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2026-02-12T12:04:03.019 DEBUG:teuthology.task.kernel:src is distro, skipping download 2026-02-12T12:04:03.019 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2026-02-12T12:04:03.019 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@trial127.front.sepia.ceph.com, path=None, version=distro) 2026-02-12T12:04:03.020 DEBUG:teuthology.orchestra.run.trial127:> rpm -q kernel | sort -rV | head -n 1 2026-02-12T12:04:03.082 INFO:teuthology.orchestra.run.trial127.stdout:kernel-5.14.0-677.el9.x86_64 2026-02-12T12:04:03.083 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-677.el9.x86_64 2026-02-12T12:04:03.083 DEBUG:teuthology.orchestra.run.trial127:> sudo rpm -qi grub2-tools 2026-02-12T12:04:03.151 INFO:teuthology.orchestra.run.trial127.stdout:Name : grub2-tools 2026-02-12T12:04:03.151 INFO:teuthology.orchestra.run.trial127.stdout:Epoch : 1 2026-02-12T12:04:03.151 INFO:teuthology.orchestra.run.trial127.stdout:Version : 2.06 2026-02-12T12:04:03.151 INFO:teuthology.orchestra.run.trial127.stdout:Release : 118.el9 2026-02-12T12:04:03.151 INFO:teuthology.orchestra.run.trial127.stdout:Architecture: x86_64 2026-02-12T12:04:03.151 INFO:teuthology.orchestra.run.trial127.stdout:Install Date: Thu 04 Dec 2025 11:54:21 PM UTC 2026-02-12T12:04:03.151 INFO:teuthology.orchestra.run.trial127.stdout:Group : Unspecified 2026-02-12T12:04:03.151 INFO:teuthology.orchestra.run.trial127.stdout:Size : 8393909 2026-02-12T12:04:03.152 INFO:teuthology.orchestra.run.trial127.stdout:License : GPLv3+ 2026-02-12T12:04:03.152 INFO:teuthology.orchestra.run.trial127.stdout:Signature : RSA/SHA256, Wed 05 Nov 2025 09:13:15 AM UTC, Key ID 05b555b38483c65d 2026-02-12T12:04:03.152 INFO:teuthology.orchestra.run.trial127.stdout:Source RPM : grub2-2.06-118.el9.src.rpm 2026-02-12T12:04:03.152 INFO:teuthology.orchestra.run.trial127.stdout:Build Date : Tue 04 Nov 2025 03:44:14 PM UTC 2026-02-12T12:04:03.152 INFO:teuthology.orchestra.run.trial127.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2026-02-12T12:04:03.152 INFO:teuthology.orchestra.run.trial127.stdout:Packager : builder@centos.org 2026-02-12T12:04:03.152 INFO:teuthology.orchestra.run.trial127.stdout:Vendor : CentOS 2026-02-12T12:04:03.152 INFO:teuthology.orchestra.run.trial127.stdout:URL : http://www.gnu.org/software/grub/ 2026-02-12T12:04:03.152 INFO:teuthology.orchestra.run.trial127.stdout:Summary : Support tools for GRUB. 2026-02-12T12:04:03.152 INFO:teuthology.orchestra.run.trial127.stdout:Description : 2026-02-12T12:04:03.152 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:04:03.153 INFO:teuthology.orchestra.run.trial127.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2026-02-12T12:04:03.153 INFO:teuthology.orchestra.run.trial127.stdout:customizable bootloader with modular architecture. It supports a rich 2026-02-12T12:04:03.153 INFO:teuthology.orchestra.run.trial127.stdout:variety of kernel formats, file systems, computer architectures and 2026-02-12T12:04:03.153 INFO:teuthology.orchestra.run.trial127.stdout:hardware devices. 2026-02-12T12:04:03.153 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:04:03.153 INFO:teuthology.orchestra.run.trial127.stdout:This subpackage provides tools for support of all platforms. 2026-02-12T12:04:03.154 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2026-02-12T12:04:03.154 INFO:teuthology.task.kernel:Updating grub on trial127 to boot 5.14.0-677.el9.x86_64 2026-02-12T12:04:03.154 DEBUG:teuthology.orchestra.run.trial127:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f -print -quit 2026-02-12T12:04:03.218 INFO:teuthology.orchestra.run.trial127.stdout:/boot/loader/entries/2734456005d743dca7f3dc77c35a2d0d-0-rescue.conf 2026-02-12T12:04:03.219 DEBUG:teuthology.orchestra.run.trial127:> sudo command -v grubby && echo yes || echo no 2026-02-12T12:04:03.282 INFO:teuthology.orchestra.run.trial127.stdout:/sbin/grubby 2026-02-12T12:04:03.284 INFO:teuthology.orchestra.run.trial127.stdout:yes 2026-02-12T12:04:03.284 DEBUG:teuthology.orchestra.run.trial127:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f | grep -F "5.14.0-677.el9.x86_64" | head -n 1 2026-02-12T12:04:03.348 INFO:teuthology.orchestra.run.trial127.stdout:/boot/loader/entries/b0373be789df49cdafedf423703d2566-5.14.0-677.el9.x86_64.conf 2026-02-12T12:04:03.349 DEBUG:teuthology.orchestra.run.trial127:> sudo grub2-set-default b0373be789df49cdafedf423703d2566-5.14.0-677.el9.x86_64 2026-02-12T12:04:03.422 DEBUG:teuthology.orchestra.run.trial127:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2026-02-12T12:04:03.633 INFO:teuthology.orchestra.run.trial127.stderr:Generating grub configuration file ... 2026-02-12T12:04:03.697 INFO:teuthology.orchestra.run.trial127.stderr:Found linux image: /boot/vmlinuz-5.14.0-677.el9.x86_64 2026-02-12T12:04:03.703 INFO:teuthology.orchestra.run.trial127.stderr:Found initrd image: /boot/initramfs-5.14.0-677.el9.x86_64.img 2026-02-12T12:04:03.952 INFO:teuthology.orchestra.run.trial127.stderr:Found linux image: /boot/vmlinuz-5.14.0-645.el9.x86_64 2026-02-12T12:04:03.958 INFO:teuthology.orchestra.run.trial127.stderr:Found initrd image: /boot/initramfs-5.14.0-645.el9.x86_64.img 2026-02-12T12:04:03.991 INFO:teuthology.orchestra.run.trial127.stderr:Found linux image: /boot/vmlinuz-0-rescue-b0373be789df49cdafedf423703d2566 2026-02-12T12:04:03.997 INFO:teuthology.orchestra.run.trial127.stderr:Found initrd image: /boot/initramfs-0-rescue-b0373be789df49cdafedf423703d2566.img 2026-02-12T12:04:04.020 INFO:teuthology.orchestra.run.trial127.stderr:Found linux image: /boot/vmlinuz-0-rescue-2734456005d743dca7f3dc77c35a2d0d 2026-02-12T12:04:04.027 INFO:teuthology.orchestra.run.trial127.stderr:Found initrd image: /boot/initramfs-0-rescue-2734456005d743dca7f3dc77c35a2d0d.img 2026-02-12T12:04:04.112 INFO:teuthology.orchestra.run.trial127.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2026-02-12T12:04:04.121 INFO:teuthology.orchestra.run.trial127.stderr:done 2026-02-12T12:04:04.123 DEBUG:teuthology.orchestra.run.trial127:> test -d /sys/firmware/efi 2026-02-12T12:04:04.139 DEBUG:teuthology.orchestra.run.trial127:> sudo find /boot/efi/EFI -mindepth 1 -maxdepth 1 -type d || true 2026-02-12T12:04:04.204 INFO:teuthology.orchestra.run.trial127.stdout:/boot/efi/EFI/BOOT 2026-02-12T12:04:04.204 INFO:teuthology.orchestra.run.trial127.stdout:/boot/efi/EFI/centos 2026-02-12T12:04:04.206 DEBUG:teuthology.orchestra.run.trial127:> sudo test -f /boot/efi/EFI/BOOT/grub.cfg && echo yes || echo no 2026-02-12T12:04:04.271 INFO:teuthology.orchestra.run.trial127.stdout:no 2026-02-12T12:04:04.271 DEBUG:teuthology.orchestra.run.trial127:> sudo test -f /boot/efi/EFI/centos/grub.cfg && echo yes || echo no 2026-02-12T12:04:04.335 INFO:teuthology.orchestra.run.trial127.stdout:yes 2026-02-12T12:04:04.335 DEBUG:teuthology.orchestra.run.trial127:> sudo cp -f /boot/grub2/grub.cfg /boot/efi/EFI/centos/grub.cfg 2026-02-12T12:04:04.399 DEBUG:teuthology.orchestra.run.trial127:> sudo shutdown -r now 2026-02-12T12:04:04.472 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2026-02-12T12:04:34.455 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2026-02-12T12:04:34.455 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial127.front.sepia.ceph.com' 2026-02-12T12:04:34.456 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial127.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-12T12:05:34.509 DEBUG:teuthology.orchestra.remote:timed out 2026-02-12T12:05:43.517 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial127.front.sepia.ceph.com' 2026-02-12T12:05:43.518 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial127.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-12T12:05:43.605 DEBUG:teuthology.orchestra.run.trial127:> true 2026-02-12T12:05:43.758 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial127.front.sepia.ceph.com' 2026-02-12T12:05:43.758 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-677.el9.x86_64"... 2026-02-12T12:05:43.758 DEBUG:teuthology.orchestra.run.trial127:> uname -r 2026-02-12T12:05:43.813 INFO:teuthology.orchestra.run.trial127.stdout:5.14.0-677.el9.x86_64 2026-02-12T12:05:43.814 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-677.el9.x86_64 vs 5.14.0-677.el9.x86_64 2026-02-12T12:05:43.814 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2026-02-12T12:05:43.814 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2026-02-12T12:05:44.815 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2026-02-12T12:05:44.815 DEBUG:teuthology.orchestra.run.trial127:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-12T12:05:44.844 INFO:teuthology.orchestra.run.trial127.stdout:ttyS1 2026-02-12T12:05:44.866 DEBUG:teuthology.parallel:result is None 2026-02-12T12:05:44.866 INFO:teuthology.run_tasks:Running task internal.base... 2026-02-12T12:05:44.877 INFO:teuthology.task.internal:Creating test directory... 2026-02-12T12:05:44.877 DEBUG:teuthology.orchestra.run.trial127:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-12T12:05:44.922 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2026-02-12T12:05:44.934 INFO:teuthology.run_tasks:Running task internal.archive... 2026-02-12T12:05:44.949 INFO:teuthology.task.internal:Creating archive directory... 2026-02-12T12:05:44.949 DEBUG:teuthology.orchestra.run.trial127:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-12T12:05:44.985 INFO:teuthology.run_tasks:Running task internal.coredump... 2026-02-12T12:05:44.997 INFO:teuthology.task.internal:Enabling coredump saving... 2026-02-12T12:05:44.998 DEBUG:teuthology.orchestra.run.trial127:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-12T12:05:45.044 INFO:teuthology.orchestra.run.trial127.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-12T12:05:45.052 INFO:teuthology.orchestra.run.trial127.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-12T12:05:45.054 INFO:teuthology.run_tasks:Running task internal.sudo... 2026-02-12T12:05:45.066 INFO:teuthology.task.internal:Configuring sudo... 2026-02-12T12:05:45.067 DEBUG:teuthology.orchestra.run.trial127:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-12T12:05:45.118 INFO:teuthology.run_tasks:Running task internal.syslog... 2026-02-12T12:05:45.132 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2026-02-12T12:05:45.132 DEBUG:teuthology.orchestra.run.trial127:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-12T12:05:45.174 DEBUG:teuthology.orchestra.run.trial127:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-12T12:05:45.331 DEBUG:teuthology.orchestra.run.trial127:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-12T12:05:45.357 DEBUG:teuthology.orchestra.run.trial127:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-12T12:05:45.475 DEBUG:teuthology.orchestra.run.trial127:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-12T12:05:45.499 DEBUG:teuthology.orchestra.run.trial127:> set -ex 2026-02-12T12:05:45.499 DEBUG:teuthology.orchestra.run.trial127:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-12T12:05:45.562 DEBUG:teuthology.orchestra.run.trial127:> sudo service rsyslog restart 2026-02-12T12:05:45.631 INFO:teuthology.orchestra.run.trial127.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-12T12:05:45.833 INFO:teuthology.run_tasks:Running task internal.timer... 2026-02-12T12:05:45.848 INFO:teuthology.task.internal:Starting timer... 2026-02-12T12:05:45.848 INFO:teuthology.run_tasks:Running task pcp... 2026-02-12T12:05:45.877 INFO:teuthology.run_tasks:Running task selinux... 2026-02-12T12:05:45.896 DEBUG:teuthology.orchestra.run.trial127:> sudo service auditd rotate 2026-02-12T12:05:45.924 INFO:teuthology.orchestra.run.trial127.stdout:Rotating logs: 2026-02-12T12:05:45.925 DEBUG:teuthology.task.selinux:Getting current SELinux state 2026-02-12T12:05:45.925 DEBUG:teuthology.orchestra.run.trial127:> /usr/sbin/getenforce 2026-02-12T12:05:45.982 INFO:teuthology.orchestra.run.trial127.stdout:Permissive 2026-02-12T12:05:45.982 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@trial127.front.sepia.ceph.com': 'permissive'} 2026-02-12T12:05:45.983 DEBUG:teuthology.orchestra.run.trial127:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-12T12:05:46.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-12T12:05:46.046 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2026-02-12T12:05:46.046 DEBUG:teuthology.orchestra.run.trial127:> sudo /usr/sbin/setenforce permissive 2026-02-12T12:05:46.110 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2026-02-12T12:05:46.127 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2026-02-12T12:05:46.137 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2026-02-12T12:05:46.139 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit trial127.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2026-02-12T12:06:52.261 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@trial127.front.sepia.ceph.com')] 2026-02-12T12:06:52.263 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial127.front.sepia.ceph.com' 2026-02-12T12:06:52.266 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial127.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-12T12:06:52.409 DEBUG:teuthology.orchestra.run.trial127:> true 2026-02-12T12:06:52.489 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial127.front.sepia.ceph.com' 2026-02-12T12:06:52.490 INFO:teuthology.run_tasks:Running task clock... 2026-02-12T12:06:52.504 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2026-02-12T12:06:52.504 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-12T12:06:52.505 DEBUG:teuthology.orchestra.run.trial127:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-12T12:06:52.556 INFO:teuthology.orchestra.run.trial127.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2026-02-12T12:06:52.567 INFO:teuthology.orchestra.run.trial127.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2026-02-12T12:06:52.586 INFO:teuthology.orchestra.run.trial127.stderr:sudo: ntpd: command not found 2026-02-12T12:06:52.597 INFO:teuthology.orchestra.run.trial127.stdout:506 Cannot talk to daemon 2026-02-12T12:06:52.608 INFO:teuthology.orchestra.run.trial127.stderr:Failed to start ntp.service: Unit ntp.service not found. 2026-02-12T12:06:52.621 INFO:teuthology.orchestra.run.trial127.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2026-02-12T12:06:52.668 INFO:teuthology.orchestra.run.trial127.stderr:bash: line 1: ntpq: command not found 2026-02-12T12:06:52.670 INFO:teuthology.orchestra.run.trial127.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-12T12:06:52.670 INFO:teuthology.orchestra.run.trial127.stdout:=============================================================================== 2026-02-12T12:06:52.670 INFO:teuthology.orchestra.run.trial127.stdout:^? t1.time.bf1.yahoo.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-12T12:06:52.670 INFO:teuthology.orchestra.run.trial127.stdout:^? dug.arpnet.net 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-12T12:06:52.670 INFO:teuthology.orchestra.run.trial127.stdout:^? ntp.shastacoe.net 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-12T12:06:52.670 INFO:teuthology.orchestra.run.trial127.stdout:^? jane.qotw.net 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-12T12:06:52.671 INFO:teuthology.run_tasks:Running task install... 2026-02-12T12:06:52.685 DEBUG:teuthology.task.install:project ceph 2026-02-12T12:06:52.685 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'debug', 'sha1': '5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0'}} 2026-02-12T12:06:52.685 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'flavor': 'debug', 'sha1': '5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0'} 2026-02-12T12:06:52.685 INFO:teuthology.task.install:Using flavor: debug 2026-02-12T12:06:52.688 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-osd', 'ceph-osd-classic', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-osd', 'ceph-osd-classic', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-12T12:06:52.688 INFO:teuthology.task.install:extra packages: {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']} 2026-02-12T12:06:52.688 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'debug', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0', 'tag': None, 'wait_for_package': False} 2026-02-12T12:06:52.689 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0 2026-02-12T12:06:52.816 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-moagrawa-seastore-io-uring-again-crimson-only/5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0/centos/9/flavors/debug/ 2026-02-12T12:06:52.816 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5145.g5fdb7d8f 2026-02-12T12:06:53.169 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-moagrawa-seastore-io-uring-again-crimson-only/5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0/centos/9/flavors/debug/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-moagrawa-seastore-io-uring-again-crimson-only/5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0/centos/9/flavors/debug/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-moagrawa-seastore-io-uring-again-crimson-only/5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0/centos/9/flavors/debug/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-12T12:06:53.170 DEBUG:teuthology.orchestra.run.trial127:> set -ex 2026-02-12T12:06:53.170 DEBUG:teuthology.orchestra.run.trial127:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-12T12:06:53.194 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-osd-crimson, ceph-osd-crimson-debuginfo on remote rpm x86_64 2026-02-12T12:06:53.194 DEBUG:teuthology.orchestra.run.trial127:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-12T12:06:53.261 DEBUG:teuthology.orchestra.run.trial127:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-12T12:06:53.334 DEBUG:teuthology.orchestra.run.trial127:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-12T12:06:53.398 INFO:teuthology.orchestra.run.trial127.stdout:check_obsoletes = 1 2026-02-12T12:06:53.400 DEBUG:teuthology.orchestra.run.trial127:> sudo yum clean all 2026-02-12T12:06:53.550 INFO:teuthology.orchestra.run.trial127.stdout:41 files removed 2026-02-12T12:06:53.565 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install ceph-radosgw 2026-02-12T12:06:54.348 INFO:teuthology.orchestra.run.trial127.stdout:ceph packages for x86_64 146 kB/s | 94 kB 00:00 2026-02-12T12:06:54.958 INFO:teuthology.orchestra.run.trial127.stdout:ceph noarch packages 42 kB/s | 24 kB 00:00 2026-02-12T12:06:55.488 INFO:teuthology.orchestra.run.trial127.stdout:ceph source packages 4.1 kB/s | 2.1 kB 00:00 2026-02-12T12:06:56.304 INFO:teuthology.orchestra.run.trial127.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.9 MB 00:00 2026-02-12T12:06:59.462 INFO:teuthology.orchestra.run.trial127.stdout:CentOS Stream 9 - AppStream 10 MB/s | 27 MB 00:02 2026-02-12T12:07:02.751 INFO:teuthology.orchestra.run.trial127.stdout:CentOS Stream 9 - CRB 8.7 MB/s | 7.8 MB 00:00 2026-02-12T12:07:07.477 INFO:teuthology.orchestra.run.trial127.stdout:CentOS Stream 9 - Extras packages 5.0 kB/s | 20 kB 00:04 2026-02-12T12:07:08.425 INFO:teuthology.orchestra.run.trial127.stdout:Extra Packages for Enterprise Linux 23 MB/s | 20 MB 00:00 2026-02-12T12:07:11.614 INFO:teuthology.orchestra.run.trial127.stdout:lab-extras 2.6 MB/s | 50 kB 00:00 2026-02-12T12:07:12.901 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-12T12:07:12.902 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-12T12:07:12.902 INFO:teuthology.orchestra.run.trial127.stdout: Package Arch Version Repository Size 2026-02-12T12:07:12.902 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-12T12:07:12.902 INFO:teuthology.orchestra.run.trial127.stdout:Installing: 2026-02-12T12:07:12.902 INFO:teuthology.orchestra.run.trial127.stdout: ceph-radosgw x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 11 M 2026-02-12T12:07:12.902 INFO:teuthology.orchestra.run.trial127.stdout:Upgrading: 2026-02-12T12:07:12.902 INFO:teuthology.orchestra.run.trial127.stdout: librados2 x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 3.3 M 2026-02-12T12:07:12.902 INFO:teuthology.orchestra.run.trial127.stdout: librbd1 x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 2.7 M 2026-02-12T12:07:12.903 INFO:teuthology.orchestra.run.trial127.stdout:Installing dependencies: 2026-02-12T12:07:12.903 INFO:teuthology.orchestra.run.trial127.stdout: boost-program-options x86_64 1.75.0-13.el9 appstream 104 k 2026-02-12T12:07:12.903 INFO:teuthology.orchestra.run.trial127.stdout: ceph-base x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 5.1 M 2026-02-12T12:07:12.903 INFO:teuthology.orchestra.run.trial127.stdout: ceph-common x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 21 M 2026-02-12T12:07:12.903 INFO:teuthology.orchestra.run.trial127.stdout: ceph-selinux x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 25 k 2026-02-12T12:07:12.903 INFO:teuthology.orchestra.run.trial127.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2026-02-12T12:07:12.903 INFO:teuthology.orchestra.run.trial127.stdout: libarrow x86_64 9.0.0-15.el9 epel 4.4 M 2026-02-12T12:07:12.903 INFO:teuthology.orchestra.run.trial127.stdout: libarrow-doc noarch 9.0.0-15.el9 epel 25 k 2026-02-12T12:07:12.903 INFO:teuthology.orchestra.run.trial127.stdout: libcephfs2 x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 860 k 2026-02-12T12:07:12.903 INFO:teuthology.orchestra.run.trial127.stdout: libnbd x86_64 1.20.3-4.el9 appstream 164 k 2026-02-12T12:07:12.903 INFO:teuthology.orchestra.run.trial127.stdout: liboath x86_64 2.6.12-1.el9 epel 49 k 2026-02-12T12:07:12.903 INFO:teuthology.orchestra.run.trial127.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2026-02-12T12:07:12.903 INFO:teuthology.orchestra.run.trial127.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2026-02-12T12:07:12.903 INFO:teuthology.orchestra.run.trial127.stdout: libradosstriper1 x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 263 k 2026-02-12T12:07:12.903 INFO:teuthology.orchestra.run.trial127.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2026-02-12T12:07:12.903 INFO:teuthology.orchestra.run.trial127.stdout: librgw2 x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 5.6 M 2026-02-12T12:07:12.903 INFO:teuthology.orchestra.run.trial127.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2026-02-12T12:07:12.903 INFO:teuthology.orchestra.run.trial127.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2026-02-12T12:07:12.903 INFO:teuthology.orchestra.run.trial127.stdout: mailcap noarch 2.1.49-5.el9 baseos 33 k 2026-02-12T12:07:12.903 INFO:teuthology.orchestra.run.trial127.stdout: parquet-libs x86_64 9.0.0-15.el9 epel 838 k 2026-02-12T12:07:12.903 INFO:teuthology.orchestra.run.trial127.stdout: pciutils x86_64 3.7.0-7.el9 baseos 93 k 2026-02-12T12:07:12.903 INFO:teuthology.orchestra.run.trial127.stdout: python3-ceph-argparse x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 45 k 2026-02-12T12:07:12.904 INFO:teuthology.orchestra.run.trial127.stdout: python3-ceph-common x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 184 k 2026-02-12T12:07:12.904 INFO:teuthology.orchestra.run.trial127.stdout: python3-cephfs x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 203 k 2026-02-12T12:07:12.904 INFO:teuthology.orchestra.run.trial127.stdout: python3-rados x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 324 k 2026-02-12T12:07:12.904 INFO:teuthology.orchestra.run.trial127.stdout: python3-rbd x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 305 k 2026-02-12T12:07:12.904 INFO:teuthology.orchestra.run.trial127.stdout: python3-rgw x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 100 k 2026-02-12T12:07:12.904 INFO:teuthology.orchestra.run.trial127.stdout: qatlib x86_64 25.08.0-2.el9 appstream 240 k 2026-02-12T12:07:12.904 INFO:teuthology.orchestra.run.trial127.stdout: qatzip-libs x86_64 1.3.1-1.el9 appstream 66 k 2026-02-12T12:07:12.904 INFO:teuthology.orchestra.run.trial127.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2026-02-12T12:07:12.904 INFO:teuthology.orchestra.run.trial127.stdout: thrift x86_64 0.15.0-4.el9 epel 1.6 M 2026-02-12T12:07:12.904 INFO:teuthology.orchestra.run.trial127.stdout:Installing weak dependencies: 2026-02-12T12:07:12.904 INFO:teuthology.orchestra.run.trial127.stdout: libcephfs-daemon x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 35 k 2026-02-12T12:07:12.904 INFO:teuthology.orchestra.run.trial127.stdout: libcephfs-proxy2 x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 24 k 2026-02-12T12:07:12.904 INFO:teuthology.orchestra.run.trial127.stdout: nvme-cli x86_64 2.13-1.el9 baseos 1.1 M 2026-02-12T12:07:12.904 INFO:teuthology.orchestra.run.trial127.stdout: qatlib-service x86_64 25.08.0-2.el9 appstream 37 k 2026-02-12T12:07:12.904 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:07:12.904 INFO:teuthology.orchestra.run.trial127.stdout:Transaction Summary 2026-02-12T12:07:12.904 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-12T12:07:12.904 INFO:teuthology.orchestra.run.trial127.stdout:Install 35 Packages 2026-02-12T12:07:12.904 INFO:teuthology.orchestra.run.trial127.stdout:Upgrade 2 Packages 2026-02-12T12:07:12.904 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:07:12.905 INFO:teuthology.orchestra.run.trial127.stdout:Total download size: 61 M 2026-02-12T12:07:12.905 INFO:teuthology.orchestra.run.trial127.stdout:Downloading Packages: 2026-02-12T12:07:15.208 INFO:teuthology.orchestra.run.trial127.stdout:(1/37): ceph-base-20.3.0-5145.g5fdb7d8f.el9.x86 2.8 MB/s | 5.1 MB 00:01 2026-02-12T12:07:15.268 INFO:teuthology.orchestra.run.trial127.stdout:(2/37): ceph-selinux-20.3.0-5145.g5fdb7d8f.el9. 414 kB/s | 25 kB 00:00 2026-02-12T12:07:15.330 INFO:teuthology.orchestra.run.trial127.stdout:(3/37): libcephfs-daemon-20.3.0-5145.g5fdb7d8f. 566 kB/s | 35 kB 00:00 2026-02-12T12:07:15.393 INFO:teuthology.orchestra.run.trial127.stdout:(4/37): libcephfs-proxy2-20.3.0-5145.g5fdb7d8f. 387 kB/s | 24 kB 00:00 2026-02-12T12:07:15.598 INFO:teuthology.orchestra.run.trial127.stdout:(5/37): libcephfs2-20.3.0-5145.g5fdb7d8f.el9.x8 4.1 MB/s | 860 kB 00:00 2026-02-12T12:07:15.668 INFO:teuthology.orchestra.run.trial127.stdout:(6/37): libradosstriper1-20.3.0-5145.g5fdb7d8f. 3.6 MB/s | 263 kB 00:00 2026-02-12T12:07:17.397 INFO:teuthology.orchestra.run.trial127.stdout:(7/37): librgw2-20.3.0-5145.g5fdb7d8f.el9.x86_6 3.3 MB/s | 5.6 MB 00:01 2026-02-12T12:07:17.460 INFO:teuthology.orchestra.run.trial127.stdout:(8/37): python3-ceph-argparse-20.3.0-5145.g5fdb 719 kB/s | 45 kB 00:00 2026-02-12T12:07:17.526 INFO:teuthology.orchestra.run.trial127.stdout:(9/37): python3-ceph-common-20.3.0-5145.g5fdb7d 2.7 MB/s | 184 kB 00:00 2026-02-12T12:07:17.593 INFO:teuthology.orchestra.run.trial127.stdout:(10/37): python3-cephfs-20.3.0-5145.g5fdb7d8f.e 3.0 MB/s | 203 kB 00:00 2026-02-12T12:07:17.713 INFO:teuthology.orchestra.run.trial127.stdout:(11/37): ceph-radosgw-20.3.0-5145.g5fdb7d8f.el9 2.6 MB/s | 11 MB 00:04 2026-02-12T12:07:17.721 INFO:teuthology.orchestra.run.trial127.stdout:(12/37): python3-rados-20.3.0-5145.g5fdb7d8f.el 2.5 MB/s | 324 kB 00:00 2026-02-12T12:07:17.786 INFO:teuthology.orchestra.run.trial127.stdout:(13/37): python3-rgw-20.3.0-5145.g5fdb7d8f.el9. 1.5 MB/s | 100 kB 00:00 2026-02-12T12:07:17.830 INFO:teuthology.orchestra.run.trial127.stdout:(14/37): mailcap-2.1.49-5.el9.noarch.rpm 754 kB/s | 33 kB 00:00 2026-02-12T12:07:17.900 INFO:teuthology.orchestra.run.trial127.stdout:(15/37): python3-rbd-20.3.0-5145.g5fdb7d8f.el9. 1.6 MB/s | 305 kB 00:00 2026-02-12T12:07:17.909 INFO:teuthology.orchestra.run.trial127.stdout:(16/37): nvme-cli-2.13-1.el9.x86_64.rpm 14 MB/s | 1.1 MB 00:00 2026-02-12T12:07:17.926 INFO:teuthology.orchestra.run.trial127.stdout:(17/37): boost-program-options-1.75.0-13.el9.x8 5.9 MB/s | 104 kB 00:00 2026-02-12T12:07:17.935 INFO:teuthology.orchestra.run.trial127.stdout:(18/37): libnbd-1.20.3-4.el9.x86_64.rpm 18 MB/s | 164 kB 00:00 2026-02-12T12:07:17.938 INFO:teuthology.orchestra.run.trial127.stdout:(19/37): pciutils-3.7.0-7.el9.x86_64.rpm 2.4 MB/s | 93 kB 00:00 2026-02-12T12:07:17.942 INFO:teuthology.orchestra.run.trial127.stdout:(20/37): libpmemobj-1.12.1-1.el9.x86_64.rpm 22 MB/s | 160 kB 00:00 2026-02-12T12:07:17.953 INFO:teuthology.orchestra.run.trial127.stdout:(21/37): librdkafka-1.6.1-102.el9.x86_64.rpm 62 MB/s | 662 kB 00:00 2026-02-12T12:07:17.959 INFO:teuthology.orchestra.run.trial127.stdout:(22/37): lttng-ust-2.12.0-6.el9.x86_64.rpm 42 MB/s | 292 kB 00:00 2026-02-12T12:07:17.965 INFO:teuthology.orchestra.run.trial127.stdout:(23/37): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.6 MB/s | 45 kB 00:00 2026-02-12T12:07:17.966 INFO:teuthology.orchestra.run.trial127.stdout:(24/37): qatlib-25.08.0-2.el9.x86_64.rpm 35 MB/s | 240 kB 00:00 2026-02-12T12:07:17.971 INFO:teuthology.orchestra.run.trial127.stdout:(25/37): qatzip-libs-1.3.1-1.el9.x86_64.rpm 14 MB/s | 66 kB 00:00 2026-02-12T12:07:17.978 INFO:teuthology.orchestra.run.trial127.stdout:(26/37): qatlib-service-25.08.0-2.el9.x86_64.rp 2.8 MB/s | 37 kB 00:00 2026-02-12T12:07:18.008 INFO:teuthology.orchestra.run.trial127.stdout:(27/37): gperftools-libs-2.9.1-3.el9.x86_64.rpm 8.1 MB/s | 308 kB 00:00 2026-02-12T12:07:18.012 INFO:teuthology.orchestra.run.trial127.stdout:(28/37): libarrow-doc-9.0.0-15.el9.noarch.rpm 5.9 MB/s | 25 kB 00:00 2026-02-12T12:07:18.017 INFO:teuthology.orchestra.run.trial127.stdout:(29/37): liboath-2.6.12-1.el9.x86_64.rpm 11 MB/s | 49 kB 00:00 2026-02-12T12:07:18.021 INFO:teuthology.orchestra.run.trial127.stdout:(30/37): libunwind-1.6.2-1.el9.x86_64.rpm 15 MB/s | 67 kB 00:00 2026-02-12T12:07:18.033 INFO:teuthology.orchestra.run.trial127.stdout:(31/37): parquet-libs-9.0.0-15.el9.x86_64.rpm 68 MB/s | 838 kB 00:00 2026-02-12T12:07:18.038 INFO:teuthology.orchestra.run.trial127.stdout:(32/37): re2-20211101-20.el9.x86_64.rpm 37 MB/s | 191 kB 00:00 2026-02-12T12:07:18.051 INFO:teuthology.orchestra.run.trial127.stdout:(33/37): thrift-0.15.0-4.el9.x86_64.rpm 131 MB/s | 1.6 MB 00:00 2026-02-12T12:07:18.067 INFO:teuthology.orchestra.run.trial127.stdout:(34/37): libarrow-9.0.0-15.el9.x86_64.rpm 50 MB/s | 4.4 MB 00:00 2026-02-12T12:07:19.025 INFO:teuthology.orchestra.run.trial127.stdout:(35/37): librados2-20.3.0-5145.g5fdb7d8f.el9.x8 3.4 MB/s | 3.3 MB 00:00 2026-02-12T12:07:19.519 INFO:teuthology.orchestra.run.trial127.stdout:(36/37): librbd1-20.3.0-5145.g5fdb7d8f.el9.x86_ 1.9 MB/s | 2.7 MB 00:01 2026-02-12T12:07:22.999 INFO:teuthology.orchestra.run.trial127.stdout:(37/37): ceph-common-20.3.0-5145.g5fdb7d8f.el9. 2.2 MB/s | 21 MB 00:09 2026-02-12T12:07:23.002 INFO:teuthology.orchestra.run.trial127.stdout:-------------------------------------------------------------------------------- 2026-02-12T12:07:23.002 INFO:teuthology.orchestra.run.trial127.stdout:Total 6.1 MB/s | 61 MB 00:10 2026-02-12T12:07:23.101 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction check 2026-02-12T12:07:23.121 INFO:teuthology.orchestra.run.trial127.stdout:Transaction check succeeded. 2026-02-12T12:07:23.121 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction test 2026-02-12T12:07:23.368 INFO:teuthology.orchestra.run.trial127.stdout:Transaction test succeeded. 2026-02-12T12:07:23.368 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction 2026-02-12T12:07:23.657 INFO:teuthology.orchestra.run.trial127.stdout: Preparing : 1/1 2026-02-12T12:07:23.800 INFO:teuthology.orchestra.run.trial127.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/39 2026-02-12T12:07:23.805 INFO:teuthology.orchestra.run.trial127.stdout: Upgrading : librados2-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2/39 2026-02-12T12:07:23.854 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: librados2-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2/39 2026-02-12T12:07:23.855 INFO:teuthology.orchestra.run.trial127.stdout: Installing : libcephfs2-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 3/39 2026-02-12T12:07:23.867 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: libcephfs2-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 3/39 2026-02-12T12:07:23.869 INFO:teuthology.orchestra.run.trial127.stdout: Installing : libcephfs-daemon-2:20.3.0-5145.g5fdb7d8f.el9.x86_6 4/39 2026-02-12T12:07:23.869 INFO:teuthology.orchestra.run.trial127.stdout: Installing : libcephfs-proxy2-2:20.3.0-5145.g5fdb7d8f.el9.x86_6 5/39 2026-02-12T12:07:23.897 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: libcephfs-proxy2-2:20.3.0-5145.g5fdb7d8f.el9.x86_6 5/39 2026-02-12T12:07:23.899 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-rados-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 6/39 2026-02-12T12:07:23.907 INFO:teuthology.orchestra.run.trial127.stdout: Installing : liboath-2.6.12-1.el9.x86_64 7/39 2026-02-12T12:07:23.909 INFO:teuthology.orchestra.run.trial127.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/39 2026-02-12T12:07:23.912 INFO:teuthology.orchestra.run.trial127.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/39 2026-02-12T12:07:23.916 INFO:teuthology.orchestra.run.trial127.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/39 2026-02-12T12:07:23.927 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-ceph-argparse-2:20.3.0-5145.g5fdb7d8f.el9. 11/39 2026-02-12T12:07:23.940 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-cephfs-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 12/39 2026-02-12T12:07:23.941 INFO:teuthology.orchestra.run.trial127.stdout: Installing : libradosstriper1-2:20.3.0-5145.g5fdb7d8f.el9.x86_6 13/39 2026-02-12T12:07:23.966 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: libradosstriper1-2:20.3.0-5145.g5fdb7d8f.el9.x86_6 13/39 2026-02-12T12:07:23.969 INFO:teuthology.orchestra.run.trial127.stdout: Installing : thrift-0.15.0-4.el9.x86_64 14/39 2026-02-12T12:07:24.002 INFO:teuthology.orchestra.run.trial127.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/39 2026-02-12T12:07:24.008 INFO:teuthology.orchestra.run.trial127.stdout: Installing : libarrow-9.0.0-15.el9.x86_64 16/39 2026-02-12T12:07:24.014 INFO:teuthology.orchestra.run.trial127.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/39 2026-02-12T12:07:24.016 INFO:teuthology.orchestra.run.trial127.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/39 2026-02-12T12:07:24.020 INFO:teuthology.orchestra.run.trial127.stdout: Installing : libarrow-doc-9.0.0-15.el9.noarch 19/39 2026-02-12T12:07:24.133 INFO:teuthology.orchestra.run.trial127.stdout: Installing : libnbd-1.20.3-4.el9.x86_64 20/39 2026-02-12T12:07:24.135 INFO:teuthology.orchestra.run.trial127.stdout: Upgrading : librbd1-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 21/39 2026-02-12T12:07:24.160 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: librbd1-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 21/39 2026-02-12T12:07:24.163 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-rbd-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 22/39 2026-02-12T12:07:24.169 INFO:teuthology.orchestra.run.trial127.stdout: Installing : boost-program-options-1.75.0-13.el9.x86_64 23/39 2026-02-12T12:07:24.414 INFO:teuthology.orchestra.run.trial127.stdout: Installing : parquet-libs-9.0.0-15.el9.x86_64 24/39 2026-02-12T12:07:24.416 INFO:teuthology.orchestra.run.trial127.stdout: Installing : librgw2-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 25/39 2026-02-12T12:07:24.433 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: librgw2-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 25/39 2026-02-12T12:07:24.437 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-rgw-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 26/39 2026-02-12T12:07:24.444 INFO:teuthology.orchestra.run.trial127.stdout: Installing : pciutils-3.7.0-7.el9.x86_64 27/39 2026-02-12T12:07:24.447 INFO:teuthology.orchestra.run.trial127.stdout: Installing : qatlib-25.08.0-2.el9.x86_64 28/39 2026-02-12T12:07:24.448 INFO:teuthology.orchestra.run.trial127.stdout: Installing : qatlib-service-25.08.0-2.el9.x86_64 29/39 2026-02-12T12:07:24.468 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: qatlib-service-25.08.0-2.el9.x86_64 29/39 2026-02-12T12:07:24.577 INFO:teuthology.orchestra.run.trial127.stdout: Installing : qatzip-libs-1.3.1-1.el9.x86_64 30/39 2026-02-12T12:07:24.580 INFO:teuthology.orchestra.run.trial127.stdout: Installing : nvme-cli-2.13-1.el9.x86_64 31/39 2026-02-12T12:07:24.857 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: nvme-cli-2.13-1.el9.x86_64 31/39 2026-02-12T12:07:24.858 INFO:teuthology.orchestra.run.trial127.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service → /usr/lib/systemd/system/nvmefc-boot-connections.service. 2026-02-12T12:07:24.858 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:07:24.901 INFO:teuthology.orchestra.run.trial127.stdout: Installing : mailcap-2.1.49-5.el9.noarch 32/39 2026-02-12T12:07:24.903 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-ceph-common-2:20.3.0-5145.g5fdb7d8f.el9.x8 33/39 2026-02-12T12:07:25.843 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: ceph-common-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 34/39 2026-02-12T12:07:25.846 INFO:teuthology.orchestra.run.trial127.stdout: Installing : ceph-common-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 34/39 2026-02-12T12:07:26.109 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: ceph-common-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 34/39 2026-02-12T12:07:26.113 INFO:teuthology.orchestra.run.trial127.stdout: Installing : ceph-base-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 35/39 2026-02-12T12:07:26.152 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: ceph-base-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 35/39 2026-02-12T12:07:26.152 INFO:teuthology.orchestra.run.trial127.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2026-02-12T12:07:26.153 INFO:teuthology.orchestra.run.trial127.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2026-02-12T12:07:26.153 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:07:26.156 INFO:teuthology.orchestra.run.trial127.stdout: Installing : ceph-selinux-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 36/39 2026-02-12T12:07:30.709 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: ceph-selinux-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 36/39 2026-02-12T12:07:30.709 INFO:teuthology.orchestra.run.trial127.stdout:skipping the directory /sys 2026-02-12T12:07:30.709 INFO:teuthology.orchestra.run.trial127.stdout:skipping the directory /proc 2026-02-12T12:07:30.709 INFO:teuthology.orchestra.run.trial127.stdout:skipping the directory /mnt 2026-02-12T12:07:30.709 INFO:teuthology.orchestra.run.trial127.stdout:skipping the directory /var/tmp 2026-02-12T12:07:30.709 INFO:teuthology.orchestra.run.trial127.stdout:skipping the directory /home 2026-02-12T12:07:30.709 INFO:teuthology.orchestra.run.trial127.stdout:skipping the directory /root 2026-02-12T12:07:30.709 INFO:teuthology.orchestra.run.trial127.stdout:skipping the directory /tmp 2026-02-12T12:07:30.709 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:07:31.200 INFO:teuthology.orchestra.run.trial127.stdout: Installing : ceph-radosgw-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 37/39 2026-02-12T12:07:31.224 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: ceph-radosgw-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 37/39 2026-02-12T12:07:31.224 INFO:teuthology.orchestra.run.trial127.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-12T12:07:31.224 INFO:teuthology.orchestra.run.trial127.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2026-02-12T12:07:31.224 INFO:teuthology.orchestra.run.trial127.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-12T12:07:31.224 INFO:teuthology.orchestra.run.trial127.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-12T12:07:31.224 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:07:31.226 INFO:teuthology.orchestra.run.trial127.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 38/39 2026-02-12T12:07:31.237 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 38/39 2026-02-12T12:07:31.237 INFO:teuthology.orchestra.run.trial127.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 39/39 2026-02-12T12:07:31.294 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 39/39 2026-02-12T12:07:31.294 INFO:teuthology.orchestra.run.trial127.stdout:Creating group 'qat' with GID 992. 2026-02-12T12:07:31.294 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:07:31.578 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : ceph-base-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 1/39 2026-02-12T12:07:31.578 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : ceph-common-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2/39 2026-02-12T12:07:31.578 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : ceph-radosgw-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 3/39 2026-02-12T12:07:31.578 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : ceph-selinux-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 4/39 2026-02-12T12:07:31.578 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : libcephfs-daemon-2:20.3.0-5145.g5fdb7d8f.el9.x86_6 5/39 2026-02-12T12:07:31.578 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : libcephfs-proxy2-2:20.3.0-5145.g5fdb7d8f.el9.x86_6 6/39 2026-02-12T12:07:31.578 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : libcephfs2-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 7/39 2026-02-12T12:07:31.578 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : libradosstriper1-2:20.3.0-5145.g5fdb7d8f.el9.x86_6 8/39 2026-02-12T12:07:31.579 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : librgw2-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 9/39 2026-02-12T12:07:31.579 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-ceph-argparse-2:20.3.0-5145.g5fdb7d8f.el9. 10/39 2026-02-12T12:07:31.579 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-ceph-common-2:20.3.0-5145.g5fdb7d8f.el9.x8 11/39 2026-02-12T12:07:31.579 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-cephfs-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 12/39 2026-02-12T12:07:31.579 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-rados-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 13/39 2026-02-12T12:07:31.579 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-rbd-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 14/39 2026-02-12T12:07:31.579 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-rgw-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 15/39 2026-02-12T12:07:31.579 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : mailcap-2.1.49-5.el9.noarch 16/39 2026-02-12T12:07:31.579 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : nvme-cli-2.13-1.el9.x86_64 17/39 2026-02-12T12:07:31.579 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : pciutils-3.7.0-7.el9.x86_64 18/39 2026-02-12T12:07:31.579 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : boost-program-options-1.75.0-13.el9.x86_64 19/39 2026-02-12T12:07:31.579 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : libnbd-1.20.3-4.el9.x86_64 20/39 2026-02-12T12:07:31.579 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 21/39 2026-02-12T12:07:31.579 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 22/39 2026-02-12T12:07:31.579 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 23/39 2026-02-12T12:07:31.579 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 24/39 2026-02-12T12:07:31.580 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : qatlib-25.08.0-2.el9.x86_64 25/39 2026-02-12T12:07:31.580 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : qatlib-service-25.08.0-2.el9.x86_64 26/39 2026-02-12T12:07:31.580 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : qatzip-libs-1.3.1-1.el9.x86_64 27/39 2026-02-12T12:07:31.580 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 28/39 2026-02-12T12:07:31.580 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : libarrow-9.0.0-15.el9.x86_64 29/39 2026-02-12T12:07:31.580 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : libarrow-doc-9.0.0-15.el9.noarch 30/39 2026-02-12T12:07:31.580 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : liboath-2.6.12-1.el9.x86_64 31/39 2026-02-12T12:07:31.580 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 32/39 2026-02-12T12:07:31.580 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : parquet-libs-9.0.0-15.el9.x86_64 33/39 2026-02-12T12:07:31.580 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : re2-1:20211101-20.el9.x86_64 34/39 2026-02-12T12:07:31.580 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : thrift-0.15.0-4.el9.x86_64 35/39 2026-02-12T12:07:31.580 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : librados2-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 36/39 2026-02-12T12:07:31.580 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 37/39 2026-02-12T12:07:31.580 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : librbd1-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 38/39 2026-02-12T12:07:31.640 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 39/39 2026-02-12T12:07:31.640 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:07:31.640 INFO:teuthology.orchestra.run.trial127.stdout:Upgraded: 2026-02-12T12:07:31.640 INFO:teuthology.orchestra.run.trial127.stdout: librados2-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:07:31.640 INFO:teuthology.orchestra.run.trial127.stdout: librbd1-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:07:31.640 INFO:teuthology.orchestra.run.trial127.stdout:Installed: 2026-02-12T12:07:31.640 INFO:teuthology.orchestra.run.trial127.stdout: boost-program-options-1.75.0-13.el9.x86_64 2026-02-12T12:07:31.640 INFO:teuthology.orchestra.run.trial127.stdout: ceph-base-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:07:31.640 INFO:teuthology.orchestra.run.trial127.stdout: ceph-common-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:07:31.640 INFO:teuthology.orchestra.run.trial127.stdout: ceph-radosgw-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:07:31.640 INFO:teuthology.orchestra.run.trial127.stdout: ceph-selinux-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:07:31.640 INFO:teuthology.orchestra.run.trial127.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2026-02-12T12:07:31.640 INFO:teuthology.orchestra.run.trial127.stdout: libarrow-9.0.0-15.el9.x86_64 2026-02-12T12:07:31.640 INFO:teuthology.orchestra.run.trial127.stdout: libarrow-doc-9.0.0-15.el9.noarch 2026-02-12T12:07:31.640 INFO:teuthology.orchestra.run.trial127.stdout: libcephfs-daemon-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:07:31.640 INFO:teuthology.orchestra.run.trial127.stdout: libcephfs-proxy2-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:07:31.640 INFO:teuthology.orchestra.run.trial127.stdout: libcephfs2-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:07:31.640 INFO:teuthology.orchestra.run.trial127.stdout: libnbd-1.20.3-4.el9.x86_64 2026-02-12T12:07:31.640 INFO:teuthology.orchestra.run.trial127.stdout: liboath-2.6.12-1.el9.x86_64 2026-02-12T12:07:31.640 INFO:teuthology.orchestra.run.trial127.stdout: libpmemobj-1.12.1-1.el9.x86_64 2026-02-12T12:07:31.640 INFO:teuthology.orchestra.run.trial127.stdout: librabbitmq-0.11.0-7.el9.x86_64 2026-02-12T12:07:31.640 INFO:teuthology.orchestra.run.trial127.stdout: libradosstriper1-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:07:31.640 INFO:teuthology.orchestra.run.trial127.stdout: librdkafka-1.6.1-102.el9.x86_64 2026-02-12T12:07:31.641 INFO:teuthology.orchestra.run.trial127.stdout: librgw2-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:07:31.641 INFO:teuthology.orchestra.run.trial127.stdout: libunwind-1.6.2-1.el9.x86_64 2026-02-12T12:07:31.641 INFO:teuthology.orchestra.run.trial127.stdout: lttng-ust-2.12.0-6.el9.x86_64 2026-02-12T12:07:31.641 INFO:teuthology.orchestra.run.trial127.stdout: mailcap-2.1.49-5.el9.noarch 2026-02-12T12:07:31.641 INFO:teuthology.orchestra.run.trial127.stdout: nvme-cli-2.13-1.el9.x86_64 2026-02-12T12:07:31.641 INFO:teuthology.orchestra.run.trial127.stdout: parquet-libs-9.0.0-15.el9.x86_64 2026-02-12T12:07:31.641 INFO:teuthology.orchestra.run.trial127.stdout: pciutils-3.7.0-7.el9.x86_64 2026-02-12T12:07:31.641 INFO:teuthology.orchestra.run.trial127.stdout: python3-ceph-argparse-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:07:31.641 INFO:teuthology.orchestra.run.trial127.stdout: python3-ceph-common-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:07:31.641 INFO:teuthology.orchestra.run.trial127.stdout: python3-cephfs-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:07:31.641 INFO:teuthology.orchestra.run.trial127.stdout: python3-rados-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:07:31.641 INFO:teuthology.orchestra.run.trial127.stdout: python3-rbd-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:07:31.641 INFO:teuthology.orchestra.run.trial127.stdout: python3-rgw-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:07:31.641 INFO:teuthology.orchestra.run.trial127.stdout: qatlib-25.08.0-2.el9.x86_64 2026-02-12T12:07:31.641 INFO:teuthology.orchestra.run.trial127.stdout: qatlib-service-25.08.0-2.el9.x86_64 2026-02-12T12:07:31.641 INFO:teuthology.orchestra.run.trial127.stdout: qatzip-libs-1.3.1-1.el9.x86_64 2026-02-12T12:07:31.641 INFO:teuthology.orchestra.run.trial127.stdout: re2-1:20211101-20.el9.x86_64 2026-02-12T12:07:31.641 INFO:teuthology.orchestra.run.trial127.stdout: thrift-0.15.0-4.el9.x86_64 2026-02-12T12:07:31.641 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:07:31.641 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-12T12:07:31.682 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install ceph-test 2026-02-12T12:07:31.984 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:00:20 ago on Thu 12 Feb 2026 12:07:11 PM UTC. 2026-02-12T12:07:32.068 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-12T12:07:32.069 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-12T12:07:32.069 INFO:teuthology.orchestra.run.trial127.stdout: Package Arch Version Repository Size 2026-02-12T12:07:32.069 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-12T12:07:32.069 INFO:teuthology.orchestra.run.trial127.stdout:Installing: 2026-02-12T12:07:32.069 INFO:teuthology.orchestra.run.trial127.stdout: ceph-test x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 126 M 2026-02-12T12:07:32.069 INFO:teuthology.orchestra.run.trial127.stdout:Installing dependencies: 2026-02-12T12:07:32.069 INFO:teuthology.orchestra.run.trial127.stdout: hwloc-libs x86_64 2.4.1-6.el9 baseos 2.1 M 2026-02-12T12:07:32.069 INFO:teuthology.orchestra.run.trial127.stdout: libasan8 x86_64 15.1.1-2.5.el9 appstream 523 k 2026-02-12T12:07:32.069 INFO:teuthology.orchestra.run.trial127.stdout: libatomic x86_64 11.5.0-14.el9 baseos 23 k 2026-02-12T12:07:32.069 INFO:teuthology.orchestra.run.trial127.stdout: libcephsqlite x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 153 k 2026-02-12T12:07:32.069 INFO:teuthology.orchestra.run.trial127.stdout: libubsan x86_64 11.5.0-14.el9 appstream 176 k 2026-02-12T12:07:32.069 INFO:teuthology.orchestra.run.trial127.stdout: libxslt x86_64 1.1.34-12.el9 appstream 233 k 2026-02-12T12:07:32.069 INFO:teuthology.orchestra.run.trial127.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2026-02-12T12:07:32.070 INFO:teuthology.orchestra.run.trial127.stdout: protobuf x86_64 3.14.0-17.el9 appstream 1.0 M 2026-02-12T12:07:32.070 INFO:teuthology.orchestra.run.trial127.stdout: socat x86_64 1.7.4.1-8.el9 appstream 303 k 2026-02-12T12:07:32.070 INFO:teuthology.orchestra.run.trial127.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2026-02-12T12:07:32.070 INFO:teuthology.orchestra.run.trial127.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2026-02-12T12:07:32.070 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:07:32.070 INFO:teuthology.orchestra.run.trial127.stdout:Transaction Summary 2026-02-12T12:07:32.070 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-12T12:07:32.070 INFO:teuthology.orchestra.run.trial127.stdout:Install 12 Packages 2026-02-12T12:07:32.070 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:07:32.070 INFO:teuthology.orchestra.run.trial127.stdout:Total download size: 131 M 2026-02-12T12:07:32.071 INFO:teuthology.orchestra.run.trial127.stdout:Installed size: 977 M 2026-02-12T12:07:32.071 INFO:teuthology.orchestra.run.trial127.stdout:Downloading Packages: 2026-02-12T12:07:32.603 INFO:teuthology.orchestra.run.trial127.stdout:(1/12): hwloc-libs-2.4.1-6.el9.x86_64.rpm 7.5 MB/s | 2.1 MB 00:00 2026-02-12T12:07:32.645 INFO:teuthology.orchestra.run.trial127.stdout:(2/12): libatomic-11.5.0-14.el9.x86_64.rpm 550 kB/s | 23 kB 00:00 2026-02-12T12:07:32.757 INFO:teuthology.orchestra.run.trial127.stdout:(3/12): lksctp-tools-1.0.19-2.el9.x86_64.rpm 840 kB/s | 94 kB 00:00 2026-02-12T12:07:32.819 INFO:teuthology.orchestra.run.trial127.stdout:(4/12): libcephsqlite-20.3.0-5145.g5fdb7d8f.el9 309 kB/s | 153 kB 00:00 2026-02-12T12:07:33.228 INFO:teuthology.orchestra.run.trial127.stdout:[MIRROR] libasan8-15.1.1-2.5.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/libasan8-15.1.1-2.5.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-12T12:07:33.291 INFO:teuthology.orchestra.run.trial127.stdout:[MIRROR] libubsan-11.5.0-14.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/libubsan-11.5.0-14.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-12T12:07:33.859 INFO:teuthology.orchestra.run.trial127.stdout:[MIRROR] libasan8-15.1.1-2.5.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/libasan8-15.1.1-2.5.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-12T12:07:33.913 INFO:teuthology.orchestra.run.trial127.stdout:[MIRROR] libubsan-11.5.0-14.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/libubsan-11.5.0-14.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-12T12:07:33.914 INFO:teuthology.orchestra.run.trial127.stdout:(5/12): libasan8-15.1.1-2.5.el9.x86_64.rpm 452 kB/s | 523 kB 00:01 2026-02-12T12:07:33.924 INFO:teuthology.orchestra.run.trial127.stdout:(6/12): libubsan-11.5.0-14.el9.x86_64.rpm 159 kB/s | 176 kB 00:01 2026-02-12T12:07:34.310 INFO:teuthology.orchestra.run.trial127.stdout:[MIRROR] protobuf-3.14.0-17.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/protobuf-3.14.0-17.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-12T12:07:34.315 INFO:teuthology.orchestra.run.trial127.stdout:[MIRROR] libxslt-1.1.34-12.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/libxslt-1.1.34-12.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-12T12:07:34.700 INFO:teuthology.orchestra.run.trial127.stdout:[MIRROR] libxslt-1.1.34-12.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/libxslt-1.1.34-12.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-12T12:07:34.701 INFO:teuthology.orchestra.run.trial127.stdout:[MIRROR] protobuf-3.14.0-17.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/protobuf-3.14.0-17.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-12T12:07:34.725 INFO:teuthology.orchestra.run.trial127.stdout:(7/12): libxslt-1.1.34-12.el9.x86_64.rpm 287 kB/s | 233 kB 00:00 2026-02-12T12:07:34.746 INFO:teuthology.orchestra.run.trial127.stdout:(8/12): socat-1.7.4.1-8.el9.x86_64.rpm 14 MB/s | 303 kB 00:00 2026-02-12T12:07:34.755 INFO:teuthology.orchestra.run.trial127.stdout:(9/12): xmlstarlet-1.6.1-20.el9.x86_64.rpm 6.8 MB/s | 64 kB 00:00 2026-02-12T12:07:34.762 INFO:teuthology.orchestra.run.trial127.stdout:(10/12): protobuf-3.14.0-17.el9.x86_64.rpm 1.2 MB/s | 1.0 MB 00:00 2026-02-12T12:07:34.899 INFO:teuthology.orchestra.run.trial127.stdout:(11/12): yaml-cpp-0.6.3-5.el9.x86_64.rpm 850 kB/s | 122 kB 00:00 2026-02-12T12:10:29.363 INFO:teuthology.orchestra.run.trial127.stdout:(12/12): ceph-test-20.3.0-5145.g5fdb7d8f.el9.x8 729 kB/s | 126 MB 02:57 2026-02-12T12:10:29.366 INFO:teuthology.orchestra.run.trial127.stdout:-------------------------------------------------------------------------------- 2026-02-12T12:10:29.366 INFO:teuthology.orchestra.run.trial127.stdout:Total 756 kB/s | 131 MB 02:57 2026-02-12T12:10:29.434 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction check 2026-02-12T12:10:29.446 INFO:teuthology.orchestra.run.trial127.stdout:Transaction check succeeded. 2026-02-12T12:10:29.446 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction test 2026-02-12T12:10:29.880 INFO:teuthology.orchestra.run.trial127.stdout:Transaction test succeeded. 2026-02-12T12:10:29.880 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction 2026-02-12T12:10:30.335 INFO:teuthology.orchestra.run.trial127.stdout: Preparing : 1/1 2026-02-12T12:10:30.343 INFO:teuthology.orchestra.run.trial127.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/12 2026-02-12T12:10:30.352 INFO:teuthology.orchestra.run.trial127.stdout: Installing : socat-1.7.4.1-8.el9.x86_64 2/12 2026-02-12T12:10:30.357 INFO:teuthology.orchestra.run.trial127.stdout: Installing : protobuf-3.14.0-17.el9.x86_64 3/12 2026-02-12T12:10:30.361 INFO:teuthology.orchestra.run.trial127.stdout: Installing : libxslt-1.1.34-12.el9.x86_64 4/12 2026-02-12T12:10:30.365 INFO:teuthology.orchestra.run.trial127.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 5/12 2026-02-12T12:10:30.369 INFO:teuthology.orchestra.run.trial127.stdout: Installing : libubsan-11.5.0-14.el9.x86_64 6/12 2026-02-12T12:10:30.370 INFO:teuthology.orchestra.run.trial127.stdout: Installing : libasan8-15.1.1-2.5.el9.x86_64 7/12 2026-02-12T12:10:30.386 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: libasan8-15.1.1-2.5.el9.x86_64 7/12 2026-02-12T12:10:30.388 INFO:teuthology.orchestra.run.trial127.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 8/12 2026-02-12T12:10:30.398 INFO:teuthology.orchestra.run.trial127.stdout: Installing : libatomic-11.5.0-14.el9.x86_64 9/12 2026-02-12T12:10:30.407 INFO:teuthology.orchestra.run.trial127.stdout: Installing : hwloc-libs-2.4.1-6.el9.x86_64 10/12 2026-02-12T12:10:30.407 INFO:teuthology.orchestra.run.trial127.stdout: Installing : libcephsqlite-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 11/12 2026-02-12T12:10:37.452 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: libcephsqlite-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 11/12 2026-02-12T12:10:37.470 INFO:teuthology.orchestra.run.trial127.stdout: Installing : ceph-test-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 12/12 2026-02-12T12:10:38.224 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: ceph-test-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 12/12 2026-02-12T12:10:38.224 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : ceph-test-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 1/12 2026-02-12T12:10:38.224 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : libcephsqlite-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2/12 2026-02-12T12:10:38.224 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : hwloc-libs-2.4.1-6.el9.x86_64 3/12 2026-02-12T12:10:38.224 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : libatomic-11.5.0-14.el9.x86_64 4/12 2026-02-12T12:10:38.224 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 5/12 2026-02-12T12:10:38.225 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : libasan8-15.1.1-2.5.el9.x86_64 6/12 2026-02-12T12:10:38.225 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : libubsan-11.5.0-14.el9.x86_64 7/12 2026-02-12T12:10:38.225 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : libxslt-1.1.34-12.el9.x86_64 8/12 2026-02-12T12:10:38.225 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : protobuf-3.14.0-17.el9.x86_64 9/12 2026-02-12T12:10:38.225 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : socat-1.7.4.1-8.el9.x86_64 10/12 2026-02-12T12:10:38.225 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 11/12 2026-02-12T12:10:38.277 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 12/12 2026-02-12T12:10:38.277 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:10:38.278 INFO:teuthology.orchestra.run.trial127.stdout:Installed: 2026-02-12T12:10:38.278 INFO:teuthology.orchestra.run.trial127.stdout: ceph-test-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:10:38.278 INFO:teuthology.orchestra.run.trial127.stdout: hwloc-libs-2.4.1-6.el9.x86_64 2026-02-12T12:10:38.278 INFO:teuthology.orchestra.run.trial127.stdout: libasan8-15.1.1-2.5.el9.x86_64 2026-02-12T12:10:38.278 INFO:teuthology.orchestra.run.trial127.stdout: libatomic-11.5.0-14.el9.x86_64 2026-02-12T12:10:38.278 INFO:teuthology.orchestra.run.trial127.stdout: libcephsqlite-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:10:38.278 INFO:teuthology.orchestra.run.trial127.stdout: libubsan-11.5.0-14.el9.x86_64 2026-02-12T12:10:38.278 INFO:teuthology.orchestra.run.trial127.stdout: libxslt-1.1.34-12.el9.x86_64 2026-02-12T12:10:38.278 INFO:teuthology.orchestra.run.trial127.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2026-02-12T12:10:38.278 INFO:teuthology.orchestra.run.trial127.stdout: protobuf-3.14.0-17.el9.x86_64 2026-02-12T12:10:38.278 INFO:teuthology.orchestra.run.trial127.stdout: socat-1.7.4.1-8.el9.x86_64 2026-02-12T12:10:38.279 INFO:teuthology.orchestra.run.trial127.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2026-02-12T12:10:38.279 INFO:teuthology.orchestra.run.trial127.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2026-02-12T12:10:38.279 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:10:38.279 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-12T12:10:38.318 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install ceph 2026-02-12T12:10:38.613 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:03:27 ago on Thu 12 Feb 2026 12:07:11 PM UTC. 2026-02-12T12:10:38.700 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-12T12:10:38.703 INFO:teuthology.orchestra.run.trial127.stdout:======================================================================================= 2026-02-12T12:10:38.703 INFO:teuthology.orchestra.run.trial127.stdout: Package Arch Version Repository Size 2026-02-12T12:10:38.704 INFO:teuthology.orchestra.run.trial127.stdout:======================================================================================= 2026-02-12T12:10:38.704 INFO:teuthology.orchestra.run.trial127.stdout:Installing: 2026-02-12T12:10:38.704 INFO:teuthology.orchestra.run.trial127.stdout: ceph x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 6.5 k 2026-02-12T12:10:38.704 INFO:teuthology.orchestra.run.trial127.stdout:Installing dependencies: 2026-02-12T12:10:38.704 INFO:teuthology.orchestra.run.trial127.stdout: abseil-cpp x86_64 20211102.0-4.el9 epel 551 k 2026-02-12T12:10:38.704 INFO:teuthology.orchestra.run.trial127.stdout: ceph-grafana-dashboards noarch 2:20.3.0-5145.g5fdb7d8f.el9 ceph-noarch 57 k 2026-02-12T12:10:38.704 INFO:teuthology.orchestra.run.trial127.stdout: ceph-mds x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 2.0 M 2026-02-12T12:10:38.704 INFO:teuthology.orchestra.run.trial127.stdout: ceph-mgr x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 937 k 2026-02-12T12:10:38.704 INFO:teuthology.orchestra.run.trial127.stdout: ceph-mgr-modules-core noarch 2:20.3.0-5145.g5fdb7d8f.el9 ceph-noarch 297 k 2026-02-12T12:10:38.704 INFO:teuthology.orchestra.run.trial127.stdout: ceph-mon x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 4.8 M 2026-02-12T12:10:38.704 INFO:teuthology.orchestra.run.trial127.stdout: ceph-osd x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 163 k 2026-02-12T12:10:38.704 INFO:teuthology.orchestra.run.trial127.stdout: ceph-osd-classic x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 15 M 2026-02-12T12:10:38.705 INFO:teuthology.orchestra.run.trial127.stdout: ceph-osd-crimson x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 87 M 2026-02-12T12:10:38.705 INFO:teuthology.orchestra.run.trial127.stdout: ceph-prometheus-alerts noarch 2:20.3.0-5145.g5fdb7d8f.el9 ceph-noarch 17 k 2026-02-12T12:10:38.705 INFO:teuthology.orchestra.run.trial127.stdout: cephadm noarch 2:20.3.0-5145.g5fdb7d8f.el9 ceph-noarch 1.0 M 2026-02-12T12:10:38.705 INFO:teuthology.orchestra.run.trial127.stdout: cryptsetup x86_64 2.8.1-2.el9 baseos 350 k 2026-02-12T12:10:38.705 INFO:teuthology.orchestra.run.trial127.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2026-02-12T12:10:38.705 INFO:teuthology.orchestra.run.trial127.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2026-02-12T12:10:38.705 INFO:teuthology.orchestra.run.trial127.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2026-02-12T12:10:38.705 INFO:teuthology.orchestra.run.trial127.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2026-02-12T12:10:38.705 INFO:teuthology.orchestra.run.trial127.stdout: ledmon-libs x86_64 1.1.0-3.el9 baseos 40 k 2026-02-12T12:10:38.705 INFO:teuthology.orchestra.run.trial127.stdout: libconfig x86_64 1.7.2-9.el9 baseos 72 k 2026-02-12T12:10:38.705 INFO:teuthology.orchestra.run.trial127.stdout: libgfortran x86_64 11.5.0-14.el9 baseos 794 k 2026-02-12T12:10:38.705 INFO:teuthology.orchestra.run.trial127.stdout: libquadmath x86_64 11.5.0-14.el9 baseos 184 k 2026-02-12T12:10:38.706 INFO:teuthology.orchestra.run.trial127.stdout: libstoragemgmt x86_64 1.10.1-1.el9 appstream 246 k 2026-02-12T12:10:38.706 INFO:teuthology.orchestra.run.trial127.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2026-02-12T12:10:38.706 INFO:teuthology.orchestra.run.trial127.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2026-02-12T12:10:38.706 INFO:teuthology.orchestra.run.trial127.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2026-02-12T12:10:38.706 INFO:teuthology.orchestra.run.trial127.stdout: openblas x86_64 0.3.29-1.el9 appstream 42 k 2026-02-12T12:10:38.706 INFO:teuthology.orchestra.run.trial127.stdout: openblas-openmp x86_64 0.3.29-1.el9 appstream 5.3 M 2026-02-12T12:10:38.706 INFO:teuthology.orchestra.run.trial127.stdout: protobuf-compiler x86_64 3.14.0-17.el9 crb 862 k 2026-02-12T12:10:38.706 INFO:teuthology.orchestra.run.trial127.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2026-02-12T12:10:38.706 INFO:teuthology.orchestra.run.trial127.stdout: python3-autocommand noarch 2.2.2-8.el9 epel 29 k 2026-02-12T12:10:38.706 INFO:teuthology.orchestra.run.trial127.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2026-02-12T12:10:38.706 INFO:teuthology.orchestra.run.trial127.stdout: python3-backports-tarfile noarch 1.2.0-1.el9 epel 60 k 2026-02-12T12:10:38.706 INFO:teuthology.orchestra.run.trial127.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2026-02-12T12:10:38.706 INFO:teuthology.orchestra.run.trial127.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2026-02-12T12:10:38.707 INFO:teuthology.orchestra.run.trial127.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2026-02-12T12:10:38.707 INFO:teuthology.orchestra.run.trial127.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2026-02-12T12:10:38.707 INFO:teuthology.orchestra.run.trial127.stdout: python3-cheroot noarch 10.0.1-4.el9 epel 173 k 2026-02-12T12:10:38.707 INFO:teuthology.orchestra.run.trial127.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2026-02-12T12:10:38.707 INFO:teuthology.orchestra.run.trial127.stdout: python3-cryptography x86_64 36.0.1-5.el9 baseos 1.2 M 2026-02-12T12:10:38.707 INFO:teuthology.orchestra.run.trial127.stdout: python3-devel x86_64 3.9.25-2.el9 appstream 244 k 2026-02-12T12:10:38.707 INFO:teuthology.orchestra.run.trial127.stdout: python3-google-auth noarch 1:2.45.0-1.el9 epel 254 k 2026-02-12T12:10:38.707 INFO:teuthology.orchestra.run.trial127.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2026-02-12T12:10:38.707 INFO:teuthology.orchestra.run.trial127.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2026-02-12T12:10:38.707 INFO:teuthology.orchestra.run.trial127.stdout: python3-isodate noarch 0.6.1-3.el9 epel 56 k 2026-02-12T12:10:38.707 INFO:teuthology.orchestra.run.trial127.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2026-02-12T12:10:38.707 INFO:teuthology.orchestra.run.trial127.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2026-02-12T12:10:38.707 INFO:teuthology.orchestra.run.trial127.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2026-02-12T12:10:38.708 INFO:teuthology.orchestra.run.trial127.stdout: python3-jaraco-context noarch 6.0.1-3.el9 epel 20 k 2026-02-12T12:10:38.708 INFO:teuthology.orchestra.run.trial127.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2026-02-12T12:10:38.708 INFO:teuthology.orchestra.run.trial127.stdout: python3-jaraco-text noarch 4.0.0-2.el9 epel 26 k 2026-02-12T12:10:38.708 INFO:teuthology.orchestra.run.trial127.stdout: python3-jinja2 noarch 2.11.3-8.el9 appstream 249 k 2026-02-12T12:10:38.708 INFO:teuthology.orchestra.run.trial127.stdout: python3-jmespath noarch 1.0.1-1.el9 appstream 48 k 2026-02-12T12:10:38.708 INFO:teuthology.orchestra.run.trial127.stdout: python3-kubernetes noarch 1:26.1.0-3.el9 epel 1.0 M 2026-02-12T12:10:38.708 INFO:teuthology.orchestra.run.trial127.stdout: python3-libstoragemgmt x86_64 1.10.1-1.el9 appstream 177 k 2026-02-12T12:10:38.708 INFO:teuthology.orchestra.run.trial127.stdout: python3-lxml x86_64 4.6.5-3.el9 appstream 1.2 M 2026-02-12T12:10:38.708 INFO:teuthology.orchestra.run.trial127.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2026-02-12T12:10:38.708 INFO:teuthology.orchestra.run.trial127.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2026-02-12T12:10:38.708 INFO:teuthology.orchestra.run.trial127.stdout: python3-msgpack x86_64 1.0.3-2.el9 epel 86 k 2026-02-12T12:10:38.708 INFO:teuthology.orchestra.run.trial127.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2026-02-12T12:10:38.708 INFO:teuthology.orchestra.run.trial127.stdout: python3-numpy x86_64 1:1.23.5-2.el9 appstream 6.1 M 2026-02-12T12:10:38.709 INFO:teuthology.orchestra.run.trial127.stdout: python3-numpy-f2py x86_64 1:1.23.5-2.el9 appstream 442 k 2026-02-12T12:10:38.709 INFO:teuthology.orchestra.run.trial127.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2026-02-12T12:10:38.709 INFO:teuthology.orchestra.run.trial127.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2026-02-12T12:10:38.709 INFO:teuthology.orchestra.run.trial127.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2026-02-12T12:10:38.709 INFO:teuthology.orchestra.run.trial127.stdout: python3-protobuf noarch 3.14.0-17.el9 appstream 267 k 2026-02-12T12:10:38.709 INFO:teuthology.orchestra.run.trial127.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2026-02-12T12:10:38.709 INFO:teuthology.orchestra.run.trial127.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2026-02-12T12:10:38.709 INFO:teuthology.orchestra.run.trial127.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2026-02-12T12:10:38.709 INFO:teuthology.orchestra.run.trial127.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2026-02-12T12:10:38.709 INFO:teuthology.orchestra.run.trial127.stdout: python3-pyparsing noarch 2.4.7-9.el9 baseos 150 k 2026-02-12T12:10:38.709 INFO:teuthology.orchestra.run.trial127.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2026-02-12T12:10:38.709 INFO:teuthology.orchestra.run.trial127.stdout: python3-requests noarch 2.25.1-10.el9 baseos 126 k 2026-02-12T12:10:38.710 INFO:teuthology.orchestra.run.trial127.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2026-02-12T12:10:38.710 INFO:teuthology.orchestra.run.trial127.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2026-02-12T12:10:38.710 INFO:teuthology.orchestra.run.trial127.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2026-02-12T12:10:38.710 INFO:teuthology.orchestra.run.trial127.stdout: python3-scipy x86_64 1.9.3-2.el9 appstream 19 M 2026-02-12T12:10:38.710 INFO:teuthology.orchestra.run.trial127.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2026-02-12T12:10:38.710 INFO:teuthology.orchestra.run.trial127.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2026-02-12T12:10:38.710 INFO:teuthology.orchestra.run.trial127.stdout: python3-typing-extensions noarch 4.15.0-1.el9 epel 86 k 2026-02-12T12:10:38.710 INFO:teuthology.orchestra.run.trial127.stdout: python3-urllib3 noarch 1.26.5-7.el9 baseos 218 k 2026-02-12T12:10:38.710 INFO:teuthology.orchestra.run.trial127.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2026-02-12T12:10:38.710 INFO:teuthology.orchestra.run.trial127.stdout: python3-xmlsec x86_64 1.3.13-1.el9 epel 48 k 2026-02-12T12:10:38.710 INFO:teuthology.orchestra.run.trial127.stdout: python3-xmltodict noarch 0.12.0-15.el9 epel 22 k 2026-02-12T12:10:38.710 INFO:teuthology.orchestra.run.trial127.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2026-02-12T12:10:38.710 INFO:teuthology.orchestra.run.trial127.stdout: unzip x86_64 6.0-59.el9 baseos 182 k 2026-02-12T12:10:38.710 INFO:teuthology.orchestra.run.trial127.stdout: xmlsec1 x86_64 1.2.29-13.el9 appstream 189 k 2026-02-12T12:10:38.711 INFO:teuthology.orchestra.run.trial127.stdout: xmlsec1-openssl x86_64 1.2.29-13.el9 appstream 90 k 2026-02-12T12:10:38.711 INFO:teuthology.orchestra.run.trial127.stdout: zip x86_64 3.0-35.el9 baseos 266 k 2026-02-12T12:10:38.711 INFO:teuthology.orchestra.run.trial127.stdout:Installing weak dependencies: 2026-02-12T12:10:38.711 INFO:teuthology.orchestra.run.trial127.stdout: ceph-mgr-cephadm noarch 2:20.3.0-5145.g5fdb7d8f.el9 ceph-noarch 183 k 2026-02-12T12:10:38.711 INFO:teuthology.orchestra.run.trial127.stdout: ceph-mgr-dashboard noarch 2:20.3.0-5145.g5fdb7d8f.el9 ceph-noarch 15 M 2026-02-12T12:10:38.711 INFO:teuthology.orchestra.run.trial127.stdout: ceph-mgr-diskprediction-local noarch 2:20.3.0-5145.g5fdb7d8f.el9 ceph-noarch 7.4 M 2026-02-12T12:10:38.711 INFO:teuthology.orchestra.run.trial127.stdout: ceph-mgr-k8sevents noarch 2:20.3.0-5145.g5fdb7d8f.el9 ceph-noarch 22 k 2026-02-12T12:10:38.711 INFO:teuthology.orchestra.run.trial127.stdout: ceph-mgr-rook noarch 2:20.3.0-5145.g5fdb7d8f.el9 ceph-noarch 50 k 2026-02-12T12:10:38.711 INFO:teuthology.orchestra.run.trial127.stdout: ceph-volume noarch 2:20.3.0-5145.g5fdb7d8f.el9 ceph-noarch 299 k 2026-02-12T12:10:38.711 INFO:teuthology.orchestra.run.trial127.stdout: python3-influxdb noarch 5.3.1-1.el9 epel 139 k 2026-02-12T12:10:38.711 INFO:teuthology.orchestra.run.trial127.stdout: python3-saml noarch 1.16.0-1.el9 epel 125 k 2026-02-12T12:10:38.712 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:10:38.712 INFO:teuthology.orchestra.run.trial127.stdout:Transaction Summary 2026-02-12T12:10:38.712 INFO:teuthology.orchestra.run.trial127.stdout:======================================================================================= 2026-02-12T12:10:38.712 INFO:teuthology.orchestra.run.trial127.stdout:Install 96 Packages 2026-02-12T12:10:38.712 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:10:38.713 INFO:teuthology.orchestra.run.trial127.stdout:Total download size: 189 M 2026-02-12T12:10:38.713 INFO:teuthology.orchestra.run.trial127.stdout:Installed size: 1.4 G 2026-02-12T12:10:38.713 INFO:teuthology.orchestra.run.trial127.stdout:Downloading Packages: 2026-02-12T12:10:39.576 INFO:teuthology.orchestra.run.trial127.stdout:(1/96): ceph-20.3.0-5145.g5fdb7d8f.el9.x86_64.r 22 kB/s | 6.5 kB 00:00 2026-02-12T12:10:40.952 INFO:teuthology.orchestra.run.trial127.stdout:(2/96): ceph-mgr-20.3.0-5145.g5fdb7d8f.el9.x86_ 562 kB/s | 937 kB 00:01 2026-02-12T12:10:41.360 INFO:teuthology.orchestra.run.trial127.stdout:(3/96): ceph-osd-20.3.0-5145.g5fdb7d8f.el9.x86_ 400 kB/s | 163 kB 00:00 2026-02-12T12:10:41.911 INFO:teuthology.orchestra.run.trial127.stdout:(4/96): ceph-mds-20.3.0-5145.g5fdb7d8f.el9.x86_ 788 kB/s | 2.0 MB 00:02 2026-02-12T12:10:50.477 INFO:teuthology.orchestra.run.trial127.stdout:(5/96): ceph-mon-20.3.0-5145.g5fdb7d8f.el9.x86_ 455 kB/s | 4.8 MB 00:10 2026-02-12T12:10:50.709 INFO:teuthology.orchestra.run.trial127.stdout:(6/96): ceph-grafana-dashboards-20.3.0-5145.g5f 247 kB/s | 57 kB 00:00 2026-02-12T12:10:51.402 INFO:teuthology.orchestra.run.trial127.stdout:(7/96): ceph-mgr-cephadm-20.3.0-5145.g5fdb7d8f. 264 kB/s | 183 kB 00:00 2026-02-12T12:11:24.562 INFO:teuthology.orchestra.run.trial127.stdout:(8/96): ceph-mgr-dashboard-20.3.0-5145.g5fdb7d8 461 kB/s | 15 MB 00:33 2026-02-12T12:11:24.707 INFO:teuthology.orchestra.run.trial127.stdout:(9/96): ceph-osd-classic-20.3.0-5145.g5fdb7d8f. 348 kB/s | 15 MB 00:43 2026-02-12T12:11:24.848 INFO:teuthology.orchestra.run.trial127.stdout:(10/96): ceph-mgr-k8sevents-20.3.0-5145.g5fdb7d 159 kB/s | 22 kB 00:00 2026-02-12T12:11:26.780 INFO:teuthology.orchestra.run.trial127.stdout:(11/96): ceph-mgr-modules-core-20.3.0-5145.g5fd 154 kB/s | 297 kB 00:01 2026-02-12T12:11:27.060 INFO:teuthology.orchestra.run.trial127.stdout:(12/96): ceph-mgr-rook-20.3.0-5145.g5fdb7d8f.el 179 kB/s | 50 kB 00:00 2026-02-12T12:11:27.128 INFO:teuthology.orchestra.run.trial127.stdout:(13/96): ceph-prometheus-alerts-20.3.0-5145.g5f 256 kB/s | 17 kB 00:00 2026-02-12T12:11:28.096 INFO:teuthology.orchestra.run.trial127.stdout:(14/96): ceph-volume-20.3.0-5145.g5fdb7d8f.el9. 309 kB/s | 299 kB 00:00 2026-02-12T12:11:31.219 INFO:teuthology.orchestra.run.trial127.stdout:(15/96): cephadm-20.3.0-5145.g5fdb7d8f.el9.noar 333 kB/s | 1.0 MB 00:03 2026-02-12T12:11:32.222 INFO:teuthology.orchestra.run.trial127.stdout:(16/96): cryptsetup-2.8.1-2.el9.x86_64.rpm 350 kB/s | 350 kB 00:01 2026-02-12T12:11:32.376 INFO:teuthology.orchestra.run.trial127.stdout:(17/96): ledmon-libs-1.1.0-3.el9.x86_64.rpm 263 kB/s | 40 kB 00:00 2026-02-12T12:11:32.606 INFO:teuthology.orchestra.run.trial127.stdout:(18/96): libconfig-1.7.2-9.el9.x86_64.rpm 313 kB/s | 72 kB 00:00 2026-02-12T12:11:33.602 INFO:teuthology.orchestra.run.trial127.stdout:(19/96): libgfortran-11.5.0-14.el9.x86_64.rpm 798 kB/s | 794 kB 00:00 2026-02-12T12:11:33.908 INFO:teuthology.orchestra.run.trial127.stdout:(20/96): libquadmath-11.5.0-14.el9.x86_64.rpm 602 kB/s | 184 kB 00:00 2026-02-12T12:11:34.293 INFO:teuthology.orchestra.run.trial127.stdout:(21/96): python3-cffi-1.14.5-5.el9.x86_64.rpm 658 kB/s | 253 kB 00:00 2026-02-12T12:11:34.908 INFO:teuthology.orchestra.run.trial127.stdout:(22/96): python3-cryptography-36.0.1-5.el9.x86_ 2.0 MB/s | 1.2 MB 00:00 2026-02-12T12:11:35.063 INFO:teuthology.orchestra.run.trial127.stdout:(23/96): python3-ply-3.11-14.el9.noarch.rpm 688 kB/s | 106 kB 00:00 2026-02-12T12:11:35.218 INFO:teuthology.orchestra.run.trial127.stdout:(24/96): python3-pycparser-2.20-6.el9.noarch.rp 873 kB/s | 135 kB 00:00 2026-02-12T12:11:35.373 INFO:teuthology.orchestra.run.trial127.stdout:(25/96): python3-pyparsing-2.4.7-9.el9.noarch.r 971 kB/s | 150 kB 00:00 2026-02-12T12:11:35.528 INFO:teuthology.orchestra.run.trial127.stdout:(26/96): python3-requests-2.25.1-10.el9.noarch. 816 kB/s | 126 kB 00:00 2026-02-12T12:11:35.684 INFO:teuthology.orchestra.run.trial127.stdout:(27/96): python3-urllib3-1.26.5-7.el9.noarch.rp 1.4 MB/s | 218 kB 00:00 2026-02-12T12:11:35.839 INFO:teuthology.orchestra.run.trial127.stdout:(28/96): unzip-6.0-59.el9.x86_64.rpm 1.1 MB/s | 182 kB 00:00 2026-02-12T12:11:35.994 INFO:teuthology.orchestra.run.trial127.stdout:(29/96): zip-3.0-35.el9.x86_64.rpm 1.7 MB/s | 266 kB 00:00 2026-02-12T12:11:36.110 INFO:teuthology.orchestra.run.trial127.stdout:(30/96): flexiblas-3.0.4-8.el9.x86_64.rpm 280 kB/s | 33 kB 00:00 2026-02-12T12:11:36.368 INFO:teuthology.orchestra.run.trial127.stdout:(31/96): flexiblas-netlib-3.0.4-8.el9.x86_64.rp 12 MB/s | 3.0 MB 00:00 2026-02-12T12:11:36.465 INFO:teuthology.orchestra.run.trial127.stdout:(32/96): flexiblas-openblas-openmp-3.0.4-8.el9. 187 kB/s | 18 kB 00:00 2026-02-12T12:11:36.564 INFO:teuthology.orchestra.run.trial127.stdout:(33/96): libstoragemgmt-1.10.1-1.el9.x86_64.rpm 2.4 MB/s | 246 kB 00:00 2026-02-12T12:11:36.662 INFO:teuthology.orchestra.run.trial127.stdout:(34/96): lua-5.4.4-4.el9.x86_64.rpm 1.9 MB/s | 188 kB 00:00 2026-02-12T12:11:36.760 INFO:teuthology.orchestra.run.trial127.stdout:(35/96): openblas-0.3.29-1.el9.x86_64.rpm 432 kB/s | 42 kB 00:00 2026-02-12T12:11:37.175 INFO:teuthology.orchestra.run.trial127.stdout:(36/96): openblas-openmp-0.3.29-1.el9.x86_64.rp 13 MB/s | 5.3 MB 00:00 2026-02-12T12:11:37.594 INFO:teuthology.orchestra.run.trial127.stdout:(37/96): python3-babel-2.9.1-2.el9.noarch.rpm 14 MB/s | 6.0 MB 00:00 2026-02-12T12:11:37.693 INFO:teuthology.orchestra.run.trial127.stdout:(38/96): python3-devel-3.9.25-2.el9.x86_64.rpm 2.4 MB/s | 244 kB 00:00 2026-02-12T12:11:37.792 INFO:teuthology.orchestra.run.trial127.stdout:(39/96): python3-jinja2-2.11.3-8.el9.noarch.rpm 2.5 MB/s | 249 kB 00:00 2026-02-12T12:11:37.889 INFO:teuthology.orchestra.run.trial127.stdout:(40/96): python3-jmespath-1.0.1-1.el9.noarch.rp 487 kB/s | 48 kB 00:00 2026-02-12T12:11:37.988 INFO:teuthology.orchestra.run.trial127.stdout:(41/96): python3-libstoragemgmt-1.10.1-1.el9.x8 1.8 MB/s | 177 kB 00:00 2026-02-12T12:11:38.165 INFO:teuthology.orchestra.run.trial127.stdout:(42/96): python3-lxml-4.6.5-3.el9.x86_64.rpm 6.9 MB/s | 1.2 MB 00:00 2026-02-12T12:11:38.263 INFO:teuthology.orchestra.run.trial127.stdout:(43/96): python3-markupsafe-1.1.1-12.el9.x86_64 356 kB/s | 35 kB 00:00 2026-02-12T12:11:38.756 INFO:teuthology.orchestra.run.trial127.stdout:(44/96): python3-numpy-1.23.5-2.el9.x86_64.rpm 12 MB/s | 6.1 MB 00:00 2026-02-12T12:11:38.856 INFO:teuthology.orchestra.run.trial127.stdout:(45/96): python3-numpy-f2py-1.23.5-2.el9.x86_64 4.3 MB/s | 442 kB 00:00 2026-02-12T12:11:38.954 INFO:teuthology.orchestra.run.trial127.stdout:(46/96): python3-packaging-20.9-5.el9.noarch.rp 787 kB/s | 77 kB 00:00 2026-02-12T12:11:39.053 INFO:teuthology.orchestra.run.trial127.stdout:(47/96): python3-protobuf-3.14.0-17.el9.noarch. 2.6 MB/s | 267 kB 00:00 2026-02-12T12:11:39.151 INFO:teuthology.orchestra.run.trial127.stdout:(48/96): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.6 MB/s | 159 kB 00:00 2026-02-12T12:11:39.250 INFO:teuthology.orchestra.run.trial127.stdout:(49/96): python3-pyasn1-modules-0.4.8-6.el9.noa 2.7 MB/s | 279 kB 00:00 2026-02-12T12:11:39.349 INFO:teuthology.orchestra.run.trial127.stdout:(50/96): python3-requests-oauthlib-1.3.0-12.el9 545 kB/s | 54 kB 00:00 2026-02-12T12:11:40.136 INFO:teuthology.orchestra.run.trial127.stdout:(51/96): ceph-mgr-diskprediction-local-20.3.0-5 488 kB/s | 7.4 MB 00:15 2026-02-12T12:11:40.543 INFO:teuthology.orchestra.run.trial127.stdout:(52/96): python3-toml-0.10.2-6.el9.noarch.rpm 102 kB/s | 42 kB 00:00 2026-02-12T12:11:40.714 INFO:teuthology.orchestra.run.trial127.stdout:(53/96): python3-scipy-1.9.3-2.el9.x86_64.rpm 14 MB/s | 19 MB 00:01 2026-02-12T12:11:40.811 INFO:teuthology.orchestra.run.trial127.stdout:(54/96): xmlsec1-openssl-1.2.29-13.el9.x86_64.r 925 kB/s | 90 kB 00:00 2026-02-12T12:11:40.822 INFO:teuthology.orchestra.run.trial127.stdout:(55/96): lua-devel-5.4.4-4.el9.x86_64.rpm 2.0 MB/s | 22 kB 00:00 2026-02-12T12:11:40.842 INFO:teuthology.orchestra.run.trial127.stdout:(56/96): protobuf-compiler-3.14.0-17.el9.x86_64 44 MB/s | 862 kB 00:00 2026-02-12T12:11:40.867 INFO:teuthology.orchestra.run.trial127.stdout:(57/96): xmlsec1-1.2.29-13.el9.x86_64.rpm 583 kB/s | 189 kB 00:00 2026-02-12T12:11:40.978 INFO:teuthology.orchestra.run.trial127.stdout:(58/96): grpc-data-1.46.7-10.el9.noarch.rpm 176 kB/s | 19 kB 00:00 2026-02-12T12:11:41.062 INFO:teuthology.orchestra.run.trial127.stdout:(59/96): abseil-cpp-20211102.0-4.el9.x86_64.rpm 2.4 MB/s | 551 kB 00:00 2026-02-12T12:11:41.069 INFO:teuthology.orchestra.run.trial127.stdout:(60/96): luarocks-3.9.2-5.el9.noarch.rpm 1.6 MB/s | 151 kB 00:00 2026-02-12T12:11:41.097 INFO:teuthology.orchestra.run.trial127.stdout:(61/96): python3-autocommand-2.2.2-8.el9.noarch 1.0 MB/s | 29 kB 00:00 2026-02-12T12:11:41.105 INFO:teuthology.orchestra.run.trial127.stdout:(62/96): python3-asyncssh-2.13.2-5.el9.noarch.r 12 MB/s | 548 kB 00:00 2026-02-12T12:11:41.130 INFO:teuthology.orchestra.run.trial127.stdout:(63/96): python3-backports-tarfile-1.2.0-1.el9. 1.8 MB/s | 60 kB 00:00 2026-02-12T12:11:41.131 INFO:teuthology.orchestra.run.trial127.stdout:(64/96): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.6 MB/s | 43 kB 00:00 2026-02-12T12:11:41.157 INFO:teuthology.orchestra.run.trial127.stdout:(65/96): python3-certifi-2023.05.07-4.el9.noarc 545 kB/s | 14 kB 00:00 2026-02-12T12:11:41.159 INFO:teuthology.orchestra.run.trial127.stdout:(66/96): python3-cachetools-4.2.4-1.el9.noarch. 1.1 MB/s | 32 kB 00:00 2026-02-12T12:11:41.188 INFO:teuthology.orchestra.run.trial127.stdout:(67/96): python3-cheroot-10.0.1-4.el9.noarch.rp 5.6 MB/s | 173 kB 00:00 2026-02-12T12:11:41.221 INFO:teuthology.orchestra.run.trial127.stdout:(68/96): python3-google-auth-2.45.0-1.el9.noarc 7.6 MB/s | 254 kB 00:00 2026-02-12T12:11:41.223 INFO:teuthology.orchestra.run.trial127.stdout:(69/96): python3-cherrypy-18.6.1-2.el9.noarch.r 5.5 MB/s | 358 kB 00:00 2026-02-12T12:11:41.254 INFO:teuthology.orchestra.run.trial127.stdout:(70/96): python3-grpcio-tools-1.46.7-10.el9.x86 4.5 MB/s | 144 kB 00:00 2026-02-12T12:11:41.286 INFO:teuthology.orchestra.run.trial127.stdout:(71/96): python3-influxdb-5.3.1-1.el9.noarch.rp 4.3 MB/s | 139 kB 00:00 2026-02-12T12:11:41.300 INFO:teuthology.orchestra.run.trial127.stdout:(72/96): python3-grpcio-1.46.7-10.el9.x86_64.rp 26 MB/s | 2.0 MB 00:00 2026-02-12T12:11:41.314 INFO:teuthology.orchestra.run.trial127.stdout:(73/96): python3-isodate-0.6.1-3.el9.noarch.rpm 1.9 MB/s | 56 kB 00:00 2026-02-12T12:11:41.326 INFO:teuthology.orchestra.run.trial127.stdout:(74/96): python3-jaraco-8.2.1-3.el9.noarch.rpm 412 kB/s | 11 kB 00:00 2026-02-12T12:11:41.341 INFO:teuthology.orchestra.run.trial127.stdout:(75/96): python3-jaraco-classes-3.2.1-5.el9.noa 662 kB/s | 18 kB 00:00 2026-02-12T12:11:41.353 INFO:teuthology.orchestra.run.trial127.stdout:(76/96): python3-jaraco-collections-3.0.0-8.el9 888 kB/s | 23 kB 00:00 2026-02-12T12:11:41.368 INFO:teuthology.orchestra.run.trial127.stdout:(77/96): python3-jaraco-context-6.0.1-3.el9.noa 734 kB/s | 20 kB 00:00 2026-02-12T12:11:41.379 INFO:teuthology.orchestra.run.trial127.stdout:(78/96): python3-jaraco-functools-3.5.0-2.el9.n 744 kB/s | 19 kB 00:00 2026-02-12T12:11:41.395 INFO:teuthology.orchestra.run.trial127.stdout:(79/96): python3-jaraco-text-4.0.0-2.el9.noarch 981 kB/s | 26 kB 00:00 2026-02-12T12:11:41.414 INFO:teuthology.orchestra.run.trial127.stdout:(80/96): python3-kubernetes-26.1.0-3.el9.noarch 29 MB/s | 1.0 MB 00:00 2026-02-12T12:11:41.424 INFO:teuthology.orchestra.run.trial127.stdout:(81/96): python3-more-itertools-8.12.0-2.el9.no 2.7 MB/s | 79 kB 00:00 2026-02-12T12:11:41.441 INFO:teuthology.orchestra.run.trial127.stdout:(82/96): python3-msgpack-1.0.3-2.el9.x86_64.rpm 3.2 MB/s | 86 kB 00:00 2026-02-12T12:11:41.451 INFO:teuthology.orchestra.run.trial127.stdout:(83/96): python3-natsort-7.1.1-5.el9.noarch.rpm 2.0 MB/s | 58 kB 00:00 2026-02-12T12:11:41.467 INFO:teuthology.orchestra.run.trial127.stdout:(84/96): python3-portend-3.1.0-2.el9.noarch.rpm 629 kB/s | 16 kB 00:00 2026-02-12T12:11:41.481 INFO:teuthology.orchestra.run.trial127.stdout:(85/96): python3-pyOpenSSL-21.0.0-1.el9.noarch. 3.0 MB/s | 90 kB 00:00 2026-02-12T12:11:41.494 INFO:teuthology.orchestra.run.trial127.stdout:(86/96): python3-repoze-lru-0.7-16.el9.noarch.r 1.1 MB/s | 31 kB 00:00 2026-02-12T12:11:41.513 INFO:teuthology.orchestra.run.trial127.stdout:(87/96): python3-routes-2.5.1-5.el9.noarch.rpm 5.6 MB/s | 188 kB 00:00 2026-02-12T12:11:41.520 INFO:teuthology.orchestra.run.trial127.stdout:(88/96): python3-rsa-4.9-2.el9.noarch.rpm 2.2 MB/s | 59 kB 00:00 2026-02-12T12:11:41.544 INFO:teuthology.orchestra.run.trial127.stdout:(89/96): python3-saml-1.16.0-1.el9.noarch.rpm 4.0 MB/s | 125 kB 00:00 2026-02-12T12:11:41.546 INFO:teuthology.orchestra.run.trial127.stdout:(90/96): python3-tempora-5.0.0-2.el9.noarch.rpm 1.4 MB/s | 36 kB 00:00 2026-02-12T12:11:41.573 INFO:teuthology.orchestra.run.trial127.stdout:(91/96): python3-typing-extensions-4.15.0-1.el9 2.9 MB/s | 86 kB 00:00 2026-02-12T12:11:41.573 INFO:teuthology.orchestra.run.trial127.stdout:(92/96): python3-websocket-client-1.2.3-2.el9.n 3.2 MB/s | 90 kB 00:00 2026-02-12T12:11:41.600 INFO:teuthology.orchestra.run.trial127.stdout:(93/96): python3-xmltodict-0.12.0-15.el9.noarch 838 kB/s | 22 kB 00:00 2026-02-12T12:11:41.601 INFO:teuthology.orchestra.run.trial127.stdout:(94/96): python3-xmlsec-1.3.13-1.el9.x86_64.rpm 1.7 MB/s | 48 kB 00:00 2026-02-12T12:11:41.628 INFO:teuthology.orchestra.run.trial127.stdout:(95/96): python3-zc-lockfile-2.0-10.el9.noarch. 713 kB/s | 20 kB 00:00 2026-02-12T12:15:00.203 INFO:teuthology.orchestra.run.trial127.stdout:(96/96): ceph-osd-crimson-20.3.0-5145.g5fdb7d8f 344 kB/s | 87 MB 04:18 2026-02-12T12:15:00.207 INFO:teuthology.orchestra.run.trial127.stdout:-------------------------------------------------------------------------------- 2026-02-12T12:15:00.207 INFO:teuthology.orchestra.run.trial127.stdout:Total 740 kB/s | 189 MB 04:21 2026-02-12T12:15:00.571 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction check 2026-02-12T12:15:00.612 INFO:teuthology.orchestra.run.trial127.stdout:Transaction check succeeded. 2026-02-12T12:15:00.612 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction test 2026-02-12T12:15:01.305 INFO:teuthology.orchestra.run.trial127.stdout:Transaction test succeeded. 2026-02-12T12:15:01.305 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction 2026-02-12T12:15:02.026 INFO:teuthology.orchestra.run.trial127.stdout: Preparing : 1/1 2026-02-12T12:15:02.028 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/96 2026-02-12T12:15:02.032 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/96 2026-02-12T12:15:02.053 INFO:teuthology.orchestra.run.trial127.stdout: Installing : xmlsec1-1.2.29-13.el9.x86_64 3/96 2026-02-12T12:15:02.086 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 4/96 2026-02-12T12:15:02.093 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-lxml-4.6.5-3.el9.x86_64 5/96 2026-02-12T12:15:02.096 INFO:teuthology.orchestra.run.trial127.stdout: Installing : lua-5.4.4-4.el9.x86_64 6/96 2026-02-12T12:15:02.100 INFO:teuthology.orchestra.run.trial127.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 7/96 2026-02-12T12:15:02.123 INFO:teuthology.orchestra.run.trial127.stdout: Installing : unzip-6.0-59.el9.x86_64 8/96 2026-02-12T12:15:02.136 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-urllib3-1.26.5-7.el9.noarch 9/96 2026-02-12T12:15:02.139 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-requests-2.25.1-10.el9.noarch 10/96 2026-02-12T12:15:02.145 INFO:teuthology.orchestra.run.trial127.stdout: Installing : libquadmath-11.5.0-14.el9.x86_64 11/96 2026-02-12T12:15:02.147 INFO:teuthology.orchestra.run.trial127.stdout: Installing : libgfortran-11.5.0-14.el9.x86_64 12/96 2026-02-12T12:15:02.158 INFO:teuthology.orchestra.run.trial127.stdout: Installing : ledmon-libs-1.1.0-3.el9.x86_64 13/96 2026-02-12T12:15:02.164 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 14/96 2026-02-12T12:15:02.188 INFO:teuthology.orchestra.run.trial127.stdout: Installing : zip-3.0-35.el9.x86_64 15/96 2026-02-12T12:15:02.191 INFO:teuthology.orchestra.run.trial127.stdout: Installing : luarocks-3.9.2-5.el9.noarch 16/96 2026-02-12T12:15:02.241 INFO:teuthology.orchestra.run.trial127.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 17/96 2026-02-12T12:15:02.254 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 18/96 2026-02-12T12:15:02.257 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-rsa-4.9-2.el9.noarch 19/96 2026-02-12T12:15:02.261 INFO:teuthology.orchestra.run.trial127.stdout: Installing : xmlsec1-openssl-1.2.29-13.el9.x86_64 20/96 2026-02-12T12:15:02.267 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-xmlsec-1.3.13-1.el9.x86_64 21/96 2026-02-12T12:15:02.271 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 22/96 2026-02-12T12:15:02.274 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 23/96 2026-02-12T12:15:02.289 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-xmltodict-0.12.0-15.el9.noarch 24/96 2026-02-12T12:15:02.293 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 25/96 2026-02-12T12:15:02.298 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-typing-extensions-4.15.0-1.el9.noarch 26/96 2026-02-12T12:15:02.309 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 27/96 2026-02-12T12:15:02.318 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 28/96 2026-02-12T12:15:02.325 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 29/96 2026-02-12T12:15:02.344 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-msgpack-1.0.3-2.el9.x86_64 30/96 2026-02-12T12:15:02.355 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-influxdb-5.3.1-1.el9.noarch 31/96 2026-02-12T12:15:02.370 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-isodate-0.6.1-3.el9.noarch 32/96 2026-02-12T12:15:02.375 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-saml-1.16.0-1.el9.noarch 33/96 2026-02-12T12:15:02.383 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 34/96 2026-02-12T12:15:02.423 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 35/96 2026-02-12T12:15:02.746 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-google-auth-1:2.45.0-1.el9.noarch 36/96 2026-02-12T12:15:02.756 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-kubernetes-1:26.1.0-3.el9.noarch 37/96 2026-02-12T12:15:02.760 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-backports-tarfile-1.2.0-1.el9.noarch 38/96 2026-02-12T12:15:02.766 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-jaraco-context-6.0.1-3.el9.noarch 39/96 2026-02-12T12:15:02.768 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-autocommand-2.2.2-8.el9.noarch 40/96 2026-02-12T12:15:02.794 INFO:teuthology.orchestra.run.trial127.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 41/96 2026-02-12T12:15:02.838 INFO:teuthology.orchestra.run.trial127.stdout: Installing : abseil-cpp-20211102.0-4.el9.x86_64 42/96 2026-02-12T12:15:02.846 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 43/96 2026-02-12T12:15:02.852 INFO:teuthology.orchestra.run.trial127.stdout: Installing : protobuf-compiler-3.14.0-17.el9.x86_64 44/96 2026-02-12T12:15:02.855 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 45/96 2026-02-12T12:15:02.861 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 46/96 2026-02-12T12:15:02.864 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-jaraco-text-4.0.0-2.el9.noarch 47/96 2026-02-12T12:15:02.871 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 48/96 2026-02-12T12:15:02.875 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 49/96 2026-02-12T12:15:02.903 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 50/96 2026-02-12T12:15:02.912 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-protobuf-3.14.0-17.el9.noarch 51/96 2026-02-12T12:15:02.918 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 52/96 2026-02-12T12:15:02.925 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 53/96 2026-02-12T12:15:02.960 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-jmespath-1.0.1-1.el9.noarch 54/96 2026-02-12T12:15:03.184 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-devel-3.9.25-2.el9.x86_64 55/96 2026-02-12T12:15:03.207 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 56/96 2026-02-12T12:15:03.211 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-jinja2-2.11.3-8.el9.noarch 57/96 2026-02-12T12:15:03.270 INFO:teuthology.orchestra.run.trial127.stdout: Installing : openblas-0.3.29-1.el9.x86_64 58/96 2026-02-12T12:15:03.272 INFO:teuthology.orchestra.run.trial127.stdout: Installing : openblas-openmp-0.3.29-1.el9.x86_64 59/96 2026-02-12T12:15:03.293 INFO:teuthology.orchestra.run.trial127.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 60/96 2026-02-12T12:15:03.617 INFO:teuthology.orchestra.run.trial127.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 61/96 2026-02-12T12:15:03.690 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-numpy-1:1.23.5-2.el9.x86_64 62/96 2026-02-12T12:15:04.358 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 63/96 2026-02-12T12:15:04.375 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-scipy-1.9.3-2.el9.x86_64 64/96 2026-02-12T12:15:04.385 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-pyparsing-2.4.7-9.el9.noarch 65/96 2026-02-12T12:15:04.392 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-packaging-20.9-5.el9.noarch 66/96 2026-02-12T12:15:04.406 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-ply-3.11-14.el9.noarch 67/96 2026-02-12T12:15:04.421 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 68/96 2026-02-12T12:15:04.497 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 69/96 2026-02-12T12:15:04.506 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-cryptography-36.0.1-5.el9.x86_64 70/96 2026-02-12T12:15:04.529 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 71/96 2026-02-12T12:15:04.558 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-cheroot-10.0.1-4.el9.noarch 72/96 2026-02-12T12:15:04.609 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 73/96 2026-02-12T12:15:04.615 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 74/96 2026-02-12T12:15:04.620 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 75/96 2026-02-12T12:15:04.621 INFO:teuthology.orchestra.run.trial127.stdout: Installing : libconfig-1.7.2-9.el9.x86_64 76/96 2026-02-12T12:15:04.640 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-12T12:15:04.640 INFO:teuthology.orchestra.run.trial127.stdout:Creating group 'libstoragemgmt' with GID 991. 2026-02-12T12:15:04.640 INFO:teuthology.orchestra.run.trial127.stdout:Creating user 'libstoragemgmt' (daemon account for libstoragemgmt) with UID 991 and GID 991. 2026-02-12T12:15:04.640 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:15:04.649 INFO:teuthology.orchestra.run.trial127.stdout: Installing : libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-12T12:15:04.673 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-12T12:15:04.674 INFO:teuthology.orchestra.run.trial127.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/libstoragemgmt.service → /usr/lib/systemd/system/libstoragemgmt.service. 2026-02-12T12:15:04.674 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:15:04.709 INFO:teuthology.orchestra.run.trial127.stdout: Installing : python3-libstoragemgmt-1.10.1-1.el9.x86_64 78/96 2026-02-12T12:15:05.312 INFO:teuthology.orchestra.run.trial127.stdout: Installing : cryptsetup-2.8.1-2.el9.x86_64 79/96 2026-02-12T12:15:10.762 INFO:teuthology.orchestra.run.trial127.stdout: Installing : ceph-osd-classic-2:20.3.0-5145.g5fdb7d8f.el9.x86_6 80/96 2026-02-12T12:15:10.825 INFO:teuthology.orchestra.run.trial127.stdout: Installing : ceph-osd-crimson-2:20.3.0-5145.g5fdb7d8f.el9.x86_6 81/96 2026-02-12T12:15:10.827 INFO:teuthology.orchestra.run.trial127.stdout: Installing : ceph-volume-2:20.3.0-5145.g5fdb7d8f.el9.noarch 82/96 2026-02-12T12:15:10.841 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: ceph-volume-2:20.3.0-5145.g5fdb7d8f.el9.noarch 82/96 2026-02-12T12:15:10.841 INFO:teuthology.orchestra.run.trial127.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-12T12:15:10.841 INFO:teuthology.orchestra.run.trial127.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2026-02-12T12:15:10.841 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:15:10.853 INFO:teuthology.orchestra.run.trial127.stdout: Installing : ceph-osd-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 83/96 2026-02-12T12:15:10.876 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: ceph-osd-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 83/96 2026-02-12T12:15:10.876 INFO:teuthology.orchestra.run.trial127.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-12T12:15:10.876 INFO:teuthology.orchestra.run.trial127.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2026-02-12T12:15:10.876 INFO:teuthology.orchestra.run.trial127.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-12T12:15:10.877 INFO:teuthology.orchestra.run.trial127.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-12T12:15:10.877 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:15:10.936 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: cephadm-2:20.3.0-5145.g5fdb7d8f.el9.noarch 84/96 2026-02-12T12:15:10.938 INFO:teuthology.orchestra.run.trial127.stdout: Installing : cephadm-2:20.3.0-5145.g5fdb7d8f.el9.noarch 84/96 2026-02-12T12:15:10.947 INFO:teuthology.orchestra.run.trial127.stdout: Installing : ceph-prometheus-alerts-2:20.3.0-5145.g5fdb7d8f.el9 85/96 2026-02-12T12:15:10.971 INFO:teuthology.orchestra.run.trial127.stdout: Installing : ceph-grafana-dashboards-2:20.3.0-5145.g5fdb7d8f.el 86/96 2026-02-12T12:15:10.973 INFO:teuthology.orchestra.run.trial127.stdout: Installing : ceph-mgr-cephadm-2:20.3.0-5145.g5fdb7d8f.el9.noarc 87/96 2026-02-12T12:15:12.559 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: ceph-mgr-cephadm-2:20.3.0-5145.g5fdb7d8f.el9.noarc 87/96 2026-02-12T12:15:12.566 INFO:teuthology.orchestra.run.trial127.stdout: Installing : ceph-mgr-dashboard-2:20.3.0-5145.g5fdb7d8f.el9.noa 88/96 2026-02-12T12:15:13.027 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: ceph-mgr-dashboard-2:20.3.0-5145.g5fdb7d8f.el9.noa 88/96 2026-02-12T12:15:13.028 INFO:teuthology.orchestra.run.trial127.stdout: Installing : ceph-mgr-diskprediction-local-2:20.3.0-5145.g5fdb7 89/96 2026-02-12T12:15:13.040 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:20.3.0-5145.g5fdb7 89/96 2026-02-12T12:15:13.040 INFO:teuthology.orchestra.run.trial127.stdout: Installing : ceph-mgr-k8sevents-2:20.3.0-5145.g5fdb7d8f.el9.noa 90/96 2026-02-12T12:15:13.102 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: ceph-mgr-k8sevents-2:20.3.0-5145.g5fdb7d8f.el9.noa 90/96 2026-02-12T12:15:13.143 INFO:teuthology.orchestra.run.trial127.stdout: Installing : ceph-mgr-modules-core-2:20.3.0-5145.g5fdb7d8f.el9. 91/96 2026-02-12T12:15:13.144 INFO:teuthology.orchestra.run.trial127.stdout: Installing : ceph-mgr-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 92/96 2026-02-12T12:15:13.165 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: ceph-mgr-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 92/96 2026-02-12T12:15:13.166 INFO:teuthology.orchestra.run.trial127.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-12T12:15:13.166 INFO:teuthology.orchestra.run.trial127.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2026-02-12T12:15:13.166 INFO:teuthology.orchestra.run.trial127.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-12T12:15:13.166 INFO:teuthology.orchestra.run.trial127.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-12T12:15:13.166 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:15:13.177 INFO:teuthology.orchestra.run.trial127.stdout: Installing : ceph-mgr-rook-2:20.3.0-5145.g5fdb7d8f.el9.noarch 93/96 2026-02-12T12:15:13.397 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: ceph-mgr-rook-2:20.3.0-5145.g5fdb7d8f.el9.noarch 93/96 2026-02-12T12:15:13.399 INFO:teuthology.orchestra.run.trial127.stdout: Installing : ceph-mon-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 94/96 2026-02-12T12:15:13.419 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: ceph-mon-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 94/96 2026-02-12T12:15:13.419 INFO:teuthology.orchestra.run.trial127.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-12T12:15:13.419 INFO:teuthology.orchestra.run.trial127.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2026-02-12T12:15:13.419 INFO:teuthology.orchestra.run.trial127.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-12T12:15:13.420 INFO:teuthology.orchestra.run.trial127.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-12T12:15:13.420 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:15:13.504 INFO:teuthology.orchestra.run.trial127.stdout: Installing : ceph-mds-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 95/96 2026-02-12T12:15:13.524 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: ceph-mds-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 95/96 2026-02-12T12:15:13.524 INFO:teuthology.orchestra.run.trial127.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-12T12:15:13.524 INFO:teuthology.orchestra.run.trial127.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2026-02-12T12:15:13.524 INFO:teuthology.orchestra.run.trial127.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-12T12:15:13.524 INFO:teuthology.orchestra.run.trial127.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-12T12:15:13.524 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:15:13.525 INFO:teuthology.orchestra.run.trial127.stdout: Installing : ceph-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 96/96 2026-02-12T12:15:13.531 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: ceph-osd-classic-2:20.3.0-5145.g5fdb7d8f.el9.x86_6 96/96 2026-02-12T12:15:13.554 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: ceph-osd-crimson-2:20.3.0-5145.g5fdb7d8f.el9.x86_6 96/96 2026-02-12T12:15:14.842 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: ceph-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 96/96 2026-02-12T12:15:14.842 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : ceph-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 1/96 2026-02-12T12:15:14.842 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : ceph-mds-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2/96 2026-02-12T12:15:14.843 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : ceph-mgr-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 3/96 2026-02-12T12:15:14.843 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : ceph-mon-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 4/96 2026-02-12T12:15:14.843 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : ceph-osd-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 5/96 2026-02-12T12:15:14.843 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : ceph-osd-classic-2:20.3.0-5145.g5fdb7d8f.el9.x86_6 6/96 2026-02-12T12:15:14.843 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : ceph-osd-crimson-2:20.3.0-5145.g5fdb7d8f.el9.x86_6 7/96 2026-02-12T12:15:14.843 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : ceph-grafana-dashboards-2:20.3.0-5145.g5fdb7d8f.el 8/96 2026-02-12T12:15:14.843 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : ceph-mgr-cephadm-2:20.3.0-5145.g5fdb7d8f.el9.noarc 9/96 2026-02-12T12:15:14.843 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : ceph-mgr-dashboard-2:20.3.0-5145.g5fdb7d8f.el9.noa 10/96 2026-02-12T12:15:14.843 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : ceph-mgr-diskprediction-local-2:20.3.0-5145.g5fdb7 11/96 2026-02-12T12:15:14.843 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : ceph-mgr-k8sevents-2:20.3.0-5145.g5fdb7d8f.el9.noa 12/96 2026-02-12T12:15:14.843 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : ceph-mgr-modules-core-2:20.3.0-5145.g5fdb7d8f.el9. 13/96 2026-02-12T12:15:14.843 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : ceph-mgr-rook-2:20.3.0-5145.g5fdb7d8f.el9.noarch 14/96 2026-02-12T12:15:14.844 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : ceph-prometheus-alerts-2:20.3.0-5145.g5fdb7d8f.el9 15/96 2026-02-12T12:15:14.844 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : ceph-volume-2:20.3.0-5145.g5fdb7d8f.el9.noarch 16/96 2026-02-12T12:15:14.844 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : cephadm-2:20.3.0-5145.g5fdb7d8f.el9.noarch 17/96 2026-02-12T12:15:14.844 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : cryptsetup-2.8.1-2.el9.x86_64 18/96 2026-02-12T12:15:14.844 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : ledmon-libs-1.1.0-3.el9.x86_64 19/96 2026-02-12T12:15:14.844 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : libconfig-1.7.2-9.el9.x86_64 20/96 2026-02-12T12:15:14.844 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : libgfortran-11.5.0-14.el9.x86_64 21/96 2026-02-12T12:15:14.844 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : libquadmath-11.5.0-14.el9.x86_64 22/96 2026-02-12T12:15:14.844 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 23/96 2026-02-12T12:15:14.844 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-cryptography-36.0.1-5.el9.x86_64 24/96 2026-02-12T12:15:14.844 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-ply-3.11-14.el9.noarch 25/96 2026-02-12T12:15:14.844 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 26/96 2026-02-12T12:15:14.845 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-pyparsing-2.4.7-9.el9.noarch 27/96 2026-02-12T12:15:14.845 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-requests-2.25.1-10.el9.noarch 28/96 2026-02-12T12:15:14.845 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-urllib3-1.26.5-7.el9.noarch 29/96 2026-02-12T12:15:14.845 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : unzip-6.0-59.el9.x86_64 30/96 2026-02-12T12:15:14.845 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : zip-3.0-35.el9.x86_64 31/96 2026-02-12T12:15:14.845 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 32/96 2026-02-12T12:15:14.845 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 33/96 2026-02-12T12:15:14.845 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 34/96 2026-02-12T12:15:14.845 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : libstoragemgmt-1.10.1-1.el9.x86_64 35/96 2026-02-12T12:15:14.845 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : lua-5.4.4-4.el9.x86_64 36/96 2026-02-12T12:15:14.845 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : openblas-0.3.29-1.el9.x86_64 37/96 2026-02-12T12:15:14.845 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : openblas-openmp-0.3.29-1.el9.x86_64 38/96 2026-02-12T12:15:14.846 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 39/96 2026-02-12T12:15:14.846 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-devel-3.9.25-2.el9.x86_64 40/96 2026-02-12T12:15:14.846 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-jinja2-2.11.3-8.el9.noarch 41/96 2026-02-12T12:15:14.846 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-jmespath-1.0.1-1.el9.noarch 42/96 2026-02-12T12:15:14.846 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-libstoragemgmt-1.10.1-1.el9.x86_64 43/96 2026-02-12T12:15:14.846 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-lxml-4.6.5-3.el9.x86_64 44/96 2026-02-12T12:15:14.846 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 45/96 2026-02-12T12:15:14.846 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-numpy-1:1.23.5-2.el9.x86_64 46/96 2026-02-12T12:15:14.846 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 47/96 2026-02-12T12:15:14.846 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 48/96 2026-02-12T12:15:14.846 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-protobuf-3.14.0-17.el9.noarch 49/96 2026-02-12T12:15:14.846 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 50/96 2026-02-12T12:15:14.846 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 51/96 2026-02-12T12:15:14.847 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 52/96 2026-02-12T12:15:14.847 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-scipy-1.9.3-2.el9.x86_64 53/96 2026-02-12T12:15:14.847 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 54/96 2026-02-12T12:15:14.847 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : xmlsec1-1.2.29-13.el9.x86_64 55/96 2026-02-12T12:15:14.847 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : xmlsec1-openssl-1.2.29-13.el9.x86_64 56/96 2026-02-12T12:15:14.847 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 57/96 2026-02-12T12:15:14.847 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : protobuf-compiler-3.14.0-17.el9.x86_64 58/96 2026-02-12T12:15:14.847 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : abseil-cpp-20211102.0-4.el9.x86_64 59/96 2026-02-12T12:15:14.847 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 60/96 2026-02-12T12:15:14.847 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 61/96 2026-02-12T12:15:14.847 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 62/96 2026-02-12T12:15:14.847 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-autocommand-2.2.2-8.el9.noarch 63/96 2026-02-12T12:15:14.847 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-backports-tarfile-1.2.0-1.el9.noarch 64/96 2026-02-12T12:15:14.848 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 65/96 2026-02-12T12:15:14.848 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 66/96 2026-02-12T12:15:14.848 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 67/96 2026-02-12T12:15:14.848 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-cheroot-10.0.1-4.el9.noarch 68/96 2026-02-12T12:15:14.848 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 69/96 2026-02-12T12:15:14.848 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-google-auth-1:2.45.0-1.el9.noarch 70/96 2026-02-12T12:15:14.848 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 71/96 2026-02-12T12:15:14.848 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 72/96 2026-02-12T12:15:14.848 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-influxdb-5.3.1-1.el9.noarch 73/96 2026-02-12T12:15:14.848 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-isodate-0.6.1-3.el9.noarch 74/96 2026-02-12T12:15:14.848 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 75/96 2026-02-12T12:15:14.848 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 76/96 2026-02-12T12:15:14.848 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 77/96 2026-02-12T12:15:14.849 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-jaraco-context-6.0.1-3.el9.noarch 78/96 2026-02-12T12:15:14.849 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 79/96 2026-02-12T12:15:14.849 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-jaraco-text-4.0.0-2.el9.noarch 80/96 2026-02-12T12:15:14.849 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-kubernetes-1:26.1.0-3.el9.noarch 81/96 2026-02-12T12:15:14.849 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 82/96 2026-02-12T12:15:14.849 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-msgpack-1.0.3-2.el9.x86_64 83/96 2026-02-12T12:15:14.849 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 84/96 2026-02-12T12:15:14.849 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 85/96 2026-02-12T12:15:14.849 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/96 2026-02-12T12:15:14.849 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 87/96 2026-02-12T12:15:14.849 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 88/96 2026-02-12T12:15:14.849 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 89/96 2026-02-12T12:15:14.849 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-saml-1.16.0-1.el9.noarch 90/96 2026-02-12T12:15:14.850 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 91/96 2026-02-12T12:15:14.850 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-typing-extensions-4.15.0-1.el9.noarch 92/96 2026-02-12T12:15:14.850 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 93/96 2026-02-12T12:15:14.850 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-xmlsec-1.3.13-1.el9.x86_64 94/96 2026-02-12T12:15:14.850 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-xmltodict-0.12.0-15.el9.noarch 95/96 2026-02-12T12:15:14.906 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 96/96 2026-02-12T12:15:14.906 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:15:14.907 INFO:teuthology.orchestra.run.trial127.stdout:Installed: 2026-02-12T12:15:14.907 INFO:teuthology.orchestra.run.trial127.stdout: abseil-cpp-20211102.0-4.el9.x86_64 2026-02-12T12:15:14.907 INFO:teuthology.orchestra.run.trial127.stdout: ceph-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:15:14.907 INFO:teuthology.orchestra.run.trial127.stdout: ceph-grafana-dashboards-2:20.3.0-5145.g5fdb7d8f.el9.noarch 2026-02-12T12:15:14.907 INFO:teuthology.orchestra.run.trial127.stdout: ceph-mds-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:15:14.907 INFO:teuthology.orchestra.run.trial127.stdout: ceph-mgr-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:15:14.907 INFO:teuthology.orchestra.run.trial127.stdout: ceph-mgr-cephadm-2:20.3.0-5145.g5fdb7d8f.el9.noarch 2026-02-12T12:15:14.907 INFO:teuthology.orchestra.run.trial127.stdout: ceph-mgr-dashboard-2:20.3.0-5145.g5fdb7d8f.el9.noarch 2026-02-12T12:15:14.907 INFO:teuthology.orchestra.run.trial127.stdout: ceph-mgr-diskprediction-local-2:20.3.0-5145.g5fdb7d8f.el9.noarch 2026-02-12T12:15:14.907 INFO:teuthology.orchestra.run.trial127.stdout: ceph-mgr-k8sevents-2:20.3.0-5145.g5fdb7d8f.el9.noarch 2026-02-12T12:15:14.907 INFO:teuthology.orchestra.run.trial127.stdout: ceph-mgr-modules-core-2:20.3.0-5145.g5fdb7d8f.el9.noarch 2026-02-12T12:15:14.908 INFO:teuthology.orchestra.run.trial127.stdout: ceph-mgr-rook-2:20.3.0-5145.g5fdb7d8f.el9.noarch 2026-02-12T12:15:14.908 INFO:teuthology.orchestra.run.trial127.stdout: ceph-mon-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:15:14.908 INFO:teuthology.orchestra.run.trial127.stdout: ceph-osd-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:15:14.908 INFO:teuthology.orchestra.run.trial127.stdout: ceph-osd-classic-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:15:14.908 INFO:teuthology.orchestra.run.trial127.stdout: ceph-osd-crimson-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:15:14.908 INFO:teuthology.orchestra.run.trial127.stdout: ceph-prometheus-alerts-2:20.3.0-5145.g5fdb7d8f.el9.noarch 2026-02-12T12:15:14.908 INFO:teuthology.orchestra.run.trial127.stdout: ceph-volume-2:20.3.0-5145.g5fdb7d8f.el9.noarch 2026-02-12T12:15:14.908 INFO:teuthology.orchestra.run.trial127.stdout: cephadm-2:20.3.0-5145.g5fdb7d8f.el9.noarch 2026-02-12T12:15:14.908 INFO:teuthology.orchestra.run.trial127.stdout: cryptsetup-2.8.1-2.el9.x86_64 2026-02-12T12:15:14.908 INFO:teuthology.orchestra.run.trial127.stdout: flexiblas-3.0.4-8.el9.x86_64 2026-02-12T12:15:14.908 INFO:teuthology.orchestra.run.trial127.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2026-02-12T12:15:14.908 INFO:teuthology.orchestra.run.trial127.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2026-02-12T12:15:14.908 INFO:teuthology.orchestra.run.trial127.stdout: grpc-data-1.46.7-10.el9.noarch 2026-02-12T12:15:14.909 INFO:teuthology.orchestra.run.trial127.stdout: ledmon-libs-1.1.0-3.el9.x86_64 2026-02-12T12:15:14.909 INFO:teuthology.orchestra.run.trial127.stdout: libconfig-1.7.2-9.el9.x86_64 2026-02-12T12:15:14.909 INFO:teuthology.orchestra.run.trial127.stdout: libgfortran-11.5.0-14.el9.x86_64 2026-02-12T12:15:14.909 INFO:teuthology.orchestra.run.trial127.stdout: libquadmath-11.5.0-14.el9.x86_64 2026-02-12T12:15:14.909 INFO:teuthology.orchestra.run.trial127.stdout: libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-12T12:15:14.909 INFO:teuthology.orchestra.run.trial127.stdout: lua-5.4.4-4.el9.x86_64 2026-02-12T12:15:14.909 INFO:teuthology.orchestra.run.trial127.stdout: lua-devel-5.4.4-4.el9.x86_64 2026-02-12T12:15:14.909 INFO:teuthology.orchestra.run.trial127.stdout: luarocks-3.9.2-5.el9.noarch 2026-02-12T12:15:14.909 INFO:teuthology.orchestra.run.trial127.stdout: openblas-0.3.29-1.el9.x86_64 2026-02-12T12:15:14.909 INFO:teuthology.orchestra.run.trial127.stdout: openblas-openmp-0.3.29-1.el9.x86_64 2026-02-12T12:15:14.909 INFO:teuthology.orchestra.run.trial127.stdout: protobuf-compiler-3.14.0-17.el9.x86_64 2026-02-12T12:15:14.910 INFO:teuthology.orchestra.run.trial127.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2026-02-12T12:15:14.910 INFO:teuthology.orchestra.run.trial127.stdout: python3-autocommand-2.2.2-8.el9.noarch 2026-02-12T12:15:14.910 INFO:teuthology.orchestra.run.trial127.stdout: python3-babel-2.9.1-2.el9.noarch 2026-02-12T12:15:14.910 INFO:teuthology.orchestra.run.trial127.stdout: python3-backports-tarfile-1.2.0-1.el9.noarch 2026-02-12T12:15:14.910 INFO:teuthology.orchestra.run.trial127.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2026-02-12T12:15:14.910 INFO:teuthology.orchestra.run.trial127.stdout: python3-cachetools-4.2.4-1.el9.noarch 2026-02-12T12:15:14.910 INFO:teuthology.orchestra.run.trial127.stdout: python3-certifi-2023.05.07-4.el9.noarch 2026-02-12T12:15:14.910 INFO:teuthology.orchestra.run.trial127.stdout: python3-cffi-1.14.5-5.el9.x86_64 2026-02-12T12:15:14.910 INFO:teuthology.orchestra.run.trial127.stdout: python3-cheroot-10.0.1-4.el9.noarch 2026-02-12T12:15:14.910 INFO:teuthology.orchestra.run.trial127.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2026-02-12T12:15:14.910 INFO:teuthology.orchestra.run.trial127.stdout: python3-cryptography-36.0.1-5.el9.x86_64 2026-02-12T12:15:14.911 INFO:teuthology.orchestra.run.trial127.stdout: python3-devel-3.9.25-2.el9.x86_64 2026-02-12T12:15:14.911 INFO:teuthology.orchestra.run.trial127.stdout: python3-google-auth-1:2.45.0-1.el9.noarch 2026-02-12T12:15:14.911 INFO:teuthology.orchestra.run.trial127.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2026-02-12T12:15:14.911 INFO:teuthology.orchestra.run.trial127.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2026-02-12T12:15:14.911 INFO:teuthology.orchestra.run.trial127.stdout: python3-influxdb-5.3.1-1.el9.noarch 2026-02-12T12:15:14.911 INFO:teuthology.orchestra.run.trial127.stdout: python3-isodate-0.6.1-3.el9.noarch 2026-02-12T12:15:14.911 INFO:teuthology.orchestra.run.trial127.stdout: python3-jaraco-8.2.1-3.el9.noarch 2026-02-12T12:15:14.911 INFO:teuthology.orchestra.run.trial127.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2026-02-12T12:15:14.911 INFO:teuthology.orchestra.run.trial127.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2026-02-12T12:15:14.911 INFO:teuthology.orchestra.run.trial127.stdout: python3-jaraco-context-6.0.1-3.el9.noarch 2026-02-12T12:15:14.911 INFO:teuthology.orchestra.run.trial127.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2026-02-12T12:15:14.911 INFO:teuthology.orchestra.run.trial127.stdout: python3-jaraco-text-4.0.0-2.el9.noarch 2026-02-12T12:15:14.911 INFO:teuthology.orchestra.run.trial127.stdout: python3-jinja2-2.11.3-8.el9.noarch 2026-02-12T12:15:14.912 INFO:teuthology.orchestra.run.trial127.stdout: python3-jmespath-1.0.1-1.el9.noarch 2026-02-12T12:15:14.912 INFO:teuthology.orchestra.run.trial127.stdout: python3-kubernetes-1:26.1.0-3.el9.noarch 2026-02-12T12:15:14.912 INFO:teuthology.orchestra.run.trial127.stdout: python3-libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-12T12:15:14.912 INFO:teuthology.orchestra.run.trial127.stdout: python3-lxml-4.6.5-3.el9.x86_64 2026-02-12T12:15:14.912 INFO:teuthology.orchestra.run.trial127.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2026-02-12T12:15:14.912 INFO:teuthology.orchestra.run.trial127.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2026-02-12T12:15:14.912 INFO:teuthology.orchestra.run.trial127.stdout: python3-msgpack-1.0.3-2.el9.x86_64 2026-02-12T12:15:14.912 INFO:teuthology.orchestra.run.trial127.stdout: python3-natsort-7.1.1-5.el9.noarch 2026-02-12T12:15:14.912 INFO:teuthology.orchestra.run.trial127.stdout: python3-numpy-1:1.23.5-2.el9.x86_64 2026-02-12T12:15:14.912 INFO:teuthology.orchestra.run.trial127.stdout: python3-numpy-f2py-1:1.23.5-2.el9.x86_64 2026-02-12T12:15:14.912 INFO:teuthology.orchestra.run.trial127.stdout: python3-packaging-20.9-5.el9.noarch 2026-02-12T12:15:14.912 INFO:teuthology.orchestra.run.trial127.stdout: python3-ply-3.11-14.el9.noarch 2026-02-12T12:15:14.912 INFO:teuthology.orchestra.run.trial127.stdout: python3-portend-3.1.0-2.el9.noarch 2026-02-12T12:15:14.913 INFO:teuthology.orchestra.run.trial127.stdout: python3-protobuf-3.14.0-17.el9.noarch 2026-02-12T12:15:14.913 INFO:teuthology.orchestra.run.trial127.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2026-02-12T12:15:14.913 INFO:teuthology.orchestra.run.trial127.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2026-02-12T12:15:14.913 INFO:teuthology.orchestra.run.trial127.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2026-02-12T12:15:14.913 INFO:teuthology.orchestra.run.trial127.stdout: python3-pycparser-2.20-6.el9.noarch 2026-02-12T12:15:14.913 INFO:teuthology.orchestra.run.trial127.stdout: python3-pyparsing-2.4.7-9.el9.noarch 2026-02-12T12:15:14.913 INFO:teuthology.orchestra.run.trial127.stdout: python3-repoze-lru-0.7-16.el9.noarch 2026-02-12T12:15:14.913 INFO:teuthology.orchestra.run.trial127.stdout: python3-requests-2.25.1-10.el9.noarch 2026-02-12T12:15:14.913 INFO:teuthology.orchestra.run.trial127.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2026-02-12T12:15:14.913 INFO:teuthology.orchestra.run.trial127.stdout: python3-routes-2.5.1-5.el9.noarch 2026-02-12T12:15:14.913 INFO:teuthology.orchestra.run.trial127.stdout: python3-rsa-4.9-2.el9.noarch 2026-02-12T12:15:14.913 INFO:teuthology.orchestra.run.trial127.stdout: python3-saml-1.16.0-1.el9.noarch 2026-02-12T12:15:14.913 INFO:teuthology.orchestra.run.trial127.stdout: python3-scipy-1.9.3-2.el9.x86_64 2026-02-12T12:15:14.914 INFO:teuthology.orchestra.run.trial127.stdout: python3-tempora-5.0.0-2.el9.noarch 2026-02-12T12:15:14.914 INFO:teuthology.orchestra.run.trial127.stdout: python3-toml-0.10.2-6.el9.noarch 2026-02-12T12:15:14.914 INFO:teuthology.orchestra.run.trial127.stdout: python3-typing-extensions-4.15.0-1.el9.noarch 2026-02-12T12:15:14.914 INFO:teuthology.orchestra.run.trial127.stdout: python3-urllib3-1.26.5-7.el9.noarch 2026-02-12T12:15:14.914 INFO:teuthology.orchestra.run.trial127.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2026-02-12T12:15:14.914 INFO:teuthology.orchestra.run.trial127.stdout: python3-xmlsec-1.3.13-1.el9.x86_64 2026-02-12T12:15:14.914 INFO:teuthology.orchestra.run.trial127.stdout: python3-xmltodict-0.12.0-15.el9.noarch 2026-02-12T12:15:14.914 INFO:teuthology.orchestra.run.trial127.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2026-02-12T12:15:14.914 INFO:teuthology.orchestra.run.trial127.stdout: unzip-6.0-59.el9.x86_64 2026-02-12T12:15:14.914 INFO:teuthology.orchestra.run.trial127.stdout: xmlsec1-1.2.29-13.el9.x86_64 2026-02-12T12:15:14.914 INFO:teuthology.orchestra.run.trial127.stdout: xmlsec1-openssl-1.2.29-13.el9.x86_64 2026-02-12T12:15:14.914 INFO:teuthology.orchestra.run.trial127.stdout: zip-3.0-35.el9.x86_64 2026-02-12T12:15:14.915 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:15:14.915 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-12T12:15:14.970 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install ceph-base 2026-02-12T12:15:15.284 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:08:04 ago on Thu 12 Feb 2026 12:07:11 PM UTC. 2026-02-12T12:15:15.349 INFO:teuthology.orchestra.run.trial127.stdout:Package ceph-base-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:15:15.368 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-12T12:15:15.368 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-12T12:15:15.368 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-12T12:15:15.390 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install cephadm 2026-02-12T12:15:15.695 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:08:04 ago on Thu 12 Feb 2026 12:07:11 PM UTC. 2026-02-12T12:15:15.760 INFO:teuthology.orchestra.run.trial127.stdout:Package cephadm-2:20.3.0-5145.g5fdb7d8f.el9.noarch is already installed. 2026-02-12T12:15:15.779 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-12T12:15:15.779 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-12T12:15:15.779 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-12T12:15:15.800 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install ceph-immutable-object-cache 2026-02-12T12:15:16.096 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:08:05 ago on Thu 12 Feb 2026 12:07:11 PM UTC. 2026-02-12T12:15:16.179 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-12T12:15:16.180 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-12T12:15:16.180 INFO:teuthology.orchestra.run.trial127.stdout: Package Arch Version Repo Size 2026-02-12T12:15:16.181 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-12T12:15:16.181 INFO:teuthology.orchestra.run.trial127.stdout:Installing: 2026-02-12T12:15:16.181 INFO:teuthology.orchestra.run.trial127.stdout: ceph-immutable-object-cache x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 149 k 2026-02-12T12:15:16.181 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:15:16.181 INFO:teuthology.orchestra.run.trial127.stdout:Transaction Summary 2026-02-12T12:15:16.181 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-12T12:15:16.181 INFO:teuthology.orchestra.run.trial127.stdout:Install 1 Package 2026-02-12T12:15:16.181 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:15:16.181 INFO:teuthology.orchestra.run.trial127.stdout:Total download size: 149 k 2026-02-12T12:15:16.181 INFO:teuthology.orchestra.run.trial127.stdout:Installed size: 455 k 2026-02-12T12:15:16.181 INFO:teuthology.orchestra.run.trial127.stdout:Downloading Packages: 2026-02-12T12:15:16.681 INFO:teuthology.orchestra.run.trial127.stdout:ceph-immutable-object-cache-20.3.0-5145.g5fdb7d 298 kB/s | 149 kB 00:00 2026-02-12T12:15:16.683 INFO:teuthology.orchestra.run.trial127.stdout:-------------------------------------------------------------------------------- 2026-02-12T12:15:16.683 INFO:teuthology.orchestra.run.trial127.stdout:Total 296 kB/s | 149 kB 00:00 2026-02-12T12:15:16.683 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction check 2026-02-12T12:15:16.688 INFO:teuthology.orchestra.run.trial127.stdout:Transaction check succeeded. 2026-02-12T12:15:16.688 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction test 2026-02-12T12:15:16.720 INFO:teuthology.orchestra.run.trial127.stdout:Transaction test succeeded. 2026-02-12T12:15:16.720 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction 2026-02-12T12:15:16.775 INFO:teuthology.orchestra.run.trial127.stdout: Preparing : 1/1 2026-02-12T12:15:16.777 INFO:teuthology.orchestra.run.trial127.stdout: Installing : ceph-immutable-object-cache-2:20.3.0-5145.g5fdb7d8f. 1/1 2026-02-12T12:15:16.799 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: ceph-immutable-object-cache-2:20.3.0-5145.g5fdb7d8f. 1/1 2026-02-12T12:15:16.799 INFO:teuthology.orchestra.run.trial127.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-12T12:15:16.799 INFO:teuthology.orchestra.run.trial127.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2026-02-12T12:15:16.799 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:15:17.095 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : ceph-immutable-object-cache-2:20.3.0-5145.g5fdb7d8f. 1/1 2026-02-12T12:15:17.095 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:15:17.095 INFO:teuthology.orchestra.run.trial127.stdout:Installed: 2026-02-12T12:15:17.095 INFO:teuthology.orchestra.run.trial127.stdout: ceph-immutable-object-cache-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:15:17.095 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:15:17.095 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-12T12:15:17.121 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install ceph-mgr 2026-02-12T12:15:17.424 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:08:06 ago on Thu 12 Feb 2026 12:07:11 PM UTC. 2026-02-12T12:15:17.489 INFO:teuthology.orchestra.run.trial127.stdout:Package ceph-mgr-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:15:17.507 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-12T12:15:17.508 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-12T12:15:17.508 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-12T12:15:17.529 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install ceph-mgr-dashboard 2026-02-12T12:15:17.829 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:08:06 ago on Thu 12 Feb 2026 12:07:11 PM UTC. 2026-02-12T12:15:17.895 INFO:teuthology.orchestra.run.trial127.stdout:Package ceph-mgr-dashboard-2:20.3.0-5145.g5fdb7d8f.el9.noarch is already installed. 2026-02-12T12:15:17.913 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-12T12:15:17.914 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-12T12:15:17.914 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-12T12:15:17.935 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-12T12:15:18.232 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:08:07 ago on Thu 12 Feb 2026 12:07:11 PM UTC. 2026-02-12T12:15:18.296 INFO:teuthology.orchestra.run.trial127.stdout:Package ceph-mgr-diskprediction-local-2:20.3.0-5145.g5fdb7d8f.el9.noarch is already installed. 2026-02-12T12:15:18.316 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-12T12:15:18.316 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-12T12:15:18.316 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-12T12:15:18.337 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install ceph-mgr-rook 2026-02-12T12:15:18.640 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:08:07 ago on Thu 12 Feb 2026 12:07:11 PM UTC. 2026-02-12T12:15:18.705 INFO:teuthology.orchestra.run.trial127.stdout:Package ceph-mgr-rook-2:20.3.0-5145.g5fdb7d8f.el9.noarch is already installed. 2026-02-12T12:15:18.723 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-12T12:15:18.724 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-12T12:15:18.724 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-12T12:15:18.745 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install ceph-mgr-cephadm 2026-02-12T12:15:19.051 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:08:08 ago on Thu 12 Feb 2026 12:07:11 PM UTC. 2026-02-12T12:15:19.117 INFO:teuthology.orchestra.run.trial127.stdout:Package ceph-mgr-cephadm-2:20.3.0-5145.g5fdb7d8f.el9.noarch is already installed. 2026-02-12T12:15:19.135 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-12T12:15:19.136 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-12T12:15:19.136 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-12T12:15:19.158 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install ceph-osd 2026-02-12T12:15:19.457 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:08:08 ago on Thu 12 Feb 2026 12:07:11 PM UTC. 2026-02-12T12:15:19.522 INFO:teuthology.orchestra.run.trial127.stdout:Package ceph-osd-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:15:19.541 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-12T12:15:19.541 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-12T12:15:19.541 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-12T12:15:19.563 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install ceph-osd-classic 2026-02-12T12:15:19.863 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:08:08 ago on Thu 12 Feb 2026 12:07:11 PM UTC. 2026-02-12T12:15:19.929 INFO:teuthology.orchestra.run.trial127.stdout:Package ceph-osd-classic-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:15:19.947 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-12T12:15:19.948 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-12T12:15:19.948 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-12T12:15:19.968 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install ceph-fuse 2026-02-12T12:15:20.274 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:08:09 ago on Thu 12 Feb 2026 12:07:11 PM UTC. 2026-02-12T12:15:20.359 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-12T12:15:20.360 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-12T12:15:20.360 INFO:teuthology.orchestra.run.trial127.stdout: Package Arch Version Repo Size 2026-02-12T12:15:20.360 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-12T12:15:20.360 INFO:teuthology.orchestra.run.trial127.stdout:Installing: 2026-02-12T12:15:20.360 INFO:teuthology.orchestra.run.trial127.stdout: ceph-fuse x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 914 k 2026-02-12T12:15:20.360 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:15:20.361 INFO:teuthology.orchestra.run.trial127.stdout:Transaction Summary 2026-02-12T12:15:20.361 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-12T12:15:20.361 INFO:teuthology.orchestra.run.trial127.stdout:Install 1 Package 2026-02-12T12:15:20.361 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:15:20.361 INFO:teuthology.orchestra.run.trial127.stdout:Total download size: 914 k 2026-02-12T12:15:20.361 INFO:teuthology.orchestra.run.trial127.stdout:Installed size: 2.9 M 2026-02-12T12:15:20.361 INFO:teuthology.orchestra.run.trial127.stdout:Downloading Packages: 2026-02-12T12:15:21.126 INFO:teuthology.orchestra.run.trial127.stdout:ceph-fuse-20.3.0-5145.g5fdb7d8f.el9.x86_64.rpm 1.2 MB/s | 914 kB 00:00 2026-02-12T12:15:21.127 INFO:teuthology.orchestra.run.trial127.stdout:-------------------------------------------------------------------------------- 2026-02-12T12:15:21.127 INFO:teuthology.orchestra.run.trial127.stdout:Total 1.2 MB/s | 914 kB 00:00 2026-02-12T12:15:21.127 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction check 2026-02-12T12:15:21.132 INFO:teuthology.orchestra.run.trial127.stdout:Transaction check succeeded. 2026-02-12T12:15:21.132 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction test 2026-02-12T12:15:21.167 INFO:teuthology.orchestra.run.trial127.stdout:Transaction test succeeded. 2026-02-12T12:15:21.167 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction 2026-02-12T12:15:21.261 INFO:teuthology.orchestra.run.trial127.stdout: Preparing : 1/1 2026-02-12T12:15:21.278 INFO:teuthology.orchestra.run.trial127.stdout: Installing : ceph-fuse-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 1/1 2026-02-12T12:15:21.522 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: ceph-fuse-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 1/1 2026-02-12T12:15:21.580 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : ceph-fuse-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 1/1 2026-02-12T12:15:21.580 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:15:21.580 INFO:teuthology.orchestra.run.trial127.stdout:Installed: 2026-02-12T12:15:21.580 INFO:teuthology.orchestra.run.trial127.stdout: ceph-fuse-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:15:21.580 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:15:21.580 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-12T12:15:21.608 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install ceph-volume 2026-02-12T12:15:21.913 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:08:10 ago on Thu 12 Feb 2026 12:07:11 PM UTC. 2026-02-12T12:15:21.977 INFO:teuthology.orchestra.run.trial127.stdout:Package ceph-volume-2:20.3.0-5145.g5fdb7d8f.el9.noarch is already installed. 2026-02-12T12:15:21.995 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-12T12:15:21.995 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-12T12:15:21.995 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-12T12:15:22.017 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install librados-devel 2026-02-12T12:15:22.319 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:08:11 ago on Thu 12 Feb 2026 12:07:11 PM UTC. 2026-02-12T12:15:22.404 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-12T12:15:22.405 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-12T12:15:22.405 INFO:teuthology.orchestra.run.trial127.stdout: Package Arch Version Repo Size 2026-02-12T12:15:22.405 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-12T12:15:22.405 INFO:teuthology.orchestra.run.trial127.stdout:Installing: 2026-02-12T12:15:22.405 INFO:teuthology.orchestra.run.trial127.stdout: librados-devel x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 124 k 2026-02-12T12:15:22.405 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:15:22.405 INFO:teuthology.orchestra.run.trial127.stdout:Transaction Summary 2026-02-12T12:15:22.405 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-12T12:15:22.405 INFO:teuthology.orchestra.run.trial127.stdout:Install 1 Package 2026-02-12T12:15:22.405 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:15:22.406 INFO:teuthology.orchestra.run.trial127.stdout:Total download size: 124 k 2026-02-12T12:15:22.406 INFO:teuthology.orchestra.run.trial127.stdout:Installed size: 434 k 2026-02-12T12:15:22.406 INFO:teuthology.orchestra.run.trial127.stdout:Downloading Packages: 2026-02-12T12:15:22.878 INFO:teuthology.orchestra.run.trial127.stdout:librados-devel-20.3.0-5145.g5fdb7d8f.el9.x86_64 261 kB/s | 124 kB 00:00 2026-02-12T12:15:22.880 INFO:teuthology.orchestra.run.trial127.stdout:-------------------------------------------------------------------------------- 2026-02-12T12:15:22.880 INFO:teuthology.orchestra.run.trial127.stdout:Total 260 kB/s | 124 kB 00:00 2026-02-12T12:15:22.881 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction check 2026-02-12T12:15:22.885 INFO:teuthology.orchestra.run.trial127.stdout:Transaction check succeeded. 2026-02-12T12:15:22.885 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction test 2026-02-12T12:15:22.918 INFO:teuthology.orchestra.run.trial127.stdout:Transaction test succeeded. 2026-02-12T12:15:22.918 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction 2026-02-12T12:15:22.973 INFO:teuthology.orchestra.run.trial127.stdout: Preparing : 1/1 2026-02-12T12:15:22.989 INFO:teuthology.orchestra.run.trial127.stdout: Installing : librados-devel-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 1/1 2026-02-12T12:15:23.069 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: librados-devel-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 1/1 2026-02-12T12:15:23.126 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : librados-devel-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 1/1 2026-02-12T12:15:23.126 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:15:23.126 INFO:teuthology.orchestra.run.trial127.stdout:Installed: 2026-02-12T12:15:23.127 INFO:teuthology.orchestra.run.trial127.stdout: librados-devel-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:15:23.127 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:15:23.127 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-12T12:15:23.154 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install libcephfs2 2026-02-12T12:15:23.458 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:08:12 ago on Thu 12 Feb 2026 12:07:11 PM UTC. 2026-02-12T12:15:23.524 INFO:teuthology.orchestra.run.trial127.stdout:Package libcephfs2-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:15:23.541 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-12T12:15:23.542 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-12T12:15:23.542 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-12T12:15:23.563 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install libcephfs-devel 2026-02-12T12:15:23.873 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:08:12 ago on Thu 12 Feb 2026 12:07:11 PM UTC. 2026-02-12T12:15:23.956 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-12T12:15:23.957 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-12T12:15:23.957 INFO:teuthology.orchestra.run.trial127.stdout: Package Arch Version Repo Size 2026-02-12T12:15:23.957 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-12T12:15:23.958 INFO:teuthology.orchestra.run.trial127.stdout:Installing: 2026-02-12T12:15:23.958 INFO:teuthology.orchestra.run.trial127.stdout: libcephfs-devel x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 37 k 2026-02-12T12:15:23.958 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:15:23.958 INFO:teuthology.orchestra.run.trial127.stdout:Transaction Summary 2026-02-12T12:15:23.958 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-12T12:15:23.958 INFO:teuthology.orchestra.run.trial127.stdout:Install 1 Package 2026-02-12T12:15:23.958 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:15:23.958 INFO:teuthology.orchestra.run.trial127.stdout:Total download size: 37 k 2026-02-12T12:15:23.958 INFO:teuthology.orchestra.run.trial127.stdout:Installed size: 171 k 2026-02-12T12:15:23.958 INFO:teuthology.orchestra.run.trial127.stdout:Downloading Packages: 2026-02-12T12:15:24.384 INFO:teuthology.orchestra.run.trial127.stdout:libcephfs-devel-20.3.0-5145.g5fdb7d8f.el9.x86_6 87 kB/s | 37 kB 00:00 2026-02-12T12:15:24.385 INFO:teuthology.orchestra.run.trial127.stdout:-------------------------------------------------------------------------------- 2026-02-12T12:15:24.385 INFO:teuthology.orchestra.run.trial127.stdout:Total 87 kB/s | 37 kB 00:00 2026-02-12T12:15:24.385 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction check 2026-02-12T12:15:24.388 INFO:teuthology.orchestra.run.trial127.stdout:Transaction check succeeded. 2026-02-12T12:15:24.388 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction test 2026-02-12T12:15:24.391 INFO:teuthology.orchestra.run.trial127.stdout:Transaction test succeeded. 2026-02-12T12:15:24.391 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction 2026-02-12T12:15:24.413 INFO:teuthology.orchestra.run.trial127.stdout: Preparing : 1/1 2026-02-12T12:15:24.430 INFO:teuthology.orchestra.run.trial127.stdout: Installing : libcephfs-devel-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 1/1 2026-02-12T12:15:24.487 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: libcephfs-devel-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 1/1 2026-02-12T12:15:24.545 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : libcephfs-devel-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 1/1 2026-02-12T12:15:24.545 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:15:24.545 INFO:teuthology.orchestra.run.trial127.stdout:Installed: 2026-02-12T12:15:24.545 INFO:teuthology.orchestra.run.trial127.stdout: libcephfs-devel-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:15:24.545 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:15:24.545 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-12T12:15:24.573 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install librados2 2026-02-12T12:15:24.881 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:08:13 ago on Thu 12 Feb 2026 12:07:11 PM UTC. 2026-02-12T12:15:24.945 INFO:teuthology.orchestra.run.trial127.stdout:Package librados2-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:15:24.962 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-12T12:15:24.963 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-12T12:15:24.963 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-12T12:15:24.985 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install librbd1 2026-02-12T12:15:25.290 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:08:14 ago on Thu 12 Feb 2026 12:07:11 PM UTC. 2026-02-12T12:15:25.354 INFO:teuthology.orchestra.run.trial127.stdout:Package librbd1-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:15:25.371 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-12T12:15:25.371 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-12T12:15:25.372 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-12T12:15:25.393 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install python3-rados 2026-02-12T12:15:25.693 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:08:14 ago on Thu 12 Feb 2026 12:07:11 PM UTC. 2026-02-12T12:15:25.758 INFO:teuthology.orchestra.run.trial127.stdout:Package python3-rados-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:15:25.775 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-12T12:15:25.776 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-12T12:15:25.776 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-12T12:15:25.797 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install python3-rgw 2026-02-12T12:15:26.105 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:08:15 ago on Thu 12 Feb 2026 12:07:11 PM UTC. 2026-02-12T12:15:26.170 INFO:teuthology.orchestra.run.trial127.stdout:Package python3-rgw-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:15:26.187 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-12T12:15:26.188 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-12T12:15:26.188 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-12T12:15:26.210 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install python3-cephfs 2026-02-12T12:15:26.513 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:08:15 ago on Thu 12 Feb 2026 12:07:11 PM UTC. 2026-02-12T12:15:26.579 INFO:teuthology.orchestra.run.trial127.stdout:Package python3-cephfs-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:15:26.596 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-12T12:15:26.597 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-12T12:15:26.597 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-12T12:15:26.618 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install python3-rbd 2026-02-12T12:15:26.920 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:08:15 ago on Thu 12 Feb 2026 12:07:11 PM UTC. 2026-02-12T12:15:26.985 INFO:teuthology.orchestra.run.trial127.stdout:Package python3-rbd-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:15:27.002 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-12T12:15:27.003 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-12T12:15:27.003 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-12T12:15:27.025 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install rbd-fuse 2026-02-12T12:15:27.329 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:08:16 ago on Thu 12 Feb 2026 12:07:11 PM UTC. 2026-02-12T12:15:27.412 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-12T12:15:27.413 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-12T12:15:27.413 INFO:teuthology.orchestra.run.trial127.stdout: Package Architecture Version Repository Size 2026-02-12T12:15:27.413 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-12T12:15:27.413 INFO:teuthology.orchestra.run.trial127.stdout:Installing: 2026-02-12T12:15:27.413 INFO:teuthology.orchestra.run.trial127.stdout: rbd-fuse x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 87 k 2026-02-12T12:15:27.413 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:15:27.413 INFO:teuthology.orchestra.run.trial127.stdout:Transaction Summary 2026-02-12T12:15:27.414 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-12T12:15:27.414 INFO:teuthology.orchestra.run.trial127.stdout:Install 1 Package 2026-02-12T12:15:27.414 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:15:27.414 INFO:teuthology.orchestra.run.trial127.stdout:Total download size: 87 k 2026-02-12T12:15:27.414 INFO:teuthology.orchestra.run.trial127.stdout:Installed size: 236 k 2026-02-12T12:15:27.414 INFO:teuthology.orchestra.run.trial127.stdout:Downloading Packages: 2026-02-12T12:15:27.833 INFO:teuthology.orchestra.run.trial127.stdout:rbd-fuse-20.3.0-5145.g5fdb7d8f.el9.x86_64.rpm 206 kB/s | 87 kB 00:00 2026-02-12T12:15:27.834 INFO:teuthology.orchestra.run.trial127.stdout:-------------------------------------------------------------------------------- 2026-02-12T12:15:27.834 INFO:teuthology.orchestra.run.trial127.stdout:Total 206 kB/s | 87 kB 00:00 2026-02-12T12:15:27.834 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction check 2026-02-12T12:15:27.838 INFO:teuthology.orchestra.run.trial127.stdout:Transaction check succeeded. 2026-02-12T12:15:27.838 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction test 2026-02-12T12:15:27.870 INFO:teuthology.orchestra.run.trial127.stdout:Transaction test succeeded. 2026-02-12T12:15:27.870 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction 2026-02-12T12:15:27.923 INFO:teuthology.orchestra.run.trial127.stdout: Preparing : 1/1 2026-02-12T12:15:27.940 INFO:teuthology.orchestra.run.trial127.stdout: Installing : rbd-fuse-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 1/1 2026-02-12T12:15:28.018 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: rbd-fuse-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 1/1 2026-02-12T12:15:28.076 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : rbd-fuse-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 1/1 2026-02-12T12:15:28.076 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:15:28.076 INFO:teuthology.orchestra.run.trial127.stdout:Installed: 2026-02-12T12:15:28.076 INFO:teuthology.orchestra.run.trial127.stdout: rbd-fuse-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:15:28.076 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:15:28.076 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-12T12:15:28.104 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install rbd-mirror 2026-02-12T12:15:28.430 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:08:17 ago on Thu 12 Feb 2026 12:07:11 PM UTC. 2026-02-12T12:15:28.512 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-12T12:15:28.513 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-12T12:15:28.513 INFO:teuthology.orchestra.run.trial127.stdout: Package Arch Version Repo Size 2026-02-12T12:15:28.513 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-12T12:15:28.513 INFO:teuthology.orchestra.run.trial127.stdout:Installing: 2026-02-12T12:15:28.513 INFO:teuthology.orchestra.run.trial127.stdout: rbd-mirror x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 2.6 M 2026-02-12T12:15:28.514 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:15:28.514 INFO:teuthology.orchestra.run.trial127.stdout:Transaction Summary 2026-02-12T12:15:28.514 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-12T12:15:28.514 INFO:teuthology.orchestra.run.trial127.stdout:Install 1 Package 2026-02-12T12:15:28.514 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:15:28.514 INFO:teuthology.orchestra.run.trial127.stdout:Total download size: 2.6 M 2026-02-12T12:15:28.514 INFO:teuthology.orchestra.run.trial127.stdout:Installed size: 11 M 2026-02-12T12:15:28.514 INFO:teuthology.orchestra.run.trial127.stdout:Downloading Packages: 2026-02-12T12:15:29.835 INFO:teuthology.orchestra.run.trial127.stdout:rbd-mirror-20.3.0-5145.g5fdb7d8f.el9.x86_64.rpm 2.0 MB/s | 2.6 MB 00:01 2026-02-12T12:15:29.836 INFO:teuthology.orchestra.run.trial127.stdout:-------------------------------------------------------------------------------- 2026-02-12T12:15:29.836 INFO:teuthology.orchestra.run.trial127.stdout:Total 2.0 MB/s | 2.6 MB 00:01 2026-02-12T12:15:29.836 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction check 2026-02-12T12:15:29.841 INFO:teuthology.orchestra.run.trial127.stdout:Transaction check succeeded. 2026-02-12T12:15:29.841 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction test 2026-02-12T12:15:29.880 INFO:teuthology.orchestra.run.trial127.stdout:Transaction test succeeded. 2026-02-12T12:15:29.880 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction 2026-02-12T12:15:30.042 INFO:teuthology.orchestra.run.trial127.stdout: Preparing : 1/1 2026-02-12T12:15:30.044 INFO:teuthology.orchestra.run.trial127.stdout: Installing : rbd-mirror-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 1/1 2026-02-12T12:15:30.066 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: rbd-mirror-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 1/1 2026-02-12T12:15:30.066 INFO:teuthology.orchestra.run.trial127.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-12T12:15:30.066 INFO:teuthology.orchestra.run.trial127.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2026-02-12T12:15:30.066 INFO:teuthology.orchestra.run.trial127.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-12T12:15:30.066 INFO:teuthology.orchestra.run.trial127.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-12T12:15:30.066 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:15:30.361 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : rbd-mirror-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 1/1 2026-02-12T12:15:30.361 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:15:30.361 INFO:teuthology.orchestra.run.trial127.stdout:Installed: 2026-02-12T12:15:30.361 INFO:teuthology.orchestra.run.trial127.stdout: rbd-mirror-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:15:30.362 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:15:30.362 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-12T12:15:30.392 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install rbd-nbd 2026-02-12T12:15:30.703 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:08:19 ago on Thu 12 Feb 2026 12:07:11 PM UTC. 2026-02-12T12:15:30.786 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-12T12:15:30.787 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-12T12:15:30.787 INFO:teuthology.orchestra.run.trial127.stdout: Package Architecture Version Repository Size 2026-02-12T12:15:30.787 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-12T12:15:30.787 INFO:teuthology.orchestra.run.trial127.stdout:Installing: 2026-02-12T12:15:30.788 INFO:teuthology.orchestra.run.trial127.stdout: rbd-nbd x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 167 k 2026-02-12T12:15:30.788 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:15:30.788 INFO:teuthology.orchestra.run.trial127.stdout:Transaction Summary 2026-02-12T12:15:30.788 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-12T12:15:30.788 INFO:teuthology.orchestra.run.trial127.stdout:Install 1 Package 2026-02-12T12:15:30.788 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:15:30.788 INFO:teuthology.orchestra.run.trial127.stdout:Total download size: 167 k 2026-02-12T12:15:30.788 INFO:teuthology.orchestra.run.trial127.stdout:Installed size: 496 k 2026-02-12T12:15:30.788 INFO:teuthology.orchestra.run.trial127.stdout:Downloading Packages: 2026-02-12T12:15:31.706 INFO:teuthology.orchestra.run.trial127.stdout:rbd-nbd-20.3.0-5145.g5fdb7d8f.el9.x86_64.rpm 181 kB/s | 167 kB 00:00 2026-02-12T12:15:31.707 INFO:teuthology.orchestra.run.trial127.stdout:-------------------------------------------------------------------------------- 2026-02-12T12:15:31.707 INFO:teuthology.orchestra.run.trial127.stdout:Total 181 kB/s | 167 kB 00:00 2026-02-12T12:15:31.708 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction check 2026-02-12T12:15:31.712 INFO:teuthology.orchestra.run.trial127.stdout:Transaction check succeeded. 2026-02-12T12:15:31.712 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction test 2026-02-12T12:15:31.744 INFO:teuthology.orchestra.run.trial127.stdout:Transaction test succeeded. 2026-02-12T12:15:31.744 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction 2026-02-12T12:15:31.800 INFO:teuthology.orchestra.run.trial127.stdout: Preparing : 1/1 2026-02-12T12:15:31.817 INFO:teuthology.orchestra.run.trial127.stdout: Installing : rbd-nbd-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 1/1 2026-02-12T12:15:31.899 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: rbd-nbd-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 1/1 2026-02-12T12:15:31.957 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : rbd-nbd-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 1/1 2026-02-12T12:15:31.957 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:15:31.957 INFO:teuthology.orchestra.run.trial127.stdout:Installed: 2026-02-12T12:15:31.957 INFO:teuthology.orchestra.run.trial127.stdout: rbd-nbd-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:15:31.957 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:15:31.958 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-12T12:15:31.987 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install ceph-osd-crimson 2026-02-12T12:15:32.293 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:08:21 ago on Thu 12 Feb 2026 12:07:11 PM UTC. 2026-02-12T12:15:32.357 INFO:teuthology.orchestra.run.trial127.stdout:Package ceph-osd-crimson-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:15:32.375 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-12T12:15:32.376 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-12T12:15:32.376 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-12T12:15:32.397 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install ceph-osd-crimson-debuginfo 2026-02-12T12:15:32.699 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:08:21 ago on Thu 12 Feb 2026 12:07:11 PM UTC. 2026-02-12T12:15:32.783 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-12T12:15:32.784 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-12T12:15:32.784 INFO:teuthology.orchestra.run.trial127.stdout: Package Arch Version Repo Size 2026-02-12T12:15:32.784 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-12T12:15:32.784 INFO:teuthology.orchestra.run.trial127.stdout:Installing: 2026-02-12T12:15:32.784 INFO:teuthology.orchestra.run.trial127.stdout: ceph-osd-crimson-debuginfo x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 429 M 2026-02-12T12:15:32.784 INFO:teuthology.orchestra.run.trial127.stdout:Installing weak dependencies: 2026-02-12T12:15:32.785 INFO:teuthology.orchestra.run.trial127.stdout: ceph-debugsource x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 12 M 2026-02-12T12:15:32.785 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:15:32.785 INFO:teuthology.orchestra.run.trial127.stdout:Transaction Summary 2026-02-12T12:15:32.785 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-12T12:15:32.785 INFO:teuthology.orchestra.run.trial127.stdout:Install 2 Packages 2026-02-12T12:15:32.785 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:15:32.785 INFO:teuthology.orchestra.run.trial127.stdout:Total download size: 441 M 2026-02-12T12:15:32.785 INFO:teuthology.orchestra.run.trial127.stdout:Installed size: 2.7 G 2026-02-12T12:15:32.785 INFO:teuthology.orchestra.run.trial127.stdout:Downloading Packages: 2026-02-12T12:15:48.591 INFO:teuthology.orchestra.run.trial127.stdout:(1/2): ceph-debugsource-20.3.0-5145.g5fdb7d8f.e 779 kB/s | 12 MB 00:15 2026-02-12T12:28:15.380 INFO:teuthology.orchestra.run.trial127.stdout:(2/2): ceph-osd-crimson-debuginfo-20.3.0-5145.g 576 kB/s | 429 MB 12:42 2026-02-12T12:28:15.381 INFO:teuthology.orchestra.run.trial127.stdout:-------------------------------------------------------------------------------- 2026-02-12T12:28:15.381 INFO:teuthology.orchestra.run.trial127.stdout:Total 592 kB/s | 441 MB 12:42 2026-02-12T12:28:15.382 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction check 2026-02-12T12:28:15.387 INFO:teuthology.orchestra.run.trial127.stdout:Transaction check succeeded. 2026-02-12T12:28:15.387 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction test 2026-02-12T12:28:16.758 INFO:teuthology.orchestra.run.trial127.stdout:Transaction test succeeded. 2026-02-12T12:28:16.759 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction 2026-02-12T12:28:20.334 INFO:teuthology.orchestra.run.trial127.stdout: Preparing : 1/1 2026-02-12T12:28:42.431 INFO:teuthology.orchestra.run.trial127.stdout: Installing : ceph-debugsource-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 1/2 2026-02-12T12:28:42.448 INFO:teuthology.orchestra.run.trial127.stdout: Installing : ceph-osd-crimson-debuginfo-2:20.3.0-5145.g5fdb7d8f.e 2/2 2026-02-12T12:28:44.435 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: ceph-osd-crimson-debuginfo-2:20.3.0-5145.g5fdb7d8f.e 2/2 2026-02-12T12:28:44.435 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : ceph-debugsource-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 1/2 2026-02-12T12:28:44.493 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : ceph-osd-crimson-debuginfo-2:20.3.0-5145.g5fdb7d8f.e 2/2 2026-02-12T12:28:44.494 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:28:44.494 INFO:teuthology.orchestra.run.trial127.stdout:Installed: 2026-02-12T12:28:44.494 INFO:teuthology.orchestra.run.trial127.stdout: ceph-debugsource-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:28:44.494 INFO:teuthology.orchestra.run.trial127.stdout: ceph-osd-crimson-debuginfo-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:28:44.494 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:28:44.494 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-12T12:28:44.573 DEBUG:teuthology.parallel:result is None 2026-02-12T12:28:44.574 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0 2026-02-12T12:28:44.736 DEBUG:teuthology.orchestra.run.trial127:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-12T12:28:44.758 INFO:teuthology.orchestra.run.trial127.stdout:20.3.0-5145.g5fdb7d8f.el9 2026-02-12T12:28:44.759 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5145.g5fdb7d8f.el9 2026-02-12T12:28:44.759 INFO:teuthology.task.install:The correct ceph version 20.3.0-5145.g5fdb7d8f is installed. 2026-02-12T12:28:44.762 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2026-02-12T12:28:44.762 DEBUG:teuthology.orchestra.run.trial127:> set -ex 2026-02-12T12:28:44.762 DEBUG:teuthology.orchestra.run.trial127:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-12T12:28:44.825 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2026-02-12T12:28:44.825 DEBUG:teuthology.orchestra.run.trial127:> set -ex 2026-02-12T12:28:44.825 DEBUG:teuthology.orchestra.run.trial127:> sudo dd of=/usr/bin/daemon-helper 2026-02-12T12:28:44.891 DEBUG:teuthology.orchestra.run.trial127:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-12T12:28:44.957 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2026-02-12T12:28:44.957 DEBUG:teuthology.orchestra.run.trial127:> set -ex 2026-02-12T12:28:44.957 DEBUG:teuthology.orchestra.run.trial127:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-12T12:28:45.023 DEBUG:teuthology.orchestra.run.trial127:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-12T12:28:45.087 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2026-02-12T12:28:45.087 DEBUG:teuthology.orchestra.run.trial127:> set -ex 2026-02-12T12:28:45.088 DEBUG:teuthology.orchestra.run.trial127:> sudo dd of=/usr/bin/stdin-killer 2026-02-12T12:28:45.150 DEBUG:teuthology.orchestra.run.trial127:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-12T12:28:45.214 INFO:teuthology.run_tasks:Running task pexec... 2026-02-12T12:28:45.233 INFO:teuthology.task.pexec:Executing custom commands... 2026-02-12T12:28:45.234 DEBUG:teuthology.orchestra.run.trial127:> TESTDIR=/home/ubuntu/cephtest bash -s 2026-02-12T12:28:45.257 INFO:teuthology.task.pexec:Running commands on host ubuntu@trial127.front.sepia.ceph.com 2026-02-12T12:28:45.257 INFO:teuthology.task.pexec:sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson 2026-02-12T12:28:45.308 DEBUG:teuthology.parallel:result is None 2026-02-12T12:28:45.308 INFO:teuthology.run_tasks:Running task install... 2026-02-12T12:28:45.322 DEBUG:teuthology.task.install:project ceph 2026-02-12T12:28:45.322 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'debug', 'sha1': '5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0'}} 2026-02-12T12:28:45.322 DEBUG:teuthology.task.install:config {'flavor': 'debug', 'sha1': '5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0'} 2026-02-12T12:28:45.322 INFO:teuthology.task.install:Using flavor: debug 2026-02-12T12:28:45.330 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-osd', 'ceph-osd-classic', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-osd', 'ceph-osd-classic', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-12T12:28:45.331 INFO:teuthology.task.install:extra packages: [] 2026-02-12T12:28:45.331 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'debug', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0', 'tag': None, 'wait_for_package': False} 2026-02-12T12:28:45.331 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0 2026-02-12T12:28:45.486 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-moagrawa-seastore-io-uring-again-crimson-only/5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0/centos/9/flavors/debug/ 2026-02-12T12:28:45.486 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5145.g5fdb7d8f 2026-02-12T12:28:45.848 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-moagrawa-seastore-io-uring-again-crimson-only/5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0/centos/9/flavors/debug/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-moagrawa-seastore-io-uring-again-crimson-only/5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0/centos/9/flavors/debug/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-moagrawa-seastore-io-uring-again-crimson-only/5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0/centos/9/flavors/debug/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-12T12:28:45.848 DEBUG:teuthology.orchestra.run.trial127:> set -ex 2026-02-12T12:28:45.848 DEBUG:teuthology.orchestra.run.trial127:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-12T12:28:45.874 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2026-02-12T12:28:45.874 DEBUG:teuthology.orchestra.run.trial127:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-12T12:28:45.939 DEBUG:teuthology.orchestra.run.trial127:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-12T12:28:46.004 DEBUG:teuthology.orchestra.run.trial127:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-12T12:28:46.061 INFO:teuthology.orchestra.run.trial127.stdout:check_obsoletes = 1 2026-02-12T12:28:46.071 DEBUG:teuthology.orchestra.run.trial127:> sudo yum clean all 2026-02-12T12:28:46.231 INFO:teuthology.orchestra.run.trial127.stdout:56 files removed 2026-02-12T12:28:46.247 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install ceph-radosgw 2026-02-12T12:28:47.104 INFO:teuthology.orchestra.run.trial127.stdout:ceph packages for x86_64 133 kB/s | 94 kB 00:00 2026-02-12T12:28:47.719 INFO:teuthology.orchestra.run.trial127.stdout:ceph noarch packages 41 kB/s | 24 kB 00:00 2026-02-12T12:28:48.293 INFO:teuthology.orchestra.run.trial127.stdout:ceph source packages 3.8 kB/s | 2.1 kB 00:00 2026-02-12T12:28:49.154 INFO:teuthology.orchestra.run.trial127.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.9 MB 00:00 2026-02-12T12:28:52.179 INFO:teuthology.orchestra.run.trial127.stdout:CentOS Stream 9 - AppStream 11 MB/s | 27 MB 00:02 2026-02-12T12:28:56.131 INFO:teuthology.orchestra.run.trial127.stdout:CentOS Stream 9 - CRB 4.9 MB/s | 7.8 MB 00:01 2026-02-12T12:28:57.638 INFO:teuthology.orchestra.run.trial127.stdout:CentOS Stream 9 - Extras packages 25 kB/s | 20 kB 00:00 2026-02-12T12:28:58.588 INFO:teuthology.orchestra.run.trial127.stdout:Extra Packages for Enterprise Linux 23 MB/s | 20 MB 00:00 2026-02-12T12:29:01.867 INFO:teuthology.orchestra.run.trial127.stdout:lab-extras 995 kB/s | 50 kB 00:00 2026-02-12T12:29:03.117 INFO:teuthology.orchestra.run.trial127.stdout:Package ceph-radosgw-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:29:03.135 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-12T12:29:03.136 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-12T12:29:03.136 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-12T12:29:03.160 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install ceph-test 2026-02-12T12:29:03.461 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:00:02 ago on Thu 12 Feb 2026 12:29:01 PM UTC. 2026-02-12T12:29:03.527 INFO:teuthology.orchestra.run.trial127.stdout:Package ceph-test-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:29:03.544 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-12T12:29:03.545 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-12T12:29:03.545 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-12T12:29:03.567 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install ceph 2026-02-12T12:29:03.873 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:00:02 ago on Thu 12 Feb 2026 12:29:01 PM UTC. 2026-02-12T12:29:03.937 INFO:teuthology.orchestra.run.trial127.stdout:Package ceph-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:29:03.954 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-12T12:29:03.955 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-12T12:29:03.955 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-12T12:29:03.978 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install ceph-base 2026-02-12T12:29:04.276 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:00:03 ago on Thu 12 Feb 2026 12:29:01 PM UTC. 2026-02-12T12:29:04.340 INFO:teuthology.orchestra.run.trial127.stdout:Package ceph-base-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:29:04.358 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-12T12:29:04.358 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-12T12:29:04.359 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-12T12:29:04.380 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install cephadm 2026-02-12T12:29:04.680 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:00:03 ago on Thu 12 Feb 2026 12:29:01 PM UTC. 2026-02-12T12:29:04.744 INFO:teuthology.orchestra.run.trial127.stdout:Package cephadm-2:20.3.0-5145.g5fdb7d8f.el9.noarch is already installed. 2026-02-12T12:29:04.761 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-12T12:29:04.762 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-12T12:29:04.762 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-12T12:29:04.784 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install ceph-immutable-object-cache 2026-02-12T12:29:05.087 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:00:04 ago on Thu 12 Feb 2026 12:29:01 PM UTC. 2026-02-12T12:29:05.152 INFO:teuthology.orchestra.run.trial127.stdout:Package ceph-immutable-object-cache-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:29:05.169 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-12T12:29:05.170 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-12T12:29:05.170 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-12T12:29:05.192 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install ceph-mgr 2026-02-12T12:29:05.495 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:00:04 ago on Thu 12 Feb 2026 12:29:01 PM UTC. 2026-02-12T12:29:05.560 INFO:teuthology.orchestra.run.trial127.stdout:Package ceph-mgr-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:29:05.577 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-12T12:29:05.578 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-12T12:29:05.578 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-12T12:29:05.599 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install ceph-mgr-dashboard 2026-02-12T12:29:05.902 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:00:04 ago on Thu 12 Feb 2026 12:29:01 PM UTC. 2026-02-12T12:29:05.967 INFO:teuthology.orchestra.run.trial127.stdout:Package ceph-mgr-dashboard-2:20.3.0-5145.g5fdb7d8f.el9.noarch is already installed. 2026-02-12T12:29:05.984 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-12T12:29:05.985 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-12T12:29:05.985 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-12T12:29:06.007 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-12T12:29:06.318 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:00:05 ago on Thu 12 Feb 2026 12:29:01 PM UTC. 2026-02-12T12:29:06.385 INFO:teuthology.orchestra.run.trial127.stdout:Package ceph-mgr-diskprediction-local-2:20.3.0-5145.g5fdb7d8f.el9.noarch is already installed. 2026-02-12T12:29:06.402 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-12T12:29:06.403 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-12T12:29:06.403 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-12T12:29:06.426 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install ceph-mgr-rook 2026-02-12T12:29:06.723 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:00:05 ago on Thu 12 Feb 2026 12:29:01 PM UTC. 2026-02-12T12:29:06.788 INFO:teuthology.orchestra.run.trial127.stdout:Package ceph-mgr-rook-2:20.3.0-5145.g5fdb7d8f.el9.noarch is already installed. 2026-02-12T12:29:06.805 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-12T12:29:06.806 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-12T12:29:06.806 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-12T12:29:06.827 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install ceph-mgr-cephadm 2026-02-12T12:29:07.130 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:00:06 ago on Thu 12 Feb 2026 12:29:01 PM UTC. 2026-02-12T12:29:07.194 INFO:teuthology.orchestra.run.trial127.stdout:Package ceph-mgr-cephadm-2:20.3.0-5145.g5fdb7d8f.el9.noarch is already installed. 2026-02-12T12:29:07.212 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-12T12:29:07.212 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-12T12:29:07.213 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-12T12:29:07.234 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install ceph-osd 2026-02-12T12:29:07.537 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:00:06 ago on Thu 12 Feb 2026 12:29:01 PM UTC. 2026-02-12T12:29:07.601 INFO:teuthology.orchestra.run.trial127.stdout:Package ceph-osd-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:29:07.619 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-12T12:29:07.619 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-12T12:29:07.620 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-12T12:29:07.641 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install ceph-osd-classic 2026-02-12T12:29:07.943 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:00:06 ago on Thu 12 Feb 2026 12:29:01 PM UTC. 2026-02-12T12:29:08.009 INFO:teuthology.orchestra.run.trial127.stdout:Package ceph-osd-classic-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:29:08.027 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-12T12:29:08.027 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-12T12:29:08.027 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-12T12:29:08.049 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install ceph-fuse 2026-02-12T12:29:08.353 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:00:07 ago on Thu 12 Feb 2026 12:29:01 PM UTC. 2026-02-12T12:29:08.418 INFO:teuthology.orchestra.run.trial127.stdout:Package ceph-fuse-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:29:08.436 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-12T12:29:08.436 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-12T12:29:08.437 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-12T12:29:08.459 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install ceph-volume 2026-02-12T12:29:08.762 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:00:07 ago on Thu 12 Feb 2026 12:29:01 PM UTC. 2026-02-12T12:29:08.827 INFO:teuthology.orchestra.run.trial127.stdout:Package ceph-volume-2:20.3.0-5145.g5fdb7d8f.el9.noarch is already installed. 2026-02-12T12:29:08.844 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-12T12:29:08.845 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-12T12:29:08.845 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-12T12:29:08.866 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install librados-devel 2026-02-12T12:29:09.168 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:00:08 ago on Thu 12 Feb 2026 12:29:01 PM UTC. 2026-02-12T12:29:09.234 INFO:teuthology.orchestra.run.trial127.stdout:Package librados-devel-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:29:09.251 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-12T12:29:09.252 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-12T12:29:09.252 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-12T12:29:09.274 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install libcephfs2 2026-02-12T12:29:09.579 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:00:08 ago on Thu 12 Feb 2026 12:29:01 PM UTC. 2026-02-12T12:29:09.643 INFO:teuthology.orchestra.run.trial127.stdout:Package libcephfs2-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:29:09.661 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-12T12:29:09.662 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-12T12:29:09.662 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-12T12:29:09.684 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install libcephfs-devel 2026-02-12T12:29:09.983 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:00:08 ago on Thu 12 Feb 2026 12:29:01 PM UTC. 2026-02-12T12:29:10.048 INFO:teuthology.orchestra.run.trial127.stdout:Package libcephfs-devel-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:29:10.065 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-12T12:29:10.066 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-12T12:29:10.066 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-12T12:29:10.088 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install librados2 2026-02-12T12:29:10.392 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:00:09 ago on Thu 12 Feb 2026 12:29:01 PM UTC. 2026-02-12T12:29:10.456 INFO:teuthology.orchestra.run.trial127.stdout:Package librados2-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:29:10.473 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-12T12:29:10.474 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-12T12:29:10.474 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-12T12:29:10.496 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install librbd1 2026-02-12T12:29:10.795 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:00:09 ago on Thu 12 Feb 2026 12:29:01 PM UTC. 2026-02-12T12:29:10.860 INFO:teuthology.orchestra.run.trial127.stdout:Package librbd1-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:29:10.878 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-12T12:29:10.878 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-12T12:29:10.878 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-12T12:29:10.901 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install python3-rados 2026-02-12T12:29:11.204 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:00:10 ago on Thu 12 Feb 2026 12:29:01 PM UTC. 2026-02-12T12:29:11.268 INFO:teuthology.orchestra.run.trial127.stdout:Package python3-rados-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:29:11.286 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-12T12:29:11.286 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-12T12:29:11.287 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-12T12:29:11.309 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install python3-rgw 2026-02-12T12:29:11.609 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:00:10 ago on Thu 12 Feb 2026 12:29:01 PM UTC. 2026-02-12T12:29:11.675 INFO:teuthology.orchestra.run.trial127.stdout:Package python3-rgw-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:29:11.693 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-12T12:29:11.693 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-12T12:29:11.693 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-12T12:29:11.715 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install python3-cephfs 2026-02-12T12:29:12.019 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:00:11 ago on Thu 12 Feb 2026 12:29:01 PM UTC. 2026-02-12T12:29:12.083 INFO:teuthology.orchestra.run.trial127.stdout:Package python3-cephfs-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:29:12.101 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-12T12:29:12.101 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-12T12:29:12.101 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-12T12:29:12.123 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install python3-rbd 2026-02-12T12:29:12.428 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:00:11 ago on Thu 12 Feb 2026 12:29:01 PM UTC. 2026-02-12T12:29:12.492 INFO:teuthology.orchestra.run.trial127.stdout:Package python3-rbd-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:29:12.510 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-12T12:29:12.511 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-12T12:29:12.511 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-12T12:29:12.531 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install rbd-fuse 2026-02-12T12:29:12.833 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:00:11 ago on Thu 12 Feb 2026 12:29:01 PM UTC. 2026-02-12T12:29:12.897 INFO:teuthology.orchestra.run.trial127.stdout:Package rbd-fuse-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:29:12.915 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-12T12:29:12.915 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-12T12:29:12.915 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-12T12:29:12.937 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install rbd-mirror 2026-02-12T12:29:13.237 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:00:12 ago on Thu 12 Feb 2026 12:29:01 PM UTC. 2026-02-12T12:29:13.303 INFO:teuthology.orchestra.run.trial127.stdout:Package rbd-mirror-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:29:13.321 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-12T12:29:13.322 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-12T12:29:13.322 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-12T12:29:13.344 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install rbd-nbd 2026-02-12T12:29:13.646 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:00:12 ago on Thu 12 Feb 2026 12:29:01 PM UTC. 2026-02-12T12:29:13.711 INFO:teuthology.orchestra.run.trial127.stdout:Package rbd-nbd-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:29:13.729 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-12T12:29:13.729 INFO:teuthology.orchestra.run.trial127.stdout:Nothing to do. 2026-02-12T12:29:13.729 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-12T12:29:13.755 DEBUG:teuthology.parallel:result is None 2026-02-12T12:29:13.756 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0 2026-02-12T12:29:13.918 DEBUG:teuthology.orchestra.run.trial127:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-12T12:29:13.939 INFO:teuthology.orchestra.run.trial127.stdout:20.3.0-5145.g5fdb7d8f.el9 2026-02-12T12:29:13.939 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5145.g5fdb7d8f.el9 2026-02-12T12:29:13.939 INFO:teuthology.task.install:The correct ceph version 20.3.0-5145.g5fdb7d8f is installed. 2026-02-12T12:29:13.942 INFO:teuthology.task.install.util:Utilities already shipped, skip it... 2026-02-12T12:29:13.942 INFO:teuthology.run_tasks:Running task ceph... 2026-02-12T12:29:14.050 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2026-02-12T12:29:14.050 DEBUG:teuthology.orchestra.run.trial127:> sudo chmod 777 /var/log/ceph 2026-02-12T12:29:14.076 INFO:tasks.ceph:Disabling ceph logrotate... 2026-02-12T12:29:14.076 DEBUG:teuthology.orchestra.run.trial127:> sudo rm -f -- /etc/logrotate.d/ceph 2026-02-12T12:29:14.141 INFO:tasks.ceph:Creating extra log directories... 2026-02-12T12:29:14.141 DEBUG:teuthology.orchestra.run.trial127:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2026-02-12T12:29:14.215 INFO:tasks.ceph:Creating ceph cluster ceph... 2026-02-12T12:29:14.215 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore main device type': 'RANDOM_BLOCK_SSD', 'seastore max concurrent transactions': 8}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2026-02-12T12:29:14.215 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/mnt/teuthology/moagrawa-2026-02-10_06:35:43-crimson-rados-main-distro-debug-trial/42503', 'branch': 'main', 'description': 'crimson-rados/perf/{0-crimson_install clusters/{crimson-fixed-1} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/rbm$/{crimson_seastore_rbm} perf_ignorelist settings/optimized workloads/fio_4M_rand_rw}', 'email': None, 'first_in_suite': False, 'flavor': 'debug', 'job_id': '42503', 'kernel': {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'trial', 'name': 'moagrawa-2026-02-10_06:35:43-crimson-rados-main-distro-debug-trial', 'no_nested_subset': False, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd': {'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore main device type': 'RANDOM_BLOCK_SSD', 'seastore max concurrent transactions': 8}}, 'crimson_compat': True, 'flavor': 'debug', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH'], 'sha1': '5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'global': {'osd crush chooseleaf type': 0, 'osd pool default pg num': 128, 'osd pool default pgp num': 128, 'osd pool default size': 2}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'debug', 'sha1': '5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0'}}, 'workunit': {'branch': 'seastore_io_uring', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0'}}, 'owner': 'scheduled_moagrawa@soko04.front.sepia.ceph.com', 'priority': 1000, 'repo': 'https://github.com/ceph/ceph.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 2425, 'sha1': '5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'seastore_io_uring', 'suite_path': '/home/teuthworker/src/github.com_mohit84_ceph_5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/mohit84/ceph.git', 'suite_sha1': '5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0', 'targets': {'trial127.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGdObeFTFkrlBKhCkAGcyYnDwtG7jTAb01C177Aek7c7uMRXKelJQnjrbSpKUeabk2CzMIsbCcy8Niz0Wz8/ERI='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'flavor': 'debug', 'sha1': '5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0'}}, {'pexec': {'all': ['sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson']}}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore main device type': 'RANDOM_BLOCK_SSD', 'seastore max concurrent transactions': 8}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}}, 'wait-for-scrub': False, 'crimson_compat': True, 'flavor': 'debug', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH'], 'sha1': '5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0', 'cluster': 'ceph'}}, {'ssh_keys': None}, {'cbt': {'benchmarks': {'librbdfio': {'iodepth': [32], 'log_avg_msec': 100, 'mode': ['randrw'], 'norandommap': True, 'op_size': [4194304], 'osd_ra': [4096], 'pool_profile': 'rbd', 'procs_per_volume': [1], 'time': 60, 'vol_size': 4096, 'volumes_per_client': [2]}}, 'cluster': {'iterations': 1, 'osds_per_node': 3, 'pool_profiles': {'rbd': {'pg_size': 128, 'pgp_size': 128, 'replication': 3}}, 'user': 'ubuntu'}, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'c433f1062990a0488dc29a553589bc609a460691', 'timestamp': '2026-02-10_06:35:43', 'tube': 'trial', 'user': 'moagrawa', 'verbose': False, 'worker_log': '/home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.2161924'} 2026-02-12T12:29:14.216 DEBUG:teuthology.orchestra.run.trial127:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2026-02-12T12:29:14.260 DEBUG:teuthology.orchestra.run.trial127:> sudo install -d -m0777 -- /var/run/ceph 2026-02-12T12:29:14.325 DEBUG:teuthology.orchestra.run.trial127:> set -ex 2026-02-12T12:29:14.325 DEBUG:teuthology.orchestra.run.trial127:> dd if=/scratch_devs of=/dev/stdout 2026-02-12T12:29:14.383 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2026-02-12T12:29:14.383 DEBUG:teuthology.orchestra.run.trial127:> stat /dev/vg_nvme/lv_1 2026-02-12T12:29:14.439 INFO:teuthology.orchestra.run.trial127.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2026-02-12T12:29:14.439 INFO:teuthology.orchestra.run.trial127.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-12T12:29:14.439 INFO:teuthology.orchestra.run.trial127.stdout:Device: 6h/6d Inode: 1312 Links: 1 2026-02-12T12:29:14.440 INFO:teuthology.orchestra.run.trial127.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-12T12:29:14.440 INFO:teuthology.orchestra.run.trial127.stdout:Context: system_u:object_r:device_t:s0 2026-02-12T12:29:14.440 INFO:teuthology.orchestra.run.trial127.stdout:Access: 2026-02-12 12:15:30.228276401 +0000 2026-02-12T12:29:14.440 INFO:teuthology.orchestra.run.trial127.stdout:Modify: 2026-02-12 12:07:24.895747954 +0000 2026-02-12T12:29:14.440 INFO:teuthology.orchestra.run.trial127.stdout:Change: 2026-02-12 12:07:24.895747954 +0000 2026-02-12T12:29:14.440 INFO:teuthology.orchestra.run.trial127.stdout: Birth: 2026-02-12 12:07:24.895747954 +0000 2026-02-12T12:29:14.440 DEBUG:teuthology.orchestra.run.trial127:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2026-02-12T12:29:14.503 INFO:teuthology.orchestra.run.trial127.stderr:1+0 records in 2026-02-12T12:29:14.503 INFO:teuthology.orchestra.run.trial127.stderr:1+0 records out 2026-02-12T12:29:14.503 INFO:teuthology.orchestra.run.trial127.stderr:512 bytes copied, 0.000156792 s, 3.3 MB/s 2026-02-12T12:29:14.505 DEBUG:teuthology.orchestra.run.trial127:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2026-02-12T12:29:14.562 DEBUG:teuthology.orchestra.run.trial127:> stat /dev/vg_nvme/lv_2 2026-02-12T12:29:14.617 INFO:teuthology.orchestra.run.trial127.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2026-02-12T12:29:14.618 INFO:teuthology.orchestra.run.trial127.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-12T12:29:14.618 INFO:teuthology.orchestra.run.trial127.stdout:Device: 6h/6d Inode: 1299 Links: 1 2026-02-12T12:29:14.618 INFO:teuthology.orchestra.run.trial127.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-12T12:29:14.618 INFO:teuthology.orchestra.run.trial127.stdout:Context: system_u:object_r:device_t:s0 2026-02-12T12:29:14.618 INFO:teuthology.orchestra.run.trial127.stdout:Access: 2026-02-12 12:15:30.229276369 +0000 2026-02-12T12:29:14.618 INFO:teuthology.orchestra.run.trial127.stdout:Modify: 2026-02-12 12:07:24.895747954 +0000 2026-02-12T12:29:14.618 INFO:teuthology.orchestra.run.trial127.stdout:Change: 2026-02-12 12:07:24.895747954 +0000 2026-02-12T12:29:14.618 INFO:teuthology.orchestra.run.trial127.stdout: Birth: 2026-02-12 12:07:24.895747954 +0000 2026-02-12T12:29:14.619 DEBUG:teuthology.orchestra.run.trial127:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2026-02-12T12:29:14.681 INFO:teuthology.orchestra.run.trial127.stderr:1+0 records in 2026-02-12T12:29:14.681 INFO:teuthology.orchestra.run.trial127.stderr:1+0 records out 2026-02-12T12:29:14.681 INFO:teuthology.orchestra.run.trial127.stderr:512 bytes copied, 0.000136043 s, 3.8 MB/s 2026-02-12T12:29:14.682 DEBUG:teuthology.orchestra.run.trial127:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2026-02-12T12:29:14.740 DEBUG:teuthology.orchestra.run.trial127:> stat /dev/vg_nvme/lv_3 2026-02-12T12:29:14.797 INFO:teuthology.orchestra.run.trial127.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2026-02-12T12:29:14.797 INFO:teuthology.orchestra.run.trial127.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-12T12:29:14.797 INFO:teuthology.orchestra.run.trial127.stdout:Device: 6h/6d Inode: 1295 Links: 1 2026-02-12T12:29:14.797 INFO:teuthology.orchestra.run.trial127.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-12T12:29:14.797 INFO:teuthology.orchestra.run.trial127.stdout:Context: system_u:object_r:device_t:s0 2026-02-12T12:29:14.797 INFO:teuthology.orchestra.run.trial127.stdout:Access: 2026-02-12 12:15:30.229276369 +0000 2026-02-12T12:29:14.797 INFO:teuthology.orchestra.run.trial127.stdout:Modify: 2026-02-12 12:07:24.895747954 +0000 2026-02-12T12:29:14.797 INFO:teuthology.orchestra.run.trial127.stdout:Change: 2026-02-12 12:07:24.895747954 +0000 2026-02-12T12:29:14.797 INFO:teuthology.orchestra.run.trial127.stdout: Birth: 2026-02-12 12:07:24.895747954 +0000 2026-02-12T12:29:14.798 DEBUG:teuthology.orchestra.run.trial127:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2026-02-12T12:29:14.860 INFO:teuthology.orchestra.run.trial127.stderr:1+0 records in 2026-02-12T12:29:14.860 INFO:teuthology.orchestra.run.trial127.stderr:1+0 records out 2026-02-12T12:29:14.860 INFO:teuthology.orchestra.run.trial127.stderr:512 bytes copied, 0.000152193 s, 3.4 MB/s 2026-02-12T12:29:14.861 DEBUG:teuthology.orchestra.run.trial127:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2026-02-12T12:29:14.920 DEBUG:teuthology.orchestra.run.trial127:> stat /dev/vg_nvme/lv_4 2026-02-12T12:29:14.976 INFO:teuthology.orchestra.run.trial127.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2026-02-12T12:29:14.977 INFO:teuthology.orchestra.run.trial127.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-12T12:29:14.977 INFO:teuthology.orchestra.run.trial127.stdout:Device: 6h/6d Inode: 1283 Links: 1 2026-02-12T12:29:14.977 INFO:teuthology.orchestra.run.trial127.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-12T12:29:14.977 INFO:teuthology.orchestra.run.trial127.stdout:Context: system_u:object_r:device_t:s0 2026-02-12T12:29:14.977 INFO:teuthology.orchestra.run.trial127.stdout:Access: 2026-02-12 12:15:30.229276369 +0000 2026-02-12T12:29:14.977 INFO:teuthology.orchestra.run.trial127.stdout:Modify: 2026-02-12 12:07:24.894747972 +0000 2026-02-12T12:29:14.977 INFO:teuthology.orchestra.run.trial127.stdout:Change: 2026-02-12 12:07:24.894747972 +0000 2026-02-12T12:29:14.977 INFO:teuthology.orchestra.run.trial127.stdout: Birth: 2026-02-12 12:07:24.894747972 +0000 2026-02-12T12:29:14.978 DEBUG:teuthology.orchestra.run.trial127:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2026-02-12T12:29:15.041 INFO:teuthology.orchestra.run.trial127.stderr:1+0 records in 2026-02-12T12:29:15.041 INFO:teuthology.orchestra.run.trial127.stderr:1+0 records out 2026-02-12T12:29:15.042 INFO:teuthology.orchestra.run.trial127.stderr:512 bytes copied, 0.000153646 s, 3.3 MB/s 2026-02-12T12:29:15.043 DEBUG:teuthology.orchestra.run.trial127:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2026-02-12T12:29:15.100 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-12T12:29:15.100 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@trial127.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2026-02-12T12:29:15.100 INFO:tasks.ceph:Generating config... 2026-02-12T12:29:15.103 INFO:tasks.ceph:[osd] debug monc = 20 2026-02-12T12:29:15.103 INFO:tasks.ceph:[osd] crimson cpu num = 4 2026-02-12T12:29:15.103 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2026-02-12T12:29:15.103 INFO:tasks.ceph:[osd] debug filestore = 0/0 2026-02-12T12:29:15.103 INFO:tasks.ceph:[osd] debug journal = 0/0 2026-02-12T12:29:15.103 INFO:tasks.ceph:[osd] debug ms = 0/0 2026-02-12T12:29:15.103 INFO:tasks.ceph:[osd] debug osd = 0/0 2026-02-12T12:29:15.104 INFO:tasks.ceph:[osd] debug seastore = 20 2026-02-12T12:29:15.104 INFO:tasks.ceph:[osd] debug seastore backref = 20 2026-02-12T12:29:15.104 INFO:tasks.ceph:[osd] debug seastore cache = 20 2026-02-12T12:29:15.104 INFO:tasks.ceph:[osd] debug seastore cleaner = 20 2026-02-12T12:29:15.104 INFO:tasks.ceph:[osd] debug seastore device = 20 2026-02-12T12:29:15.104 INFO:tasks.ceph:[osd] debug seastore epm = 20 2026-02-12T12:29:15.104 INFO:tasks.ceph:[osd] debug seastore fixedkv tree = 20 2026-02-12T12:29:15.104 INFO:tasks.ceph:[osd] debug seastore journal = 20 2026-02-12T12:29:15.104 INFO:tasks.ceph:[osd] debug seastore lba = 20 2026-02-12T12:29:15.104 INFO:tasks.ceph:[osd] debug seastore odata = 20 2026-02-12T12:29:15.104 INFO:tasks.ceph:[osd] debug seastore omap = 20 2026-02-12T12:29:15.105 INFO:tasks.ceph:[osd] debug seastore onode = 20 2026-02-12T12:29:15.105 INFO:tasks.ceph:[osd] debug seastore t = 20 2026-02-12T12:29:15.105 INFO:tasks.ceph:[osd] debug seastore tm = 20 2026-02-12T12:29:15.105 INFO:tasks.ceph:[osd] osd objectstore = seastore 2026-02-12T12:29:15.105 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2026-02-12T12:29:15.105 INFO:tasks.ceph:[osd] seastore cachepin size pershard = 64M 2026-02-12T12:29:15.105 INFO:tasks.ceph:[osd] seastore main device type = RANDOM_BLOCK_SSD 2026-02-12T12:29:15.105 INFO:tasks.ceph:[osd] seastore max concurrent transactions = 8 2026-02-12T12:29:15.105 INFO:tasks.ceph:[global] auth client required = none 2026-02-12T12:29:15.106 INFO:tasks.ceph:[global] auth cluster required = none 2026-02-12T12:29:15.106 INFO:tasks.ceph:[global] auth service required = none 2026-02-12T12:29:15.106 INFO:tasks.ceph:[global] auth supported = none 2026-02-12T12:29:15.106 INFO:tasks.ceph:[global] debug asok = 0/0 2026-02-12T12:29:15.106 INFO:tasks.ceph:[global] debug auth = 0/0 2026-02-12T12:29:15.106 INFO:tasks.ceph:[global] debug bdev = 0/0 2026-02-12T12:29:15.106 INFO:tasks.ceph:[global] debug bluefs = 0/0 2026-02-12T12:29:15.106 INFO:tasks.ceph:[global] debug bluestore = 0/0 2026-02-12T12:29:15.106 INFO:tasks.ceph:[global] debug buffer = 0/0 2026-02-12T12:29:15.106 INFO:tasks.ceph:[global] debug civetweb = 0/0 2026-02-12T12:29:15.106 INFO:tasks.ceph:[global] debug client = 0/0 2026-02-12T12:29:15.106 INFO:tasks.ceph:[global] debug compressor = 0/0 2026-02-12T12:29:15.106 INFO:tasks.ceph:[global] debug context = 0/0 2026-02-12T12:29:15.107 INFO:tasks.ceph:[global] debug crush = 0/0 2026-02-12T12:29:15.107 INFO:tasks.ceph:[global] debug crypto = 0/0 2026-02-12T12:29:15.107 INFO:tasks.ceph:[global] debug dpdk = 0/0 2026-02-12T12:29:15.107 INFO:tasks.ceph:[global] debug eventtrace = 0/0 2026-02-12T12:29:15.107 INFO:tasks.ceph:[global] debug filer = 0/0 2026-02-12T12:29:15.107 INFO:tasks.ceph:[global] debug filestore = 0/0 2026-02-12T12:29:15.107 INFO:tasks.ceph:[global] debug finisher = 0/0 2026-02-12T12:29:15.107 INFO:tasks.ceph:[global] debug fuse = 0/0 2026-02-12T12:29:15.107 INFO:tasks.ceph:[global] debug heartbeatmap = 0/0 2026-02-12T12:29:15.107 INFO:tasks.ceph:[global] debug javaclient = 0/0 2026-02-12T12:29:15.107 INFO:tasks.ceph:[global] debug journal = 0/0 2026-02-12T12:29:15.107 INFO:tasks.ceph:[global] debug journaler = 0/0 2026-02-12T12:29:15.108 INFO:tasks.ceph:[global] debug kstore = 0/0 2026-02-12T12:29:15.108 INFO:tasks.ceph:[global] debug leveldb = 0/0 2026-02-12T12:29:15.108 INFO:tasks.ceph:[global] debug lockdep = 0/0 2026-02-12T12:29:15.108 INFO:tasks.ceph:[global] debug mds = 0/0 2026-02-12T12:29:15.108 INFO:tasks.ceph:[global] debug mds balancer = 0/0 2026-02-12T12:29:15.108 INFO:tasks.ceph:[global] debug mds locker = 0/0 2026-02-12T12:29:15.108 INFO:tasks.ceph:[global] debug mds log = 0/0 2026-02-12T12:29:15.108 INFO:tasks.ceph:[global] debug mds log expire = 0/0 2026-02-12T12:29:15.108 INFO:tasks.ceph:[global] debug mds migrator = 0/0 2026-02-12T12:29:15.108 INFO:tasks.ceph:[global] debug memdb = 0/0 2026-02-12T12:29:15.108 INFO:tasks.ceph:[global] debug mgr = 0/0 2026-02-12T12:29:15.108 INFO:tasks.ceph:[global] debug mgrc = 0/0 2026-02-12T12:29:15.108 INFO:tasks.ceph:[global] debug mon = 0/0 2026-02-12T12:29:15.108 INFO:tasks.ceph:[global] debug monc = 0/0 2026-02-12T12:29:15.109 INFO:tasks.ceph:[global] debug ms = 0/0 2026-02-12T12:29:15.109 INFO:tasks.ceph:[global] debug objclass = 0/0 2026-02-12T12:29:15.109 INFO:tasks.ceph:[global] debug objectcacher = 0/0 2026-02-12T12:29:15.109 INFO:tasks.ceph:[global] debug objecter = 0/0 2026-02-12T12:29:15.109 INFO:tasks.ceph:[global] debug optracker = 0/0 2026-02-12T12:29:15.109 INFO:tasks.ceph:[global] debug osd = 0/0 2026-02-12T12:29:15.109 INFO:tasks.ceph:[global] debug paxos = 0/0 2026-02-12T12:29:15.109 INFO:tasks.ceph:[global] debug perfcounter = 0/0 2026-02-12T12:29:15.109 INFO:tasks.ceph:[global] debug rados = 0/0 2026-02-12T12:29:15.109 INFO:tasks.ceph:[global] debug rbd = 0/0 2026-02-12T12:29:15.109 INFO:tasks.ceph:[global] debug rbd mirror = 0/0 2026-02-12T12:29:15.109 INFO:tasks.ceph:[global] debug rbd replay = 0/0 2026-02-12T12:29:15.109 INFO:tasks.ceph:[global] debug refs = 0/0 2026-02-12T12:29:15.110 INFO:tasks.ceph:[global] debug rgw = 0/0 2026-02-12T12:29:15.110 INFO:tasks.ceph:[global] debug rgw lifecycle = 0/0 2026-02-12T12:29:15.110 INFO:tasks.ceph:[global] debug rgw notification = 0/0 2026-02-12T12:29:15.110 INFO:tasks.ceph:[global] debug rgw sync = 0/0 2026-02-12T12:29:15.110 INFO:tasks.ceph:[global] debug rocksdb = 0/0 2026-02-12T12:29:15.110 INFO:tasks.ceph:[global] debug striper = 0/0 2026-02-12T12:29:15.110 INFO:tasks.ceph:[global] debug throttle = 0/0 2026-02-12T12:29:15.110 INFO:tasks.ceph:[global] debug timer = 0/0 2026-02-12T12:29:15.110 INFO:tasks.ceph:[global] debug tp = 0/0 2026-02-12T12:29:15.110 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2026-02-12T12:29:15.110 INFO:tasks.ceph:[global] ms client mode = crc 2026-02-12T12:29:15.110 INFO:tasks.ceph:[global] ms cluster mode = crc 2026-02-12T12:29:15.111 INFO:tasks.ceph:[global] ms mon client mode = crc 2026-02-12T12:29:15.111 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2026-02-12T12:29:15.111 INFO:tasks.ceph:[global] ms mon service mode = crc 2026-02-12T12:29:15.111 INFO:tasks.ceph:[global] ms service mode = crc 2026-02-12T12:29:15.111 INFO:tasks.ceph:[mgr] debug mgr = 20 2026-02-12T12:29:15.111 INFO:tasks.ceph:[mgr] debug ms = 1 2026-02-12T12:29:15.111 INFO:tasks.ceph:[mon] debug mon = 0/0 2026-02-12T12:29:15.111 INFO:tasks.ceph:[mon] debug ms = 0/0 2026-02-12T12:29:15.111 INFO:tasks.ceph:[mon] debug paxos = 0/0 2026-02-12T12:29:15.111 INFO:tasks.ceph:[mon] osd pool default crimson = True 2026-02-12T12:29:15.112 INFO:tasks.ceph:Setting up mon.a... 2026-02-12T12:29:15.112 DEBUG:teuthology.orchestra.run.trial127:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2026-02-12T12:29:15.183 INFO:teuthology.orchestra.run.trial127.stdout:creating /etc/ceph/ceph.keyring 2026-02-12T12:29:15.186 DEBUG:teuthology.orchestra.run.trial127:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2026-02-12T12:29:15.273 DEBUG:teuthology.orchestra.run.trial127:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-12T12:29:15.297 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '10.20.193.127')] 2026-02-12T12:29:15.298 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'none', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '10.20.193.127', 'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': 'true', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore main device type': 'RANDOM_BLOCK_SSD', 'seastore max concurrent transactions': 8}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': '0/0', 'debug mon': '0/0', 'debug paxos': '0/0', 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}} 2026-02-12T12:29:15.300 DEBUG:teuthology.orchestra.run.trial127:> set -ex 2026-02-12T12:29:15.300 DEBUG:teuthology.orchestra.run.trial127:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2026-02-12T12:29:15.355 DEBUG:teuthology.orchestra.run.trial127:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 10.20.193.127 --print /home/ubuntu/cephtest/ceph.monmap 2026-02-12T12:29:15.435 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-12T12:29:15.434+0000 7fc5e2088f40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:29:15.436 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-12T12:29:15.434+0000 7fc5e2088f40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:29:15.436 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-12T12:29:15.434+0000 7fc5e2088f40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:29:15.437 INFO:teuthology.orchestra.run.trial127.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2026-02-12T12:29:15.437 INFO:teuthology.orchestra.run.trial127.stdout:monmaptool: generated fsid 0d04f462-0e99-4bbd-8f83-79681663314a 2026-02-12T12:29:15.437 INFO:teuthology.orchestra.run.trial127.stdout:setting min_mon_release = tentacle 2026-02-12T12:29:15.437 INFO:teuthology.orchestra.run.trial127.stdout:epoch 0 2026-02-12T12:29:15.437 INFO:teuthology.orchestra.run.trial127.stdout:fsid 0d04f462-0e99-4bbd-8f83-79681663314a 2026-02-12T12:29:15.437 INFO:teuthology.orchestra.run.trial127.stdout:last_changed 2026-02-12T12:29:15.435140+0000 2026-02-12T12:29:15.438 INFO:teuthology.orchestra.run.trial127.stdout:created 2026-02-12T12:29:15.435140+0000 2026-02-12T12:29:15.438 INFO:teuthology.orchestra.run.trial127.stdout:min_mon_release 20 (tentacle) 2026-02-12T12:29:15.438 INFO:teuthology.orchestra.run.trial127.stdout:election_strategy: 1 2026-02-12T12:29:15.438 INFO:teuthology.orchestra.run.trial127.stdout:0: [v2:10.20.193.127:3300/0,v1:10.20.193.127:6789/0] mon.a 2026-02-12T12:29:15.438 INFO:teuthology.orchestra.run.trial127.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2026-02-12T12:29:15.439 DEBUG:teuthology.orchestra.run.trial127:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2026-02-12T12:29:15.495 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 0d04f462-0e99-4bbd-8f83-79681663314a... 2026-02-12T12:29:15.496 DEBUG:teuthology.orchestra.run.trial127:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2026-02-12T12:29:15.576 INFO:teuthology.orchestra.run.trial127.stdout:[global] 2026-02-12T12:29:15.576 INFO:teuthology.orchestra.run.trial127.stdout: chdir = "" 2026-02-12T12:29:15.576 INFO:teuthology.orchestra.run.trial127.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2026-02-12T12:29:15.576 INFO:teuthology.orchestra.run.trial127.stdout: auth supported = none 2026-02-12T12:29:15.576 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:29:15.577 INFO:teuthology.orchestra.run.trial127.stdout: filestore xattr use omap = true 2026-02-12T12:29:15.577 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:29:15.577 INFO:teuthology.orchestra.run.trial127.stdout: mon clock drift allowed = 1.000 2026-02-12T12:29:15.577 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:29:15.577 INFO:teuthology.orchestra.run.trial127.stdout: osd crush chooseleaf type = 0 2026-02-12T12:29:15.577 INFO:teuthology.orchestra.run.trial127.stdout: auth debug = true 2026-02-12T12:29:15.577 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:29:15.577 INFO:teuthology.orchestra.run.trial127.stdout: ms die on old message = true 2026-02-12T12:29:15.577 INFO:teuthology.orchestra.run.trial127.stdout: ms die on bug = true 2026-02-12T12:29:15.577 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:29:15.577 INFO:teuthology.orchestra.run.trial127.stdout: mon max pg per osd = 10000 # >= luminous 2026-02-12T12:29:15.577 INFO:teuthology.orchestra.run.trial127.stdout: mon pg warn max object skew = 0 2026-02-12T12:29:15.578 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:29:15.578 INFO:teuthology.orchestra.run.trial127.stdout: # disable pg_autoscaler by default for new pools 2026-02-12T12:29:15.578 INFO:teuthology.orchestra.run.trial127.stdout: osd_pool_default_pg_autoscale_mode = off 2026-02-12T12:29:15.578 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:29:15.578 INFO:teuthology.orchestra.run.trial127.stdout: osd pool default size = 2 2026-02-12T12:29:15.578 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:29:15.578 INFO:teuthology.orchestra.run.trial127.stdout: mon osd allow primary affinity = true 2026-02-12T12:29:15.578 INFO:teuthology.orchestra.run.trial127.stdout: mon osd allow pg remap = true 2026-02-12T12:29:15.578 INFO:teuthology.orchestra.run.trial127.stdout: mon warn on legacy crush tunables = false 2026-02-12T12:29:15.578 INFO:teuthology.orchestra.run.trial127.stdout: mon warn on crush straw calc version zero = false 2026-02-12T12:29:15.578 INFO:teuthology.orchestra.run.trial127.stdout: mon warn on no sortbitwise = false 2026-02-12T12:29:15.578 INFO:teuthology.orchestra.run.trial127.stdout: mon warn on osd down out interval zero = false 2026-02-12T12:29:15.579 INFO:teuthology.orchestra.run.trial127.stdout: mon warn on too few osds = false 2026-02-12T12:29:15.579 INFO:teuthology.orchestra.run.trial127.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2026-02-12T12:29:15.579 INFO:teuthology.orchestra.run.trial127.stdout: mon_warn_on_pool_no_redundancy = false 2026-02-12T12:29:15.579 INFO:teuthology.orchestra.run.trial127.stdout: mon_allow_pool_size_one = true 2026-02-12T12:29:15.579 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:29:15.579 INFO:teuthology.orchestra.run.trial127.stdout: osd pool default erasure code profile = plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2026-02-12T12:29:15.579 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:29:15.579 INFO:teuthology.orchestra.run.trial127.stdout: osd default data pool replay window = 5 2026-02-12T12:29:15.579 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:29:15.579 INFO:teuthology.orchestra.run.trial127.stdout: mon allow pool delete = true 2026-02-12T12:29:15.579 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:29:15.579 INFO:teuthology.orchestra.run.trial127.stdout: mon cluster log file level = debug 2026-02-12T12:29:15.580 INFO:teuthology.orchestra.run.trial127.stdout: debug asserts on shutdown = true 2026-02-12T12:29:15.580 INFO:teuthology.orchestra.run.trial127.stdout: mon health detail to clog = false 2026-02-12T12:29:15.580 INFO:teuthology.orchestra.run.trial127.stdout: mon host = 10.20.193.127 2026-02-12T12:29:15.580 INFO:teuthology.orchestra.run.trial127.stdout: auth client required = none 2026-02-12T12:29:15.580 INFO:teuthology.orchestra.run.trial127.stdout: auth cluster required = none 2026-02-12T12:29:15.580 INFO:teuthology.orchestra.run.trial127.stdout: auth service required = none 2026-02-12T12:29:15.580 INFO:teuthology.orchestra.run.trial127.stdout: debug asok = 0/0 2026-02-12T12:29:15.580 INFO:teuthology.orchestra.run.trial127.stdout: debug auth = 0/0 2026-02-12T12:29:15.580 INFO:teuthology.orchestra.run.trial127.stdout: debug bdev = 0/0 2026-02-12T12:29:15.580 INFO:teuthology.orchestra.run.trial127.stdout: debug bluefs = 0/0 2026-02-12T12:29:15.580 INFO:teuthology.orchestra.run.trial127.stdout: debug bluestore = 0/0 2026-02-12T12:29:15.580 INFO:teuthology.orchestra.run.trial127.stdout: debug buffer = 0/0 2026-02-12T12:29:15.581 INFO:teuthology.orchestra.run.trial127.stdout: debug civetweb = 0/0 2026-02-12T12:29:15.581 INFO:teuthology.orchestra.run.trial127.stdout: debug client = 0/0 2026-02-12T12:29:15.581 INFO:teuthology.orchestra.run.trial127.stdout: debug compressor = 0/0 2026-02-12T12:29:15.581 INFO:teuthology.orchestra.run.trial127.stdout: debug context = 0/0 2026-02-12T12:29:15.581 INFO:teuthology.orchestra.run.trial127.stdout: debug crush = 0/0 2026-02-12T12:29:15.581 INFO:teuthology.orchestra.run.trial127.stdout: debug crypto = 0/0 2026-02-12T12:29:15.581 INFO:teuthology.orchestra.run.trial127.stdout: debug dpdk = 0/0 2026-02-12T12:29:15.581 INFO:teuthology.orchestra.run.trial127.stdout: debug eventtrace = 0/0 2026-02-12T12:29:15.581 INFO:teuthology.orchestra.run.trial127.stdout: debug filer = 0/0 2026-02-12T12:29:15.581 INFO:teuthology.orchestra.run.trial127.stdout: debug filestore = 0/0 2026-02-12T12:29:15.581 INFO:teuthology.orchestra.run.trial127.stdout: debug finisher = 0/0 2026-02-12T12:29:15.581 INFO:teuthology.orchestra.run.trial127.stdout: debug fuse = 0/0 2026-02-12T12:29:15.581 INFO:teuthology.orchestra.run.trial127.stdout: debug heartbeatmap = 0/0 2026-02-12T12:29:15.581 INFO:teuthology.orchestra.run.trial127.stdout: debug javaclient = 0/0 2026-02-12T12:29:15.582 INFO:teuthology.orchestra.run.trial127.stdout: debug journal = 0/0 2026-02-12T12:29:15.582 INFO:teuthology.orchestra.run.trial127.stdout: debug journaler = 0/0 2026-02-12T12:29:15.582 INFO:teuthology.orchestra.run.trial127.stdout: debug kstore = 0/0 2026-02-12T12:29:15.582 INFO:teuthology.orchestra.run.trial127.stdout: debug leveldb = 0/0 2026-02-12T12:29:15.582 INFO:teuthology.orchestra.run.trial127.stdout: debug lockdep = 0/0 2026-02-12T12:29:15.582 INFO:teuthology.orchestra.run.trial127.stdout: debug mds = 0/0 2026-02-12T12:29:15.582 INFO:teuthology.orchestra.run.trial127.stdout: debug mds balancer = 0/0 2026-02-12T12:29:15.582 INFO:teuthology.orchestra.run.trial127.stdout: debug mds locker = 0/0 2026-02-12T12:29:15.582 INFO:teuthology.orchestra.run.trial127.stdout: debug mds log = 0/0 2026-02-12T12:29:15.582 INFO:teuthology.orchestra.run.trial127.stdout: debug mds log expire = 0/0 2026-02-12T12:29:15.582 INFO:teuthology.orchestra.run.trial127.stdout: debug mds migrator = 0/0 2026-02-12T12:29:15.582 INFO:teuthology.orchestra.run.trial127.stdout: debug memdb = 0/0 2026-02-12T12:29:15.582 INFO:teuthology.orchestra.run.trial127.stdout: debug mgr = 0/0 2026-02-12T12:29:15.583 INFO:teuthology.orchestra.run.trial127.stdout: debug mgrc = 0/0 2026-02-12T12:29:15.583 INFO:teuthology.orchestra.run.trial127.stdout: debug mon = 0/0 2026-02-12T12:29:15.583 INFO:teuthology.orchestra.run.trial127.stdout: debug monc = 0/0 2026-02-12T12:29:15.583 INFO:teuthology.orchestra.run.trial127.stdout: debug ms = 0/0 2026-02-12T12:29:15.583 INFO:teuthology.orchestra.run.trial127.stdout: debug objclass = 0/0 2026-02-12T12:29:15.583 INFO:teuthology.orchestra.run.trial127.stdout: debug objectcacher = 0/0 2026-02-12T12:29:15.583 INFO:teuthology.orchestra.run.trial127.stdout: debug objecter = 0/0 2026-02-12T12:29:15.583 INFO:teuthology.orchestra.run.trial127.stdout: debug optracker = 0/0 2026-02-12T12:29:15.583 INFO:teuthology.orchestra.run.trial127.stdout: debug osd = 0/0 2026-02-12T12:29:15.583 INFO:teuthology.orchestra.run.trial127.stdout: debug paxos = 0/0 2026-02-12T12:29:15.583 INFO:teuthology.orchestra.run.trial127.stdout: debug perfcounter = 0/0 2026-02-12T12:29:15.583 INFO:teuthology.orchestra.run.trial127.stdout: debug rados = 0/0 2026-02-12T12:29:15.583 INFO:teuthology.orchestra.run.trial127.stdout: debug rbd = 0/0 2026-02-12T12:29:15.584 INFO:teuthology.orchestra.run.trial127.stdout: debug rbd mirror = 0/0 2026-02-12T12:29:15.584 INFO:teuthology.orchestra.run.trial127.stdout: debug rbd replay = 0/0 2026-02-12T12:29:15.584 INFO:teuthology.orchestra.run.trial127.stdout: debug refs = 0/0 2026-02-12T12:29:15.584 INFO:teuthology.orchestra.run.trial127.stdout: debug rgw = 0/0 2026-02-12T12:29:15.584 INFO:teuthology.orchestra.run.trial127.stdout: debug rgw lifecycle = 0/0 2026-02-12T12:29:15.584 INFO:teuthology.orchestra.run.trial127.stdout: debug rgw notification = 0/0 2026-02-12T12:29:15.584 INFO:teuthology.orchestra.run.trial127.stdout: debug rgw sync = 0/0 2026-02-12T12:29:15.584 INFO:teuthology.orchestra.run.trial127.stdout: debug rocksdb = 0/0 2026-02-12T12:29:15.584 INFO:teuthology.orchestra.run.trial127.stdout: debug striper = 0/0 2026-02-12T12:29:15.584 INFO:teuthology.orchestra.run.trial127.stdout: debug throttle = 0/0 2026-02-12T12:29:15.584 INFO:teuthology.orchestra.run.trial127.stdout: debug timer = 0/0 2026-02-12T12:29:15.584 INFO:teuthology.orchestra.run.trial127.stdout: debug tp = 0/0 2026-02-12T12:29:15.584 INFO:teuthology.orchestra.run.trial127.stdout: enable experimental unrecoverable data corrupting features = crimson 2026-02-12T12:29:15.585 INFO:teuthology.orchestra.run.trial127.stdout: ms client mode = crc 2026-02-12T12:29:15.585 INFO:teuthology.orchestra.run.trial127.stdout: ms cluster mode = crc 2026-02-12T12:29:15.585 INFO:teuthology.orchestra.run.trial127.stdout: ms mon client mode = crc 2026-02-12T12:29:15.585 INFO:teuthology.orchestra.run.trial127.stdout: ms mon cluster mode = crc 2026-02-12T12:29:15.585 INFO:teuthology.orchestra.run.trial127.stdout: ms mon service mode = crc 2026-02-12T12:29:15.585 INFO:teuthology.orchestra.run.trial127.stdout: ms service mode = crc 2026-02-12T12:29:15.585 INFO:teuthology.orchestra.run.trial127.stdout: fsid = 0d04f462-0e99-4bbd-8f83-79681663314a 2026-02-12T12:29:15.585 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:29:15.585 INFO:teuthology.orchestra.run.trial127.stdout:[osd] 2026-02-12T12:29:15.585 INFO:teuthology.orchestra.run.trial127.stdout: osd journal size = 100 2026-02-12T12:29:15.585 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:29:15.585 INFO:teuthology.orchestra.run.trial127.stdout: osd scrub load threshold = 5.0 2026-02-12T12:29:15.585 INFO:teuthology.orchestra.run.trial127.stdout: osd scrub max interval = 600 2026-02-12T12:29:15.586 INFO:teuthology.orchestra.run.trial127.stdout: osd mclock profile = high_recovery_ops 2026-02-12T12:29:15.586 INFO:teuthology.orchestra.run.trial127.stdout: osd mclock skip benchmark = true 2026-02-12T12:29:15.586 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:29:15.586 INFO:teuthology.orchestra.run.trial127.stdout: osd recover clone overlap = true 2026-02-12T12:29:15.586 INFO:teuthology.orchestra.run.trial127.stdout: osd recovery max chunk = 1048576 2026-02-12T12:29:15.586 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:29:15.586 INFO:teuthology.orchestra.run.trial127.stdout: osd debug shutdown = true 2026-02-12T12:29:15.586 INFO:teuthology.orchestra.run.trial127.stdout: osd debug op order = true 2026-02-12T12:29:15.586 INFO:teuthology.orchestra.run.trial127.stdout: osd debug verify stray on activate = true 2026-02-12T12:29:15.586 INFO:teuthology.orchestra.run.trial127.stdout: osd debug trim objects = true 2026-02-12T12:29:15.586 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:29:15.586 INFO:teuthology.orchestra.run.trial127.stdout: osd open classes on start = true 2026-02-12T12:29:15.587 INFO:teuthology.orchestra.run.trial127.stdout: osd debug pg log writeout = true 2026-02-12T12:29:15.587 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:29:15.587 INFO:teuthology.orchestra.run.trial127.stdout: osd deep scrub update digest min age = 30 2026-02-12T12:29:15.587 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:29:15.587 INFO:teuthology.orchestra.run.trial127.stdout: osd map max advance = 10 2026-02-12T12:29:15.587 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:29:15.587 INFO:teuthology.orchestra.run.trial127.stdout: journal zero on create = true 2026-02-12T12:29:15.587 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:29:15.587 INFO:teuthology.orchestra.run.trial127.stdout: filestore ondisk finisher threads = 3 2026-02-12T12:29:15.587 INFO:teuthology.orchestra.run.trial127.stdout: filestore apply finisher threads = 3 2026-02-12T12:29:15.587 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:29:15.587 INFO:teuthology.orchestra.run.trial127.stdout: bdev debug aio = true 2026-02-12T12:29:15.587 INFO:teuthology.orchestra.run.trial127.stdout: osd debug misdirected ops = true 2026-02-12T12:29:15.588 INFO:teuthology.orchestra.run.trial127.stdout: debug monc = 20 2026-02-12T12:29:15.588 INFO:teuthology.orchestra.run.trial127.stdout: crimson cpu num = 4 2026-02-12T12:29:15.588 INFO:teuthology.orchestra.run.trial127.stdout: crimson osd obc lru size = 10 2026-02-12T12:29:15.588 INFO:teuthology.orchestra.run.trial127.stdout: debug filestore = 0/0 2026-02-12T12:29:15.588 INFO:teuthology.orchestra.run.trial127.stdout: debug journal = 0/0 2026-02-12T12:29:15.588 INFO:teuthology.orchestra.run.trial127.stdout: debug ms = 0/0 2026-02-12T12:29:15.588 INFO:teuthology.orchestra.run.trial127.stdout: debug osd = 0/0 2026-02-12T12:29:15.588 INFO:teuthology.orchestra.run.trial127.stdout: debug seastore = 20 2026-02-12T12:29:15.588 INFO:teuthology.orchestra.run.trial127.stdout: debug seastore backref = 20 2026-02-12T12:29:15.588 INFO:teuthology.orchestra.run.trial127.stdout: debug seastore cache = 20 2026-02-12T12:29:15.588 INFO:teuthology.orchestra.run.trial127.stdout: debug seastore cleaner = 20 2026-02-12T12:29:15.588 INFO:teuthology.orchestra.run.trial127.stdout: debug seastore device = 20 2026-02-12T12:29:15.588 INFO:teuthology.orchestra.run.trial127.stdout: debug seastore epm = 20 2026-02-12T12:29:15.589 INFO:teuthology.orchestra.run.trial127.stdout: debug seastore fixedkv tree = 20 2026-02-12T12:29:15.589 INFO:teuthology.orchestra.run.trial127.stdout: debug seastore journal = 20 2026-02-12T12:29:15.589 INFO:teuthology.orchestra.run.trial127.stdout: debug seastore lba = 20 2026-02-12T12:29:15.589 INFO:teuthology.orchestra.run.trial127.stdout: debug seastore odata = 20 2026-02-12T12:29:15.589 INFO:teuthology.orchestra.run.trial127.stdout: debug seastore omap = 20 2026-02-12T12:29:15.589 INFO:teuthology.orchestra.run.trial127.stdout: debug seastore onode = 20 2026-02-12T12:29:15.589 INFO:teuthology.orchestra.run.trial127.stdout: debug seastore t = 20 2026-02-12T12:29:15.589 INFO:teuthology.orchestra.run.trial127.stdout: debug seastore tm = 20 2026-02-12T12:29:15.589 INFO:teuthology.orchestra.run.trial127.stdout: osd objectstore = seastore 2026-02-12T12:29:15.589 INFO:teuthology.orchestra.run.trial127.stdout: osd shutdown pgref assert = True 2026-02-12T12:29:15.589 INFO:teuthology.orchestra.run.trial127.stdout: seastore cachepin size pershard = 64M 2026-02-12T12:29:15.589 INFO:teuthology.orchestra.run.trial127.stdout: seastore main device type = RANDOM_BLOCK_SSD 2026-02-12T12:29:15.589 INFO:teuthology.orchestra.run.trial127.stdout: seastore max concurrent transactions = 8 2026-02-12T12:29:15.590 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:29:15.590 INFO:teuthology.orchestra.run.trial127.stdout:[mgr] 2026-02-12T12:29:15.590 INFO:teuthology.orchestra.run.trial127.stdout: debug ms = 1 2026-02-12T12:29:15.590 INFO:teuthology.orchestra.run.trial127.stdout: debug mgr = 20 2026-02-12T12:29:15.590 INFO:teuthology.orchestra.run.trial127.stdout: debug mon = 20 2026-02-12T12:29:15.590 INFO:teuthology.orchestra.run.trial127.stdout: debug auth = 20 2026-02-12T12:29:15.590 INFO:teuthology.orchestra.run.trial127.stdout: mon reweight min pgs per osd = 4 2026-02-12T12:29:15.590 INFO:teuthology.orchestra.run.trial127.stdout: mon reweight min bytes per osd = 10 2026-02-12T12:29:15.590 INFO:teuthology.orchestra.run.trial127.stdout: mgr/telemetry/nag = false 2026-02-12T12:29:15.590 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:29:15.590 INFO:teuthology.orchestra.run.trial127.stdout:[mon] 2026-02-12T12:29:15.590 INFO:teuthology.orchestra.run.trial127.stdout: debug ms = 0/0 2026-02-12T12:29:15.591 INFO:teuthology.orchestra.run.trial127.stdout: debug mon = 0/0 2026-02-12T12:29:15.591 INFO:teuthology.orchestra.run.trial127.stdout: debug paxos = 0/0 2026-02-12T12:29:15.591 INFO:teuthology.orchestra.run.trial127.stdout: debug auth = 20 2026-02-12T12:29:15.591 INFO:teuthology.orchestra.run.trial127.stdout: mon data avail warn = 5 2026-02-12T12:29:15.591 INFO:teuthology.orchestra.run.trial127.stdout: mon mgr mkfs grace = 240 2026-02-12T12:29:15.591 INFO:teuthology.orchestra.run.trial127.stdout: mon reweight min pgs per osd = 4 2026-02-12T12:29:15.591 INFO:teuthology.orchestra.run.trial127.stdout: mon osd reporter subtree level = osd 2026-02-12T12:29:15.591 INFO:teuthology.orchestra.run.trial127.stdout: mon osd prime pg temp = true 2026-02-12T12:29:15.591 INFO:teuthology.orchestra.run.trial127.stdout: mon reweight min bytes per osd = 10 2026-02-12T12:29:15.591 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:29:15.591 INFO:teuthology.orchestra.run.trial127.stdout: # rotate auth tickets quickly to exercise renewal paths 2026-02-12T12:29:15.591 INFO:teuthology.orchestra.run.trial127.stdout: auth mon ticket ttl = 660 # 11m 2026-02-12T12:29:15.591 INFO:teuthology.orchestra.run.trial127.stdout: auth service ticket ttl = 240 # 4m 2026-02-12T12:29:15.592 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:29:15.592 INFO:teuthology.orchestra.run.trial127.stdout: # don't complain about insecure global_id in the test suite 2026-02-12T12:29:15.592 INFO:teuthology.orchestra.run.trial127.stdout: mon_warn_on_insecure_global_id_reclaim = false 2026-02-12T12:29:15.592 INFO:teuthology.orchestra.run.trial127.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2026-02-12T12:29:15.592 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:29:15.592 INFO:teuthology.orchestra.run.trial127.stdout: # 1m isn't quite enough 2026-02-12T12:29:15.592 INFO:teuthology.orchestra.run.trial127.stdout: mon_down_mkfs_grace = 2m 2026-02-12T12:29:15.592 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:29:15.592 INFO:teuthology.orchestra.run.trial127.stdout: mon_warn_on_filestore_osds = false 2026-02-12T12:29:15.592 INFO:teuthology.orchestra.run.trial127.stdout: osd pool default crimson = True 2026-02-12T12:29:15.592 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:29:15.592 INFO:teuthology.orchestra.run.trial127.stdout:[client] 2026-02-12T12:29:15.592 INFO:teuthology.orchestra.run.trial127.stdout: rgw cache enabled = true 2026-02-12T12:29:15.593 INFO:teuthology.orchestra.run.trial127.stdout: rgw enable ops log = true 2026-02-12T12:29:15.593 INFO:teuthology.orchestra.run.trial127.stdout: rgw enable usage log = true 2026-02-12T12:29:15.593 INFO:teuthology.orchestra.run.trial127.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2026-02-12T12:29:15.593 INFO:teuthology.orchestra.run.trial127.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2026-02-12T12:29:15.593 INFO:teuthology.orchestra.run.trial127.stdout:[mon.a] 2026-02-12T12:29:15.594 INFO:tasks.ceph:Creating admin key on mon.a... 2026-02-12T12:29:15.594 DEBUG:teuthology.orchestra.run.trial127:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2026-02-12T12:29:15.686 INFO:tasks.ceph:Copying monmap to all nodes... 2026-02-12T12:29:15.686 DEBUG:teuthology.orchestra.run.trial127:> set -ex 2026-02-12T12:29:15.686 DEBUG:teuthology.orchestra.run.trial127:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2026-02-12T12:29:15.703 DEBUG:teuthology.orchestra.run.trial127:> set -ex 2026-02-12T12:29:15.703 DEBUG:teuthology.orchestra.run.trial127:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2026-02-12T12:29:15.761 INFO:tasks.ceph:Sending monmap to node ubuntu@trial127.front.sepia.ceph.com 2026-02-12T12:29:15.761 DEBUG:teuthology.orchestra.run.trial127:> set -ex 2026-02-12T12:29:15.762 DEBUG:teuthology.orchestra.run.trial127:> sudo dd of=/etc/ceph/ceph.keyring 2026-02-12T12:29:15.762 DEBUG:teuthology.orchestra.run.trial127:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-12T12:29:15.835 DEBUG:teuthology.orchestra.run.trial127:> set -ex 2026-02-12T12:29:15.835 DEBUG:teuthology.orchestra.run.trial127:> dd of=/home/ubuntu/cephtest/ceph.monmap 2026-02-12T12:29:15.892 INFO:tasks.ceph:Setting up mon nodes... 2026-02-12T12:29:15.892 INFO:tasks.ceph:Setting up mgr nodes... 2026-02-12T12:29:15.893 DEBUG:teuthology.orchestra.run.trial127:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2026-02-12T12:29:15.981 INFO:teuthology.orchestra.run.trial127.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2026-02-12T12:29:15.984 INFO:tasks.ceph:Setting up mds nodes... 2026-02-12T12:29:15.984 INFO:tasks.ceph_client:Setting up client nodes... 2026-02-12T12:29:15.985 DEBUG:teuthology.orchestra.run.trial127:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2026-02-12T12:29:16.025 INFO:teuthology.orchestra.run.trial127.stdout:creating /etc/ceph/ceph.client.0.keyring 2026-02-12T12:29:16.037 INFO:tasks.ceph:Running mkfs on osd nodes... 2026-02-12T12:29:16.038 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@trial127.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2026-02-12T12:29:16.038 DEBUG:teuthology.orchestra.run.trial127:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2026-02-12T12:29:16.101 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-12T12:29:16.101 INFO:tasks.ceph:role: osd.0 2026-02-12T12:29:16.102 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@trial127.front.sepia.ceph.com 2026-02-12T12:29:16.102 DEBUG:teuthology.orchestra.run.trial127:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2026-02-12T12:29:16.166 INFO:teuthology.orchestra.run.trial127.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=45875200 blks 2026-02-12T12:29:16.167 INFO:teuthology.orchestra.run.trial127.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-12T12:29:16.167 INFO:teuthology.orchestra.run.trial127.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-12T12:29:16.167 INFO:teuthology.orchestra.run.trial127.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-12T12:29:16.167 INFO:teuthology.orchestra.run.trial127.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-12T12:29:16.167 INFO:teuthology.orchestra.run.trial127.stdout: = sunit=0 swidth=0 blks 2026-02-12T12:29:16.167 INFO:teuthology.orchestra.run.trial127.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-12T12:29:16.167 INFO:teuthology.orchestra.run.trial127.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-12T12:29:16.167 INFO:teuthology.orchestra.run.trial127.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-12T12:29:16.167 INFO:teuthology.orchestra.run.trial127.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-12T12:29:16.746 INFO:teuthology.orchestra.run.trial127.stdout:Discarding blocks...Done. 2026-02-12T12:29:16.748 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@trial127.front.sepia.ceph.com -o noatime 2026-02-12T12:29:16.748 DEBUG:teuthology.orchestra.run.trial127:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2026-02-12T12:29:16.816 DEBUG:teuthology.orchestra.run.trial127:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2026-02-12T12:29:16.884 DEBUG:teuthology.orchestra.run.trial127:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2026-02-12T12:29:16.949 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-12T12:29:16.949 INFO:tasks.ceph:role: osd.1 2026-02-12T12:29:16.949 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@trial127.front.sepia.ceph.com 2026-02-12T12:29:16.949 DEBUG:teuthology.orchestra.run.trial127:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2026-02-12T12:29:17.014 INFO:teuthology.orchestra.run.trial127.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=45875200 blks 2026-02-12T12:29:17.014 INFO:teuthology.orchestra.run.trial127.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-12T12:29:17.014 INFO:teuthology.orchestra.run.trial127.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-12T12:29:17.014 INFO:teuthology.orchestra.run.trial127.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-12T12:29:17.014 INFO:teuthology.orchestra.run.trial127.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-12T12:29:17.015 INFO:teuthology.orchestra.run.trial127.stdout: = sunit=0 swidth=0 blks 2026-02-12T12:29:17.015 INFO:teuthology.orchestra.run.trial127.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-12T12:29:17.015 INFO:teuthology.orchestra.run.trial127.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-12T12:29:17.015 INFO:teuthology.orchestra.run.trial127.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-12T12:29:17.015 INFO:teuthology.orchestra.run.trial127.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-12T12:29:17.573 INFO:teuthology.orchestra.run.trial127.stdout:Discarding blocks...Done. 2026-02-12T12:29:17.575 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@trial127.front.sepia.ceph.com -o noatime 2026-02-12T12:29:17.575 DEBUG:teuthology.orchestra.run.trial127:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2026-02-12T12:29:17.643 DEBUG:teuthology.orchestra.run.trial127:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2026-02-12T12:29:17.710 DEBUG:teuthology.orchestra.run.trial127:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2026-02-12T12:29:17.775 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-12T12:29:17.775 INFO:tasks.ceph:role: osd.2 2026-02-12T12:29:17.775 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@trial127.front.sepia.ceph.com 2026-02-12T12:29:17.775 DEBUG:teuthology.orchestra.run.trial127:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2026-02-12T12:29:17.840 INFO:teuthology.orchestra.run.trial127.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=45875200 blks 2026-02-12T12:29:17.840 INFO:teuthology.orchestra.run.trial127.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-12T12:29:17.840 INFO:teuthology.orchestra.run.trial127.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-12T12:29:17.840 INFO:teuthology.orchestra.run.trial127.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-12T12:29:17.840 INFO:teuthology.orchestra.run.trial127.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-12T12:29:17.841 INFO:teuthology.orchestra.run.trial127.stdout: = sunit=0 swidth=0 blks 2026-02-12T12:29:17.841 INFO:teuthology.orchestra.run.trial127.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-12T12:29:17.841 INFO:teuthology.orchestra.run.trial127.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-12T12:29:17.841 INFO:teuthology.orchestra.run.trial127.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-12T12:29:17.841 INFO:teuthology.orchestra.run.trial127.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-12T12:29:18.409 INFO:teuthology.orchestra.run.trial127.stdout:Discarding blocks...Done. 2026-02-12T12:29:18.411 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@trial127.front.sepia.ceph.com -o noatime 2026-02-12T12:29:18.411 DEBUG:teuthology.orchestra.run.trial127:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2026-02-12T12:29:18.479 DEBUG:teuthology.orchestra.run.trial127:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2026-02-12T12:29:18.546 DEBUG:teuthology.orchestra.run.trial127:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-12T12:29:18.697 INFO:teuthology.orchestra.run.trial127.stderr:INFO  osd - OSD::main: parsing early config 2026-02-12T12:29:18.755 INFO:teuthology.orchestra.run.trial127.stderr:INFO 2026-02-12 12:29:18,754 seastar - Reactor backend: linux-aio 2026-02-12T12:29:18.770 INFO:teuthology.orchestra.run.trial127.stderr:INFO 2026-02-12 12:29:18,769 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-12T12:29:18.770 INFO:teuthology.orchestra.run.trial127.stderr:INFO 2026-02-12 12:29:18,769 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-12T12:29:18.772 INFO:teuthology.orchestra.run.trial127.stderr:==21109==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-12T12:29:18.832 INFO:teuthology.orchestra.run.trial127.stderr:DEBUG 2026-02-12 12:29:18,831 [shard 0:main] seastar - reactor::drain 2026-02-12T12:29:20.924 INFO:teuthology.orchestra.run.trial127.stderr:----------------------------------------------------- 2026-02-12T12:29:20.924 INFO:teuthology.orchestra.run.trial127.stderr:Suppressions used: 2026-02-12T12:29:20.924 INFO:teuthology.orchestra.run.trial127.stderr: count bytes template 2026-02-12T12:29:20.924 INFO:teuthology.orchestra.run.trial127.stderr: 1 8 MallocExtension::Register 2026-02-12T12:29:20.924 INFO:teuthology.orchestra.run.trial127.stderr:----------------------------------------------------- 2026-02-12T12:29:20.925 INFO:teuthology.orchestra.run.trial127.stderr: 2026-02-12T12:29:20.943 INFO:teuthology.orchestra.run.trial127.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-12T12:29:20.943 INFO:teuthology.orchestra.run.trial127.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-12T12:29:20.987 INFO:teuthology.orchestra.run.trial127.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-12T12:29:20.988 INFO:teuthology.orchestra.run.trial127.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-12T12:29:20.994 INFO:teuthology.orchestra.run.trial127.stderr:INFO 2026-02-12 12:29:20,994 seastar - Reactor backend: linux-aio 2026-02-12T12:29:21.004 INFO:teuthology.orchestra.run.trial127.stderr:INFO 2026-02-12 12:29:21,003 [shard 2:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-12T12:29:21.004 INFO:teuthology.orchestra.run.trial127.stderr:INFO 2026-02-12 12:29:21,003 [shard 2:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-12T12:29:21.007 INFO:teuthology.orchestra.run.trial127.stderr:==21107==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-12T12:29:21.008 INFO:teuthology.orchestra.run.trial127.stderr:INFO 2026-02-12 12:29:21,007 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-12T12:29:21.146 INFO:teuthology.orchestra.run.trial127.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e4796c 0x9e47c55 0x9ea85f3 0x9ea8de6 0x9ea9054 0x9ea919c 0x9ea965b /lib64/libc.so.6+0x3fc2f 0x9c711c2 0x9c796df 0x9c79dbf 0x9d6290c 0x9d66fb7 0x9d27585 0x9d26585 0x7bb28d7 0x7bbc3a0 0x7a3ca55 0x7a7cdbd 0x7113237 0x6f93ace 0x6f978cf 0x6f9d479 0x74115d6 0x7433c3e 0x7283c37 0x7283d2f 0x7354fe2 0x735509b 0x7355252 0x7433cbf 0x7283dac 0x7283e31 0x72afa7b 0x741c466 0x741c4eb 0x741c5c9 0x729b554 0x73a9088 0x74287fc 0x7428877 0x74288f2 0x74ec4a2 0x75da55c 0x9f024bb 0x9e56790 0x9f2ff2b 0x9f9870e 0x9fb319e 0x9fb381a 0x26ed857 0xa5c1720 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T12:29:21.146 INFO:teuthology.orchestra.run.trial127.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e4796c 0x9e47c55 0x9ea85f3 0x9ea8de6 0x9ea9054 0x9ea919c 0x9ea965b /lib64/libc.so.6+0x3fc2f 0x9d681b1 0x9d68320 0x9d683b3 0x9d691f8 0x78a0be9 0x7bb280e 0x7bbc3a0 0x7a3ca55 0x7a7cdbd 0x7113237 0x6f93ace 0x6f978cf 0x6f9d479 0x74115d6 0x7433c3e 0x7283c37 0x7283d2f 0x7354fe2 0x735509b 0x7355252 0x7433cbf 0x7283dac 0x7283e31 0x72afa7b 0x741c466 0x741c4eb 0x741c5c9 0x729b554 0x73a9088 0x74287fc 0x7428877 0x74288f2 0x74ec4a2 0x75da55c 0x9f024bb 0x9e56790 0x9f2ff2b 0x9f9870e 0x9fb319e 0x9fb381a 0x26ed857 0xa5c1720 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T12:29:21.146 INFO:teuthology.orchestra.run.trial127.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e4796c 0x9e47c55 0x9ea85f3 0x9ea8de6 0x9ea9054 0x9ea919c 0x9ea965b /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x16b364 /lib64/libasan.so.8+0xb0581 /lib64/libasan.so.8+0xb1477 0xb6a317 0x1092c80 0xf2fdae 0x9c70450 0x9c7074d 0x9c709c0 0x9c70c76 0x9c71725 0x9c71c23 0x9c72253 0x9c723bd 0x9d5272d 0x9d65530 0x9d27585 0x9d26585 0x7bb28d7 0x7bbc3a0 0x7a3ca55 0x7a7cdbd 0x7113237 0x6f93ace 0x6f978cf 0x6f9d479 0x74115d6 0x7433c3e 0x7283c37 0x7283d2f 0x7354fe2 0x735509b 0x7355252 0x7433cbf 0x7283dac 0x7283e31 0x72afa7b 0x741c466 0x741c4eb 0x741c5c9 0x729b554 0x73a9088 0x74287fc 0x7428877 0x74288f2 0x74ec4a2 0x75da55c 0x9f024bb 0x9e56790 0x9f2ff2b 0x9f9870e 0x9fb319e 0x9fb381a 0x26ed857 0xa5c1720 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T12:29:21.293 INFO:teuthology.orchestra.run.trial127.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e4796c 0x9e47c55 0x9ea85f3 0x9ea8de6 0x9ea9054 0x9ea919c 0x9ea965b /lib64/libc.so.6+0x3fc2f 0x9c304d5 0x9c69a18 0x9d3ada8 0x9d67224 0x9d27585 0x9d26585 0x7bb28d7 0x7bbc3a0 0x7a3ca55 0x7a7cdbd 0x7113237 0x6f3791e 0x6f39c86 0x747c276 0x729b1dd 0x729b2df 0x729b3d7 0x73541ae 0x7354267 0x735441a 0x7354552 0x73545cd 0x7354652 0x73547ad 0x741bee3 0x747c2ff 0x7282672 0x729b554 0x73a9088 0x74287fc 0x7428877 0x74288f2 0x74ec4a2 0x75da55c 0x9f024bb 0x9e56790 0x9f2ff2b 0x9f9870e 0x9fb319e 0x9fb381a 0x26ed857 0xa5c1720 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T12:29:21.293 INFO:teuthology.orchestra.run.trial127.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e4796c 0x9e47c55 0x9ea85f3 0x9ea8de6 0x9ea9054 0x9ea919c 0x9ea965b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x129841 /lib64/libasan.so.8+0xc6631 /lib64/libasan.so.8+0x151b7 0x9c7229b 0x9c723bd 0x9d5272d 0x9d65530 0x9d27585 0x9d26585 0x7bb28d7 0x7bbc3a0 0x7a3ca55 0x7a7cdbd 0x7113237 0x6f3791e 0x6f39c86 0x747c276 0x729b1dd 0x729b2df 0x729b3d7 0x73541ae 0x7354267 0x735441a 0x7354552 0x73545cd 0x7354652 0x73547ad 0x741bee3 0x747c2ff 0x7282672 0x729b554 0x73a9088 0x74287fc 0x7428877 0x74288f2 0x74ec4a2 0x75da55c 0x9f024bb 0x9e56790 0x9f2ff2b 0x9f9870e 0x9fb319e 0x9fb381a 0x26ed857 0xa5c1720 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T12:29:21.293 INFO:teuthology.orchestra.run.trial127.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e4796c 0x9e47c55 0x9ea85f3 0x9ea8de6 0x9ea9054 0x9ea919c 0x9ea965b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc597 /lib64/libasan.so.8+0x91c1 /lib64/libasan.so.8+0xee7fc 0x9b44d0b 0x9bc24a6 0x9bc25bc 0x9bc2698 0x9bc2d54 0x9bc2ebb 0x9bc32bf 0x9bc357c 0x9d65185 0x9d27585 0x9d26585 0x7bb28d7 0x7bbc3a0 0x7a3ca55 0x7a7cdbd 0x7113237 0x6f3791e 0x6f39c86 0x747c276 0x729b1dd 0x729b2df 0x729b3d7 0x73541ae 0x7354267 0x735441a 0x7354552 0x73545cd 0x7354652 0x73547ad 0x741bee3 0x747c2ff 0x7282672 0x729b554 0x73a9088 0x74287fc 0x7428877 0x74288f2 0x74ec4a2 0x75da55c 0x9f024bb 0x9e56790 0x9f2ff2b 0x9f9870e 0x9fb319e 0x9fb381a 0x26ed857 0xa5c1720 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T12:29:21.294 INFO:teuthology.orchestra.run.trial127.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff9d5d3fde 0xffffffff9d2910d2 0xffffffff9df4d451 0xffffffff9e000ba2 2026-02-12T12:29:23.472 INFO:teuthology.orchestra.run.trial127.stderr:----------------------------------------------------- 2026-02-12T12:29:23.472 INFO:teuthology.orchestra.run.trial127.stderr:Suppressions used: 2026-02-12T12:29:23.472 INFO:teuthology.orchestra.run.trial127.stderr: count bytes template 2026-02-12T12:29:23.472 INFO:teuthology.orchestra.run.trial127.stderr: 1 8 MallocExtension::Register 2026-02-12T12:29:23.473 INFO:teuthology.orchestra.run.trial127.stderr:----------------------------------------------------- 2026-02-12T12:29:23.473 INFO:teuthology.orchestra.run.trial127.stderr: 2026-02-12T12:29:23.503 DEBUG:teuthology.orchestra.run.trial127:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2026-02-12T12:29:23.528 DEBUG:teuthology.orchestra.run.trial127:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-12T12:29:23.679 INFO:teuthology.orchestra.run.trial127.stderr:INFO  osd - OSD::main: parsing early config 2026-02-12T12:29:23.735 INFO:teuthology.orchestra.run.trial127.stderr:INFO 2026-02-12 12:29:23,735 seastar - Reactor backend: linux-aio 2026-02-12T12:29:23.744 INFO:teuthology.orchestra.run.trial127.stderr:INFO 2026-02-12 12:29:23,744 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-12T12:29:23.745 INFO:teuthology.orchestra.run.trial127.stderr:INFO 2026-02-12 12:29:23,744 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-12T12:29:23.747 INFO:teuthology.orchestra.run.trial127.stderr:==21179==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-12T12:29:23.806 INFO:teuthology.orchestra.run.trial127.stderr:DEBUG 2026-02-12 12:29:23,805 [shard 0:main] seastar - reactor::drain 2026-02-12T12:29:25.904 INFO:teuthology.orchestra.run.trial127.stderr:----------------------------------------------------- 2026-02-12T12:29:25.905 INFO:teuthology.orchestra.run.trial127.stderr:Suppressions used: 2026-02-12T12:29:25.905 INFO:teuthology.orchestra.run.trial127.stderr: count bytes template 2026-02-12T12:29:25.905 INFO:teuthology.orchestra.run.trial127.stderr: 1 8 MallocExtension::Register 2026-02-12T12:29:25.905 INFO:teuthology.orchestra.run.trial127.stderr:----------------------------------------------------- 2026-02-12T12:29:25.905 INFO:teuthology.orchestra.run.trial127.stderr: 2026-02-12T12:29:25.924 INFO:teuthology.orchestra.run.trial127.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-12T12:29:25.924 INFO:teuthology.orchestra.run.trial127.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-12T12:29:25.968 INFO:teuthology.orchestra.run.trial127.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-12T12:29:25.969 INFO:teuthology.orchestra.run.trial127.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-12T12:29:25.975 INFO:teuthology.orchestra.run.trial127.stderr:INFO 2026-02-12 12:29:25,975 seastar - Reactor backend: linux-aio 2026-02-12T12:29:25.983 INFO:teuthology.orchestra.run.trial127.stderr:INFO 2026-02-12 12:29:25,982 [shard 2:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-12T12:29:25.983 INFO:teuthology.orchestra.run.trial127.stderr:INFO 2026-02-12 12:29:25,982 [shard 2:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-12T12:29:25.988 INFO:teuthology.orchestra.run.trial127.stderr:==21177==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-12T12:29:25.988 INFO:teuthology.orchestra.run.trial127.stderr:INFO 2026-02-12 12:29:25,987 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-12T12:29:26.126 INFO:teuthology.orchestra.run.trial127.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e4796c 0x9e47c55 0x9ea85f3 0x9ea8de6 0x9ea9054 0x9ea919c 0x9ea965b /lib64/libc.so.6+0x3fc2f 0x9c2c7ee 0x9d58ce0 0x9d5e3d0 0x9d5ea0e 0x9d68edb 0x78a0be9 0x7bb2209 0x7bbc3a0 0x7a3ca55 0x7a7cdbd 0x7113237 0x6f93ace 0x6f978cf 0x6f9d479 0x74115d6 0x7433c3e 0x7283c37 0x7283d2f 0x7354fe2 0x735509b 0x7355252 0x7433cbf 0x7283dac 0x7283e31 0x72afa7b 0x741c466 0x741c4eb 0x741c5c9 0x729b554 0x73a9088 0x74287fc 0x7428877 0x74288f2 0x74ec4a2 0x75da55c 0x9f024bb 0x9e56790 0x9f2ff2b 0x9f9870e 0x9fb319e 0x9fb381a 0x26ed857 0xa5c1720 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T12:29:26.126 INFO:teuthology.orchestra.run.trial127.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e4796c 0x9e47c55 0x9ea85f3 0x9ea8de6 0x9ea9054 0x9ea919c 0x9ea965b /lib64/libc.so.6+0x3fc2f 0xba17c3 0x9d59d8e 0x9d5e3d0 0x9d5ea0e 0x9d68edb 0x78a0be9 0x7bb280e 0x7bbc3a0 0x7a3ca55 0x7a7cdbd 0x7113237 0x6f93ace 0x6f978cf 0x6f9d479 0x74115d6 0x7433c3e 0x7283c37 0x7283d2f 0x7354fe2 0x735509b 0x7355252 0x7433cbf 0x7283dac 0x7283e31 0x72afa7b 0x741c466 0x741c4eb 0x741c5c9 0x729b554 0x73a9088 0x74287fc 0x7428877 0x74288f2 0x74ec4a2 0x75da55c 0x9f024bb 0x9e56790 0x9f2ff2b 0x9f9870e 0x9fb319e 0x9fb381a 0x26ed857 0xa5c1720 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T12:29:26.126 INFO:teuthology.orchestra.run.trial127.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e4796c 0x9e47c55 0x9ea85f3 0x9ea8de6 0x9ea9054 0x9ea919c 0x9ea965b /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x191cbf /lib64/libasan.so.8+0xb9ea /lib64/libasan.so.8+0xbf98 /lib64/libasan.so.8+0xc08c /lib64/libasan.so.8+0xc360 /lib64/libasan.so.8+0xc553 /lib64/libasan.so.8+0x91c1 /lib64/libasan.so.8+0xee7fc 0x9cc9255 0x9cc9314 0x9cc95bc 0x9d57bab 0x9d5836a 0x9d588b0 0x9d62b3e 0x9d66fb7 0x9d27585 0x9d26585 0x7bb28d7 0x7bbc3a0 0x7a3ca55 0x7a7cdbd 0x7113237 0x6f93ace 0x6f978cf 0x6f9d479 0x74115d6 0x7433c3e 0x7283c37 0x7283d2f 0x7354fe2 0x735509b 0x7355252 0x7433cbf 0x7283dac 0x7283e31 0x72afa7b 0x741c466 0x741c4eb 0x741c5c9 0x729b554 0x73a9088 0x74287fc 0x7428877 0x74288f2 0x74ec4a2 0x75da55c 0x9f024bb 0x9e56790 0x9f2ff2b 0x9f9870e 0x9fb319e 0x9fb381a 0x26ed857 0xa5c1720 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T12:29:26.127 INFO:teuthology.orchestra.run.trial127.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff9d39cee9 0xffffffff9d5d0d73 0xffffffff9d2910a4 0xffffffff9df4d451 0xffffffff9e000ba2 2026-02-12T12:29:26.274 INFO:teuthology.orchestra.run.trial127.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e4796c 0x9e47c55 0x9ea85f3 0x9ea8de6 0x9ea9054 0x9ea919c 0x9ea965b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc597 /lib64/libasan.so.8+0x91c1 /lib64/libasan.so.8+0xee7fc 0x7485195 0x73cb426 0x714d9d4 0x7bb1f60 0x7bbc3a0 0x7a3ca55 0x7a7cdbd 0x7113237 0x6f3791e 0x6f39c86 0x747c276 0x729b1dd 0x729b2df 0x729b3d7 0x73541ae 0x7354267 0x735441a 0x7354552 0x73545cd 0x7354652 0x73547ad 0x741bee3 0x747c2ff 0x7282672 0x729b554 0x73a9088 0x74287fc 0x7428877 0x74288f2 0x74ec4a2 0x75da55c 0x9f024bb 0x9e56790 0x9f2ff2b 0x9f9870e 0x9fb319e 0x9fb381a 0x26ed857 0xa5c1720 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T12:29:26.274 INFO:teuthology.orchestra.run.trial127.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff9d41f704 0xffffffff9d41fb70 0xffffffff9d29525e 0xffffffff9d5e6239 0xffffffff9d5c97a5 0xffffffff9d5d3b7b 0xffffffff9d5d3fdb 0xffffffff9d2910d2 0xffffffff9df4d451 0xffffffff9e000ba2 2026-02-12T12:29:26.275 INFO:teuthology.orchestra.run.trial127.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e4796c 0x9e47c55 0x9ea85f3 0x9ea8de6 0x9ea9054 0x9ea919c 0x9ea965b /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x16b364 /lib64/libasan.so.8+0xb0581 /lib64/libasan.so.8+0xb1477 0xb6a317 0x1092c80 0xf2fdae 0x9c70283 0x9c7074d 0x9c709c0 0x9c70c76 0x9c71238 0x9c796df 0x9c79dbf 0x9d6290c 0x9d66fb7 0x9d27585 0x9d26585 0x7bb28d7 0x7bbc3a0 0x7a3ca55 0x7a7cdbd 0x7113237 0x6f3791e 0x6f39c86 0x747c276 0x729b1dd 0x729b2df 0x729b3d7 0x73541ae 0x7354267 0x735441a 0x7354552 0x73545cd 0x7354652 0x73547ad 0x741bee3 0x747c2ff 0x7282672 0x729b554 0x73a9088 0x74287fc 0x7428877 0x74288f2 0x74ec4a2 0x75da55c 0x9f024bb 0x9e56790 0x9f2ff2b 0x9f9870e 0x9fb319e 0x9fb381a 0x26ed857 0xa5c1720 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T12:29:26.275 INFO:teuthology.orchestra.run.trial127.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e4796c 0x9e47c55 0x9ea85f3 0x9ea8de6 0x9ea9054 0x9ea919c 0x9ea965b /lib64/libc.so.6+0x3fc2f 0xe07591 0x11277c5 0xf2fda3 0x9c70283 0x9c7074d 0x9c709c0 0x9c726e7 0x9d58641 0x9d588b0 0x9d62b3e 0x9d66fb7 0x9d27585 0x9d26585 0x7bb28d7 0x7bbc3a0 0x7a3ca55 0x7a7cdbd 0x7113237 0x6f3791e 0x6f39c86 0x747c276 0x729b1dd 0x729b2df 0x729b3d7 0x73541ae 0x7354267 0x735441a 0x7354552 0x73545cd 0x7354652 0x73547ad 0x741bee3 0x747c2ff 0x7282672 0x729b554 0x73a9088 0x74287fc 0x7428877 0x74288f2 0x74ec4a2 0x75da55c 0x9f024bb 0x9e56790 0x9f2ff2b 0x9f9870e 0x9fb319e 0x9fb381a 0x26ed857 0xa5c1720 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T12:29:28.455 INFO:teuthology.orchestra.run.trial127.stderr:----------------------------------------------------- 2026-02-12T12:29:28.455 INFO:teuthology.orchestra.run.trial127.stderr:Suppressions used: 2026-02-12T12:29:28.455 INFO:teuthology.orchestra.run.trial127.stderr: count bytes template 2026-02-12T12:29:28.455 INFO:teuthology.orchestra.run.trial127.stderr: 1 8 MallocExtension::Register 2026-02-12T12:29:28.456 INFO:teuthology.orchestra.run.trial127.stderr:----------------------------------------------------- 2026-02-12T12:29:28.456 INFO:teuthology.orchestra.run.trial127.stderr: 2026-02-12T12:29:28.486 DEBUG:teuthology.orchestra.run.trial127:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2026-02-12T12:29:28.511 DEBUG:teuthology.orchestra.run.trial127:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-12T12:29:28.662 INFO:teuthology.orchestra.run.trial127.stderr:INFO  osd - OSD::main: parsing early config 2026-02-12T12:29:28.715 INFO:teuthology.orchestra.run.trial127.stderr:INFO 2026-02-12 12:29:28,714 seastar - Reactor backend: linux-aio 2026-02-12T12:29:28.725 INFO:teuthology.orchestra.run.trial127.stderr:INFO 2026-02-12 12:29:28,724 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-12T12:29:28.725 INFO:teuthology.orchestra.run.trial127.stderr:INFO 2026-02-12 12:29:28,724 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-12T12:29:28.727 INFO:teuthology.orchestra.run.trial127.stderr:==21245==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-12T12:29:28.787 INFO:teuthology.orchestra.run.trial127.stderr:DEBUG 2026-02-12 12:29:28,787 [shard 0:main] seastar - reactor::drain 2026-02-12T12:29:30.885 INFO:teuthology.orchestra.run.trial127.stderr:----------------------------------------------------- 2026-02-12T12:29:30.885 INFO:teuthology.orchestra.run.trial127.stderr:Suppressions used: 2026-02-12T12:29:30.885 INFO:teuthology.orchestra.run.trial127.stderr: count bytes template 2026-02-12T12:29:30.885 INFO:teuthology.orchestra.run.trial127.stderr: 1 8 MallocExtension::Register 2026-02-12T12:29:30.886 INFO:teuthology.orchestra.run.trial127.stderr:----------------------------------------------------- 2026-02-12T12:29:30.886 INFO:teuthology.orchestra.run.trial127.stderr: 2026-02-12T12:29:30.905 INFO:teuthology.orchestra.run.trial127.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-12T12:29:30.905 INFO:teuthology.orchestra.run.trial127.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-12T12:29:30.949 INFO:teuthology.orchestra.run.trial127.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-12T12:29:30.950 INFO:teuthology.orchestra.run.trial127.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-12T12:29:30.957 INFO:teuthology.orchestra.run.trial127.stderr:INFO 2026-02-12 12:29:30,956 seastar - Reactor backend: linux-aio 2026-02-12T12:29:30.967 INFO:teuthology.orchestra.run.trial127.stderr:INFO 2026-02-12 12:29:30,966 [shard 2:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-12T12:29:30.967 INFO:teuthology.orchestra.run.trial127.stderr:INFO 2026-02-12 12:29:30,966 [shard 2:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-12T12:29:30.971 INFO:teuthology.orchestra.run.trial127.stderr:==21243==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-12T12:29:30.971 INFO:teuthology.orchestra.run.trial127.stderr:INFO 2026-02-12 12:29:30,970 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-12T12:29:31.111 INFO:teuthology.orchestra.run.trial127.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e4796c 0x9e47c55 0x9ea85f3 0x9ea8de6 0x9ea9054 0x9ea919c 0x9ea965b /lib64/libc.so.6+0x3fc2f 0x9c2c7ee 0x9d58ce0 0x9d5e3d0 0x9d5ea0e 0x9d68edb 0x78a0be9 0x7bb280e 0x7bbc3a0 0x7a3ca55 0x7a7cdbd 0x7113237 0x6f93ace 0x6f978cf 0x6f9d479 0x74115d6 0x7433c3e 0x7283c37 0x7283d2f 0x7354fe2 0x735509b 0x7355252 0x7433cbf 0x7283dac 0x7283e31 0x72afa7b 0x741c466 0x741c4eb 0x741c5c9 0x729b554 0x73a9088 0x74287fc 0x7428877 0x74288f2 0x74ec4a2 0x75da55c 0x9f024bb 0x9e56790 0x9f2ff2b 0x9f9870e 0x9fb319e 0x9fb381a 0x26ed857 0xa5c1720 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T12:29:31.111 INFO:teuthology.orchestra.run.trial127.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e4796c 0x9e47c55 0x9ea85f3 0x9ea8de6 0x9ea9054 0x9ea919c 0x9ea965b /lib64/libc.so.6+0x3fc2f 0xf2fdae 0x9c70450 0x9c7074d 0x9c709c0 0x9c70c76 0x9c71725 0x9c71c23 0x9c72253 0x9c723bd 0x9d5272d 0x9d65530 0x9d27585 0x9d26585 0x7bb28d7 0x7bbc3a0 0x7a3ca55 0x7a7cdbd 0x7113237 0x6f93ace 0x6f978cf 0x6f9d479 0x74115d6 0x7433c3e 0x7283c37 0x7283d2f 0x7354fe2 0x735509b 0x7355252 0x7433cbf 0x7283dac 0x7283e31 0x72afa7b 0x741c466 0x741c4eb 0x741c5c9 0x729b554 0x73a9088 0x74287fc 0x7428877 0x74288f2 0x74ec4a2 0x75da55c 0x9f024bb 0x9e56790 0x9f2ff2b 0x9f9870e 0x9fb319e 0x9fb381a 0x26ed857 0xa5c1720 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T12:29:31.111 INFO:teuthology.orchestra.run.trial127.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e4796c 0x9e47c55 0x9ea85f3 0x9ea8de6 0x9ea9054 0x9ea919c 0x9ea965b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xe93d /lib64/libasan.so.8+0xef925 0x7477dba 0x714d70b 0x714dcca 0x7bb2b9c 0x7bbc3a0 0x7a3ca55 0x7a7cdbd 0x7113237 0x6f93ace 0x6f978cf 0x6f9d479 0x74115d6 0x7433c3e 0x7283c37 0x7283d2f 0x7354fe2 0x735509b 0x7355252 0x7433cbf 0x7283dac 0x7283e31 0x72afa7b 0x741c466 0x741c4eb 0x741c5c9 0x729b554 0x73a9088 0x74287fc 0x7428877 0x74288f2 0x74ec4a2 0x75da55c 0x9f024bb 0x9e56790 0x9f2ff2b 0x9f9870e 0x9fb319e 0x9fb381a 0x26ed857 0xa5c1720 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T12:29:31.111 INFO:teuthology.orchestra.run.trial127.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff9d5d0ddc 0xffffffff9d2910a4 0xffffffff9df4d451 0xffffffff9e000ba2 2026-02-12T12:29:31.258 INFO:teuthology.orchestra.run.trial127.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e4796c 0x9e47c55 0x9ea85f3 0x9ea8de6 0x9ea9054 0x9ea919c 0x9ea965b /lib64/libc.so.6+0x3fc2f 0xc969b4 0xca7fdd 0x9cb8149 0x9d39a5a 0x9d39c42 0x9d39cee 0x9d61e0d 0x9d66fb7 0x9d27585 0x9d26585 0x7bb28d7 0x7bbc3a0 0x7a3ca55 0x7a7cdbd 0x7113237 0x6f3791e 0x6f39c86 0x747c276 0x729b1dd 0x729b2df 0x729b3d7 0x73541ae 0x7354267 0x735441a 0x7354552 0x73545cd 0x7354652 0x73547ad 0x741bee3 0x747c2ff 0x7282672 0x729b554 0x73a9088 0x74287fc 0x7428877 0x74288f2 0x74ec4a2 0x75da55c 0x9f024bb 0x9e56790 0x9f2ff2b 0x9f9870e 0x9fb319e 0x9fb381a 0x26ed857 0xa5c1720 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T12:29:31.258 INFO:teuthology.orchestra.run.trial127.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e4796c 0x9e47c55 0x9ea85f3 0x9ea8de6 0x9ea9054 0x9ea919c 0x9ea965b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1006bf /lib64/libasan.so.8+0xfd1ca /lib64/libasan.so.8+0xee879 0x9cc9255 0x9cc9314 0x9cc95bc 0x9d57bab 0x9d5836a 0x9d588b0 0x9d62b3e 0x9d66fb7 0x9d27585 0x9d26585 0x7bb28d7 0x7bbc3a0 0x7a3ca55 0x7a7cdbd 0x7113237 0x6f3791e 0x6f39c86 0x747c276 0x729b1dd 0x729b2df 0x729b3d7 0x73541ae 0x7354267 0x735441a 0x7354552 0x73545cd 0x7354652 0x73547ad 0x741bee3 0x747c2ff 0x7282672 0x729b554 0x73a9088 0x74287fc 0x7428877 0x74288f2 0x74ec4a2 0x75da55c 0x9f024bb 0x9e56790 0x9f2ff2b 0x9f9870e 0x9fb319e 0x9fb381a 0x26ed857 0xa5c1720 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T12:29:31.258 INFO:teuthology.orchestra.run.trial127.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e4796c 0x9e47c55 0x9ea85f3 0x9ea8de6 0x9ea9054 0x9ea919c 0x9ea965b /lib64/libc.so.6+0x3fc2f /lib64/ld-linux-x86-64.so.2+0x16ec6 /lib64/libasan.so.8+0xc6611 /lib64/libasan.so.8+0x151b7 0x112781c 0xf2fda3 0x9c70450 0x9c7074d 0x9c709c0 0x9c726e7 0x9c72c34 0x9c72e33 0x9d58155 0x9d588b0 0x9d62b3e 0x9d66fb7 0x9d27585 0x9d26585 0x7bb28d7 0x7bbc3a0 0x7a3ca55 0x7a7cdbd 0x7113237 0x6f3791e 0x6f39c86 0x747c276 0x729b1dd 0x729b2df 0x729b3d7 0x73541ae 0x7354267 0x735441a 0x7354552 0x73545cd 0x7354652 0x73547ad 0x741bee3 0x747c2ff 0x7282672 0x729b554 0x73a9088 0x74287fc 0x7428877 0x74288f2 0x74ec4a2 0x75da55c 0x9f024bb 0x9e56790 0x9f2ff2b 0x9f9870e 0x9fb319e 0x9fb381a 0x26ed857 0xa5c1720 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T12:29:33.418 INFO:teuthology.orchestra.run.trial127.stderr:----------------------------------------------------- 2026-02-12T12:29:33.418 INFO:teuthology.orchestra.run.trial127.stderr:Suppressions used: 2026-02-12T12:29:33.418 INFO:teuthology.orchestra.run.trial127.stderr: count bytes template 2026-02-12T12:29:33.418 INFO:teuthology.orchestra.run.trial127.stderr: 1 8 MallocExtension::Register 2026-02-12T12:29:33.419 INFO:teuthology.orchestra.run.trial127.stderr:----------------------------------------------------- 2026-02-12T12:29:33.419 INFO:teuthology.orchestra.run.trial127.stderr: 2026-02-12T12:29:33.449 DEBUG:teuthology.orchestra.run.trial127:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2026-02-12T12:29:33.474 INFO:tasks.ceph:Reading keys from all nodes... 2026-02-12T12:29:33.475 DEBUG:teuthology.orchestra.run.trial127:> set -ex 2026-02-12T12:29:33.475 DEBUG:teuthology.orchestra.run.trial127:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2026-02-12T12:29:33.539 DEBUG:teuthology.orchestra.run.trial127:> set -ex 2026-02-12T12:29:33.539 DEBUG:teuthology.orchestra.run.trial127:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2026-02-12T12:29:33.603 DEBUG:teuthology.orchestra.run.trial127:> set -ex 2026-02-12T12:29:33.603 DEBUG:teuthology.orchestra.run.trial127:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2026-02-12T12:29:33.667 DEBUG:teuthology.orchestra.run.trial127:> set -ex 2026-02-12T12:29:33.668 DEBUG:teuthology.orchestra.run.trial127:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2026-02-12T12:29:33.731 DEBUG:teuthology.orchestra.run.trial127:> set -ex 2026-02-12T12:29:33.731 DEBUG:teuthology.orchestra.run.trial127:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2026-02-12T12:29:33.787 INFO:tasks.ceph:Adding keys to all mons... 2026-02-12T12:29:33.788 DEBUG:teuthology.orchestra.run.trial127:> sudo tee -a /etc/ceph/ceph.keyring 2026-02-12T12:29:33.850 INFO:teuthology.orchestra.run.trial127.stdout:[mgr.x] 2026-02-12T12:29:33.850 INFO:teuthology.orchestra.run.trial127.stdout: key = AQCbx41pOht9OhAAgJb8jIz3u2uXI/ef1lw5bg== 2026-02-12T12:29:33.850 INFO:teuthology.orchestra.run.trial127.stdout:[osd.0] 2026-02-12T12:29:33.850 INFO:teuthology.orchestra.run.trial127.stdout: key = AQChx41pw3NGBBAAZdU4JG4qxRiVLC+v65hgGA== 2026-02-12T12:29:33.850 INFO:teuthology.orchestra.run.trial127.stdout:[osd.1] 2026-02-12T12:29:33.850 INFO:teuthology.orchestra.run.trial127.stdout: key = AQCmx41pwpwWAxAAFiS13NtqLNumevWUPtDxZg== 2026-02-12T12:29:33.850 INFO:teuthology.orchestra.run.trial127.stdout:[osd.2] 2026-02-12T12:29:33.850 INFO:teuthology.orchestra.run.trial127.stdout: key = AQCrx41p/nMgAhAArCiuvYMu0NpYzNxGGdM/Ig== 2026-02-12T12:29:33.850 INFO:teuthology.orchestra.run.trial127.stdout:[client.0] 2026-02-12T12:29:33.851 INFO:teuthology.orchestra.run.trial127.stdout: key = AQCcx41pNjSDARAApJKNutzyiFYJ/UUxWQGZ1w== 2026-02-12T12:29:33.852 DEBUG:teuthology.orchestra.run.trial127:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2026-02-12T12:29:33.939 DEBUG:teuthology.orchestra.run.trial127:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-12T12:29:33.988 DEBUG:teuthology.orchestra.run.trial127:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-12T12:29:34.036 DEBUG:teuthology.orchestra.run.trial127:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-12T12:29:34.084 DEBUG:teuthology.orchestra.run.trial127:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-12T12:29:34.136 INFO:tasks.ceph:Running mkfs on mon nodes... 2026-02-12T12:29:34.136 DEBUG:teuthology.orchestra.run.trial127:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2026-02-12T12:29:34.161 DEBUG:teuthology.orchestra.run.trial127:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2026-02-12T12:29:34.248 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-12T12:29:34.246+0000 7fef6462cc00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:29:34.248 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-12T12:29:34.246+0000 7fef6462cc00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:29:34.255 DEBUG:teuthology.orchestra.run.trial127:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2026-02-12T12:29:34.280 DEBUG:teuthology.orchestra.run.trial127:> rm -- /home/ubuntu/cephtest/ceph.monmap 2026-02-12T12:29:34.336 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2026-02-12T12:29:34.336 INFO:tasks.ceph.mon.a:Restarting daemon 2026-02-12T12:29:34.336 DEBUG:teuthology.orchestra.run.trial127:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2026-02-12T12:29:34.378 INFO:tasks.ceph.mon.a:Started 2026-02-12T12:29:34.378 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2026-02-12T12:29:34.379 INFO:tasks.ceph.mgr.x:Restarting daemon 2026-02-12T12:29:34.379 DEBUG:teuthology.orchestra.run.trial127:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2026-02-12T12:29:34.381 INFO:tasks.ceph.mgr.x:Started 2026-02-12T12:29:34.381 DEBUG:tasks.ceph:set 0 configs 2026-02-12T12:29:34.381 DEBUG:teuthology.orchestra.run.trial127:> sudo ceph --cluster ceph config dump 2026-02-12T12:29:34.440 INFO:tasks.ceph.mon.a.trial127.stderr:2026-02-12T12:29:34.438+0000 7f11eba2fc00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:29:34.441 INFO:tasks.ceph.mon.a.trial127.stderr:2026-02-12T12:29:34.438+0000 7f11eba2fc00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:29:34.441 INFO:tasks.ceph.mon.a.trial127.stderr:2026-02-12T12:29:34.438+0000 7f11eba2fc00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:29:34.445 INFO:tasks.ceph.mgr.x.trial127.stderr:2026-02-12T12:29:34.442+0000 7fc9ba237fc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:29:34.458 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-12T12:29:34.454+0000 7ff6c165a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:29:34.459 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-12T12:29:34.457+0000 7ff6c165a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:29:34.490 INFO:teuthology.orchestra.run.trial127.stdout:WHO MASK LEVEL OPTION VALUE RO 2026-02-12T12:29:34.498 INFO:tasks.ceph:Setting crush tunables to default 2026-02-12T12:29:34.498 DEBUG:teuthology.orchestra.run.trial127:> sudo ceph --cluster ceph osd crush tunables default 2026-02-12T12:29:34.574 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-12T12:29:34.569+0000 7fc3e2e11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:29:34.576 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-12T12:29:34.574+0000 7fc3e2e11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:29:34.609 INFO:teuthology.orchestra.run.trial127.stderr:adjusted tunables profile to default 2026-02-12T12:29:34.617 INFO:tasks.ceph:check_enable_crimson: True 2026-02-12T12:29:34.617 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2026-02-12T12:29:34.617 DEBUG:teuthology.orchestra.run.trial127:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2026-02-12T12:29:34.647 INFO:tasks.ceph.mgr.x.trial127.stderr:2026-02-12T12:29:34.645+0000 7fc9ba237fc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:29:34.647 INFO:tasks.ceph.mgr.x.trial127.stderr:2026-02-12T12:29:34.645+0000 7fc9ba237fc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:29:34.694 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-12T12:29:34.689+0000 7f3487451640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:29:34.695 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-12T12:29:34.693+0000 7f3487451640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:29:34.733 INFO:tasks.ceph.mon.a.trial127.stderr:2026-02-12T12:29:34.731+0000 7f11e2df1640 -1 WARNING: experimental feature 'crimson' is enabled 2026-02-12T12:29:34.733 INFO:tasks.ceph.mon.a.trial127.stderr:Please be aware that this feature is experimental, untested, 2026-02-12T12:29:34.733 INFO:tasks.ceph.mon.a.trial127.stderr:unsupported, and may result in data corruption, data loss, 2026-02-12T12:29:34.733 INFO:tasks.ceph.mon.a.trial127.stderr:and/or irreparable damage to your cluster. Do not use 2026-02-12T12:29:34.733 INFO:tasks.ceph.mon.a.trial127.stderr:feature with important data. 2026-02-12T12:29:34.733 INFO:tasks.ceph.mon.a.trial127.stderr: 2026-02-12T12:29:35.491 INFO:tasks.ceph.mgr.x.trial127.stderr:/usr/lib64/python3.9/site-packages/scipy/__init__.py:73: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2026-02-12T12:29:35.491 INFO:tasks.ceph.mgr.x.trial127.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2026-02-12T12:29:35.491 INFO:tasks.ceph.mgr.x.trial127.stderr: from numpy import show_config as show_numpy_config 2026-02-12T12:29:35.619 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2026-02-12T12:29:35.619 DEBUG:teuthology.orchestra.run.trial127:> set -ex 2026-02-12T12:29:35.620 DEBUG:teuthology.orchestra.run.trial127:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2026-02-12T12:29:35.684 DEBUG:teuthology.orchestra.run.trial127:> set -ex 2026-02-12T12:29:35.684 DEBUG:teuthology.orchestra.run.trial127:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2026-02-12T12:29:35.747 DEBUG:teuthology.orchestra.run.trial127:> set -ex 2026-02-12T12:29:35.748 DEBUG:teuthology.orchestra.run.trial127:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2026-02-12T12:29:35.811 DEBUG:teuthology.orchestra.run.trial127:> sudo ceph --cluster ceph osd new 2a4d13c1-66d8-4d4a-b315-0cec79a35697 0 2026-02-12T12:29:35.927 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-12T12:29:35.922+0000 7f62e150b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:29:35.928 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-12T12:29:35.926+0000 7f62dbfff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:29:35.964 INFO:teuthology.orchestra.run.trial127.stdout:0 2026-02-12T12:29:35.972 DEBUG:teuthology.orchestra.run.trial127:> sudo ceph --cluster ceph osd new e1358829-b6fd-41ae-9d9c-33e8b112e759 1 2026-02-12T12:29:36.046 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-12T12:29:36.041+0000 7fde520cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:29:36.047 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-12T12:29:36.045+0000 7fde520cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:29:36.084 INFO:teuthology.orchestra.run.trial127.stdout:1 2026-02-12T12:29:36.093 DEBUG:teuthology.orchestra.run.trial127:> sudo ceph --cluster ceph osd new f6658fec-6534-46b7-801c-d3afd86ad127 2 2026-02-12T12:29:36.166 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-12T12:29:36.161+0000 7f1d2be06640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:29:36.167 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-12T12:29:36.165+0000 7f1d2be06640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:29:36.204 INFO:teuthology.orchestra.run.trial127.stdout:2 2026-02-12T12:29:36.212 DEBUG:teuthology.orchestra.run.trial127:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2026-02-12T12:29:36.241 INFO:tasks.ceph.osd.0:Restarting daemon 2026-02-12T12:29:36.241 DEBUG:teuthology.orchestra.run.trial127:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2026-02-12T12:29:36.283 INFO:tasks.ceph.osd.0:Started 2026-02-12T12:29:36.283 DEBUG:teuthology.orchestra.run.trial127:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2026-02-12T12:29:36.311 INFO:tasks.ceph.osd.1:Restarting daemon 2026-02-12T12:29:36.312 DEBUG:teuthology.orchestra.run.trial127:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2026-02-12T12:29:36.353 INFO:tasks.ceph.osd.1:Started 2026-02-12T12:29:36.354 DEBUG:teuthology.orchestra.run.trial127:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2026-02-12T12:29:36.380 INFO:tasks.ceph.osd.2:Restarting daemon 2026-02-12T12:29:36.381 DEBUG:teuthology.orchestra.run.trial127:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2026-02-12T12:29:36.424 INFO:tasks.ceph.osd.2:Started 2026-02-12T12:29:36.425 DEBUG:teuthology.orchestra.run.trial127:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-12T12:29:36.501 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-12T12:29:36.497+0000 7fb0b32cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:29:36.503 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-12T12:29:36.501+0000 7fb0b32cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:29:36.536 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:29:36.536 INFO:teuthology.orchestra.run.trial127.stdout:{"epoch":6,"fsid":"0d04f462-0e99-4bbd-8f83-79681663314a","created":"2026-02-12T12:29:34.450566+0000","modified":"2026-02-12T12:29:36.203121+0000","last_up_change":"0.000000","last_in_change":"2026-02-12T12:29:36.203121+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"2a4d13c1-66d8-4d4a-b315-0cec79a35697","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1358829-b6fd-41ae-9d9c-33e8b112e759","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f6658fec-6534-46b7-801c-d3afd86ad127","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-12T12:29:36.547 INFO:tasks.ceph.ceph_manager.ceph:[] 2026-02-12T12:29:36.547 INFO:tasks.ceph:Waiting for OSDs to come up 2026-02-12T12:29:36.849 DEBUG:teuthology.orchestra.run.trial127:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-12T12:29:36.920 INFO:teuthology.misc.health.trial127.stderr:2026-02-12T12:29:36.916+0000 7fbfef45a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:29:36.921 INFO:teuthology.misc.health.trial127.stderr:2026-02-12T12:29:36.919+0000 7fbfef45a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:29:36.953 INFO:teuthology.misc.health.trial127.stdout: 2026-02-12T12:29:36.953 INFO:teuthology.misc.health.trial127.stdout:{"epoch":6,"fsid":"0d04f462-0e99-4bbd-8f83-79681663314a","created":"2026-02-12T12:29:34.450566+0000","modified":"2026-02-12T12:29:36.203121+0000","last_up_change":"0.000000","last_in_change":"2026-02-12T12:29:36.203121+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"2a4d13c1-66d8-4d4a-b315-0cec79a35697","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1358829-b6fd-41ae-9d9c-33e8b112e759","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f6658fec-6534-46b7-801c-d3afd86ad127","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-12T12:29:36.960 DEBUG:teuthology.misc:0 of 3 OSDs are up 2026-02-12T12:29:37.889 INFO:tasks.ceph.mgr.x.trial127.stderr:2026-02-12T12:29:37.887+0000 7fc8d4e5e640 -1 mgr.server handle_report got status from non-daemon mon.a 2026-02-12T12:29:43.262 DEBUG:teuthology.orchestra.run.trial127:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-12T12:29:43.329 INFO:teuthology.misc.health.trial127.stderr:2026-02-12T12:29:43.326+0000 7f1dff9e4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:29:43.330 INFO:teuthology.misc.health.trial127.stderr:2026-02-12T12:29:43.329+0000 7f1dff9e4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:29:43.440 INFO:teuthology.misc.health.trial127.stdout: 2026-02-12T12:29:43.440 INFO:teuthology.misc.health.trial127.stdout:{"epoch":8,"fsid":"0d04f462-0e99-4bbd-8f83-79681663314a","created":"2026-02-12T12:29:34.450566+0000","modified":"2026-02-12T12:29:43.158307+0000","last_up_change":"0.000000","last_in_change":"2026-02-12T12:29:36.203121+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"2a4d13c1-66d8-4d4a-b315-0cec79a35697","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1358829-b6fd-41ae-9d9c-33e8b112e759","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f6658fec-6534-46b7-801c-d3afd86ad127","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-12T12:29:43.448 DEBUG:teuthology.misc:0 of 3 OSDs are up 2026-02-12T12:29:49.751 DEBUG:teuthology.orchestra.run.trial127:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-12T12:29:49.818 INFO:teuthology.misc.health.trial127.stderr:2026-02-12T12:29:49.815+0000 7f1a73f0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:29:49.819 INFO:teuthology.misc.health.trial127.stderr:2026-02-12T12:29:49.818+0000 7f1a73f0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:29:49.928 INFO:teuthology.misc.health.trial127.stdout: 2026-02-12T12:29:49.929 INFO:teuthology.misc.health.trial127.stdout:{"epoch":13,"fsid":"0d04f462-0e99-4bbd-8f83-79681663314a","created":"2026-02-12T12:29:34.450566+0000","modified":"2026-02-12T12:29:49.178882+0000","last_up_change":"2026-02-12T12:29:45.164984+0000","last_in_change":"2026-02-12T12:29:36.203121+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-12T12:29:46.901373+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2a4d13c1-66d8-4d4a-b315-0cec79a35697","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.127:6807","nonce":4197616344}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.127:6808","nonce":4197616344}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.127:6810","nonce":4197616344}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.127:6813","nonce":4197616344}]},"public_addr":"10.20.193.127:6807/4197616344","cluster_addr":"10.20.193.127:6808/4197616344","heartbeat_back_addr":"10.20.193.127:6810/4197616344","heartbeat_front_addr":"10.20.193.127:6813/4197616344","state":["exists","up"]},{"osd":1,"uuid":"e1358829-b6fd-41ae-9d9c-33e8b112e759","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.127:6803","nonce":3431701742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.127:6804","nonce":3431701742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.127:6812","nonce":3431701742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.127:6811","nonce":3431701742}]},"public_addr":"10.20.193.127:6803/3431701742","cluster_addr":"10.20.193.127:6804/3431701742","heartbeat_back_addr":"10.20.193.127:6812/3431701742","heartbeat_front_addr":"10.20.193.127:6811/3431701742","state":["exists","up"]},{"osd":2,"uuid":"f6658fec-6534-46b7-801c-d3afd86ad127","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.127:6805","nonce":2058343392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.127:6806","nonce":2058343392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.127:6809","nonce":2058343392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.127:6802","nonce":2058343392}]},"public_addr":"10.20.193.127:6805/2058343392","cluster_addr":"10.20.193.127:6806/2058343392","heartbeat_back_addr":"10.20.193.127:6809/2058343392","heartbeat_front_addr":"10.20.193.127:6802/2058343392","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-12T12:29:49.937 DEBUG:teuthology.misc:3 of 3 OSDs are up 2026-02-12T12:29:49.937 INFO:tasks.ceph:Creating RBD pool 2026-02-12T12:29:49.937 DEBUG:teuthology.orchestra.run.trial127:> sudo ceph --cluster ceph osd pool create rbd 8 2026-02-12T12:29:50.012 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-12T12:29:50.007+0000 7f7503ecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:29:50.013 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-12T12:29:50.012+0000 7f7503ecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:29:50.513 INFO:teuthology.orchestra.run.trial127.stderr:pool 'rbd' created 2026-02-12T12:29:50.529 DEBUG:teuthology.orchestra.run.trial127:> rbd --cluster ceph pool init rbd 2026-02-12T12:29:50.573 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-12T12:29:50.572+0000 7f08072df240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:29:50.574 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-12T12:29:50.572+0000 7f08072df240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:29:50.575 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-12T12:29:50.574+0000 7f08072df240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:29:53.553 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2026-02-12T12:29:53.554 DEBUG:teuthology.orchestra.run.trial127:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2026-02-12T12:29:53.554 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2026-02-12T12:29:53.673 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-12T12:29:53.669+0000 7f4c3405a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:29:53.674 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-12T12:29:53.673+0000 7f4c3405a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:29:53.783 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:29:53.793 INFO:teuthology.orchestra.run.trial127.stdout:[{"version":1,"timestamp":"0.000000","name":"","changes":[]}] 2026-02-12T12:29:53.793 INFO:tasks.ceph_manager:config epoch is 1 2026-02-12T12:29:53.793 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2026-02-12T12:29:53.793 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2026-02-12T12:29:53.793 DEBUG:teuthology.orchestra.run.trial127:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2026-02-12T12:29:53.872 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-12T12:29:53.869+0000 7f52305a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:29:53.873 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-12T12:29:53.872+0000 7f52305a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:29:54.014 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:29:54.023 INFO:teuthology.orchestra.run.trial127.stdout:{"epoch":5,"flags":0,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.127:6800","nonce":1182003022},{"type":"v1","addr":"10.20.193.127:6801","nonce":1182003022}]},"active_addr":"10.20.193.127:6801/1182003022","active_change":"2026-02-12T12:29:36.882865+0000","active_mgr_features":4544132024016699391,"available":true,"standbys":[],"modules":["iostat","nfs"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_pg_upmap_activity":{"name":"update_pg_upmap_activity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Updates pg_upmap activity stats to be used in `balancer status detail`","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often the agent on each host will gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"The first TCP port the agent will try to bind to. Up to 1000 subsequent ports will be attempted if this port cannot be bound","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this option can facilitate debugging daemons that encounter runtime problems.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"How frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"Fraction of total system memory to divide among autotuned daemons. Converged systems hosting both Ceph and compute should set a lower value; systems with more than 10 GiB of memory per OSD may benefit from a slightly larger value so that the abundance of RAM can be fully utilized","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command persistent logging","long_desc":"","tags":[],"see_also":[]},"certificate_automated_rotation_enabled":{"name":"certificate_automated_rotation_enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"This flag controls whether cephadm automatically rotates certificates upon expiration.","long_desc":"","tags":[],"see_also":[]},"certificate_check_debug_mode":{"name":"certificate_check_debug_mode","type":"bool","level":"dev","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"FOR TESTING ONLY: This flag forces the certificate check instead of waiting for certificate_check_period.","long_desc":"","tags":[],"see_also":[]},"certificate_check_period":{"name":"certificate_check_period","type":"int","level":"advanced","flags":0,"default_value":"1","min":"0","max":"30","enum_allowed":[],"desc":"Specifies how often (in days) the certificate should be checked for validity.","long_desc":"","tags":[],"see_also":[]},"certificate_duration_days":{"name":"certificate_duration_days","type":"int","level":"advanced","flags":0,"default_value":"1095","min":"90","max":"3650","enum_allowed":[],"desc":"Specifies the duration of self-signed certificates generated and signed by the cephadm root CA","long_desc":"","tags":[],"see_also":[]},"certificate_renewal_threshold_days":{"name":"certificate_renewal_threshold_days","type":"int","level":"advanced","flags":0,"default_value":"30","min":"10","max":"90","enum_allowed":[],"desc":"Specifies the lead time in days to initiate certificate renewal before expiration.","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently Podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.28.1","min":"","max":"","enum_allowed":[],"desc":"Alertmanager container image","long_desc":"","tags":[],"see_also":[]},"container_image_alloy":{"name":"container_image_alloy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/alloy:latest","min":"","max":"","enum_allowed":[],"desc":"Alloy container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"Elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:12.3.1","min":"","max":"","enum_allowed":[],"desc":"Grafana container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"Haproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.9.1","min":"","max":"","enum_allowed":[],"desc":"Node exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.6","min":"","max":"","enum_allowed":[],"desc":"Nvmeof container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"Oauth2 proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v3.6.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba metrics container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"Snmp gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"quay.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"Location of dashboards to include in Grafana","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"How frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether the hardware monitoring daemon be deployed on every host?","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at the debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"Max number of daemons per service per host: guard against mistakes and runaways","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"Max number of OSDs that will be drained simultaneously when OSDs are removed","long_desc":"","tags":[],"see_also":[]},"max_parallel_osd_upgrades":{"name":"max_parallel_osd_upgrades","type":"int","level":"advanced","flags":0,"default_value":"16","min":"","max":"","enum_allowed":[],"desc":"Maximum number of OSD daemons upgraded in parallel.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"Mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default IP address for RedFish API (OOB management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"Location of alerts to include in Prometheus Alertmanager","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry URL for login purposes. This is not the default registry.","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"Cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Customized SSH config file to distribute to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many sequential SSH connection liveness check failures before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often SSH connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"stray_daemon_check_interval":{"name":"stray_daemon_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"How frequently cephadm should check for the presence of stray daemons","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use the cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest to ensure that all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROM_ALERT_CREDENTIAL_CACHE_TTL":{"name":"PROM_ALERT_CREDENTIAL_CACHE_TTL","type":"int","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_HOSTNAME_PER_DAEMON":{"name":"RGW_HOSTNAME_PER_DAEMON","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crypto_caller":{"name":"crypto_caller","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":true,"error_string":"","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"healthcheck_history_max_entries":{"name":"healthcheck_history_max_entries","type":"int","level":"advanced","flags":1,"default_value":"1000","min":"","max":"","enum_allowed":[],"desc":"Maximum number of health check history entries to keep","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"prometheus_tls_secret_name":{"name":"prometheus_tls_secret_name","type":"str","level":"advanced","flags":0,"default_value":"rook-ceph-prometheus-server-tls","min":"","max":"","enum_allowed":[],"desc":"name of tls secret in k8s for prometheus","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_cloning":{"name":"pause_cloning","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_purging":{"name":"pause_purging","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous subvolume purge threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"tentacle":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"force_disabled_modules":{},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"10.20.193.127:0","nonce":2499306877}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"10.20.193.127:0","nonce":1016197420}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"10.20.193.127:0","nonce":2034777030}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"10.20.193.127:0","nonce":3027887331}]}]} 2026-02-12T12:29:54.025 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2026-02-12T12:29:54.025 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2026-02-12T12:29:54.025 DEBUG:teuthology.orchestra.run.trial127:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-12T12:29:54.100 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-12T12:29:54.097+0000 7f17fe4cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:29:54.101 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-12T12:29:54.101+0000 7f17fd242640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:29:54.212 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:29:54.213 INFO:teuthology.orchestra.run.trial127.stdout:{"epoch":17,"fsid":"0d04f462-0e99-4bbd-8f83-79681663314a","created":"2026-02-12T12:29:34.450566+0000","modified":"2026-02-12T12:29:53.525617+0000","last_up_change":"2026-02-12T12:29:45.164984+0000","last_in_change":"2026-02-12T12:29:36.203121+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-12T12:29:46.901373+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-12T12:29:50.124309+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2a4d13c1-66d8-4d4a-b315-0cec79a35697","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.127:6807","nonce":4197616344}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.127:6808","nonce":4197616344}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.127:6810","nonce":4197616344}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.127:6813","nonce":4197616344}]},"public_addr":"10.20.193.127:6807/4197616344","cluster_addr":"10.20.193.127:6808/4197616344","heartbeat_back_addr":"10.20.193.127:6810/4197616344","heartbeat_front_addr":"10.20.193.127:6813/4197616344","state":["exists","up"]},{"osd":1,"uuid":"e1358829-b6fd-41ae-9d9c-33e8b112e759","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.127:6803","nonce":3431701742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.127:6804","nonce":3431701742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.127:6812","nonce":3431701742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.127:6811","nonce":3431701742}]},"public_addr":"10.20.193.127:6803/3431701742","cluster_addr":"10.20.193.127:6804/3431701742","heartbeat_back_addr":"10.20.193.127:6812/3431701742","heartbeat_front_addr":"10.20.193.127:6811/3431701742","state":["exists","up"]},{"osd":2,"uuid":"f6658fec-6534-46b7-801c-d3afd86ad127","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.127:6805","nonce":2058343392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.127:6806","nonce":2058343392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.127:6809","nonce":2058343392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.127:6802","nonce":2058343392}]},"public_addr":"10.20.193.127:6805/2058343392","cluster_addr":"10.20.193.127:6806/2058343392","heartbeat_back_addr":"10.20.193.127:6809/2058343392","heartbeat_front_addr":"10.20.193.127:6802/2058343392","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-12T12:29:54.223 INFO:tasks.ceph.ceph_manager.ceph:all up! 2026-02-12T12:29:54.223 DEBUG:teuthology.orchestra.run.trial127:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-12T12:29:54.300 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-12T12:29:54.297+0000 7fdcfad6f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:29:54.301 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-12T12:29:54.300+0000 7fdcfad6f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:29:54.410 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:29:54.411 INFO:teuthology.orchestra.run.trial127.stdout:{"epoch":17,"fsid":"0d04f462-0e99-4bbd-8f83-79681663314a","created":"2026-02-12T12:29:34.450566+0000","modified":"2026-02-12T12:29:53.525617+0000","last_up_change":"2026-02-12T12:29:45.164984+0000","last_in_change":"2026-02-12T12:29:36.203121+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-12T12:29:46.901373+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-12T12:29:50.124309+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2a4d13c1-66d8-4d4a-b315-0cec79a35697","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.127:6807","nonce":4197616344}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.127:6808","nonce":4197616344}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.127:6810","nonce":4197616344}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.127:6813","nonce":4197616344}]},"public_addr":"10.20.193.127:6807/4197616344","cluster_addr":"10.20.193.127:6808/4197616344","heartbeat_back_addr":"10.20.193.127:6810/4197616344","heartbeat_front_addr":"10.20.193.127:6813/4197616344","state":["exists","up"]},{"osd":1,"uuid":"e1358829-b6fd-41ae-9d9c-33e8b112e759","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.127:6803","nonce":3431701742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.127:6804","nonce":3431701742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.127:6812","nonce":3431701742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.127:6811","nonce":3431701742}]},"public_addr":"10.20.193.127:6803/3431701742","cluster_addr":"10.20.193.127:6804/3431701742","heartbeat_back_addr":"10.20.193.127:6812/3431701742","heartbeat_front_addr":"10.20.193.127:6811/3431701742","state":["exists","up"]},{"osd":2,"uuid":"f6658fec-6534-46b7-801c-d3afd86ad127","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.127:6805","nonce":2058343392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.127:6806","nonce":2058343392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.127:6809","nonce":2058343392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.127:6802","nonce":2058343392}]},"public_addr":"10.20.193.127:6805/2058343392","cluster_addr":"10.20.193.127:6806/2058343392","heartbeat_back_addr":"10.20.193.127:6809/2058343392","heartbeat_front_addr":"10.20.193.127:6802/2058343392","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-12T12:29:54.420 DEBUG:teuthology.orchestra.run.trial127:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-12T12:29:54.420 DEBUG:teuthology.orchestra.run.trial127:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-12T12:29:54.420 DEBUG:teuthology.orchestra.run.trial127:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-12T12:29:54.501 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-12T12:29:54.497+0000 7f3f8d30b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:29:54.501 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-12T12:29:54.497+0000 7f0ab3e5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:29:54.501 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-12T12:29:54.498+0000 7f22ca0b7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:29:54.502 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-12T12:29:54.501+0000 7f0ab3e5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:29:54.502 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-12T12:29:54.501+0000 7f3f8d30b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:29:54.503 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-12T12:29:54.502+0000 7f22ca0b7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:29:54.516 INFO:teuthology.orchestra.run.trial127.stdout:42949672969 2026-02-12T12:29:54.517 DEBUG:teuthology.orchestra.run.trial127:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-12T12:29:54.517 INFO:teuthology.orchestra.run.trial127.stdout:42949672969 2026-02-12T12:29:54.517 DEBUG:teuthology.orchestra.run.trial127:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-12T12:29:54.518 INFO:teuthology.orchestra.run.trial127.stdout:38654705674 2026-02-12T12:29:54.518 DEBUG:teuthology.orchestra.run.trial127:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-12T12:29:54.636 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-12T12:29:54.632+0000 7f96945c1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:29:54.636 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-12T12:29:54.632+0000 7f69b1451640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:29:54.637 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-12T12:29:54.636+0000 7f96945c1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:29:54.637 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-12T12:29:54.636+0000 7f69b1451640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:29:54.679 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-12T12:29:54.676+0000 7fa9cb38c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:29:54.680 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-12T12:29:54.680+0000 7fa9cb38c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:29:54.750 INFO:teuthology.orchestra.run.trial127.stdout:42949672969 2026-02-12T12:29:54.756 INFO:teuthology.orchestra.run.trial127.stdout:42949672969 2026-02-12T12:29:54.761 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672969 got 42949672969 for osd.2 2026-02-12T12:29:54.761 DEBUG:teuthology.parallel:result is None 2026-02-12T12:29:54.766 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672969 got 42949672969 for osd.0 2026-02-12T12:29:54.767 DEBUG:teuthology.parallel:result is None 2026-02-12T12:29:54.792 INFO:teuthology.orchestra.run.trial127.stdout:38654705673 2026-02-12T12:29:54.802 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705674 got 38654705673 for osd.1 2026-02-12T12:29:55.803 DEBUG:teuthology.orchestra.run.trial127:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-12T12:29:55.882 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-12T12:29:55.879+0000 7fe1e6793640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:29:55.883 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-12T12:29:55.882+0000 7fe1e6793640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:29:55.996 INFO:teuthology.orchestra.run.trial127.stdout:38654705674 2026-02-12T12:29:56.006 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705674 got 38654705674 for osd.1 2026-02-12T12:29:56.006 DEBUG:teuthology.parallel:result is None 2026-02-12T12:29:56.007 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-12T12:29:56.007 DEBUG:teuthology.orchestra.run.trial127:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T12:29:56.084 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-12T12:29:56.081+0000 7f7033ccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:29:56.085 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-12T12:29:56.084+0000 7f7032a42640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:29:56.194 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:29:56.194 INFO:teuthology.orchestra.run.trial127.stderr:dumped all 2026-02-12T12:29:56.205 INFO:teuthology.orchestra.run.trial127.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2026-02-12T12:29:54.886909+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":157286352,"kb_used":62917052,"kb_used_data":62917052,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":94369300,"statfs":{"total":161061224448,"available":96634163200,"internally_reserved":0,"allocated":64427061248,"data_stored":64427061248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.705564"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-12T12:29:53.539747+0000","last_change":"2026-02-12T12:29:53.542589+0000","last_active":"2026-02-12T12:29:53.539747+0000","last_peered":"2026-02-12T12:29:53.539747+0000","last_clean":"2026-02-12T12:29:53.539747+0000","last_became_active":"2026-02-12T12:29:51.550707+0000","last_became_peered":"2026-02-12T12:29:51.550707+0000","last_unstale":"2026-02-12T12:29:53.539747+0000","last_undegraded":"2026-02-12T12:29:53.539747+0000","last_fullsized":"2026-02-12T12:29:53.539747+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T12:29:50.511964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T12:29:50.511964+0000","last_clean_scrub_stamp":"2026-02-12T12:29:50.511964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-12T12:29:53.541243+0000","last_change":"2026-02-12T12:29:53.543046+0000","last_active":"2026-02-12T12:29:53.541243+0000","last_peered":"2026-02-12T12:29:53.541243+0000","last_clean":"2026-02-12T12:29:53.541243+0000","last_became_active":"2026-02-12T12:29:51.548981+0000","last_became_peered":"2026-02-12T12:29:51.548981+0000","last_unstale":"2026-02-12T12:29:53.541243+0000","last_undegraded":"2026-02-12T12:29:53.541243+0000","last_fullsized":"2026-02-12T12:29:53.541243+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T12:29:50.511964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T12:29:50.511964+0000","last_clean_scrub_stamp":"2026-02-12T12:29:50.511964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-12T12:29:53.541272+0000","last_change":"2026-02-12T12:29:53.544394+0000","last_active":"2026-02-12T12:29:53.541272+0000","last_peered":"2026-02-12T12:29:53.541272+0000","last_clean":"2026-02-12T12:29:53.541272+0000","last_became_active":"2026-02-12T12:29:51.549234+0000","last_became_peered":"2026-02-12T12:29:51.549234+0000","last_unstale":"2026-02-12T12:29:53.541272+0000","last_undegraded":"2026-02-12T12:29:53.541272+0000","last_fullsized":"2026-02-12T12:29:53.541272+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T12:29:50.511964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T12:29:50.511964+0000","last_clean_scrub_stamp":"2026-02-12T12:29:50.511964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-12T12:29:53.540671+0000","last_change":"2026-02-12T12:29:53.542534+0000","last_active":"2026-02-12T12:29:53.540671+0000","last_peered":"2026-02-12T12:29:53.540671+0000","last_clean":"2026-02-12T12:29:53.540671+0000","last_became_active":"2026-02-12T12:29:51.550232+0000","last_became_peered":"2026-02-12T12:29:51.550232+0000","last_unstale":"2026-02-12T12:29:53.540671+0000","last_undegraded":"2026-02-12T12:29:53.540671+0000","last_fullsized":"2026-02-12T12:29:53.540671+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T12:29:50.511964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T12:29:50.511964+0000","last_clean_scrub_stamp":"2026-02-12T12:29:50.511964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'1","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-12T12:29:53.541109+0000","last_change":"2026-02-12T12:29:53.541109+0000","last_active":"2026-02-12T12:29:53.541109+0000","last_peered":"2026-02-12T12:29:53.541109+0000","last_clean":"2026-02-12T12:29:53.541109+0000","last_became_active":"2026-02-12T12:29:51.557304+0000","last_became_peered":"2026-02-12T12:29:51.557304+0000","last_unstale":"2026-02-12T12:29:53.541109+0000","last_undegraded":"2026-02-12T12:29:53.541109+0000","last_fullsized":"2026-02-12T12:29:53.541109+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T12:29:50.511964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T12:29:50.511964+0000","last_clean_scrub_stamp":"2026-02-12T12:29:50.511964+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-12T12:29:53.538861+0000","last_change":"2026-02-12T12:29:53.540750+0000","last_active":"2026-02-12T12:29:53.538861+0000","last_peered":"2026-02-12T12:29:53.538861+0000","last_clean":"2026-02-12T12:29:53.538861+0000","last_became_active":"2026-02-12T12:29:51.557952+0000","last_became_peered":"2026-02-12T12:29:51.557952+0000","last_unstale":"2026-02-12T12:29:53.538861+0000","last_undegraded":"2026-02-12T12:29:53.538861+0000","last_fullsized":"2026-02-12T12:29:53.538861+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T12:29:50.511964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T12:29:50.511964+0000","last_clean_scrub_stamp":"2026-02-12T12:29:50.511964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-12T12:29:53.537239+0000","last_change":"2026-02-12T12:29:53.539438+0000","last_active":"2026-02-12T12:29:53.537239+0000","last_peered":"2026-02-12T12:29:53.537239+0000","last_clean":"2026-02-12T12:29:53.537239+0000","last_became_active":"2026-02-12T12:29:51.549086+0000","last_became_peered":"2026-02-12T12:29:51.549086+0000","last_unstale":"2026-02-12T12:29:53.537239+0000","last_undegraded":"2026-02-12T12:29:53.537239+0000","last_fullsized":"2026-02-12T12:29:53.537239+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T12:29:50.511964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T12:29:50.511964+0000","last_clean_scrub_stamp":"2026-02-12T12:29:50.511964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":16,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-12T12:29:53.541205+0000","last_change":"2026-02-12T12:29:53.544105+0000","last_active":"2026-02-12T12:29:53.541205+0000","last_peered":"2026-02-12T12:29:53.541205+0000","last_clean":"2026-02-12T12:29:53.541205+0000","last_became_active":"2026-02-12T12:29:51.552719+0000","last_became_peered":"2026-02-12T12:29:51.552719+0000","last_unstale":"2026-02-12T12:29:53.541205+0000","last_undegraded":"2026-02-12T12:29:53.541205+0000","last_fullsized":"2026-02-12T12:29:53.541205+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T12:29:50.511964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T12:29:50.511964+0000","last_clean_scrub_stamp":"2026-02-12T12:29:50.511964+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":90,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-12T12:29:53.539606+0000","last_change":"2026-02-12T12:29:48.203040+0000","last_active":"2026-02-12T12:29:53.539606+0000","last_peered":"2026-02-12T12:29:53.539606+0000","last_clean":"2026-02-12T12:29:53.539606+0000","last_became_active":"2026-02-12T12:29:48.195870+0000","last_became_peered":"2026-02-12T12:29:48.195870+0000","last_unstale":"2026-02-12T12:29:53.539606+0000","last_undegraded":"2026-02-12T12:29:53.539606+0000","last_fullsized":"2026-02-12T12:29:53.539606+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T12:29:47.166668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T12:29:47.166668+0000","last_clean_scrub_stamp":"2026-02-12T12:29:47.166668+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":161061224448,"available":96634163200,"internally_reserved":0,"allocated":64427061248,"data_stored":64427061248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53687074816,"available":32211079168,"internally_reserved":0,"allocated":21475995648,"data_stored":21475995648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20971820,"kb_used_data":20971820,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456964,"statfs":{"total":53687074816,"available":32211931136,"internally_reserved":0,"allocated":21475143680,"data_stored":21475143680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972580,"kb_used_data":20972580,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456204,"statfs":{"total":53687074816,"available":32211152896,"internally_reserved":0,"allocated":21475921920,"data_stored":21475921920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972652,"kb_used_data":20972652,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456132,"statfs":{"total":53687074816,"available":32211079168,"internally_reserved":0,"allocated":21475995648,"data_stored":21475995648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53687074816,"available":32211079168,"internally_reserved":0,"allocated":21475995648,"data_stored":21475995648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53687074816,"available":32211152896,"internally_reserved":0,"allocated":21475921920,"data_stored":21475921920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53687074816,"available":32211079168,"internally_reserved":0,"allocated":21475995648,"data_stored":21475995648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53687074816,"available":32211931136,"internally_reserved":0,"allocated":21475143680,"data_stored":21475143680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T12:29:56.206 DEBUG:teuthology.orchestra.run.trial127:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T12:29:56.283 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-12T12:29:56.281+0000 7fe73dd5c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:29:56.284 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-12T12:29:56.283+0000 7fe73dd5c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:29:56.393 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:29:56.393 INFO:teuthology.orchestra.run.trial127.stderr:dumped all 2026-02-12T12:29:56.403 INFO:teuthology.orchestra.run.trial127.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2026-02-12T12:29:54.886909+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":157286352,"kb_used":62917052,"kb_used_data":62917052,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":94369300,"statfs":{"total":161061224448,"available":96634163200,"internally_reserved":0,"allocated":64427061248,"data_stored":64427061248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.705564"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-12T12:29:53.539747+0000","last_change":"2026-02-12T12:29:53.542589+0000","last_active":"2026-02-12T12:29:53.539747+0000","last_peered":"2026-02-12T12:29:53.539747+0000","last_clean":"2026-02-12T12:29:53.539747+0000","last_became_active":"2026-02-12T12:29:51.550707+0000","last_became_peered":"2026-02-12T12:29:51.550707+0000","last_unstale":"2026-02-12T12:29:53.539747+0000","last_undegraded":"2026-02-12T12:29:53.539747+0000","last_fullsized":"2026-02-12T12:29:53.539747+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T12:29:50.511964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T12:29:50.511964+0000","last_clean_scrub_stamp":"2026-02-12T12:29:50.511964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-12T12:29:53.541243+0000","last_change":"2026-02-12T12:29:53.543046+0000","last_active":"2026-02-12T12:29:53.541243+0000","last_peered":"2026-02-12T12:29:53.541243+0000","last_clean":"2026-02-12T12:29:53.541243+0000","last_became_active":"2026-02-12T12:29:51.548981+0000","last_became_peered":"2026-02-12T12:29:51.548981+0000","last_unstale":"2026-02-12T12:29:53.541243+0000","last_undegraded":"2026-02-12T12:29:53.541243+0000","last_fullsized":"2026-02-12T12:29:53.541243+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T12:29:50.511964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T12:29:50.511964+0000","last_clean_scrub_stamp":"2026-02-12T12:29:50.511964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-12T12:29:53.541272+0000","last_change":"2026-02-12T12:29:53.544394+0000","last_active":"2026-02-12T12:29:53.541272+0000","last_peered":"2026-02-12T12:29:53.541272+0000","last_clean":"2026-02-12T12:29:53.541272+0000","last_became_active":"2026-02-12T12:29:51.549234+0000","last_became_peered":"2026-02-12T12:29:51.549234+0000","last_unstale":"2026-02-12T12:29:53.541272+0000","last_undegraded":"2026-02-12T12:29:53.541272+0000","last_fullsized":"2026-02-12T12:29:53.541272+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T12:29:50.511964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T12:29:50.511964+0000","last_clean_scrub_stamp":"2026-02-12T12:29:50.511964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-12T12:29:53.540671+0000","last_change":"2026-02-12T12:29:53.542534+0000","last_active":"2026-02-12T12:29:53.540671+0000","last_peered":"2026-02-12T12:29:53.540671+0000","last_clean":"2026-02-12T12:29:53.540671+0000","last_became_active":"2026-02-12T12:29:51.550232+0000","last_became_peered":"2026-02-12T12:29:51.550232+0000","last_unstale":"2026-02-12T12:29:53.540671+0000","last_undegraded":"2026-02-12T12:29:53.540671+0000","last_fullsized":"2026-02-12T12:29:53.540671+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T12:29:50.511964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T12:29:50.511964+0000","last_clean_scrub_stamp":"2026-02-12T12:29:50.511964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'1","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-12T12:29:53.541109+0000","last_change":"2026-02-12T12:29:53.541109+0000","last_active":"2026-02-12T12:29:53.541109+0000","last_peered":"2026-02-12T12:29:53.541109+0000","last_clean":"2026-02-12T12:29:53.541109+0000","last_became_active":"2026-02-12T12:29:51.557304+0000","last_became_peered":"2026-02-12T12:29:51.557304+0000","last_unstale":"2026-02-12T12:29:53.541109+0000","last_undegraded":"2026-02-12T12:29:53.541109+0000","last_fullsized":"2026-02-12T12:29:53.541109+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T12:29:50.511964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T12:29:50.511964+0000","last_clean_scrub_stamp":"2026-02-12T12:29:50.511964+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-12T12:29:53.538861+0000","last_change":"2026-02-12T12:29:53.540750+0000","last_active":"2026-02-12T12:29:53.538861+0000","last_peered":"2026-02-12T12:29:53.538861+0000","last_clean":"2026-02-12T12:29:53.538861+0000","last_became_active":"2026-02-12T12:29:51.557952+0000","last_became_peered":"2026-02-12T12:29:51.557952+0000","last_unstale":"2026-02-12T12:29:53.538861+0000","last_undegraded":"2026-02-12T12:29:53.538861+0000","last_fullsized":"2026-02-12T12:29:53.538861+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T12:29:50.511964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T12:29:50.511964+0000","last_clean_scrub_stamp":"2026-02-12T12:29:50.511964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-12T12:29:53.537239+0000","last_change":"2026-02-12T12:29:53.539438+0000","last_active":"2026-02-12T12:29:53.537239+0000","last_peered":"2026-02-12T12:29:53.537239+0000","last_clean":"2026-02-12T12:29:53.537239+0000","last_became_active":"2026-02-12T12:29:51.549086+0000","last_became_peered":"2026-02-12T12:29:51.549086+0000","last_unstale":"2026-02-12T12:29:53.537239+0000","last_undegraded":"2026-02-12T12:29:53.537239+0000","last_fullsized":"2026-02-12T12:29:53.537239+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T12:29:50.511964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T12:29:50.511964+0000","last_clean_scrub_stamp":"2026-02-12T12:29:50.511964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":16,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-12T12:29:53.541205+0000","last_change":"2026-02-12T12:29:53.544105+0000","last_active":"2026-02-12T12:29:53.541205+0000","last_peered":"2026-02-12T12:29:53.541205+0000","last_clean":"2026-02-12T12:29:53.541205+0000","last_became_active":"2026-02-12T12:29:51.552719+0000","last_became_peered":"2026-02-12T12:29:51.552719+0000","last_unstale":"2026-02-12T12:29:53.541205+0000","last_undegraded":"2026-02-12T12:29:53.541205+0000","last_fullsized":"2026-02-12T12:29:53.541205+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T12:29:50.511964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T12:29:50.511964+0000","last_clean_scrub_stamp":"2026-02-12T12:29:50.511964+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":90,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-12T12:29:53.539606+0000","last_change":"2026-02-12T12:29:48.203040+0000","last_active":"2026-02-12T12:29:53.539606+0000","last_peered":"2026-02-12T12:29:53.539606+0000","last_clean":"2026-02-12T12:29:53.539606+0000","last_became_active":"2026-02-12T12:29:48.195870+0000","last_became_peered":"2026-02-12T12:29:48.195870+0000","last_unstale":"2026-02-12T12:29:53.539606+0000","last_undegraded":"2026-02-12T12:29:53.539606+0000","last_fullsized":"2026-02-12T12:29:53.539606+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T12:29:47.166668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T12:29:47.166668+0000","last_clean_scrub_stamp":"2026-02-12T12:29:47.166668+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":161061224448,"available":96634163200,"internally_reserved":0,"allocated":64427061248,"data_stored":64427061248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53687074816,"available":32211079168,"internally_reserved":0,"allocated":21475995648,"data_stored":21475995648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20971820,"kb_used_data":20971820,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456964,"statfs":{"total":53687074816,"available":32211931136,"internally_reserved":0,"allocated":21475143680,"data_stored":21475143680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972580,"kb_used_data":20972580,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456204,"statfs":{"total":53687074816,"available":32211152896,"internally_reserved":0,"allocated":21475921920,"data_stored":21475921920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972652,"kb_used_data":20972652,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456132,"statfs":{"total":53687074816,"available":32211079168,"internally_reserved":0,"allocated":21475995648,"data_stored":21475995648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53687074816,"available":32211079168,"internally_reserved":0,"allocated":21475995648,"data_stored":21475995648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53687074816,"available":32211152896,"internally_reserved":0,"allocated":21475921920,"data_stored":21475921920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53687074816,"available":32211079168,"internally_reserved":0,"allocated":21475995648,"data_stored":21475995648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53687074816,"available":32211931136,"internally_reserved":0,"allocated":21475143680,"data_stored":21475143680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T12:29:56.404 INFO:tasks.ceph.ceph_manager.ceph:clean! 2026-02-12T12:29:56.405 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2026-02-12T12:29:56.405 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2026-02-12T12:29:56.405 DEBUG:teuthology.orchestra.run.trial127:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-12T12:29:56.481 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-12T12:29:56.479+0000 7f0131c1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:29:56.482 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-12T12:29:56.481+0000 7f0131c1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:29:56.619 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:29:56.619 INFO:teuthology.orchestra.run.trial127.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2026-02-12T12:29:56.630 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2026-02-12T12:29:56.630 INFO:teuthology.run_tasks:Running task ssh_keys... 2026-02-12T12:29:56.735 DEBUG:teuthology.orchestra.run.trial127:> echo 'StrictHostKeyChecking no 2026-02-12T12:29:56.735 DEBUG:teuthology.orchestra.run.trial127:> ' > /home/ubuntu/.ssh/config && echo 'UserKnownHostsFile ' /dev/null >> /home/ubuntu/.ssh/config && chmod 600 /home/ubuntu/.ssh/config 2026-02-12T12:29:56.750 INFO:teuthology.task.ssh_keys:generated public key AAAAB3NzaC1yc2EAAAADAQABAAABAQDAeFCWSvIXkr9DPH2/VDpgP9PBsWQVLaxsR46ViTFEqj/eGE/MeuLFRfLv0WZtsDANrusX6n9t+A4cO73gl1wEdpRWb/74ruztucJPCvfzXffZzMV1vBf337QofSMFMizFhNMg+3es4DadERn26abE1vLBwx9EpuiefQy+x2hgPxsfBu1glR0asiewLRUlZvDiDW7Ndp9DDVRvjy7R4gUAXzI8RC6c1ICONe5Eu/SKrtHfCJ9VjKGtwI8y3c3qtMe16i9jP9abXkF53DkmWL0VX/L4eZigifiQmRW6PNenyr2Ad+xK1VZuKNPR8Mq28I8VLSx+cBQSoNADD4tRJKPb 2026-02-12T12:29:56.750 INFO:teuthology.task.ssh_keys:pushing keys to trial127.front.sepia.ceph.com for ubuntu 2026-02-12T12:29:56.750 DEBUG:teuthology.orchestra.run.trial127:> rm -f -- /home/ubuntu/.ssh/id_rsa 2026-02-12T12:29:56.806 DEBUG:teuthology.orchestra.run.trial127:> set -ex 2026-02-12T12:29:56.806 DEBUG:teuthology.orchestra.run.trial127:> dd of=/home/ubuntu/.ssh/id_rsa 2026-02-12T12:29:56.806 DEBUG:teuthology.orchestra.run.trial127:> chmod 0500 /home/ubuntu/.ssh/id_rsa 2026-02-12T12:29:56.863 DEBUG:teuthology.orchestra.run.trial127:> rm -f -- /home/ubuntu/.ssh/id_rsa.pub 2026-02-12T12:29:56.918 DEBUG:teuthology.orchestra.run.trial127:> set -ex 2026-02-12T12:29:56.918 DEBUG:teuthology.orchestra.run.trial127:> dd of=/home/ubuntu/.ssh/id_rsa.pub 2026-02-12T12:29:56.974 DEBUG:teuthology.orchestra.run.trial127:> cp -v -a /home/ubuntu/.ssh/authorized_keys /home/ubuntu/.ssh/authorized_keys_2026-02-12_12:29:56:974066 2026-02-12T12:29:57.030 INFO:teuthology.orchestra.run.trial127.stdout:'/home/ubuntu/.ssh/authorized_keys' -> '/home/ubuntu/.ssh/authorized_keys_2026-02-12_12:29:56:974066' 2026-02-12T12:29:57.031 DEBUG:teuthology.orchestra.run.trial127:> set -ex 2026-02-12T12:29:57.031 DEBUG:teuthology.orchestra.run.trial127:> dd of=/home/ubuntu/.ssh/authorized_keys conv=notrunc oflag=append 2026-02-12T12:29:57.085 INFO:teuthology.run_tasks:Running task cbt... 2026-02-12T12:29:57.112 INFO:tasks.cbt:cbt configuration is {'cluster': {'user': 'ubuntu', 'head': 'ubuntu@trial127.front.sepia.ceph.com', 'osds': ['ubuntu@trial127.front.sepia.ceph.com'], 'mons': ['ubuntu@trial127.front.sepia.ceph.com'], 'clients': ['ubuntu@trial127.front.sepia.ceph.com'], 'rgws': {'ubuntu@trial127.front.sepia.ceph.com': None}, 'osds_per_node': 3, 'rebuild_every_test': False, 'use_existing': True, 'is_teuthology': True, 'iterations': 1, 'tmp_dir': '/tmp/cbt', 'pool_profiles': {'rbd': {'pg_size': 128, 'pgp_size': 128, 'replication': 3}}, 'pid_dir': '/var/run/ceph'}, 'benchmarks': {'librbdfio': {'iodepth': [32], 'log_avg_msec': 100, 'mode': ['randrw'], 'norandommap': True, 'op_size': [4194304], 'osd_ra': [4096], 'pool_profile': 'rbd', 'procs_per_volume': [1], 'time': 60, 'vol_size': 4096, 'volumes_per_client': [2], 'cmd_path': '/home/ubuntu/cephtest/fio/fio'}}, 'client_endpoints': None, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}} 2026-02-12T12:29:57.113 DEBUG:teuthology.orchestra.run.trial127:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/cbt 2026-02-12T12:29:57.146 DEBUG:teuthology.orchestra.run.trial127:> set -ex 2026-02-12T12:29:57.146 DEBUG:teuthology.orchestra.run.trial127:> dd of=/home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2026-02-12T12:29:57.198 DEBUG:teuthology.orchestra.run.trial127:> git clone --depth 1 -b master https://github.com/ceph/cbt.git /home/ubuntu/cephtest/cbt 2026-02-12T12:29:57.261 INFO:teuthology.orchestra.run.trial127.stderr:Cloning into '/home/ubuntu/cephtest/cbt'... 2026-02-12T12:29:57.535 INFO:tasks.cbt:Installing collectl 2026-02-12T12:29:57.535 DEBUG:teuthology.orchestra.run.trial127:> sudo mkdir collectl && cd collectl && sudo wget https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download -O collectl.tar.gz && sudo tar -xvf collectl.tar.gz && cd collectl-4.3.1 && sudo ./INSTALL 2026-02-12T12:29:57.610 INFO:teuthology.orchestra.run.trial127.stderr:--2026-02-12 12:29:57-- https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download 2026-02-12T12:30:02.669 INFO:teuthology.orchestra.run.trial127.stderr:Resolving sourceforge.net (sourceforge.net)... 104.18.13.149, 104.18.12.149, 2606:4700::6812:d95, ... 2026-02-12T12:30:02.672 INFO:teuthology.orchestra.run.trial127.stderr:Connecting to sourceforge.net (sourceforge.net)|104.18.13.149|:443... connected. 2026-02-12T12:30:02.935 INFO:teuthology.orchestra.run.trial127.stderr:HTTP request sent, awaiting response... 302 Found 2026-02-12T12:30:02.936 INFO:teuthology.orchestra.run.trial127.stderr:Location: https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABpjcfKvcnRibKxQcqOHIKORopDJ5E_kyxpnD8D0I4s-ND2jXfBWV_jXawd4EuCIU6Jgt4PP6MAB4fD9KyIXiJm3cL6BQ%3D%3D&use_mirror=netactuate&r= [following] 2026-02-12T12:30:02.937 INFO:teuthology.orchestra.run.trial127.stderr:--2026-02-12 12:30:02-- https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABpjcfKvcnRibKxQcqOHIKORopDJ5E_kyxpnD8D0I4s-ND2jXfBWV_jXawd4EuCIU6Jgt4PP6MAB4fD9KyIXiJm3cL6BQ%3D%3D&use_mirror=netactuate&r= 2026-02-12T12:30:02.942 INFO:teuthology.orchestra.run.trial127.stderr:Resolving downloads.sourceforge.net (downloads.sourceforge.net)... 104.18.12.149, 104.18.13.149, 2606:4700::6812:d95, ... 2026-02-12T12:30:02.945 INFO:teuthology.orchestra.run.trial127.stderr:Connecting to downloads.sourceforge.net (downloads.sourceforge.net)|104.18.12.149|:443... connected. 2026-02-12T12:30:03.130 INFO:teuthology.orchestra.run.trial127.stderr:HTTP request sent, awaiting response... 302 Found 2026-02-12T12:30:03.130 INFO:teuthology.orchestra.run.trial127.stderr:Location: https://netactuate.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?viasf=1 [following] 2026-02-12T12:30:03.130 INFO:teuthology.orchestra.run.trial127.stderr:--2026-02-12 12:30:03-- https://netactuate.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?viasf=1 2026-02-12T12:30:03.137 INFO:teuthology.orchestra.run.trial127.stderr:Resolving netactuate.dl.sourceforge.net (netactuate.dl.sourceforge.net)... 104.225.3.66 2026-02-12T12:30:03.157 INFO:teuthology.orchestra.run.trial127.stderr:Connecting to netactuate.dl.sourceforge.net (netactuate.dl.sourceforge.net)|104.225.3.66|:443... connected. 2026-02-12T12:30:03.301 INFO:teuthology.orchestra.run.trial127.stderr:HTTP request sent, awaiting response... 200 OK 2026-02-12T12:30:03.301 INFO:teuthology.orchestra.run.trial127.stderr:Length: 620693 (606K) [application/x-gzip] 2026-02-12T12:30:03.301 INFO:teuthology.orchestra.run.trial127.stderr:Saving to: ‘collectl.tar.gz’ 2026-02-12T12:30:03.301 INFO:teuthology.orchestra.run.trial127.stderr: 2026-02-12T12:30:03.321 INFO:teuthology.orchestra.run.trial127.stderr: 0K .......... .......... .......... .......... .......... 8% 2.44M 0s 2026-02-12T12:30:03.342 INFO:teuthology.orchestra.run.trial127.stderr: 50K .......... .......... .......... .......... .......... 16% 2.49M 0s 2026-02-12T12:30:03.342 INFO:teuthology.orchestra.run.trial127.stderr: 100K .......... .......... .......... .......... .......... 24% 119M 0s 2026-02-12T12:30:03.342 INFO:teuthology.orchestra.run.trial127.stderr: 150K .......... .......... .......... .......... .......... 32% 361M 0s 2026-02-12T12:30:03.364 INFO:teuthology.orchestra.run.trial127.stderr: 200K .......... .......... .......... .......... .......... 41% 2.49M 0s 2026-02-12T12:30:03.364 INFO:teuthology.orchestra.run.trial127.stderr: 250K .......... .......... .......... .......... .......... 49% 376M 0s 2026-02-12T12:30:03.365 INFO:teuthology.orchestra.run.trial127.stderr: 300K .......... .......... .......... .......... .......... 57% 403M 0s 2026-02-12T12:30:03.365 INFO:teuthology.orchestra.run.trial127.stderr: 350K .......... .......... .......... .......... .......... 65% 329M 0s 2026-02-12T12:30:03.365 INFO:teuthology.orchestra.run.trial127.stderr: 400K .......... .......... .......... .......... .......... 74% 408M 0s 2026-02-12T12:30:03.383 INFO:teuthology.orchestra.run.trial127.stderr: 450K .......... .......... .......... .......... .......... 82% 2.51M 0s 2026-02-12T12:30:03.383 INFO:teuthology.orchestra.run.trial127.stderr: 500K .......... .......... .......... .......... .......... 90% 385M 0s 2026-02-12T12:30:03.383 INFO:teuthology.orchestra.run.trial127.stderr: 550K .......... .......... .......... .......... .......... 98% 365M 0s 2026-02-12T12:30:03.383 INFO:teuthology.orchestra.run.trial127.stderr: 600K ...... 100% 11.4T=0.08s 2026-02-12T12:30:03.383 INFO:teuthology.orchestra.run.trial127.stderr: 2026-02-12T12:30:03.383 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-12 12:30:03 (7.40 MB/s) - ‘collectl.tar.gz’ saved [620693/620693] 2026-02-12T12:30:03.383 INFO:teuthology.orchestra.run.trial127.stderr: 2026-02-12T12:30:03.415 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/ 2026-02-12T12:30:03.415 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/collectl.conf 2026-02-12T12:30:03.416 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/ 2026-02-12T12:30:03.416 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/SlowProc.html 2026-02-12T12:30:03.416 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/BuddyInfo.html 2026-02-12T12:30:03.416 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/WhySummary.html 2026-02-12T12:30:03.416 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/Socket.html 2026-02-12T12:30:03.416 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/colmux.html 2026-02-12T12:30:03.416 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/Tutorial.html 2026-02-12T12:30:03.416 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/Gexpr.html 2026-02-12T12:30:03.416 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/Import.html 2026-02-12T12:30:03.416 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/style.css 2026-02-12T12:30:03.416 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/Ganglia3.jpg 2026-02-12T12:30:03.416 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/Ganglia2.jpg 2026-02-12T12:30:03.417 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/FileNaming.html 2026-02-12T12:30:03.417 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/TheMath.html 2026-02-12T12:30:03.417 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/Environmental.html 2026-02-12T12:30:03.417 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/Process.html 2026-02-12T12:30:03.417 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/OutputFiles.html 2026-02-12T12:30:03.417 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/HiResTime.html 2026-02-12T12:30:03.417 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/OutputFormats.html 2026-02-12T12:30:03.417 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/slub.jpg 2026-02-12T12:30:03.417 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/Lustre.html 2026-02-12T12:30:03.417 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/Documentation.html 2026-02-12T12:30:03.417 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/Inodes.html 2026-02-12T12:30:03.417 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/Graphite.html 2026-02-12T12:30:03.418 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/InputFiles.html 2026-02-12T12:30:03.418 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/Export.html 2026-02-12T12:30:03.418 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/Interrupts.html 2026-02-12T12:30:03.418 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/Startup.html 2026-02-12T12:30:03.418 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/Performance.html 2026-02-12T12:30:03.418 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/Messages.html 2026-02-12T12:30:03.418 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/Logging.html 2026-02-12T12:30:03.418 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/Data-detail.html 2026-02-12T12:30:03.418 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/CPUs.html 2026-02-12T12:30:03.418 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/Matrix.html 2026-02-12T12:30:03.418 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/Data-brief.html 2026-02-12T12:30:03.418 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/SlabInfo.html 2026-02-12T12:30:03.418 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/Misc.html 2026-02-12T12:30:03.419 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/Examples.html 2026-02-12T12:30:03.419 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/OpenStack.html 2026-02-12T12:30:03.419 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/Plotfiles.html 2026-02-12T12:30:03.419 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/ColmuxCPU.jpg 2026-02-12T12:30:03.419 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/Tutorial-Lustre.html 2026-02-12T12:30:03.419 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/Data-verbose.html 2026-02-12T12:30:03.419 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/Data.html 2026-02-12T12:30:03.419 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/Exceptions.html 2026-02-12T12:30:03.419 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/RunningAsAService.html 2026-02-12T12:30:03.419 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/FAQ-collectl.html 2026-02-12T12:30:03.419 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/Ganglia.jpg 2026-02-12T12:30:03.419 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/OperationalModes.html 2026-02-12T12:30:03.419 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/Playback.html 2026-02-12T12:30:03.420 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/Hello.html 2026-02-12T12:30:03.420 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/Sockets.html 2026-02-12T12:30:03.420 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/Network.html 2026-02-12T12:30:03.420 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/Infiniband.html 2026-02-12T12:30:03.420 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/Memory.html 2026-02-12T12:30:03.420 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/ColmuxLustre.jpg 2026-02-12T12:30:03.420 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/Disks.html 2026-02-12T12:30:03.420 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/docs/NfsInfo.html 2026-02-12T12:30:03.420 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/service/ 2026-02-12T12:30:03.420 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/service/collectl.service 2026-02-12T12:30:03.420 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/initd/ 2026-02-12T12:30:03.420 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/initd/collectl-suse 2026-02-12T12:30:03.420 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/initd/collectl-debian 2026-02-12T12:30:03.420 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/initd/collectl-generic 2026-02-12T12:30:03.421 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/initd/collectl 2026-02-12T12:30:03.421 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/misc.ph 2026-02-12T12:30:03.421 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/gexpr.ph 2026-02-12T12:30:03.421 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/statsd.ph 2026-02-12T12:30:03.421 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/envrules.std 2026-02-12T12:30:03.421 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/hello.ph 2026-02-12T12:30:03.421 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/vmsum.ph 2026-02-12T12:30:03.421 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/proctree.ph 2026-02-12T12:30:03.421 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/README-WINDOWS 2026-02-12T12:30:03.421 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/formatit.ph 2026-02-12T12:30:03.421 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/colmux 2026-02-12T12:30:03.421 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/INSTALL 2026-02-12T12:30:03.421 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/graphite.ph 2026-02-12T12:30:03.422 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/RELEASE-collectl 2026-02-12T12:30:03.422 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/vmstat.ph 2026-02-12T12:30:03.422 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/README 2026-02-12T12:30:03.422 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/man1/ 2026-02-12T12:30:03.422 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/man1/collectl.1 2026-02-12T12:30:03.422 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/man1/colmux.1 2026-02-12T12:30:03.422 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/lexpr.ph 2026-02-12T12:30:03.422 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/client.pl 2026-02-12T12:30:03.422 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/collectl 2026-02-12T12:30:03.422 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/COPYING 2026-02-12T12:30:03.422 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/vnet.ph 2026-02-12T12:30:03.422 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/GPL 2026-02-12T12:30:03.422 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/UNINSTALL 2026-02-12T12:30:03.423 INFO:teuthology.orchestra.run.trial127.stdout:collectl-4.3.1/ARTISTIC 2026-02-12T12:30:03.460 INFO:teuthology.orchestra.run.trial127.stderr:./INSTALL: line 90: chkconfig: command not found 2026-02-12T12:30:03.461 DEBUG:teuthology.orchestra.run.trial127:> sudo yum -y install librbd-devel pdsh pdsh-rcmd-ssh perf 2026-02-12T12:30:03.810 INFO:teuthology.orchestra.run.trial127.stdout:Last metadata expiration check: 0:01:02 ago on Thu 12 Feb 2026 12:29:01 PM UTC. 2026-02-12T12:30:03.896 INFO:teuthology.orchestra.run.trial127.stdout:Dependencies resolved. 2026-02-12T12:30:03.897 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-12T12:30:03.897 INFO:teuthology.orchestra.run.trial127.stdout: Package Arch Version Repository Size 2026-02-12T12:30:03.897 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-12T12:30:03.897 INFO:teuthology.orchestra.run.trial127.stdout:Installing: 2026-02-12T12:30:03.897 INFO:teuthology.orchestra.run.trial127.stdout: librbd-devel x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 24 k 2026-02-12T12:30:03.897 INFO:teuthology.orchestra.run.trial127.stdout: pdsh x86_64 2.34-7.el9 epel 118 k 2026-02-12T12:30:03.897 INFO:teuthology.orchestra.run.trial127.stdout: pdsh-rcmd-ssh x86_64 2.34-7.el9 epel 13 k 2026-02-12T12:30:03.898 INFO:teuthology.orchestra.run.trial127.stdout: perf x86_64 5.14.0-677.el9 appstream 3.2 M 2026-02-12T12:30:03.898 INFO:teuthology.orchestra.run.trial127.stdout:Installing dependencies: 2026-02-12T12:30:03.898 INFO:teuthology.orchestra.run.trial127.stdout: bzip2 x86_64 1.0.8-10.el9 baseos 56 k 2026-02-12T12:30:03.898 INFO:teuthology.orchestra.run.trial127.stdout: libradospp-devel x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 31 k 2026-02-12T12:30:03.898 INFO:teuthology.orchestra.run.trial127.stdout: libtraceevent x86_64 1.8.4-2.el9 baseos 290 k 2026-02-12T12:30:03.898 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:30:03.898 INFO:teuthology.orchestra.run.trial127.stdout:Transaction Summary 2026-02-12T12:30:03.898 INFO:teuthology.orchestra.run.trial127.stdout:================================================================================ 2026-02-12T12:30:03.898 INFO:teuthology.orchestra.run.trial127.stdout:Install 7 Packages 2026-02-12T12:30:03.898 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:30:03.898 INFO:teuthology.orchestra.run.trial127.stdout:Total download size: 3.7 M 2026-02-12T12:30:03.898 INFO:teuthology.orchestra.run.trial127.stdout:Installed size: 14 M 2026-02-12T12:30:03.898 INFO:teuthology.orchestra.run.trial127.stdout:Downloading Packages: 2026-02-12T12:30:04.979 INFO:teuthology.orchestra.run.trial127.stdout:(1/7): bzip2-1.0.8-10.el9.x86_64.rpm 90 kB/s | 56 kB 00:00 2026-02-12T12:30:05.435 INFO:teuthology.orchestra.run.trial127.stdout:(2/7): libtraceevent-1.8.4-2.el9.x86_64.rpm 636 kB/s | 290 kB 00:00 2026-02-12T12:30:05.904 INFO:teuthology.orchestra.run.trial127.stdout:(3/7): perf-5.14.0-677.el9.x86_64.rpm 6.7 MB/s | 3.2 MB 00:00 2026-02-12T12:30:09.557 INFO:teuthology.orchestra.run.trial127.stdout:(4/7): libradospp-devel-20.3.0-5145.g5fdb7d8f.e 6.0 kB/s | 31 kB 00:05 2026-02-12T12:30:09.559 INFO:teuthology.orchestra.run.trial127.stdout:(5/7): librbd-devel-20.3.0-5145.g5fdb7d8f.el9.x 4.6 kB/s | 24 kB 00:05 2026-02-12T12:30:09.961 INFO:teuthology.orchestra.run.trial127.stdout:(6/7): pdsh-rcmd-ssh-2.34-7.el9.x86_64.rpm 32 kB/s | 13 kB 00:00 2026-02-12T12:30:10.054 INFO:teuthology.orchestra.run.trial127.stdout:(7/7): pdsh-2.34-7.el9.x86_64.rpm 29 kB/s | 118 kB 00:04 2026-02-12T12:30:10.058 INFO:teuthology.orchestra.run.trial127.stdout:-------------------------------------------------------------------------------- 2026-02-12T12:30:10.058 INFO:teuthology.orchestra.run.trial127.stdout:Total 611 kB/s | 3.7 MB 00:06 2026-02-12T12:30:10.090 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction check 2026-02-12T12:30:10.101 INFO:teuthology.orchestra.run.trial127.stdout:Transaction check succeeded. 2026-02-12T12:30:10.101 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction test 2026-02-12T12:30:10.159 INFO:teuthology.orchestra.run.trial127.stdout:Transaction test succeeded. 2026-02-12T12:30:10.159 INFO:teuthology.orchestra.run.trial127.stdout:Running transaction 2026-02-12T12:30:10.234 INFO:teuthology.orchestra.run.trial127.stdout: Preparing : 1/1 2026-02-12T12:30:10.239 INFO:teuthology.orchestra.run.trial127.stdout: Installing : pdsh-rcmd-ssh-2.34-7.el9.x86_64 1/7 2026-02-12T12:30:10.279 INFO:teuthology.orchestra.run.trial127.stdout: Installing : pdsh-2.34-7.el9.x86_64 2/7 2026-02-12T12:30:10.284 INFO:teuthology.orchestra.run.trial127.stdout: Installing : libtraceevent-1.8.4-2.el9.x86_64 3/7 2026-02-12T12:30:10.288 INFO:teuthology.orchestra.run.trial127.stdout: Installing : bzip2-1.0.8-10.el9.x86_64 4/7 2026-02-12T12:30:10.290 INFO:teuthology.orchestra.run.trial127.stdout: Installing : libradospp-devel-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 5/7 2026-02-12T12:30:10.451 INFO:teuthology.orchestra.run.trial127.stdout: Installing : librbd-devel-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 6/7 2026-02-12T12:30:10.471 INFO:teuthology.orchestra.run.trial127.stdout: Installing : perf-5.14.0-677.el9.x86_64 7/7 2026-02-12T12:30:10.581 INFO:teuthology.orchestra.run.trial127.stdout: Running scriptlet: perf-5.14.0-677.el9.x86_64 7/7 2026-02-12T12:30:10.582 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : libradospp-devel-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 1/7 2026-02-12T12:30:10.582 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : librbd-devel-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2/7 2026-02-12T12:30:10.582 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : bzip2-1.0.8-10.el9.x86_64 3/7 2026-02-12T12:30:10.582 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : libtraceevent-1.8.4-2.el9.x86_64 4/7 2026-02-12T12:30:10.582 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : perf-5.14.0-677.el9.x86_64 5/7 2026-02-12T12:30:10.582 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : pdsh-2.34-7.el9.x86_64 6/7 2026-02-12T12:30:10.642 INFO:teuthology.orchestra.run.trial127.stdout: Verifying : pdsh-rcmd-ssh-2.34-7.el9.x86_64 7/7 2026-02-12T12:30:10.642 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:30:10.642 INFO:teuthology.orchestra.run.trial127.stdout:Installed: 2026-02-12T12:30:10.642 INFO:teuthology.orchestra.run.trial127.stdout: bzip2-1.0.8-10.el9.x86_64 2026-02-12T12:30:10.642 INFO:teuthology.orchestra.run.trial127.stdout: libradospp-devel-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:30:10.643 INFO:teuthology.orchestra.run.trial127.stdout: librbd-devel-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:30:10.643 INFO:teuthology.orchestra.run.trial127.stdout: libtraceevent-1.8.4-2.el9.x86_64 2026-02-12T12:30:10.643 INFO:teuthology.orchestra.run.trial127.stdout: pdsh-2.34-7.el9.x86_64 2026-02-12T12:30:10.643 INFO:teuthology.orchestra.run.trial127.stdout: pdsh-rcmd-ssh-2.34-7.el9.x86_64 2026-02-12T12:30:10.643 INFO:teuthology.orchestra.run.trial127.stdout: perf-5.14.0-677.el9.x86_64 2026-02-12T12:30:10.643 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:30:10.643 INFO:teuthology.orchestra.run.trial127.stdout:Complete! 2026-02-12T12:30:10.673 DEBUG:teuthology.orchestra.run.trial127:> python3 -m pip install -r /home/ubuntu/cephtest/cbt/requirements.txt 2026-02-12T12:30:10.812 INFO:teuthology.orchestra.run.trial127.stdout:Defaulting to user installation because normal site-packages is not writeable 2026-02-12T12:30:10.824 INFO:teuthology.orchestra.run.trial127.stdout:Requirement already satisfied: pyyaml in /usr/lib64/python3.9/site-packages (from -r /home/ubuntu/cephtest/cbt/requirements.txt (line 1)) (5.4.1) 2026-02-12T12:30:10.824 INFO:teuthology.orchestra.run.trial127.stdout:Requirement already satisfied: lxml in /usr/lib64/python3.9/site-packages (from -r /home/ubuntu/cephtest/cbt/requirements.txt (line 2)) (4.6.5) 2026-02-12T12:30:11.148 INFO:teuthology.orchestra.run.trial127.stdout:Collecting matplotlib 2026-02-12T12:30:11.182 INFO:teuthology.orchestra.run.trial127.stdout: Downloading matplotlib-3.9.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (8.3 MB) 2026-02-12T12:30:11.248 INFO:teuthology.orchestra.run.trial127.stdout:Requirement already satisfied: numpy>=1.23 in /usr/lib64/python3.9/site-packages (from matplotlib->-r /home/ubuntu/cephtest/cbt/requirements.txt (line 3)) (1.23.5) 2026-02-12T12:30:11.567 INFO:teuthology.orchestra.run.trial127.stdout:Collecting fonttools>=4.22.0 2026-02-12T12:30:11.574 INFO:teuthology.orchestra.run.trial127.stdout: Downloading fonttools-4.60.2-cp39-cp39-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (4.8 MB) 2026-02-12T12:30:11.651 INFO:teuthology.orchestra.run.trial127.stdout:Collecting importlib-resources>=3.2.0 2026-02-12T12:30:11.654 INFO:teuthology.orchestra.run.trial127.stdout: Downloading importlib_resources-6.5.2-py3-none-any.whl (37 kB) 2026-02-12T12:30:11.793 INFO:teuthology.orchestra.run.trial127.stdout:Collecting contourpy>=1.0.1 2026-02-12T12:30:11.797 INFO:teuthology.orchestra.run.trial127.stdout: Downloading contourpy-1.3.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (321 kB) 2026-02-12T12:30:11.801 INFO:teuthology.orchestra.run.trial127.stdout:Requirement already satisfied: packaging>=20.0 in /usr/lib/python3.9/site-packages (from matplotlib->-r /home/ubuntu/cephtest/cbt/requirements.txt (line 3)) (20.9) 2026-02-12T12:30:11.939 INFO:teuthology.orchestra.run.trial127.stdout:Collecting kiwisolver>=1.3.1 2026-02-12T12:30:11.945 INFO:teuthology.orchestra.run.trial127.stdout: Downloading kiwisolver-1.4.7-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.6 MB) 2026-02-12T12:30:11.955 INFO:teuthology.orchestra.run.trial127.stdout:Requirement already satisfied: python-dateutil>=2.7 in /usr/lib/python3.9/site-packages (from matplotlib->-r /home/ubuntu/cephtest/cbt/requirements.txt (line 3)) (2.9.0.post0) 2026-02-12T12:30:12.367 INFO:teuthology.orchestra.run.trial127.stdout:Collecting pillow>=8 2026-02-12T12:30:12.373 INFO:teuthology.orchestra.run.trial127.stdout: Downloading pillow-11.3.0-cp39-cp39-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (6.6 MB) 2026-02-12T12:30:12.406 INFO:teuthology.orchestra.run.trial127.stdout:Requirement already satisfied: pyparsing>=2.3.1 in /usr/lib/python3.9/site-packages (from matplotlib->-r /home/ubuntu/cephtest/cbt/requirements.txt (line 3)) (2.4.7) 2026-02-12T12:30:12.416 INFO:teuthology.orchestra.run.trial127.stdout:Collecting cycler>=0.10 2026-02-12T12:30:12.420 INFO:teuthology.orchestra.run.trial127.stdout: Downloading cycler-0.12.1-py3-none-any.whl (8.3 kB) 2026-02-12T12:30:12.470 INFO:teuthology.orchestra.run.trial127.stdout:Collecting zipp>=3.1.0 2026-02-12T12:30:12.474 INFO:teuthology.orchestra.run.trial127.stdout: Downloading zipp-3.23.0-py3-none-any.whl (10 kB) 2026-02-12T12:30:12.485 INFO:teuthology.orchestra.run.trial127.stdout:Requirement already satisfied: six>=1.5 in /usr/lib/python3.9/site-packages (from python-dateutil>=2.7->matplotlib->-r /home/ubuntu/cephtest/cbt/requirements.txt (line 3)) (1.15.0) 2026-02-12T12:30:12.599 INFO:teuthology.orchestra.run.trial127.stdout:Installing collected packages: zipp, pillow, kiwisolver, importlib-resources, fonttools, cycler, contourpy, matplotlib 2026-02-12T12:30:13.660 INFO:teuthology.orchestra.run.trial127.stdout:Successfully installed contourpy-1.3.0 cycler-0.12.1 fonttools-4.60.2 importlib-resources-6.5.2 kiwisolver-1.4.7 matplotlib-3.9.4 pillow-11.3.0 zipp-3.23.0 2026-02-12T12:30:13.771 INFO:tasks.cbt:benchmark: librbdfio 2026-02-12T12:30:13.771 DEBUG:teuthology.orchestra.run.trial127:> git clone -b master https://github.com/axboe/fio.git /home/ubuntu/cephtest/fio 2026-02-12T12:30:13.788 INFO:teuthology.orchestra.run.trial127.stderr:Cloning into '/home/ubuntu/cephtest/fio'... 2026-02-12T12:30:14.591 DEBUG:teuthology.orchestra.run.trial127:> cd /home/ubuntu/cephtest/fio && ./configure && make 2026-02-12T12:30:14.764 INFO:teuthology.orchestra.run.trial127.stdout:Operating system Linux 2026-02-12T12:30:14.764 INFO:teuthology.orchestra.run.trial127.stdout:CPU x86_64 2026-02-12T12:30:14.764 INFO:teuthology.orchestra.run.trial127.stdout:Big endian no 2026-02-12T12:30:14.764 INFO:teuthology.orchestra.run.trial127.stdout:Compiler gcc 2026-02-12T12:30:14.765 INFO:teuthology.orchestra.run.trial127.stdout:Cross compile no 2026-02-12T12:30:14.765 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:30:14.765 INFO:teuthology.orchestra.run.trial127.stdout:Static build no 2026-02-12T12:30:14.813 INFO:teuthology.orchestra.run.trial127.stdout:Wordsize 64 2026-02-12T12:30:14.818 INFO:teuthology.orchestra.run.trial127.stdout:zlib no 2026-02-12T12:30:14.829 INFO:teuthology.orchestra.run.trial127.stdout:fcntl(F_FULLFSYNC) no 2026-02-12T12:30:14.840 INFO:teuthology.orchestra.run.trial127.stdout:Linux AIO support no 2026-02-12T12:30:14.840 INFO:teuthology.orchestra.run.trial127.stdout:Linux AIO support rw flags no 2026-02-12T12:30:14.863 INFO:teuthology.orchestra.run.trial127.stdout:POSIX AIO support yes 2026-02-12T12:30:14.863 INFO:teuthology.orchestra.run.trial127.stdout:POSIX AIO support needs -lrt no 2026-02-12T12:30:14.882 INFO:teuthology.orchestra.run.trial127.stdout:POSIX AIO fsync yes 2026-02-12T12:30:14.903 INFO:teuthology.orchestra.run.trial127.stdout:POSIX pshared support yes 2026-02-12T12:30:14.928 INFO:teuthology.orchestra.run.trial127.stdout:pthread_condattr_setclock() yes 2026-02-12T12:30:14.951 INFO:teuthology.orchestra.run.trial127.stdout:pthread_sigmask() yes 2026-02-12T12:30:14.974 INFO:teuthology.orchestra.run.trial127.stdout:pthread_getaffinity_np() yes 2026-02-12T12:30:14.980 INFO:teuthology.orchestra.run.trial127.stdout:Solaris AIO support no 2026-02-12T12:30:15.001 INFO:teuthology.orchestra.run.trial127.stdout:__sync_fetch_and_add yes 2026-02-12T12:30:15.022 INFO:teuthology.orchestra.run.trial127.stdout:__sync_synchronize yes 2026-02-12T12:30:15.041 INFO:teuthology.orchestra.run.trial127.stdout:__sync_val_compare_and_swap yes 2026-02-12T12:30:15.046 INFO:teuthology.orchestra.run.trial127.stdout:libverbs no 2026-02-12T12:30:15.055 INFO:teuthology.orchestra.run.trial127.stdout:rdmacm no 2026-02-12T12:30:15.075 INFO:teuthology.orchestra.run.trial127.stdout:asprintf() yes 2026-02-12T12:30:15.096 INFO:teuthology.orchestra.run.trial127.stdout:vasprintf() yes 2026-02-12T12:30:15.117 INFO:teuthology.orchestra.run.trial127.stdout:Linux fallocate yes 2026-02-12T12:30:15.138 INFO:teuthology.orchestra.run.trial127.stdout:POSIX fadvise yes 2026-02-12T12:30:15.159 INFO:teuthology.orchestra.run.trial127.stdout:POSIX fallocate yes 2026-02-12T12:30:15.179 INFO:teuthology.orchestra.run.trial127.stdout:sched_setaffinity(3 arg) yes 2026-02-12T12:30:15.180 INFO:teuthology.orchestra.run.trial127.stdout:sched_setaffinity(2 arg) no 2026-02-12T12:30:15.202 INFO:teuthology.orchestra.run.trial127.stdout:clock_gettime yes 2026-02-12T12:30:15.223 INFO:teuthology.orchestra.run.trial127.stdout:CLOCK_MONOTONIC yes 2026-02-12T12:30:15.244 INFO:teuthology.orchestra.run.trial127.stdout:gettimeofday yes 2026-02-12T12:30:15.265 INFO:teuthology.orchestra.run.trial127.stdout:fdatasync yes 2026-02-12T12:30:15.286 INFO:teuthology.orchestra.run.trial127.stdout:pipe() yes 2026-02-12T12:30:15.306 INFO:teuthology.orchestra.run.trial127.stdout:pipe2() yes 2026-02-12T12:30:15.327 INFO:teuthology.orchestra.run.trial127.stdout:pread() yes 2026-02-12T12:30:15.350 INFO:teuthology.orchestra.run.trial127.stdout:sync_file_range yes 2026-02-12T12:30:15.355 INFO:teuthology.orchestra.run.trial127.stdout:ASharedMemory_create no 2026-02-12T12:30:15.363 INFO:teuthology.orchestra.run.trial127.stdout:EXT4 move extent yes 2026-02-12T12:30:15.383 INFO:teuthology.orchestra.run.trial127.stdout:Linux splice(2) yes 2026-02-12T12:30:15.389 INFO:teuthology.orchestra.run.trial127.stdout:libnuma no 2026-02-12T12:30:15.408 INFO:teuthology.orchestra.run.trial127.stdout:strsep yes 2026-02-12T12:30:15.429 INFO:teuthology.orchestra.run.trial127.stdout:strcasestr yes 2026-02-12T12:30:15.438 INFO:teuthology.orchestra.run.trial127.stdout:strlcat no 2026-02-12T12:30:15.459 INFO:teuthology.orchestra.run.trial127.stdout:getopt_long_only() yes 2026-02-12T12:30:15.484 INFO:teuthology.orchestra.run.trial127.stdout:inet_aton yes 2026-02-12T12:30:15.505 INFO:teuthology.orchestra.run.trial127.stdout:socklen_t yes 2026-02-12T12:30:15.527 INFO:teuthology.orchestra.run.trial127.stdout:__thread yes 2026-02-12T12:30:15.548 INFO:teuthology.orchestra.run.trial127.stdout:RUSAGE_THREAD yes 2026-02-12T12:30:15.571 INFO:teuthology.orchestra.run.trial127.stdout:SCHED_IDLE yes 2026-02-12T12:30:15.594 INFO:teuthology.orchestra.run.trial127.stdout:TCP_NODELAY yes 2026-02-12T12:30:15.614 INFO:teuthology.orchestra.run.trial127.stdout:vsock yes 2026-02-12T12:30:15.637 INFO:teuthology.orchestra.run.trial127.stdout:Net engine window_size yes 2026-02-12T12:30:15.661 INFO:teuthology.orchestra.run.trial127.stdout:TCP_MAXSEG yes 2026-02-12T12:30:15.681 INFO:teuthology.orchestra.run.trial127.stdout:RLIMIT_MEMLOCK yes 2026-02-12T12:30:15.704 INFO:teuthology.orchestra.run.trial127.stdout:pwritev/preadv yes 2026-02-12T12:30:15.726 INFO:teuthology.orchestra.run.trial127.stdout:pwritev2/preadv2 yes 2026-02-12T12:30:15.752 INFO:teuthology.orchestra.run.trial127.stdout:IPv6 helpers yes 2026-02-12T12:30:15.762 INFO:teuthology.orchestra.run.trial127.stdout:http engine no 2026-02-12T12:30:15.815 INFO:teuthology.orchestra.run.trial127.stdout:Rados engine yes 2026-02-12T12:30:15.880 INFO:teuthology.orchestra.run.trial127.stdout:Rados Block Device engine yes 2026-02-12T12:30:15.944 INFO:teuthology.orchestra.run.trial127.stdout:rbd_poll yes 2026-02-12T12:30:16.009 INFO:teuthology.orchestra.run.trial127.stdout:rbd_invalidate_cache yes 2026-02-12T12:30:16.076 INFO:teuthology.orchestra.run.trial127.stdout:rbd_encryption_load yes 2026-02-12T12:30:16.097 INFO:teuthology.orchestra.run.trial127.stdout:setvbuf yes 2026-02-12T12:30:16.103 INFO:teuthology.orchestra.run.trial127.stdout:Gluster API engine no 2026-02-12T12:30:16.114 INFO:teuthology.orchestra.run.trial127.stdout:s390_z196_facilities no 2026-02-12T12:30:16.114 INFO:teuthology.orchestra.run.trial127.stdout:HDFS engine no 2026-02-12T12:30:16.136 INFO:teuthology.orchestra.run.trial127.stdout:MTD yes 2026-02-12T12:30:16.142 INFO:teuthology.orchestra.run.trial127.stdout:libpmem no 2026-02-12T12:30:16.142 INFO:teuthology.orchestra.run.trial127.stdout:libpmem1_5 no 2026-02-12T12:30:16.148 INFO:teuthology.orchestra.run.trial127.stdout:libpmem2 no 2026-02-12T12:30:16.148 INFO:teuthology.orchestra.run.trial127.stdout:PMDK dev-dax engine no 2026-02-12T12:30:16.148 INFO:teuthology.orchestra.run.trial127.stdout:PMDK libpmem engine no 2026-02-12T12:30:16.154 INFO:teuthology.orchestra.run.trial127.stdout:DDN's Infinite Memory Engine no 2026-02-12T12:30:16.154 INFO:teuthology.orchestra.run.trial127.stdout:iscsi engine no 2026-02-12T12:30:16.154 INFO:teuthology.orchestra.run.trial127.stdout:NBD engine no 2026-02-12T12:30:16.160 INFO:teuthology.orchestra.run.trial127.stdout:DAOS File System (dfs) Engine no 2026-02-12T12:30:16.166 INFO:teuthology.orchestra.run.trial127.stdout:NFS engine no 2026-02-12T12:30:16.167 INFO:teuthology.orchestra.run.trial127.stdout:lex/yacc for arithmetic no 2026-02-12T12:30:16.188 INFO:teuthology.orchestra.run.trial127.stdout:getmntent yes 2026-02-12T12:30:16.201 INFO:teuthology.orchestra.run.trial127.stdout:getmntinfo no 2026-02-12T12:30:16.232 INFO:teuthology.orchestra.run.trial127.stdout:Static Assert yes 2026-02-12T12:30:16.251 INFO:teuthology.orchestra.run.trial127.stdout:bool yes 2026-02-12T12:30:16.274 INFO:teuthology.orchestra.run.trial127.stdout:strndup yes 2026-02-12T12:30:16.280 INFO:teuthology.orchestra.run.trial127.stdout:Valgrind headers no 2026-02-12T12:30:16.299 INFO:teuthology.orchestra.run.trial127.stdout:Zoned block device support yes 2026-02-12T12:30:16.319 INFO:teuthology.orchestra.run.trial127.stdout:Zoned block device capacity yes 2026-02-12T12:30:16.325 INFO:teuthology.orchestra.run.trial127.stdout:libzbc engine no 2026-02-12T12:30:16.343 INFO:teuthology.orchestra.run.trial127.stdout:NVMe uring command support yes 2026-02-12T12:30:16.355 INFO:teuthology.orchestra.run.trial127.stdout:xnvme engine no 2026-02-12T12:30:16.361 INFO:teuthology.orchestra.run.trial127.stdout:isal no 2026-02-12T12:30:16.362 INFO:teuthology.orchestra.run.trial127.stdout:isal CRC64 no 2026-02-12T12:30:16.373 INFO:teuthology.orchestra.run.trial127.stdout:libblkio engine no 2026-02-12T12:30:16.373 INFO:teuthology.orchestra.run.trial127.stdout:march_armv8_a_crc_crypto no 2026-02-12T12:30:16.373 INFO:teuthology.orchestra.run.trial127.stdout:cuda no 2026-02-12T12:30:16.373 INFO:teuthology.orchestra.run.trial127.stdout:libcufile no 2026-02-12T12:30:16.395 INFO:teuthology.orchestra.run.trial127.stdout:Build march=native yes 2026-02-12T12:30:16.400 INFO:teuthology.orchestra.run.trial127.stdout:CUnit no 2026-02-12T12:30:16.420 INFO:teuthology.orchestra.run.trial127.stdout:__kernel_rwf_t yes 2026-02-12T12:30:16.441 INFO:teuthology.orchestra.run.trial127.stdout:-Wimplicit-fallthrough=2 yes 2026-02-12T12:30:16.462 INFO:teuthology.orchestra.run.trial127.stdout:-Wno-stringop-truncation yes 2026-02-12T12:30:16.484 INFO:teuthology.orchestra.run.trial127.stdout:MADV_HUGEPAGE yes 2026-02-12T12:30:16.505 INFO:teuthology.orchestra.run.trial127.stdout:gettid yes 2026-02-12T12:30:16.528 INFO:teuthology.orchestra.run.trial127.stdout:statx(2)/libc yes 2026-02-12T12:30:16.551 INFO:teuthology.orchestra.run.trial127.stdout:statx(2)/syscall yes 2026-02-12T12:30:16.553 INFO:teuthology.orchestra.run.trial127.stdout:Windows PDB generation no 2026-02-12T12:30:16.573 INFO:teuthology.orchestra.run.trial127.stdout:timerfd_create yes 2026-02-12T12:30:16.575 INFO:teuthology.orchestra.run.trial127.stdout:Consider installing zlib1g-dev (zlib-devel) as some fio features depend on it. 2026-02-12T12:30:16.575 INFO:teuthology.orchestra.run.trial127.stdout:Lib-based ioengines dynamic no 2026-02-12T12:30:16.616 INFO:teuthology.orchestra.run.trial127.stdout:TCMalloc support yes 2026-02-12T12:30:16.617 INFO:teuthology.orchestra.run.trial127.stdout:seed_buckets 4 2026-02-12T12:30:16.662 INFO:teuthology.orchestra.run.trial127.stderr:FIO_VERSION = fio-3.41-107-g05b2 2026-02-12T12:30:16.686 INFO:teuthology.orchestra.run.trial127.stdout: CC crc/crc16.o 2026-02-12T12:30:16.722 INFO:teuthology.orchestra.run.trial127.stdout: CC crc/crc32.o 2026-02-12T12:30:16.751 INFO:teuthology.orchestra.run.trial127.stdout: CC crc/crc32c-arm64.o 2026-02-12T12:30:16.795 INFO:teuthology.orchestra.run.trial127.stdout: CC crc/crc32c-intel.o 2026-02-12T12:30:16.826 INFO:teuthology.orchestra.run.trial127.stdout: CC crc/crc32c.o 2026-02-12T12:30:16.855 INFO:teuthology.orchestra.run.trial127.stdout: CC crc/crc64.o 2026-02-12T12:30:16.883 INFO:teuthology.orchestra.run.trial127.stdout: CC crc/crc7.o 2026-02-12T12:30:16.909 INFO:teuthology.orchestra.run.trial127.stdout: CC crc/crct10dif_common.o 2026-02-12T12:30:16.937 INFO:teuthology.orchestra.run.trial127.stdout: CC crc/fnv.o 2026-02-12T12:30:16.967 INFO:teuthology.orchestra.run.trial127.stdout: CC crc/md5.o 2026-02-12T12:30:17.026 INFO:teuthology.orchestra.run.trial127.stdout: CC crc/murmur3.o 2026-02-12T12:30:17.056 INFO:teuthology.orchestra.run.trial127.stdout: CC crc/sha1.o 2026-02-12T12:30:17.155 INFO:teuthology.orchestra.run.trial127.stdout: CC crc/sha256.o 2026-02-12T12:30:17.298 INFO:teuthology.orchestra.run.trial127.stdout: CC crc/sha3.o 2026-02-12T12:30:17.381 INFO:teuthology.orchestra.run.trial127.stdout: CC crc/sha512.o 2026-02-12T12:30:17.448 INFO:teuthology.orchestra.run.trial127.stdout: CC crc/test.o 2026-02-12T12:30:17.543 INFO:teuthology.orchestra.run.trial127.stdout: CC crc/xxhash.o 2026-02-12T12:30:17.602 INFO:teuthology.orchestra.run.trial127.stdout: CC lib/axmap.o 2026-02-12T12:30:17.675 INFO:teuthology.orchestra.run.trial127.stdout: CC lib/bloom.o 2026-02-12T12:30:17.727 INFO:teuthology.orchestra.run.trial127.stdout: CC lib/flist_sort.o 2026-02-12T12:30:17.774 INFO:teuthology.orchestra.run.trial127.stdout: CC lib/gauss.o 2026-02-12T12:30:17.828 INFO:teuthology.orchestra.run.trial127.stdout: CC lib/getrusage.o 2026-02-12T12:30:17.855 INFO:teuthology.orchestra.run.trial127.stdout: CC lib/hweight.o 2026-02-12T12:30:17.886 INFO:teuthology.orchestra.run.trial127.stdout: CC lib/ieee754.o 2026-02-12T12:30:17.920 INFO:teuthology.orchestra.run.trial127.stdout: CC lib/lfsr.o 2026-02-12T12:30:17.979 INFO:teuthology.orchestra.run.trial127.stdout: CC lib/memalign.o 2026-02-12T12:30:18.011 INFO:teuthology.orchestra.run.trial127.stdout: CC lib/memcpy.o 2026-02-12T12:30:18.097 INFO:teuthology.orchestra.run.trial127.stdout: CC lib/mountcheck.o 2026-02-12T12:30:18.128 INFO:teuthology.orchestra.run.trial127.stdout: CC lib/num2str.o 2026-02-12T12:30:18.173 INFO:teuthology.orchestra.run.trial127.stdout: CC lib/output_buffer.o 2026-02-12T12:30:18.208 INFO:teuthology.orchestra.run.trial127.stdout: CC lib/pattern.o 2026-02-12T12:30:18.291 INFO:teuthology.orchestra.run.trial127.stdout: CC lib/prio_tree.o 2026-02-12T12:30:18.361 INFO:teuthology.orchestra.run.trial127.stdout: CC lib/rand.o 2026-02-12T12:30:18.435 INFO:teuthology.orchestra.run.trial127.stdout: CC lib/rbtree.o 2026-02-12T12:30:18.505 INFO:teuthology.orchestra.run.trial127.stdout: CC lib/strntol.o 2026-02-12T12:30:18.538 INFO:teuthology.orchestra.run.trial127.stdout: CC lib/zipf.o 2026-02-12T12:30:18.604 INFO:teuthology.orchestra.run.trial127.stdout: CC gettime.o 2026-02-12T12:30:18.755 INFO:teuthology.orchestra.run.trial127.stdout: CC ioengines.o 2026-02-12T12:30:18.894 INFO:teuthology.orchestra.run.trial127.stdout: CC init.o 2026-02-12T12:30:19.271 INFO:teuthology.orchestra.run.trial127.stdout: CC stat.o 2026-02-12T12:30:19.981 INFO:teuthology.orchestra.run.trial127.stdout: CC log.o 2026-02-12T12:30:20.053 INFO:teuthology.orchestra.run.trial127.stdout: CC time.o 2026-02-12T12:30:20.141 INFO:teuthology.orchestra.run.trial127.stdout: CC filesetup.o 2026-02-12T12:30:20.481 INFO:teuthology.orchestra.run.trial127.stdout: CC eta.o 2026-02-12T12:30:20.644 INFO:teuthology.orchestra.run.trial127.stdout: CC verify.o 2026-02-12T12:30:20.987 INFO:teuthology.orchestra.run.trial127.stdout: CC memory.o 2026-02-12T12:30:21.085 INFO:teuthology.orchestra.run.trial127.stdout: CC io_u.o 2026-02-12T12:30:21.468 INFO:teuthology.orchestra.run.trial127.stdout: CC parse.o 2026-02-12T12:30:21.689 INFO:teuthology.orchestra.run.trial127.stdout: CC fio_sem.o 2026-02-12T12:30:21.752 INFO:teuthology.orchestra.run.trial127.stdout: CC rwlock.o 2026-02-12T12:30:21.805 INFO:teuthology.orchestra.run.trial127.stdout: CC pshared.o 2026-02-12T12:30:21.849 INFO:teuthology.orchestra.run.trial127.stdout: CC options.o 2026-02-12T12:30:22.251 INFO:teuthology.orchestra.run.trial127.stdout: CC fio_shared_sem.o 2026-02-12T12:30:22.285 INFO:teuthology.orchestra.run.trial127.stdout: CC smalloc.o 2026-02-12T12:30:22.389 INFO:teuthology.orchestra.run.trial127.stdout: CC filehash.o 2026-02-12T12:30:22.475 INFO:teuthology.orchestra.run.trial127.stdout: CC profile.o 2026-02-12T12:30:22.545 INFO:teuthology.orchestra.run.trial127.stdout: CC debug.o 2026-02-12T12:30:22.575 INFO:teuthology.orchestra.run.trial127.stdout: CC engines/cpu.o 2026-02-12T12:30:22.657 INFO:teuthology.orchestra.run.trial127.stdout: CC engines/mmap.o 2026-02-12T12:30:22.748 INFO:teuthology.orchestra.run.trial127.stdout: CC engines/sync.o 2026-02-12T12:30:22.872 INFO:teuthology.orchestra.run.trial127.stdout: CC engines/null.o 2026-02-12T12:30:22.942 INFO:teuthology.orchestra.run.trial127.stdout: CC engines/net.o 2026-02-12T12:30:23.158 INFO:teuthology.orchestra.run.trial127.stdout: CC engines/ftruncate.o 2026-02-12T12:30:23.220 INFO:teuthology.orchestra.run.trial127.stdout: CC engines/fileoperations.o 2026-02-12T12:30:23.308 INFO:teuthology.orchestra.run.trial127.stdout: CC engines/exec.o 2026-02-12T12:30:23.392 INFO:teuthology.orchestra.run.trial127.stdout: CC server.o 2026-02-12T12:30:23.751 INFO:teuthology.orchestra.run.trial127.stdout: CC client.o 2026-02-12T12:30:24.034 INFO:teuthology.orchestra.run.trial127.stdout: CC iolog.o 2026-02-12T12:30:24.265 INFO:teuthology.orchestra.run.trial127.stdout: CC backend.o 2026-02-12T12:30:24.678 INFO:teuthology.orchestra.run.trial127.stdout: CC libfio.o 2026-02-12T12:30:24.776 INFO:teuthology.orchestra.run.trial127.stdout: CC flow.o 2026-02-12T12:30:24.844 INFO:teuthology.orchestra.run.trial127.stdout: CC cconv.o 2026-02-12T12:30:25.070 INFO:teuthology.orchestra.run.trial127.stdout: CC gettime-thread.o 2026-02-12T12:30:25.136 INFO:teuthology.orchestra.run.trial127.stdout: CC helpers.o 2026-02-12T12:30:25.179 INFO:teuthology.orchestra.run.trial127.stdout: CC json.o 2026-02-12T12:30:25.280 INFO:teuthology.orchestra.run.trial127.stdout: CC idletime.o 2026-02-12T12:30:25.398 INFO:teuthology.orchestra.run.trial127.stdout: CC td_error.o 2026-02-12T12:30:25.459 INFO:teuthology.orchestra.run.trial127.stdout: CC profiles/tiobench.o 2026-02-12T12:30:25.520 INFO:teuthology.orchestra.run.trial127.stdout: CC profiles/act.o 2026-02-12T12:30:25.612 INFO:teuthology.orchestra.run.trial127.stdout: CC io_u_queue.o 2026-02-12T12:30:25.648 INFO:teuthology.orchestra.run.trial127.stdout: CC filelock.o 2026-02-12T12:30:25.722 INFO:teuthology.orchestra.run.trial127.stdout: CC workqueue.o 2026-02-12T12:30:25.821 INFO:teuthology.orchestra.run.trial127.stdout: CC rate-submit.o 2026-02-12T12:30:25.909 INFO:teuthology.orchestra.run.trial127.stdout: CC optgroup.o 2026-02-12T12:30:25.943 INFO:teuthology.orchestra.run.trial127.stdout: CC helper_thread.o 2026-02-12T12:30:26.048 INFO:teuthology.orchestra.run.trial127.stdout: CC steadystate.o 2026-02-12T12:30:26.195 INFO:teuthology.orchestra.run.trial127.stdout: CC zone-dist.o 2026-02-12T12:30:26.276 INFO:teuthology.orchestra.run.trial127.stdout: CC zbd.o 2026-02-12T12:30:26.646 INFO:teuthology.orchestra.run.trial127.stdout: CC dedupe.o 2026-02-12T12:30:26.728 INFO:teuthology.orchestra.run.trial127.stdout: CC dataplacement.o 2026-02-12T12:30:26.823 INFO:teuthology.orchestra.run.trial127.stdout: CC sprandom.o 2026-02-12T12:30:26.984 INFO:teuthology.orchestra.run.trial127.stdout: CC engines/posixaio.o 2026-02-12T12:30:27.064 INFO:teuthology.orchestra.run.trial127.stdout: CC engines/falloc.o 2026-02-12T12:30:27.133 INFO:teuthology.orchestra.run.trial127.stdout: CC engines/e4defrag.o 2026-02-12T12:30:27.212 INFO:teuthology.orchestra.run.trial127.stdout: CC engines/splice.o 2026-02-12T12:30:27.317 INFO:teuthology.orchestra.run.trial127.stdout: CC oslib/asprintf.o 2026-02-12T12:30:27.344 INFO:teuthology.orchestra.run.trial127.stdout: CC oslib/strlcat.o 2026-02-12T12:30:27.375 INFO:teuthology.orchestra.run.trial127.stdout: CC engines/mtd.o 2026-02-12T12:30:27.465 INFO:teuthology.orchestra.run.trial127.stdout: CC oslib/libmtd.o 2026-02-12T12:30:27.658 INFO:teuthology.orchestra.run.trial127.stdout: CC oslib/libmtd_legacy.o 2026-02-12T12:30:27.734 INFO:teuthology.orchestra.run.trial127.stdout: CC diskutil.o 2026-02-12T12:30:27.848 INFO:teuthology.orchestra.run.trial127.stdout: CC fifo.o 2026-02-12T12:30:27.886 INFO:teuthology.orchestra.run.trial127.stdout: CC blktrace.o 2026-02-12T12:30:28.056 INFO:teuthology.orchestra.run.trial127.stdout: CC cgroup.o 2026-02-12T12:30:28.142 INFO:teuthology.orchestra.run.trial127.stdout: CC trim.o 2026-02-12T12:30:28.211 INFO:teuthology.orchestra.run.trial127.stdout: CC engines/sg.o 2026-02-12T12:30:28.410 INFO:teuthology.orchestra.run.trial127.stdout: CC oslib/linux-dev-lookup.o 2026-02-12T12:30:28.449 INFO:teuthology.orchestra.run.trial127.stdout: CC engines/io_uring.o 2026-02-12T12:30:28.684 INFO:teuthology.orchestra.run.trial127.stdout: CC engines/nvme.o 2026-02-12T12:30:28.848 INFO:teuthology.orchestra.run.trial127.stdout: CC oslib/linux-blkzoned.o 2026-02-12T12:30:28.939 INFO:teuthology.orchestra.run.trial127.stdout: CC engines/cmdprio.o 2026-02-12T12:30:29.066 INFO:teuthology.orchestra.run.trial127.stdout: CC engines/rados.o 2026-02-12T12:30:29.168 INFO:teuthology.orchestra.run.trial127.stdout: CC engines/rbd.o 2026-02-12T12:30:29.276 INFO:teuthology.orchestra.run.trial127.stdout: CC fio.o 2026-02-12T12:30:29.332 INFO:teuthology.orchestra.run.trial127.stdout: LINK fio 2026-02-12T12:30:29.454 INFO:teuthology.orchestra.run.trial127.stdout: CC t/log.o 2026-02-12T12:30:29.485 INFO:teuthology.orchestra.run.trial127.stdout: CC t/genzipf.o 2026-02-12T12:30:29.563 INFO:teuthology.orchestra.run.trial127.stdout: CC oslib/strcasestr.o 2026-02-12T12:30:29.586 INFO:teuthology.orchestra.run.trial127.stdout: CC oslib/strndup.o 2026-02-12T12:30:29.605 INFO:teuthology.orchestra.run.trial127.stdout: LINK t/fio-genzipf 2026-02-12T12:30:29.660 INFO:teuthology.orchestra.run.trial127.stdout: CC t/btrace2fio.o 2026-02-12T12:30:29.858 INFO:teuthology.orchestra.run.trial127.stdout: LINK t/fio-btrace2fio 2026-02-12T12:30:29.915 INFO:teuthology.orchestra.run.trial127.stdout: CC t/verify-state.o 2026-02-12T12:30:29.975 INFO:teuthology.orchestra.run.trial127.stdout: CC t/debug.o 2026-02-12T12:30:30.001 INFO:teuthology.orchestra.run.trial127.stdout: LINK t/fio-verify-state 2026-02-12T12:30:30.056 INFO:teuthology.orchestra.run.trial127.stdout: CC t/stest.o 2026-02-12T12:30:30.107 INFO:teuthology.orchestra.run.trial127.stdout: CC t/arch.o 2026-02-12T12:30:30.131 INFO:teuthology.orchestra.run.trial127.stdout: LINK t/stest 2026-02-12T12:30:30.187 INFO:teuthology.orchestra.run.trial127.stdout: CC t/ieee754.o 2026-02-12T12:30:30.218 INFO:teuthology.orchestra.run.trial127.stdout: LINK t/ieee754 2026-02-12T12:30:30.273 INFO:teuthology.orchestra.run.trial127.stdout: CC t/axmap.o 2026-02-12T12:30:30.333 INFO:teuthology.orchestra.run.trial127.stdout: LINK t/axmap 2026-02-12T12:30:30.388 INFO:teuthology.orchestra.run.trial127.stdout: CC t/lfsr-test.o 2026-02-12T12:30:30.444 INFO:teuthology.orchestra.run.trial127.stdout: LINK t/lfsr-test 2026-02-12T12:30:30.501 INFO:teuthology.orchestra.run.trial127.stdout: CC t/gen-rand.o 2026-02-12T12:30:30.557 INFO:teuthology.orchestra.run.trial127.stdout: LINK t/gen-rand 2026-02-12T12:30:30.611 INFO:teuthology.orchestra.run.trial127.stdout: CC t/memlock.o 2026-02-12T12:30:30.651 INFO:teuthology.orchestra.run.trial127.stdout: LINK t/memlock 2026-02-12T12:30:30.704 INFO:teuthology.orchestra.run.trial127.stdout: CC t/read-to-pipe-async.o 2026-02-12T12:30:30.807 INFO:teuthology.orchestra.run.trial127.stdout: LINK t/read-to-pipe-async 2026-02-12T12:30:30.860 INFO:teuthology.orchestra.run.trial127.stdout: CC t/io_uring.o 2026-02-12T12:30:31.137 INFO:teuthology.orchestra.run.trial127.stdout: LINK t/io_uring 2026-02-12T12:30:31.195 DEBUG:teuthology.orchestra.run.trial127:> sudo /sbin/sysctl -q -w kernel.perf_event_paranoid=0 2026-02-12T12:30:31.221 DEBUG:teuthology.orchestra.run.trial127:> /home/ubuntu/cephtest/cbt/cbt.py -a /home/ubuntu/cephtest/archive/cbt /home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2026-02-12T12:30:31.334 INFO:teuthology.orchestra.run.trial127.stderr:12:30:31 - DEBUG - cbt - Settings.cluster: 2026-02-12T12:30:31.334 INFO:teuthology.orchestra.run.trial127.stderr: {'archive_dir': '/home/ubuntu/cephtest/archive/cbt', 2026-02-12T12:30:31.334 INFO:teuthology.orchestra.run.trial127.stderr: 'clients': ['ubuntu@trial127.front.sepia.ceph.com'], 2026-02-12T12:30:31.334 INFO:teuthology.orchestra.run.trial127.stderr: 'conf_file': '/etc/ceph/ceph.conf', 2026-02-12T12:30:31.334 INFO:teuthology.orchestra.run.trial127.stderr: 'head': 'ubuntu@trial127.front.sepia.ceph.com', 2026-02-12T12:30:31.334 INFO:teuthology.orchestra.run.trial127.stderr: 'is_teuthology': True, 2026-02-12T12:30:31.335 INFO:teuthology.orchestra.run.trial127.stderr: 'iterations': 1, 2026-02-12T12:30:31.335 INFO:teuthology.orchestra.run.trial127.stderr: 'mons': ['ubuntu@trial127.front.sepia.ceph.com'], 2026-02-12T12:30:31.335 INFO:teuthology.orchestra.run.trial127.stderr: 'osds': ['ubuntu@trial127.front.sepia.ceph.com'], 2026-02-12T12:30:31.335 INFO:teuthology.orchestra.run.trial127.stderr: 'osds_per_node': 3, 2026-02-12T12:30:31.335 INFO:teuthology.orchestra.run.trial127.stderr: 'pid_dir': '/var/run/ceph', 2026-02-12T12:30:31.335 INFO:teuthology.orchestra.run.trial127.stderr: 'pool_profiles': {'rbd': {'pg_size': 128, 'pgp_size': 128, 'replication': 3}}, 2026-02-12T12:30:31.335 INFO:teuthology.orchestra.run.trial127.stderr: 'rebuild_every_test': False, 2026-02-12T12:30:31.335 INFO:teuthology.orchestra.run.trial127.stderr: 'rgws': {'ubuntu@trial127.front.sepia.ceph.com': None}, 2026-02-12T12:30:31.335 INFO:teuthology.orchestra.run.trial127.stderr: 'tmp_dir': '/tmp/cbt', 2026-02-12T12:30:31.335 INFO:teuthology.orchestra.run.trial127.stderr: 'use_existing': True, 2026-02-12T12:30:31.335 INFO:teuthology.orchestra.run.trial127.stderr: 'user': 'ubuntu'} 2026-02-12T12:30:31.335 INFO:teuthology.orchestra.run.trial127.stderr:12:30:31 - INFO - cbt - Results dir: /home/ubuntu/cephtest/archive/cbt/results/00000000/id-5928d42f 2026-02-12T12:30:31.336 INFO:teuthology.orchestra.run.trial127.stderr:12:30:31 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com if [ -d "/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw" ]; then rm -rf /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw; fi 2026-02-12T12:30:31.504 INFO:teuthology.orchestra.run.trial127.stderr:12:30:31 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw 2026-02-12T12:30:31.667 INFO:teuthology.orchestra.run.trial127.stderr:12:30:31 - INFO - cbt - Pausing for 60s for idle monitoring. 2026-02-12T12:30:31.668 INFO:teuthology.orchestra.run.trial127.stderr:12:30:31 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/idle_monitoring/collectl 2026-02-12T12:30:31.836 INFO:teuthology.orchestra.run.trial127.stderr:12:30:31 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/idle_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2026-02-12T12:30:31.838 INFO:teuthology.orchestra.run.trial127.stderr:12:30:31 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/idle_monitoring/perf 2026-02-12T12:30:32.007 INFO:teuthology.orchestra.run.trial127.stderr:12:30:32 - DEBUG - cbt - PerfMonitoring: remote_node 2026-02-12T12:30:32.007 INFO:teuthology.orchestra.run.trial127.stderr:12:30:32 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com for pid in `cat /var/run/ceph/crimson-osd.*.pid`; do perf stat -p ${pid} -o /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/idle_monitoring/perf/perf_stat.${pid} & done 2026-02-12T12:31:32.030 INFO:teuthology.orchestra.run.trial127.stderr:12:31:32 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com pkill -SIGINT -f collectl 2026-02-12T12:31:32.210 INFO:teuthology.orchestra.run.trial127.stderr:12:31:32 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com sudo pkill -SIGINT -f perf\ 2026-02-12T12:31:32.394 INFO:teuthology.orchestra.run.trial127.stderr:12:31:32 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/* 2026-02-12T12:31:32.566 INFO:teuthology.orchestra.run.trial127.stderr:12:31:32 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=rpdcp -f 10 -R ssh -w ubuntu@trial127.front.sepia.ceph.com -r /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/* /home/ubuntu/cephtest/archive/cbt/results/00000000/id-5928d42f 2026-02-12T12:31:32.728 INFO:teuthology.orchestra.run.trial127.stderr:12:31:32 - INFO - cbt - Creating fio images... 2026-02-12T12:31:32.729 INFO:teuthology.orchestra.run.trial127.stderr:12:31:32 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/pool_monitoring/collectl 2026-02-12T12:31:32.894 INFO:teuthology.orchestra.run.trial127.stderr:12:31:32 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/pool_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2026-02-12T12:31:32.897 INFO:teuthology.orchestra.run.trial127.stderr:12:31:32 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/pool_monitoring/perf 2026-02-12T12:31:33.060 INFO:teuthology.orchestra.run.trial127.stderr:12:31:33 - DEBUG - cbt - PerfMonitoring: remote_node 2026-02-12T12:31:33.060 INFO:teuthology.orchestra.run.trial127.stderr:12:31:33 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com for pid in `cat /var/run/ceph/crimson-osd.*.pid`; do perf stat -p ${pid} -o /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/pool_monitoring/perf/perf_stat.${pid} & done 2026-02-12T12:31:33.061 INFO:teuthology.orchestra.run.trial127.stderr:12:31:33 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool delete cbt-librbdfio cbt-librbdfio --yes-i-really-really-mean-it 2026-02-12T12:31:33.413 INFO:teuthology.orchestra.run.trial127.stderr:12:31:33 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool create cbt-librbdfio 128 128 2026-02-12T12:31:34.611 INFO:teuthology.orchestra.run.trial127.stderr:12:31:34 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool application enable cbt-librbdfio rbd 2026-02-12T12:31:35.613 INFO:teuthology.orchestra.run.trial127.stderr:12:31:35 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool set cbt-librbdfio size 3 2026-02-12T12:31:36.620 INFO:teuthology.orchestra.run.trial127.stderr:12:31:36 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool set cbt-librbdfio min_size 2 2026-02-12T12:31:37.620 INFO:teuthology.orchestra.run.trial127.stderr:12:31:37 - INFO - cbt - Checking Health after pool creation. 2026-02-12T12:31:42.626 INFO:teuthology.orchestra.run.trial127.stderr:12:31:42 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com /usr/bin/ceph -c /etc/ceph/ceph.conf health 2026-02-12T12:31:42.988 INFO:teuthology.orchestra.run.trial127.stderr:12:31:42 - INFO - cbt - Final Pool Health Check. 2026-02-12T12:31:47.993 INFO:teuthology.orchestra.run.trial127.stderr:12:31:47 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com /usr/bin/ceph -c /etc/ceph/ceph.conf health 2026-02-12T12:31:48.358 INFO:teuthology.orchestra.run.trial127.stderr:12:31:48 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com hostname -f 2026-02-12T12:31:48.516 INFO:teuthology.orchestra.run.trial127.stderr:12:31:48 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com /usr/bin/rbd -c /etc/ceph/ceph.conf create cbt-librbdfio-trial127-0 --size 4096 --pool cbt-librbdfio --order 22 2026-02-12T12:31:50.746 INFO:teuthology.orchestra.run.trial127.stderr:12:31:50 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com /usr/bin/rbd -c /etc/ceph/ceph.conf create cbt-librbdfio-trial127-1 --size 4096 --pool cbt-librbdfio --order 22 2026-02-12T12:31:50.992 INFO:teuthology.orchestra.run.trial127.stderr:12:31:50 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com pkill -SIGINT -f collectl 2026-02-12T12:31:51.159 INFO:teuthology.orchestra.run.trial127.stderr:12:31:51 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com sudo pkill -SIGINT -f perf\ 2026-02-12T12:31:51.333 INFO:teuthology.orchestra.run.trial127.stderr:12:31:51 - INFO - cbt - Attempting to prefill fio images... 2026-02-12T12:31:51.333 INFO:teuthology.orchestra.run.trial127.stderr:12:31:51 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-0 --invalidate=0 --rw=write --numjobs=1 --bs=4M --size 4096M --name=cbt-librbdfio-`hostname -f`-file-0 --output-format=json,normal > /dev/null 2026-02-12T12:31:51.335 INFO:teuthology.orchestra.run.trial127.stderr:12:31:51 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-1 --invalidate=0 --rw=write --numjobs=1 --bs=4M --size 4096M --name=cbt-librbdfio-`hostname -f`-file-0 --output-format=json,normal > /dev/null 2026-02-12T12:32:33.893 INFO:teuthology.orchestra.run.trial127.stderr:12:32:33 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-0 --invalidate=0 --rw=write --numjobs=1 --bs=4M --size 4096M --name=cbt-librbdfio-`hostname -f`-file-0 --output-format=json,normal > /dev/null 2026-02-12T12:32:33.895 INFO:teuthology.orchestra.run.trial127.stderr:12:32:33 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-1 --invalidate=0 --rw=write --numjobs=1 --bs=4M --size 4096M --name=cbt-librbdfio-`hostname -f`-file-0 --output-format=json,normal > /dev/null 2026-02-12T12:32:56.972 INFO:teuthology.orchestra.run.trial127.stderr:12:32:56 - INFO - cbt - Results dir: /home/ubuntu/cephtest/archive/cbt/results/00000000/id-5928d42f 2026-02-12T12:32:56.973 INFO:teuthology.orchestra.run.trial127.stderr:12:32:56 - INFO - cbt - Skipping existing test in /home/ubuntu/cephtest/archive/cbt/results/00000000/id-5928d42f. 2026-02-12T12:32:56.973 INFO:teuthology.orchestra.run.trial127.stderr:12:32:56 - INFO - cbt - Running benchmark /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw 2026-02-12T12:32:56.973 INFO:teuthology.orchestra.run.trial127.stderr:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-5928d42f 2026-02-12T12:32:56.974 INFO:teuthology.orchestra.run.trial127.stderr:{'cmd_path': '/home/ubuntu/cephtest/fio/fio', 'log_avg_msec': 100, 'norandommap': True, 'pool_profile': 'rbd', 'time': 60, 'vol_size': 4096, 'iodepth': 32, 'mode': 'randrw', 'op_size': 4194304, 'osd_ra': 4096, 'procs_per_volume': 1, 'volumes_per_client': 2, 'benchmark': 'librbdfio', 'iteration': 0} == iteration 0 == 2026-02-12T12:32:56.974 INFO:teuthology.orchestra.run.trial127.stderr:12:32:56 - INFO - cbt - Setting OSD Read Ahead to: 4096 2026-02-12T12:32:56.974 INFO:teuthology.orchestra.run.trial127.stderr:12:32:56 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com find /dev/disk/by-partlabel/osd-device-*data -exec readlink {} \; | cut -d"/" -f 3 | sed "s/[0-9]$//" | xargs -I{} sudo sh -c "echo 4096 > /sys/block/'{}'/queue/read_ahead_kb" 2026-02-12T12:32:56.974 INFO:teuthology.orchestra.run.trial127.stderr:12:32:56 - DEBUG - cbt - Cleaning existing temporary run directory: /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw 2026-02-12T12:32:56.975 INFO:teuthology.orchestra.run.trial127.stderr:12:32:56 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com sudo rm -rf /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw 2026-02-12T12:32:57.152 INFO:teuthology.orchestra.run.trial127.stderr:12:32:57 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com sync 2026-02-12T12:32:57.429 INFO:teuthology.orchestra.run.trial127.stderr:12:32:57 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com echo 3 | sudo tee /proc/sys/vm/drop_caches 2026-02-12T12:32:58.581 INFO:teuthology.orchestra.run.trial127.stderr:12:32:58 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw 2026-02-12T12:32:58.777 INFO:teuthology.orchestra.run.trial127.stderr:12:32:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf daemon osd.0 config show > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/ceph_settings.out 2026-02-12T12:33:24.071 INFO:teuthology.orchestra.run.trial127.stderr:12:33:24 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com echo "[`date`] ceph progress" >> /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/pgautoscaler.log;/usr/bin/ceph -c /etc/ceph/ceph.conf progress | tee -a /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/pgautoscaler.log 2026-02-12T12:33:24.439 INFO:teuthology.orchestra.run.trial127.stderr:12:33:24 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/collectl 2026-02-12T12:33:24.603 INFO:teuthology.orchestra.run.trial127.stderr:12:33:24 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2026-02-12T12:33:24.605 INFO:teuthology.orchestra.run.trial127.stderr:12:33:24 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/perf 2026-02-12T12:33:24.766 INFO:teuthology.orchestra.run.trial127.stderr:12:33:24 - DEBUG - cbt - PerfMonitoring: remote_node 2026-02-12T12:33:24.766 INFO:teuthology.orchestra.run.trial127.stderr:12:33:24 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com for pid in `cat /var/run/ceph/crimson-osd.*.pid`; do perf stat -p ${pid} -o /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/perf/perf_stat.${pid} & done 2026-02-12T12:33:24.768 INFO:teuthology.orchestra.run.trial127.stderr:12:33:24 - INFO - cbt - Running rbd fio randrw test. 2026-02-12T12:33:24.768 INFO:teuthology.orchestra.run.trial127.stderr:12:33:24 - DEBUG - cbt - Using rbdname cbt-librbdfio-`hostname -f`-0 2026-02-12T12:33:24.768 INFO:teuthology.orchestra.run.trial127.stderr:12:33:24 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-0 --invalidate=0 --rw=randrw --output-format=json,normal --rwmixread=50 --rwmixwrite=50 --runtime=60 --numjobs=1 --direct=1 --bs=4194304B --iodepth=32 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/output.0 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/output.0 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/output.0 --log_avg_msec=100 --name=cbt-librbdfio-`hostname -f`-file-0 > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/output.0 2026-02-12T12:33:24.769 INFO:teuthology.orchestra.run.trial127.stderr:12:33:24 - DEBUG - cbt - Using rbdname cbt-librbdfio-`hostname -f`-1 2026-02-12T12:33:24.770 INFO:teuthology.orchestra.run.trial127.stderr:12:33:24 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-1 --invalidate=0 --rw=randrw --output-format=json,normal --rwmixread=50 --rwmixwrite=50 --runtime=60 --numjobs=1 --direct=1 --bs=4194304B --iodepth=32 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/output.1 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/output.1 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/output.1 --log_avg_msec=100 --name=cbt-librbdfio-`hostname -f`-file-0 > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/output.1 2026-02-12T12:33:39.025 INFO:teuthology.orchestra.run.trial127.stderr:12:33:39 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com pkill -SIGINT -f collectl 2026-02-12T12:33:39.202 INFO:teuthology.orchestra.run.trial127.stderr:12:33:39 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com sudo pkill -SIGINT -f perf\ 2026-02-12T12:33:39.376 INFO:teuthology.orchestra.run.trial127.stderr:12:33:39 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/perf/perf.data 2026-02-12T12:33:39.378 INFO:teuthology.orchestra.run.trial127.stderr:12:33:39 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/perf/perf_stat.* 2026-02-12T12:33:39.379 INFO:teuthology.orchestra.run.trial127.stderr:12:33:39 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com find /var/run/ceph/ceph-osd*.asok -maxdepth 1 -exec sudo /usr/bin/ceph --admin-daemon {} dump_historic_ops \; > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/historic_ops.out 2026-02-12T12:33:39.742 INFO:teuthology.orchestra.run.trial127.stderr:12:33:39 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial127.front.sepia.ceph.com sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/* 2026-02-12T12:33:39.921 INFO:teuthology.orchestra.run.trial127.stderr:12:33:39 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=rpdcp -f 10 -R ssh -w ubuntu@trial127.front.sepia.ceph.com -r /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/* /home/ubuntu/cephtest/archive/cbt/results/00000000/id-5928d42f 2026-02-12T12:33:40.089 INFO:teuthology.orchestra.run.trial127.stderr:12:33:40 - INFO - cbt - Convert results to json format. 2026-02-12T12:33:40.090 INFO:teuthology.orchestra.run.trial127.stdout:cleaning remote dir /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw 2026-02-12T12:33:40.090 INFO:teuthology.orchestra.run.trial127.stdout:trial127: trial127 2026-02-12T12:33:40.090 INFO:teuthology.orchestra.run.trial127.stdout: 2026-02-12T12:33:40.090 INFO:teuthology.orchestra.run.trial127.stdout:['trial127'] 2026-02-12T12:33:40.102 DEBUG:teuthology.run_tasks:Unwinding manager cbt 2026-02-12T12:33:40.116 DEBUG:teuthology.orchestra.run.trial127:> rm --one-file-system -rf -- /home/ubuntu/cephtest/cbt 2026-02-12T12:33:40.142 DEBUG:teuthology.orchestra.run.trial127:> rm --one-file-system -rf -- /home/ubuntu/cephtest/fio 2026-02-12T12:33:40.221 INFO:tasks.cbt_performance:Collecting CBT performance data config 2026-02-12T12:33:40.222 INFO:tasks.cbt_performance:reading cbt preferences 2026-02-12T12:33:40.222 DEBUG:teuthology.orchestra.run.trial127:> find /home/ubuntu/cephtest -name 'json_output.*' 2026-02-12T12:33:40.281 INFO:teuthology.orchestra.run.trial127.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-5928d42f/json_output.1.trial127.front.sepia.ceph.com 2026-02-12T12:33:40.282 INFO:teuthology.orchestra.run.trial127.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-5928d42f/json_output.0.trial127.front.sepia.ceph.com 2026-02-12T12:33:40.283 DEBUG:teuthology.orchestra.run.trial127:> set -ex 2026-02-12T12:33:40.283 DEBUG:teuthology.orchestra.run.trial127:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-5928d42f/json_output.1.trial127.front.sepia.ceph.com of=/dev/stdout 2026-02-12T12:33:40.341 INFO:tasks.cbt_performance:reading total cpu cycles 2026-02-12T12:33:40.342 DEBUG:teuthology.orchestra.run.trial127:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-5928d42f -name 'perf_stat.*' 2026-02-12T12:33:40.398 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2026-02-12T12:33:40.398 INFO:tasks.cbt_performance:total cpu cycles: 0 2026-02-12T12:33:40.398 DEBUG:teuthology.orchestra.run.trial127:> set -ex 2026-02-12T12:33:40.398 DEBUG:teuthology.orchestra.run.trial127:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-5928d42f/json_output.0.trial127.front.sepia.ceph.com of=/dev/stdout 2026-02-12T12:33:40.457 INFO:tasks.cbt_performance:reading total cpu cycles 2026-02-12T12:33:40.457 DEBUG:teuthology.orchestra.run.trial127:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-5928d42f -name 'perf_stat.*' 2026-02-12T12:33:40.512 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2026-02-12T12:33:40.513 INFO:tasks.cbt_performance:total cpu cycles: 0 2026-02-12T12:33:40.518 WARNING:tasks.cbt_performance:Could not send performance data to http://mira118.front.sepia.ceph.com:4000/cbt_performance: HTTPConnectionPool(host='mira118.front.sepia.ceph.com', port=4000): Max retries exceeded with url: /cbt_performance (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -2] Name or service not known')) 2026-02-12T12:33:40.518 WARNING:tasks.cbt_performance:Continuing without performance data collection 2026-02-12T12:33:40.519 DEBUG:teuthology.run_tasks:Unwinding manager ssh_keys 2026-02-12T12:33:40.534 INFO:teuthology.task.ssh_keys:Cleaning up SSH keys 2026-02-12T12:33:40.535 INFO:teuthology.task.ssh_keys:cleaning up keys added for testing 2026-02-12T12:33:40.535 INFO:teuthology.task.ssh_keys: cleaning up keys for user ubuntu on trial127.front.sepia.ceph.com 2026-02-12T12:33:40.535 DEBUG:teuthology.orchestra.run.trial127:> rm -- /home/ubuntu/.ssh/id_rsa 2026-02-12T12:33:40.568 DEBUG:teuthology.orchestra.run.trial127:> rm -- /home/ubuntu/.ssh/id_rsa.pub 2026-02-12T12:33:40.624 DEBUG:teuthology.orchestra.run.trial127:> stat -c '"%a"' /home/ubuntu/.ssh/authorized_keys 2026-02-12T12:33:40.682 INFO:teuthology.orchestra.run.trial127.stdout:"600" 2026-02-12T12:33:40.682 DEBUG:teuthology.orchestra.run.trial127:> mv -- /home/ubuntu/.ssh/authorized_keys_2026-02-12_12:29:56:974066 /home/ubuntu/.ssh/authorized_keys 2026-02-12T12:33:40.739 DEBUG:teuthology.orchestra.run.trial127:> chmod 600 /home/ubuntu/.ssh/authorized_keys 2026-02-12T12:33:40.796 DEBUG:teuthology.orchestra.run.trial127:> rm /home/ubuntu/.ssh/config 2026-02-12T12:33:40.852 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2026-02-12T12:33:40.868 DEBUG:teuthology.orchestra.run.trial127:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2026-02-12T12:33:40.967 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-12T12:33:40.961+0000 7fee96f64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:33:40.968 INFO:teuthology.orchestra.run.trial127.stderr:2026-02-12T12:33:40.966+0000 7fee96f64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:33:41.096 INFO:teuthology.misc:Shutting down mds daemons... 2026-02-12T12:33:41.096 INFO:teuthology.misc:Shutting down osd daemons... 2026-02-12T12:33:41.096 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2026-02-12T12:33:41.096 INFO:teuthology.orchestra.run:waiting for 300 2026-02-12T12:33:41.177 INFO:tasks.ceph.osd.0:Stopped 2026-02-12T12:33:41.177 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2026-02-12T12:33:41.177 INFO:teuthology.orchestra.run:waiting for 300 2026-02-12T12:33:41.264 INFO:tasks.ceph.osd.1:Stopped 2026-02-12T12:33:41.265 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2026-02-12T12:33:41.265 INFO:teuthology.orchestra.run:waiting for 300 2026-02-12T12:33:41.348 INFO:tasks.ceph.osd.2:Stopped 2026-02-12T12:33:41.349 INFO:teuthology.misc:Shutting down mgr daemons... 2026-02-12T12:33:41.349 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2026-02-12T12:33:41.349 INFO:teuthology.orchestra.run:waiting for 300 2026-02-12T12:33:41.386 INFO:tasks.ceph.mgr.x:Stopped 2026-02-12T12:33:41.386 INFO:teuthology.misc:Shutting down mon daemons... 2026-02-12T12:33:41.387 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2026-02-12T12:33:41.387 INFO:teuthology.orchestra.run:waiting for 300 2026-02-12T12:33:41.437 INFO:tasks.ceph.mon.a:Stopped 2026-02-12T12:33:41.437 INFO:tasks.ceph:Checking cluster log for badness... 2026-02-12T12:33:41.437 DEBUG:teuthology.orchestra.run.trial127:> sudo grep -E '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | grep -E -v '\(MDS_ALL_DOWN\)' | grep -E -v '\(MDS_UP_LESS_THAN_MAX\)' | grep -E -v '\(POOL_APP_NOT_ENABLED\)' | grep -E -v '\(PG_' | grep -E -v '\(OSD_' | grep -E -v '\(OBJECT_' | grep -E -v 'overall HEALTH' | head -n 1 2026-02-12T12:33:41.503 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@trial127.front.sepia.ceph.com 2026-02-12T12:33:41.503 DEBUG:teuthology.orchestra.run.trial127:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2026-02-12T12:33:41.661 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@trial127.front.sepia.ceph.com 2026-02-12T12:33:41.661 DEBUG:teuthology.orchestra.run.trial127:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2026-02-12T12:33:41.691 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@trial127.front.sepia.ceph.com 2026-02-12T12:33:41.692 DEBUG:teuthology.orchestra.run.trial127:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2026-02-12T12:33:41.762 INFO:tasks.ceph:Archiving mon data... 2026-02-12T12:33:41.764 DEBUG:teuthology.misc:Transferring archived files from trial127:/var/lib/ceph/mon/ceph-a to /home/teuthworker/mnt/teuthology/moagrawa-2026-02-10_06:35:43-crimson-rados-main-distro-debug-trial/42503/data/mon.a.tgz 2026-02-12T12:33:41.764 DEBUG:teuthology.orchestra.run.trial127:> mktemp 2026-02-12T12:33:41.819 INFO:teuthology.orchestra.run.trial127.stdout:/tmp/tmp.6qitkE4QWz 2026-02-12T12:33:41.819 DEBUG:teuthology.orchestra.run.trial127:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.6qitkE4QWz 2026-02-12T12:33:41.909 DEBUG:teuthology.orchestra.run.trial127:> sudo chmod 0666 /tmp/tmp.6qitkE4QWz 2026-02-12T12:33:41.992 DEBUG:teuthology.orchestra.remote:trial127:/tmp/tmp.6qitkE4QWz is 200KB 2026-02-12T12:33:42.044 DEBUG:teuthology.orchestra.run.trial127:> rm -fr /tmp/tmp.6qitkE4QWz 2026-02-12T12:33:42.060 INFO:tasks.ceph:Cleaning ceph cluster... 2026-02-12T12:33:42.061 DEBUG:teuthology.orchestra.run.trial127:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2026-02-12T12:33:42.181 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2026-02-12T12:33:42.181 INFO:tasks.ceph:Archiving crash dumps... 2026-02-12T12:33:42.183 DEBUG:teuthology.misc:Transferring archived files from trial127:/var/lib/ceph/crash to /home/teuthworker/mnt/teuthology/moagrawa-2026-02-10_06:35:43-crimson-rados-main-distro-debug-trial/42503/remote/trial127/crash 2026-02-12T12:33:42.185 DEBUG:teuthology.orchestra.run.trial127:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2026-02-12T12:33:42.245 INFO:tasks.ceph:Compressing logs... 2026-02-12T12:33:42.246 DEBUG:teuthology.orchestra.run.trial127:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2026-02-12T12:33:42.310 INFO:teuthology.orchestra.run.trial127.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2026-02-12T12:33:42.324 INFO:teuthology.orchestra.run.trial127.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28974.log 2026-02-12T12:33:42.324 INFO:teuthology.orchestra.run.trial127.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2026-02-12T12:33:42.325 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-osd.2.log: /var/log/ceph/ceph-client.admin.28974.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22094.log 2026-02-12T12:33:42.325 INFO:teuthology.orchestra.run.trial127.stderr: 66.9% -- replaced with /var/log/ceph/ceph-client.admin.28974.log.gz 2026-02-12T12:33:42.325 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2026-02-12T12:33:42.325 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-client.admin.22094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22391.log 2026-02-12T12:33:42.325 INFO:teuthology.orchestra.run.trial127.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.22094.log.gz 2026-02-12T12:33:42.325 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22914.log 2026-02-12T12:33:42.325 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-client.admin.22391.log: 91.0% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2026-02-12T12:33:42.325 INFO:teuthology.orchestra.run.trial127.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23136.log 2026-02-12T12:33:42.325 INFO:teuthology.orchestra.run.trial127.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.22391.log.gz 2026-02-12T12:33:42.325 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-client.admin.22914.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22677.log 2026-02-12T12:33:42.326 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-client.admin.23136.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.22914.log.gz 2026-02-12T12:33:42.326 INFO:teuthology.orchestra.run.trial127.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29151.log 2026-02-12T12:33:42.326 INFO:teuthology.orchestra.run.trial127.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.23136.log.gz 2026-02-12T12:33:42.326 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-client.admin.22677.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28579.log 2026-02-12T12:33:42.326 INFO:teuthology.orchestra.run.trial127.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.22677.log.gz 2026-02-12T12:33:42.326 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-client.admin.29151.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29152.log 2026-02-12T12:33:42.326 INFO:teuthology.orchestra.run.trial127.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.29151.log.gz 2026-02-12T12:33:42.326 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-client.admin.28579.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23355.log 2026-02-12T12:33:42.326 INFO:teuthology.orchestra.run.trial127.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28579.log.gz 2026-02-12T12:33:42.326 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-client.admin.29152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23306.log 2026-02-12T12:33:42.326 INFO:teuthology.orchestra.run.trial127.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29152.log.gz 2026-02-12T12:33:42.326 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-client.admin.23355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28637.log 2026-02-12T12:33:42.327 INFO:teuthology.orchestra.run.trial127.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.23355.log.gz 2026-02-12T12:33:42.327 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-client.admin.23306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21910.log 2026-02-12T12:33:42.327 INFO:teuthology.orchestra.run.trial127.stderr: 48.0% -- replaced with /var/log/ceph/ceph-client.admin.23306.log.gz 2026-02-12T12:33:42.327 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-client.admin.28637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22360.log 2026-02-12T12:33:42.327 INFO:teuthology.orchestra.run.trial127.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28637.log.gz 2026-02-12T12:33:42.327 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-client.admin.21910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28786.log 2026-02-12T12:33:42.327 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-client.admin.22360.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.21910.log.gz 2026-02-12T12:33:42.327 INFO:teuthology.orchestra.run.trial127.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23137.log 2026-02-12T12:33:42.327 INFO:teuthology.orchestra.run.trial127.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.22360.log.gz 2026-02-12T12:33:42.327 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-client.admin.28786.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28752.log 2026-02-12T12:33:42.327 INFO:teuthology.orchestra.run.trial127.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28786.log.gz 2026-02-12T12:33:42.327 INFO:teuthology.orchestra.run.trial127.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29905.log 2026-02-12T12:33:42.327 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-client.admin.23137.log: /var/log/ceph/ceph-client.admin.28752.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.23137.log.gz 2026-02-12T12:33:42.328 INFO:teuthology.orchestra.run.trial127.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28752.log.gz 2026-02-12T12:33:42.328 INFO:teuthology.orchestra.run.trial127.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23015.log 2026-02-12T12:33:42.328 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-client.admin.29905.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29905.log.gz 2026-02-12T12:33:42.328 INFO:teuthology.orchestra.run.trial127.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28840.log 2026-02-12T12:33:42.328 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-client.admin.23015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29906.log 2026-02-12T12:33:42.328 INFO:teuthology.orchestra.run.trial127.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.23015.log.gz 2026-02-12T12:33:42.328 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-client.admin.28840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23164.log 2026-02-12T12:33:42.328 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-client.admin.29906.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.28840.log.gz 2026-02-12T12:33:42.328 INFO:teuthology.orchestra.run.trial127.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.29906.log.gz 2026-02-12T12:33:42.328 INFO:teuthology.orchestra.run.trial127.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22047.log 2026-02-12T12:33:42.328 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-client.admin.23164.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.20474.log 2026-02-12T12:33:42.328 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-client.admin.22047.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.23164.log.gz 2026-02-12T12:33:42.329 INFO:teuthology.orchestra.run.trial127.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2026-02-12T12:33:42.329 INFO:teuthology.orchestra.run.trial127.stderr: 48.4% -- replaced with /var/log/ceph/ceph-client.admin.22047.log.gz 2026-02-12T12:33:42.329 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph.tmp-client.admin.20474.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23017.log 2026-02-12T12:33:42.329 INFO:teuthology.orchestra.run.trial127.stderr: 67.7% -- replaced with /var/log/ceph/ceph.tmp-client.admin.20474.log.gz 2026-02-12T12:33:42.329 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29324.log 2026-02-12T12:33:42.329 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-client.admin.23017.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22607.log 2026-02-12T12:33:42.329 INFO:teuthology.orchestra.run.trial127.stderr: 86.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2026-02-12T12:33:42.329 INFO:teuthology.orchestra.run.trial127.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.23017.log.gz 2026-02-12T12:33:42.329 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-client.admin.29324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23404.log 2026-02-12T12:33:42.329 INFO:teuthology.orchestra.run.trial127.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.29324.log.gz 2026-02-12T12:33:42.329 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-client.admin.22607.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2026-02-12T12:33:42.329 INFO:teuthology.orchestra.run.trial127.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.22607.log.gz 2026-02-12T12:33:42.329 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-client.admin.23404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22537.log 2026-02-12T12:33:42.330 INFO:teuthology.orchestra.run.trial127.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.23404.log.gz 2026-02-12T12:33:42.330 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22816.log 2026-02-12T12:33:42.330 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-client.admin.22537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28523.log 2026-02-12T12:33:42.330 INFO:teuthology.orchestra.run.trial127.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.22537.log.gz 2026-02-12T12:33:42.330 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-client.admin.22816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29325.log 2026-02-12T12:33:42.330 INFO:teuthology.orchestra.run.trial127.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.22816.log.gz 2026-02-12T12:33:42.330 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-client.admin.28523.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22702.log 2026-02-12T12:33:42.330 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-client.admin.29325.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23016.log 2026-02-12T12:33:42.330 INFO:teuthology.orchestra.run.trial127.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.28523.log.gz 2026-02-12T12:33:42.330 INFO:teuthology.orchestra.run.trial127.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29325.log.gz 2026-02-12T12:33:42.330 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-client.admin.22702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22865.log 2026-02-12T12:33:42.330 INFO:teuthology.orchestra.run.trial127.stderr: 66.9% -- replaced with /var/log/ceph/ceph-client.admin.22702.log.gz 2026-02-12T12:33:42.330 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-client.admin.23016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22767.log 2026-02-12T12:33:42.331 INFO:teuthology.orchestra.run.trial127.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.23016.log.gz 2026-02-12T12:33:42.331 INFO:teuthology.orchestra.run.trial127.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30562.log/var/log/ceph/ceph-client.admin.22865.log: 2026-02-12T12:33:42.331 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-client.admin.22767.log: 47.6% -- replaced with /var/log/ceph/ceph-client.admin.22865.log.gz 2026-02-12T12:33:42.331 INFO:teuthology.orchestra.run.trial127.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23257.log 49.6% 2026-02-12T12:33:42.331 INFO:teuthology.orchestra.run.trial127.stderr: -- replaced with /var/log/ceph/ceph-client.admin.22767.log.gz 2026-02-12T12:33:42.331 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-client.admin.30562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21863.log 2026-02-12T12:33:42.331 INFO:teuthology.orchestra.run.trial127.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.30562.log.gz 2026-02-12T12:33:42.331 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-client.admin.23257.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2026-02-12T12:33:42.331 INFO:teuthology.orchestra.run.trial127.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.23257.log.gz 2026-02-12T12:33:42.331 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-client.admin.21863.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21768.log 2026-02-12T12:33:42.331 INFO:teuthology.orchestra.run.trial127.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.21863.log.gz 2026-02-12T12:33:42.331 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22141.log 2026-02-12T12:33:42.332 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-client.admin.21768.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2026-02-12T12:33:42.332 INFO:teuthology.orchestra.run.trial127.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.21768.log.gz 2026-02-12T12:33:42.332 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-client.admin.22141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28694.log 2026-02-12T12:33:42.332 INFO:teuthology.orchestra.run.trial127.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.22141.log.gz 2026-02-12T12:33:42.332 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29657.log 2026-02-12T12:33:42.332 INFO:teuthology.orchestra.run.trial127.stderr: 86.4% -- replaced with /var/log/ceph/ceph.log.gz 2026-02-12T12:33:42.332 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-client.admin.28694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28926.log 2026-02-12T12:33:42.332 INFO:teuthology.orchestra.run.trial127.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28694.log.gz 2026-02-12T12:33:42.332 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-client.admin.29657.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29657.log.gz 2026-02-12T12:33:42.333 INFO:teuthology.orchestra.run.trial127.stderr:/var/log/ceph/ceph-client.admin.28926.log: 65.6% -- replaced with /var/log/ceph/ceph-client.admin.28926.log.gz 2026-02-12T12:33:42.333 INFO:teuthology.orchestra.run.trial127.stderr: 92.2% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2026-02-12T12:33:44.527 INFO:teuthology.orchestra.run.trial127.stderr: 92.0% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2026-02-12T12:33:44.597 INFO:teuthology.orchestra.run.trial127.stderr: 92.1% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2026-02-12T12:33:44.638 INFO:teuthology.orchestra.run.trial127.stderr: 92.1% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2026-02-12T12:33:44.640 INFO:teuthology.orchestra.run.trial127.stderr: 2026-02-12T12:33:44.640 INFO:teuthology.orchestra.run.trial127.stderr:real 0m2.338s 2026-02-12T12:33:44.640 INFO:teuthology.orchestra.run.trial127.stderr:user 0m6.356s 2026-02-12T12:33:44.641 INFO:teuthology.orchestra.run.trial127.stderr:sys 0m0.497s 2026-02-12T12:33:44.641 INFO:tasks.ceph:Archiving logs... 2026-02-12T12:33:44.641 DEBUG:teuthology.misc:Transferring archived files from trial127:/var/log/ceph to /home/teuthworker/mnt/teuthology/moagrawa-2026-02-10_06:35:43-crimson-rados-main-distro-debug-trial/42503/remote/trial127/log 2026-02-12T12:33:44.643 DEBUG:teuthology.orchestra.run.trial127:> sudo tar c -f - -C /var/log/ceph -- . 2026-02-12T12:33:45.337 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-12T12:33:45.408 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-12T12:33:45.419 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2026-02-12T12:33:45.420 DEBUG:teuthology.orchestra.run.trial127:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-12T12:33:45.446 DEBUG:teuthology.run_tasks:Unwinding manager clock 2026-02-12T12:33:45.457 INFO:teuthology.task.clock:Checking final clock skew... 2026-02-12T12:33:45.458 DEBUG:teuthology.orchestra.run.trial127:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-12T12:33:45.501 INFO:teuthology.orchestra.run.trial127.stderr:bash: line 1: ntpq: command not found 2026-02-12T12:33:45.528 INFO:teuthology.orchestra.run.trial127.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-12T12:33:45.528 INFO:teuthology.orchestra.run.trial127.stdout:=============================================================================== 2026-02-12T12:33:45.528 INFO:teuthology.orchestra.run.trial127.stdout:^- t1.time.bf1.yahoo.com 2 7 377 52 -35ms[ -35ms] +/- 42ms 2026-02-12T12:33:45.528 INFO:teuthology.orchestra.run.trial127.stdout:^- dug.arpnet.net 2 7 367 46 +56us[ +56us] +/- 39ms 2026-02-12T12:33:45.528 INFO:teuthology.orchestra.run.trial127.stdout:^* 69-172-133-130.planet.net 2 6 377 46 -45us[ -62us] +/- 3229us 2026-02-12T12:33:45.528 INFO:teuthology.orchestra.run.trial127.stdout:^- jane.qotw.net 2 7 377 117 +1202us[+1179us] +/- 25ms 2026-02-12T12:33:45.528 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2026-02-12T12:33:45.542 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2026-02-12T12:33:45.543 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2026-02-12T12:33:45.558 DEBUG:teuthology.orchestra.run.trial127:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2026-02-12T12:33:45.617 DEBUG:teuthology.orchestra.run.trial127:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-12T12:33:45.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-12T12:33:45.643 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2026-02-12T12:33:45.661 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2026-02-12T12:33:45.678 INFO:teuthology.task.internal:Duration was 1679.829902 seconds 2026-02-12T12:33:45.678 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2026-02-12T12:33:45.695 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2026-02-12T12:33:45.695 DEBUG:teuthology.orchestra.run.trial127:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-12T12:33:45.736 INFO:teuthology.orchestra.run.trial127.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-12T12:33:46.088 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2026-02-12T12:33:46.088 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial127.front.sepia.ceph.com 2026-02-12T12:33:46.088 DEBUG:teuthology.orchestra.run.trial127:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-12T12:33:46.147 INFO:teuthology.task.internal.syslog:Gathering journactl... 2026-02-12T12:33:46.147 DEBUG:teuthology.orchestra.run.trial127:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-12T12:33:46.234 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2026-02-12T12:33:46.234 DEBUG:teuthology.orchestra.run.trial127:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-12T12:33:46.258 INFO:teuthology.orchestra.run.trial127.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-12T12:33:46.258 INFO:teuthology.orchestra.run.trial127.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-12T12:33:46.258 INFO:teuthology.orchestra.run.trial127.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-02-12T12:33:46.258 INFO:teuthology.orchestra.run.trial127.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-12T12:33:46.259 INFO:teuthology.orchestra.run.trial127.stderr:/home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-12T12:33:46.262 INFO:teuthology.orchestra.run.trial127.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: 88.9% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-12T12:33:46.264 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2026-02-12T12:33:46.282 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2026-02-12T12:33:46.282 DEBUG:teuthology.orchestra.run.trial127:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-12T12:33:46.330 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2026-02-12T12:33:46.348 DEBUG:teuthology.orchestra.run.trial127:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-12T12:33:46.394 INFO:teuthology.orchestra.run.trial127.stdout:kernel.core_pattern = core 2026-02-12T12:33:46.408 DEBUG:teuthology.orchestra.run.trial127:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-12T12:33:46.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-12T12:33:46.464 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2026-02-12T12:33:46.482 INFO:teuthology.task.internal:Transferring archived files... 2026-02-12T12:33:46.483 DEBUG:teuthology.misc:Transferring archived files from trial127:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/moagrawa-2026-02-10_06:35:43-crimson-rados-main-distro-debug-trial/42503/remote/trial127 2026-02-12T12:33:46.483 DEBUG:teuthology.orchestra.run.trial127:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-12T12:33:46.574 INFO:teuthology.task.internal:Removing archive directory... 2026-02-12T12:33:46.574 DEBUG:teuthology.orchestra.run.trial127:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-12T12:33:46.592 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2026-02-12T12:33:46.612 INFO:teuthology.task.internal:Not uploading archives. 2026-02-12T12:33:46.613 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2026-02-12T12:33:46.630 INFO:teuthology.task.internal:Tidying up after the test... 2026-02-12T12:33:46.631 DEBUG:teuthology.orchestra.run.trial127:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-12T12:33:46.648 INFO:teuthology.orchestra.run.trial127.stdout: 786525 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 12 12:33 /home/ubuntu/cephtest 2026-02-12T12:33:46.649 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2026-02-12T12:33:46.668 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2026-02-12T12:33:46.710 INFO:teuthology.run:Summary data: description: crimson-rados/perf/{0-crimson_install clusters/{crimson-fixed-1} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/rbm$/{crimson_seastore_rbm} perf_ignorelist settings/optimized workloads/fio_4M_rand_rw} duration: 1679.8299024105072 flavor: debug owner: scheduled_moagrawa@soko04.front.sepia.ceph.com success: true 2026-02-12T12:33:46.710 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-12T12:33:46.819 INFO:teuthology.run:pass