2026-02-12T08:31:54.213 INFO:root:teuthology version: 0.0.1.dev29+gc433f1062 2026-02-12T08:31:54.229 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-12T08:31:54.315 INFO:teuthology.run:Config: archive_path: /home/teuthworker/mnt/teuthology/matan-2026-02-09_15:50:17-crimson-rados-main-distro-debug-trial/41545 branch: main description: crimson-rados/perf/{0-crimson_install clusters/{crimson-fixed-1} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/segmented$/{crimson_seastore_segmented} perf_ignorelist settings/optimized workloads/radosbench_4M_seq_read} email: null first_in_suite: false flavor: debug job_id: '41545' kernel: branch: distro kdb: 1 sha1: distro last_in_suite: false machine_type: trial name: matan-2026-02-09_15:50:17-crimson-rados-main-distro-debug-trial no_nested_subset: false os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: global: auth client required: none auth cluster required: none auth service required: none auth supported: none debug asok: 0/0 debug auth: 0/0 debug bdev: 0/0 debug bluefs: 0/0 debug bluestore: 0/0 debug buffer: 0/0 debug civetweb: 0/0 debug client: 0/0 debug compressor: 0/0 debug context: 0/0 debug crush: 0/0 debug crypto: 0/0 debug dpdk: 0/0 debug eventtrace: 0/0 debug filer: 0/0 debug filestore: 0/0 debug finisher: 0/0 debug fuse: 0/0 debug heartbeatmap: 0/0 debug javaclient: 0/0 debug journal: 0/0 debug journaler: 0/0 debug kstore: 0/0 debug leveldb: 0/0 debug lockdep: 0/0 debug mds: 0/0 debug mds balancer: 0/0 debug mds locker: 0/0 debug mds log: 0/0 debug mds log expire: 0/0 debug mds migrator: 0/0 debug memdb: 0/0 debug mgr: 0/0 debug mgrc: 0/0 debug mon: 0/0 debug monc: 0/0 debug ms: 0/0 debug objclass: 0/0 debug objectcacher: 0/0 debug objecter: 0/0 debug optracker: 0/0 debug osd: 0/0 debug paxos: 0/0 debug perfcounter: 0/0 debug rados: 0/0 debug rbd: 0/0 debug rbd mirror: 0/0 debug rbd replay: 0/0 debug refs: 0/0 debug rgw: 0/0 debug rgw lifecycle: 0/0 debug rgw notification: 0/0 debug rgw sync: 0/0 debug rocksdb: 0/0 debug striper: 0/0 debug throttle: 0/0 debug timer: 0/0 debug tp: 0/0 enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 0/0 debug ms: 0/0 debug paxos: 0/0 osd pool default crimson: true osd: crimson cpu num: 4 crimson osd obc lru size: 10 debug filestore: 0/0 debug journal: 0/0 debug ms: 0/0 debug osd: 0/0 debug seastore: 20 debug seastore backref: 20 debug seastore cache: 20 debug seastore cleaner: 20 debug seastore device: 20 debug seastore epm: 20 debug seastore fixedkv tree: 20 debug seastore journal: 20 debug seastore lba: 20 debug seastore odata: 20 debug seastore omap: 20 debug seastore onode: 20 debug seastore t: 20 debug seastore tm: 20 osd objectstore: seastore osd shutdown pgref assert: true seastore cachepin size pershard: 64M seastore max concurrent transactions: 8 crimson_compat: true flavor: debug log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) - \(PG_ - \(OSD_ - \(OBJECT_ - overall HEALTH sha1: 2401a109e2ac85dfa20fc57328da663aa754c875 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log global: osd crush chooseleaf type: 0 osd pool default pg num: 128 osd pool default pgp num: 128 osd pool default size: 2 mon: {} install: ceph: flavor: debug sha1: 2401a109e2ac85dfa20fc57328da663aa754c875 workunit: branch: main env: CRIMSON_COMPAT: '1' sha1: 6bf51b64d7733c58c2fe4a45746d4b9e65a38d1c owner: scheduled_matan@soko04.front.sepia.ceph.com priority: 100 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 9353 sha1: 2401a109e2ac85dfa20fc57328da663aa754c875 sleep_before_teardown: 0 suite: crimson-rados suite_branch: main suite_path: /home/teuthworker/src/github.com_ceph_ceph_6bf51b64d7733c58c2fe4a45746d4b9e65a38d1c/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph.git suite_sha1: 6bf51b64d7733c58c2fe4a45746d4b9e65a38d1c targets: trial187.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGUQF4Dh4CZ7JoVPAoUH6MZNB4MuhPX5CLuhWSf5k0gS8waXJhCD8tcGvw5PQReoIDiSI5u+TF2iKA/e4SIG90k= tasks: - install: extra_packages: deb: - ceph-osd-crimson - ceph-osd-crimson-dbg rpm: - ceph-osd-crimson - ceph-osd-crimson-debuginfo - pexec: all: - sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson - install: null - ceph: conf: osd: debug monc: 20 wait-for-scrub: false - ssh_keys: null - cbt: benchmarks: radosbench: concurrent_ops: 4 concurrent_procs: 2 op_size: - 4194304 pool_monitoring_list: - collectl pool_profile: replicated run_monitoring_list: - collectl time: 60 write_only: false cluster: iterations: 1 osds_per_node: 3 pool_profiles: replicated: pg_size: 256 pgp_size: 256 replication: replicated user: ubuntu monitoring_profiles: perf: args: stat -p {pid} -o {perf_dir}/perf_stat.{pid} nodes: - osds perf_cmd: perf pid_glob: crimson-osd.*.pid teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: c433f1062990a0488dc29a553589bc609a460691 timestamp: 2026-02-09_15:50:17 tube: trial user: matan verbose: false worker_log: /home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.2161924 2026-02-12T08:31:54.315 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph_6bf51b64d7733c58c2fe4a45746d4b9e65a38d1c/qa; will attempt to use it 2026-02-12T08:31:54.315 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph_6bf51b64d7733c58c2fe4a45746d4b9e65a38d1c/qa/tasks 2026-02-12T08:31:54.315 INFO:teuthology.run_tasks:Running task internal.check_packages... 2026-02-12T08:31:54.317 INFO:teuthology.task.internal:Checking packages... 2026-02-12T08:31:54.317 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'debug' and ceph hash '2401a109e2ac85dfa20fc57328da663aa754c875' 2026-02-12T08:31:54.317 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2026-02-12T08:31:54.317 INFO:teuthology.packaging:ref: None 2026-02-12T08:31:54.317 INFO:teuthology.packaging:tag: None 2026-02-12T08:31:54.317 INFO:teuthology.packaging:branch: main 2026-02-12T08:31:54.317 INFO:teuthology.packaging:sha1: 2401a109e2ac85dfa20fc57328da663aa754c875 2026-02-12T08:31:54.317 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&ref=main 2026-02-12T08:31:54.483 INFO:teuthology.task.internal:Found packages for ceph version 20.3.0-5271.gdbd7cb72 2026-02-12T08:31:54.484 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2026-02-12T08:31:54.496 INFO:teuthology.task.internal:no buildpackages task found 2026-02-12T08:31:54.496 INFO:teuthology.run_tasks:Running task internal.save_config... 2026-02-12T08:31:54.510 INFO:teuthology.task.internal:Saving configuration 2026-02-12T08:31:54.532 INFO:teuthology.run_tasks:Running task internal.check_lock... 2026-02-12T08:31:54.545 INFO:teuthology.task.internal.check_lock:Checking locks... 2026-02-12T08:31:54.634 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial187.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/matan-2026-02-09_15:50:17-crimson-rados-main-distro-debug-trial/41545', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-12 08:28:36.205074', 'locked_by': 'scheduled_matan@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGUQF4Dh4CZ7JoVPAoUH6MZNB4MuhPX5CLuhWSf5k0gS8waXJhCD8tcGvw5PQReoIDiSI5u+TF2iKA/e4SIG90k='} 2026-02-12T08:31:54.634 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2026-02-12T08:31:54.645 INFO:teuthology.task.internal:roles: ubuntu@trial187.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2026-02-12T08:31:54.645 INFO:teuthology.run_tasks:Running task console_log... 2026-02-12T08:31:54.735 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x73c81a015760>, signals=[15]) 2026-02-12T08:31:54.735 INFO:teuthology.run_tasks:Running task internal.connect... 2026-02-12T08:31:54.746 INFO:teuthology.task.internal:Opening connections... 2026-02-12T08:31:54.746 DEBUG:teuthology.task.internal:connecting to ubuntu@trial187.front.sepia.ceph.com 2026-02-12T08:31:54.747 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-12T08:31:54.809 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2026-02-12T08:31:54.820 DEBUG:teuthology.orchestra.run.trial187:> uname -m 2026-02-12T08:31:54.890 INFO:teuthology.orchestra.run.trial187.stdout:x86_64 2026-02-12T08:31:54.890 DEBUG:teuthology.orchestra.run.trial187:> cat /etc/os-release 2026-02-12T08:31:54.944 INFO:teuthology.orchestra.run.trial187.stdout:NAME="CentOS Stream" 2026-02-12T08:31:54.945 INFO:teuthology.orchestra.run.trial187.stdout:VERSION="9" 2026-02-12T08:31:54.945 INFO:teuthology.orchestra.run.trial187.stdout:ID="centos" 2026-02-12T08:31:54.945 INFO:teuthology.orchestra.run.trial187.stdout:ID_LIKE="rhel fedora" 2026-02-12T08:31:54.945 INFO:teuthology.orchestra.run.trial187.stdout:VERSION_ID="9" 2026-02-12T08:31:54.945 INFO:teuthology.orchestra.run.trial187.stdout:PLATFORM_ID="platform:el9" 2026-02-12T08:31:54.945 INFO:teuthology.orchestra.run.trial187.stdout:PRETTY_NAME="CentOS Stream 9" 2026-02-12T08:31:54.945 INFO:teuthology.orchestra.run.trial187.stdout:ANSI_COLOR="0;31" 2026-02-12T08:31:54.945 INFO:teuthology.orchestra.run.trial187.stdout:LOGO="fedora-logo-icon" 2026-02-12T08:31:54.945 INFO:teuthology.orchestra.run.trial187.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2026-02-12T08:31:54.945 INFO:teuthology.orchestra.run.trial187.stdout:HOME_URL="https://centos.org/" 2026-02-12T08:31:54.945 INFO:teuthology.orchestra.run.trial187.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2026-02-12T08:31:54.945 INFO:teuthology.orchestra.run.trial187.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2026-02-12T08:31:54.945 INFO:teuthology.orchestra.run.trial187.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2026-02-12T08:31:54.945 INFO:teuthology.lock.ops:Updating trial187.front.sepia.ceph.com on lock server 2026-02-12T08:31:55.021 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2026-02-12T08:31:55.034 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2026-02-12T08:31:55.045 INFO:teuthology.task.internal:Checking for old test directory... 2026-02-12T08:31:55.045 DEBUG:teuthology.orchestra.run.trial187:> test '!' -e /home/ubuntu/cephtest 2026-02-12T08:31:55.060 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2026-02-12T08:31:55.075 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2026-02-12T08:31:55.075 DEBUG:teuthology.orchestra.run.trial187:> test -z $(ls -A /var/lib/ceph) 2026-02-12T08:31:55.117 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2026-02-12T08:31:55.222 INFO:teuthology.run_tasks:Running task kernel... 2026-02-12T08:31:55.245 INFO:teuthology.task.kernel:normalize config orig: {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'} 2026-02-12T08:31:55.245 INFO:teuthology.task.kernel:config {'mon.a': {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'}}, timeout 300 2026-02-12T08:31:55.245 DEBUG:teuthology.orchestra.run.trial187:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-12T08:31:55.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-12T08:31:55.260 DEBUG:teuthology.orchestra.run.trial187:> uname -r 2026-02-12T08:31:55.316 INFO:teuthology.orchestra.run.trial187.stdout:5.14.0-645.el9.x86_64 2026-02-12T08:31:55.316 INFO:teuthology.task.kernel:Running kernel on trial187: 5.14.0-645.el9.x86_64 2026-02-12T08:31:55.316 DEBUG:teuthology.orchestra.run.trial187:> sudo yum install -y kernel 2026-02-12T08:31:58.001 INFO:teuthology.orchestra.run.trial187.stdout:CentOS Stream 9 - BaseOS 3.8 MB/s | 8.9 MB 00:02 2026-02-12T08:32:19.066 INFO:teuthology.orchestra.run.trial187.stdout:CentOS Stream 9 - AppStream 1.3 MB/s | 27 MB 00:20 2026-02-12T08:32:22.912 INFO:teuthology.orchestra.run.trial187.stdout:CentOS Stream 9 - CRB 5.3 MB/s | 7.8 MB 00:01 2026-02-12T08:32:23.843 INFO:teuthology.orchestra.run.trial187.stdout:CentOS Stream 9 - Extras packages 96 kB/s | 20 kB 00:00 2026-02-12T08:32:24.396 INFO:teuthology.orchestra.run.trial187.stdout:Extra Packages for Enterprise Linux 42 MB/s | 20 MB 00:00 2026-02-12T08:32:27.561 INFO:teuthology.orchestra.run.trial187.stdout:lab-extras 1.7 MB/s | 50 kB 00:00 2026-02-12T08:32:28.621 INFO:teuthology.orchestra.run.trial187.stdout:Package kernel-5.14.0-645.el9.x86_64 is already installed. 2026-02-12T08:32:28.641 INFO:teuthology.orchestra.run.trial187.stdout:Dependencies resolved. 2026-02-12T08:32:28.642 INFO:teuthology.orchestra.run.trial187.stdout:================================================================================ 2026-02-12T08:32:28.642 INFO:teuthology.orchestra.run.trial187.stdout: Package Architecture Version Repository Size 2026-02-12T08:32:28.643 INFO:teuthology.orchestra.run.trial187.stdout:================================================================================ 2026-02-12T08:32:28.643 INFO:teuthology.orchestra.run.trial187.stdout:Installing: 2026-02-12T08:32:28.643 INFO:teuthology.orchestra.run.trial187.stdout: kernel x86_64 5.14.0-677.el9 baseos 768 k 2026-02-12T08:32:28.643 INFO:teuthology.orchestra.run.trial187.stdout:Installing dependencies: 2026-02-12T08:32:28.643 INFO:teuthology.orchestra.run.trial187.stdout: kernel-core x86_64 5.14.0-677.el9 baseos 17 M 2026-02-12T08:32:28.643 INFO:teuthology.orchestra.run.trial187.stdout: kernel-modules x86_64 5.14.0-677.el9 baseos 40 M 2026-02-12T08:32:28.643 INFO:teuthology.orchestra.run.trial187.stdout: kernel-modules-core x86_64 5.14.0-677.el9 baseos 31 M 2026-02-12T08:32:28.643 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:32:28.643 INFO:teuthology.orchestra.run.trial187.stdout:Transaction Summary 2026-02-12T08:32:28.643 INFO:teuthology.orchestra.run.trial187.stdout:================================================================================ 2026-02-12T08:32:28.643 INFO:teuthology.orchestra.run.trial187.stdout:Install 4 Packages 2026-02-12T08:32:28.644 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:32:28.644 INFO:teuthology.orchestra.run.trial187.stdout:Total download size: 88 M 2026-02-12T08:32:28.644 INFO:teuthology.orchestra.run.trial187.stdout:Installed size: 135 M 2026-02-12T08:32:28.644 INFO:teuthology.orchestra.run.trial187.stdout:Downloading Packages: 2026-02-12T08:32:29.189 INFO:teuthology.orchestra.run.trial187.stdout:[MIRROR] kernel-core-5.14.0-677.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-677.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-12T08:32:29.193 INFO:teuthology.orchestra.run.trial187.stdout:[MIRROR] kernel-modules-5.14.0-677.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-677.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-12T08:32:29.194 INFO:teuthology.orchestra.run.trial187.stdout:[MIRROR] kernel-5.14.0-677.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-677.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-12T08:32:30.772 INFO:teuthology.orchestra.run.trial187.stdout:(1/4): kernel-5.14.0-677.el9.x86_64.rpm 376 kB/s | 768 kB 00:02 2026-02-12T08:32:30.933 INFO:teuthology.orchestra.run.trial187.stdout:[MIRROR] kernel-modules-core-5.14.0-677.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-677.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-12T08:32:31.325 INFO:teuthology.orchestra.run.trial187.stdout:[MIRROR] kernel-modules-core-5.14.0-677.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-677.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-12T08:32:32.691 INFO:teuthology.orchestra.run.trial187.stdout:(2/4): kernel-modules-core-5.14.0-677.el9.x86_6 16 MB/s | 31 MB 00:01 2026-02-12T08:32:48.626 INFO:teuthology.orchestra.run.trial187.stdout:(3/4): kernel-core-5.14.0-677.el9.x86_64.rpm 877 kB/s | 17 MB 00:19 2026-02-12T08:33:13.236 INFO:teuthology.orchestra.run.trial187.stdout:(4/4): kernel-modules-5.14.0-677.el9.x86_64.rpm 911 kB/s | 40 MB 00:44 2026-02-12T08:33:13.239 INFO:teuthology.orchestra.run.trial187.stdout:-------------------------------------------------------------------------------- 2026-02-12T08:33:13.240 INFO:teuthology.orchestra.run.trial187.stdout:Total 2.0 MB/s | 88 MB 00:44 2026-02-12T08:33:13.425 INFO:teuthology.orchestra.run.trial187.stdout:Running transaction check 2026-02-12T08:33:13.626 INFO:teuthology.orchestra.run.trial187.stdout:Transaction check succeeded. 2026-02-12T08:33:13.626 INFO:teuthology.orchestra.run.trial187.stdout:Running transaction test 2026-02-12T08:33:13.971 INFO:teuthology.orchestra.run.trial187.stdout:Transaction test succeeded. 2026-02-12T08:33:13.971 INFO:teuthology.orchestra.run.trial187.stdout:Running transaction 2026-02-12T08:33:15.056 INFO:teuthology.orchestra.run.trial187.stdout: Preparing : 1/1 2026-02-12T08:33:15.338 INFO:teuthology.orchestra.run.trial187.stdout: Installing : kernel-modules-core-5.14.0-677.el9.x86_64 1/4 2026-02-12T08:33:15.357 INFO:teuthology.orchestra.run.trial187.stdout: Installing : kernel-core-5.14.0-677.el9.x86_64 2/4 2026-02-12T08:33:17.124 INFO:teuthology.orchestra.run.trial187.stdout: Running scriptlet: kernel-core-5.14.0-677.el9.x86_64 2/4 2026-02-12T08:33:17.152 INFO:teuthology.orchestra.run.trial187.stdout: Installing : kernel-modules-5.14.0-677.el9.x86_64 3/4 2026-02-12T08:33:20.243 INFO:teuthology.orchestra.run.trial187.stdout: Running scriptlet: kernel-modules-5.14.0-677.el9.x86_64 3/4 2026-02-12T08:33:20.249 INFO:teuthology.orchestra.run.trial187.stdout: Installing : kernel-5.14.0-677.el9.x86_64 4/4 2026-02-12T08:33:23.281 INFO:teuthology.orchestra.run.trial187.stdout: Running scriptlet: kernel-modules-core-5.14.0-677.el9.x86_64 4/4 2026-02-12T08:33:53.589 INFO:teuthology.orchestra.run.trial187.stdout: Running scriptlet: kernel-core-5.14.0-677.el9.x86_64 4/4 2026-02-12T08:33:53.611 INFO:teuthology.orchestra.run.trial187.stdout: Running scriptlet: kernel-modules-5.14.0-677.el9.x86_64 4/4 2026-02-12T08:33:53.781 INFO:teuthology.orchestra.run.trial187.stdout: Running scriptlet: kernel-5.14.0-677.el9.x86_64 4/4 2026-02-12T08:33:53.781 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : kernel-5.14.0-677.el9.x86_64 1/4 2026-02-12T08:33:53.781 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : kernel-core-5.14.0-677.el9.x86_64 2/4 2026-02-12T08:33:53.782 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : kernel-modules-5.14.0-677.el9.x86_64 3/4 2026-02-12T08:33:53.833 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : kernel-modules-core-5.14.0-677.el9.x86_64 4/4 2026-02-12T08:33:53.833 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:33:53.833 INFO:teuthology.orchestra.run.trial187.stdout:Installed: 2026-02-12T08:33:53.833 INFO:teuthology.orchestra.run.trial187.stdout: kernel-5.14.0-677.el9.x86_64 2026-02-12T08:33:53.833 INFO:teuthology.orchestra.run.trial187.stdout: kernel-core-5.14.0-677.el9.x86_64 2026-02-12T08:33:53.833 INFO:teuthology.orchestra.run.trial187.stdout: kernel-modules-5.14.0-677.el9.x86_64 2026-02-12T08:33:53.833 INFO:teuthology.orchestra.run.trial187.stdout: kernel-modules-core-5.14.0-677.el9.x86_64 2026-02-12T08:33:53.834 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:33:53.834 INFO:teuthology.orchestra.run.trial187.stdout:Complete! 2026-02-12T08:33:53.890 DEBUG:teuthology.orchestra.run.trial187:> rpm -q kernel | sort -rV | head -n 1 2026-02-12T08:33:53.916 INFO:teuthology.orchestra.run.trial187.stdout:kernel-5.14.0-677.el9.x86_64 2026-02-12T08:33:53.917 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-677.el9.x86_64 2026-02-12T08:33:53.917 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-645.el9.x86_64 Expected: 5.14.0-677.el9.x86_64 2026-02-12T08:33:53.917 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2026-02-12T08:33:53.917 DEBUG:teuthology.task.kernel:src is distro, skipping download 2026-02-12T08:33:53.917 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2026-02-12T08:33:53.917 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@trial187.front.sepia.ceph.com, path=None, version=distro) 2026-02-12T08:33:53.918 DEBUG:teuthology.orchestra.run.trial187:> rpm -q kernel | sort -rV | head -n 1 2026-02-12T08:33:53.979 INFO:teuthology.orchestra.run.trial187.stdout:kernel-5.14.0-677.el9.x86_64 2026-02-12T08:33:53.981 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-677.el9.x86_64 2026-02-12T08:33:53.981 DEBUG:teuthology.orchestra.run.trial187:> sudo rpm -qi grub2-tools 2026-02-12T08:33:54.048 INFO:teuthology.orchestra.run.trial187.stdout:Name : grub2-tools 2026-02-12T08:33:54.048 INFO:teuthology.orchestra.run.trial187.stdout:Epoch : 1 2026-02-12T08:33:54.049 INFO:teuthology.orchestra.run.trial187.stdout:Version : 2.06 2026-02-12T08:33:54.049 INFO:teuthology.orchestra.run.trial187.stdout:Release : 118.el9 2026-02-12T08:33:54.049 INFO:teuthology.orchestra.run.trial187.stdout:Architecture: x86_64 2026-02-12T08:33:54.049 INFO:teuthology.orchestra.run.trial187.stdout:Install Date: Thu 04 Dec 2025 11:54:21 PM UTC 2026-02-12T08:33:54.049 INFO:teuthology.orchestra.run.trial187.stdout:Group : Unspecified 2026-02-12T08:33:54.049 INFO:teuthology.orchestra.run.trial187.stdout:Size : 8393909 2026-02-12T08:33:54.049 INFO:teuthology.orchestra.run.trial187.stdout:License : GPLv3+ 2026-02-12T08:33:54.049 INFO:teuthology.orchestra.run.trial187.stdout:Signature : RSA/SHA256, Wed 05 Nov 2025 09:13:15 AM UTC, Key ID 05b555b38483c65d 2026-02-12T08:33:54.049 INFO:teuthology.orchestra.run.trial187.stdout:Source RPM : grub2-2.06-118.el9.src.rpm 2026-02-12T08:33:54.049 INFO:teuthology.orchestra.run.trial187.stdout:Build Date : Tue 04 Nov 2025 03:44:14 PM UTC 2026-02-12T08:33:54.049 INFO:teuthology.orchestra.run.trial187.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2026-02-12T08:33:54.049 INFO:teuthology.orchestra.run.trial187.stdout:Packager : builder@centos.org 2026-02-12T08:33:54.050 INFO:teuthology.orchestra.run.trial187.stdout:Vendor : CentOS 2026-02-12T08:33:54.050 INFO:teuthology.orchestra.run.trial187.stdout:URL : http://www.gnu.org/software/grub/ 2026-02-12T08:33:54.050 INFO:teuthology.orchestra.run.trial187.stdout:Summary : Support tools for GRUB. 2026-02-12T08:33:54.050 INFO:teuthology.orchestra.run.trial187.stdout:Description : 2026-02-12T08:33:54.050 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:33:54.050 INFO:teuthology.orchestra.run.trial187.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2026-02-12T08:33:54.050 INFO:teuthology.orchestra.run.trial187.stdout:customizable bootloader with modular architecture. It supports a rich 2026-02-12T08:33:54.050 INFO:teuthology.orchestra.run.trial187.stdout:variety of kernel formats, file systems, computer architectures and 2026-02-12T08:33:54.050 INFO:teuthology.orchestra.run.trial187.stdout:hardware devices. 2026-02-12T08:33:54.050 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:33:54.050 INFO:teuthology.orchestra.run.trial187.stdout:This subpackage provides tools for support of all platforms. 2026-02-12T08:33:54.051 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2026-02-12T08:33:54.052 INFO:teuthology.task.kernel:Updating grub on trial187 to boot 5.14.0-677.el9.x86_64 2026-02-12T08:33:54.052 DEBUG:teuthology.orchestra.run.trial187:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f -print -quit 2026-02-12T08:33:54.115 INFO:teuthology.orchestra.run.trial187.stdout:/boot/loader/entries/2734456005d743dca7f3dc77c35a2d0d-0-rescue.conf 2026-02-12T08:33:54.117 DEBUG:teuthology.orchestra.run.trial187:> sudo command -v grubby && echo yes || echo no 2026-02-12T08:33:54.180 INFO:teuthology.orchestra.run.trial187.stdout:/sbin/grubby 2026-02-12T08:33:54.181 INFO:teuthology.orchestra.run.trial187.stdout:yes 2026-02-12T08:33:54.182 DEBUG:teuthology.orchestra.run.trial187:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f | grep -F "5.14.0-677.el9.x86_64" | head -n 1 2026-02-12T08:33:54.245 INFO:teuthology.orchestra.run.trial187.stdout:/boot/loader/entries/b0373be789df49cdafedf423703d2566-5.14.0-677.el9.x86_64.conf 2026-02-12T08:33:54.246 DEBUG:teuthology.orchestra.run.trial187:> sudo grub2-set-default b0373be789df49cdafedf423703d2566-5.14.0-677.el9.x86_64 2026-02-12T08:33:54.319 DEBUG:teuthology.orchestra.run.trial187:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2026-02-12T08:33:54.530 INFO:teuthology.orchestra.run.trial187.stderr:Generating grub configuration file ... 2026-02-12T08:33:54.594 INFO:teuthology.orchestra.run.trial187.stderr:Found linux image: /boot/vmlinuz-5.14.0-677.el9.x86_64 2026-02-12T08:33:54.601 INFO:teuthology.orchestra.run.trial187.stderr:Found initrd image: /boot/initramfs-5.14.0-677.el9.x86_64.img 2026-02-12T08:33:54.851 INFO:teuthology.orchestra.run.trial187.stderr:Found linux image: /boot/vmlinuz-5.14.0-645.el9.x86_64 2026-02-12T08:33:54.858 INFO:teuthology.orchestra.run.trial187.stderr:Found initrd image: /boot/initramfs-5.14.0-645.el9.x86_64.img 2026-02-12T08:33:54.890 INFO:teuthology.orchestra.run.trial187.stderr:Found linux image: /boot/vmlinuz-0-rescue-b0373be789df49cdafedf423703d2566 2026-02-12T08:33:54.896 INFO:teuthology.orchestra.run.trial187.stderr:Found initrd image: /boot/initramfs-0-rescue-b0373be789df49cdafedf423703d2566.img 2026-02-12T08:33:54.920 INFO:teuthology.orchestra.run.trial187.stderr:Found linux image: /boot/vmlinuz-0-rescue-2734456005d743dca7f3dc77c35a2d0d 2026-02-12T08:33:54.927 INFO:teuthology.orchestra.run.trial187.stderr:Found initrd image: /boot/initramfs-0-rescue-2734456005d743dca7f3dc77c35a2d0d.img 2026-02-12T08:33:55.009 INFO:teuthology.orchestra.run.trial187.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2026-02-12T08:33:55.018 INFO:teuthology.orchestra.run.trial187.stderr:done 2026-02-12T08:33:55.020 DEBUG:teuthology.orchestra.run.trial187:> test -d /sys/firmware/efi 2026-02-12T08:33:55.035 DEBUG:teuthology.orchestra.run.trial187:> sudo find /boot/efi/EFI -mindepth 1 -maxdepth 1 -type d || true 2026-02-12T08:33:55.102 INFO:teuthology.orchestra.run.trial187.stdout:/boot/efi/EFI/BOOT 2026-02-12T08:33:55.102 INFO:teuthology.orchestra.run.trial187.stdout:/boot/efi/EFI/centos 2026-02-12T08:33:55.104 DEBUG:teuthology.orchestra.run.trial187:> sudo test -f /boot/efi/EFI/BOOT/grub.cfg && echo yes || echo no 2026-02-12T08:33:55.171 INFO:teuthology.orchestra.run.trial187.stdout:no 2026-02-12T08:33:55.171 DEBUG:teuthology.orchestra.run.trial187:> sudo test -f /boot/efi/EFI/centos/grub.cfg && echo yes || echo no 2026-02-12T08:33:55.234 INFO:teuthology.orchestra.run.trial187.stdout:yes 2026-02-12T08:33:55.235 DEBUG:teuthology.orchestra.run.trial187:> sudo cp -f /boot/grub2/grub.cfg /boot/efi/EFI/centos/grub.cfg 2026-02-12T08:33:55.299 DEBUG:teuthology.orchestra.run.trial187:> sudo shutdown -r now 2026-02-12T08:33:55.375 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2026-02-12T08:34:25.372 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2026-02-12T08:34:25.373 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial187.front.sepia.ceph.com' 2026-02-12T08:34:25.374 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-12T08:35:25.434 DEBUG:teuthology.orchestra.remote:timed out 2026-02-12T08:35:34.443 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial187.front.sepia.ceph.com' 2026-02-12T08:35:34.444 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-12T08:35:34.547 DEBUG:teuthology.orchestra.run.trial187:> true 2026-02-12T08:35:34.707 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial187.front.sepia.ceph.com' 2026-02-12T08:35:34.707 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-677.el9.x86_64"... 2026-02-12T08:35:34.707 DEBUG:teuthology.orchestra.run.trial187:> uname -r 2026-02-12T08:35:34.762 INFO:teuthology.orchestra.run.trial187.stdout:5.14.0-677.el9.x86_64 2026-02-12T08:35:34.763 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-677.el9.x86_64 vs 5.14.0-677.el9.x86_64 2026-02-12T08:35:34.763 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2026-02-12T08:35:34.763 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2026-02-12T08:35:35.763 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2026-02-12T08:35:35.764 DEBUG:teuthology.orchestra.run.trial187:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-12T08:35:35.793 INFO:teuthology.orchestra.run.trial187.stdout:ttyS1 2026-02-12T08:35:35.817 DEBUG:teuthology.parallel:result is None 2026-02-12T08:35:35.817 INFO:teuthology.run_tasks:Running task internal.base... 2026-02-12T08:35:35.830 INFO:teuthology.task.internal:Creating test directory... 2026-02-12T08:35:35.830 DEBUG:teuthology.orchestra.run.trial187:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-12T08:35:35.873 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2026-02-12T08:35:35.886 INFO:teuthology.run_tasks:Running task internal.archive... 2026-02-12T08:35:35.899 INFO:teuthology.task.internal:Creating archive directory... 2026-02-12T08:35:35.899 DEBUG:teuthology.orchestra.run.trial187:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-12T08:35:35.935 INFO:teuthology.run_tasks:Running task internal.coredump... 2026-02-12T08:35:35.946 INFO:teuthology.task.internal:Enabling coredump saving... 2026-02-12T08:35:35.946 DEBUG:teuthology.orchestra.run.trial187:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-12T08:35:35.994 INFO:teuthology.orchestra.run.trial187.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-12T08:35:36.002 INFO:teuthology.orchestra.run.trial187.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-12T08:35:36.004 INFO:teuthology.run_tasks:Running task internal.sudo... 2026-02-12T08:35:36.015 INFO:teuthology.task.internal:Configuring sudo... 2026-02-12T08:35:36.016 DEBUG:teuthology.orchestra.run.trial187:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-12T08:35:36.068 INFO:teuthology.run_tasks:Running task internal.syslog... 2026-02-12T08:35:36.085 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2026-02-12T08:35:36.085 DEBUG:teuthology.orchestra.run.trial187:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-12T08:35:36.124 DEBUG:teuthology.orchestra.run.trial187:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-12T08:35:36.289 DEBUG:teuthology.orchestra.run.trial187:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-12T08:35:36.314 DEBUG:teuthology.orchestra.run.trial187:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-12T08:35:36.428 DEBUG:teuthology.orchestra.run.trial187:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-12T08:35:36.452 DEBUG:teuthology.orchestra.run.trial187:> set -ex 2026-02-12T08:35:36.452 DEBUG:teuthology.orchestra.run.trial187:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-12T08:35:36.516 DEBUG:teuthology.orchestra.run.trial187:> sudo service rsyslog restart 2026-02-12T08:35:36.583 INFO:teuthology.orchestra.run.trial187.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-12T08:35:36.744 INFO:teuthology.run_tasks:Running task internal.timer... 2026-02-12T08:35:36.759 INFO:teuthology.task.internal:Starting timer... 2026-02-12T08:35:36.759 INFO:teuthology.run_tasks:Running task pcp... 2026-02-12T08:35:36.772 INFO:teuthology.run_tasks:Running task selinux... 2026-02-12T08:35:36.784 DEBUG:teuthology.orchestra.run.trial187:> sudo service auditd rotate 2026-02-12T08:35:36.813 INFO:teuthology.orchestra.run.trial187.stdout:Rotating logs: 2026-02-12T08:35:36.814 DEBUG:teuthology.task.selinux:Getting current SELinux state 2026-02-12T08:35:36.814 DEBUG:teuthology.orchestra.run.trial187:> /usr/sbin/getenforce 2026-02-12T08:35:36.870 INFO:teuthology.orchestra.run.trial187.stdout:Permissive 2026-02-12T08:35:36.870 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@trial187.front.sepia.ceph.com': 'permissive'} 2026-02-12T08:35:36.870 DEBUG:teuthology.orchestra.run.trial187:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-12T08:35:36.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-12T08:35:36.935 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2026-02-12T08:35:36.935 DEBUG:teuthology.orchestra.run.trial187:> sudo /usr/sbin/setenforce permissive 2026-02-12T08:35:36.999 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2026-02-12T08:35:37.015 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2026-02-12T08:35:37.027 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2026-02-12T08:35:37.028 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit trial187.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2026-02-12T08:36:47.690 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@trial187.front.sepia.ceph.com')] 2026-02-12T08:36:47.691 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial187.front.sepia.ceph.com' 2026-02-12T08:36:47.693 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-12T08:36:47.800 DEBUG:teuthology.orchestra.run.trial187:> true 2026-02-12T08:36:47.875 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial187.front.sepia.ceph.com' 2026-02-12T08:36:47.875 INFO:teuthology.run_tasks:Running task clock... 2026-02-12T08:36:47.889 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2026-02-12T08:36:47.889 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-12T08:36:47.890 DEBUG:teuthology.orchestra.run.trial187:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-12T08:36:47.941 INFO:teuthology.orchestra.run.trial187.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2026-02-12T08:36:47.953 INFO:teuthology.orchestra.run.trial187.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2026-02-12T08:36:47.974 INFO:teuthology.orchestra.run.trial187.stderr:sudo: ntpd: command not found 2026-02-12T08:36:47.983 INFO:teuthology.orchestra.run.trial187.stdout:506 Cannot talk to daemon 2026-02-12T08:36:47.995 INFO:teuthology.orchestra.run.trial187.stderr:Failed to start ntp.service: Unit ntp.service not found. 2026-02-12T08:36:48.008 INFO:teuthology.orchestra.run.trial187.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2026-02-12T08:36:48.058 INFO:teuthology.orchestra.run.trial187.stderr:bash: line 1: ntpq: command not found 2026-02-12T08:36:48.226 INFO:teuthology.orchestra.run.trial187.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-12T08:36:48.227 INFO:teuthology.orchestra.run.trial187.stdout:=============================================================================== 2026-02-12T08:36:48.227 INFO:teuthology.orchestra.run.trial187.stdout:^? 23.157.160.168 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-12T08:36:48.227 INFO:teuthology.orchestra.run.trial187.stdout:^? nyc1.starstorm.net 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-12T08:36:48.227 INFO:teuthology.orchestra.run.trial187.stdout:^? time-usw1.crimpac.net 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-12T08:36:48.227 INFO:teuthology.orchestra.run.trial187.stdout:^? ip229.ip-51-81-226.us 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-12T08:36:48.228 INFO:teuthology.run_tasks:Running task install... 2026-02-12T08:36:48.239 DEBUG:teuthology.task.install:project ceph 2026-02-12T08:36:48.240 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'debug', 'sha1': '2401a109e2ac85dfa20fc57328da663aa754c875'}} 2026-02-12T08:36:48.240 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'flavor': 'debug', 'sha1': '2401a109e2ac85dfa20fc57328da663aa754c875'} 2026-02-12T08:36:48.240 INFO:teuthology.task.install:Using flavor: debug 2026-02-12T08:36:48.244 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-osd', 'ceph-osd-classic', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-osd', 'ceph-osd-classic', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-12T08:36:48.244 INFO:teuthology.task.install:extra packages: {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']} 2026-02-12T08:36:48.245 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'debug', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '2401a109e2ac85dfa20fc57328da663aa754c875', 'tag': None, 'wait_for_package': False} 2026-02-12T08:36:48.245 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=2401a109e2ac85dfa20fc57328da663aa754c875 2026-02-12T08:36:48.387 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-matanb-crimson-66488-revert/2401a109e2ac85dfa20fc57328da663aa754c875/centos/9/flavors/debug/ 2026-02-12T08:36:48.387 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5161.g2401a109 2026-02-12T08:36:48.754 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-matanb-crimson-66488-revert/2401a109e2ac85dfa20fc57328da663aa754c875/centos/9/flavors/debug/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-matanb-crimson-66488-revert/2401a109e2ac85dfa20fc57328da663aa754c875/centos/9/flavors/debug/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-matanb-crimson-66488-revert/2401a109e2ac85dfa20fc57328da663aa754c875/centos/9/flavors/debug/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-12T08:36:48.754 DEBUG:teuthology.orchestra.run.trial187:> set -ex 2026-02-12T08:36:48.754 DEBUG:teuthology.orchestra.run.trial187:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-12T08:36:48.778 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-osd-crimson, ceph-osd-crimson-debuginfo on remote rpm x86_64 2026-02-12T08:36:48.779 DEBUG:teuthology.orchestra.run.trial187:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/2401a109e2ac85dfa20fc57328da663aa754c875/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-12T08:36:48.842 DEBUG:teuthology.orchestra.run.trial187:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-12T08:36:48.913 DEBUG:teuthology.orchestra.run.trial187:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-12T08:36:48.977 INFO:teuthology.orchestra.run.trial187.stdout:check_obsoletes = 1 2026-02-12T08:36:48.978 DEBUG:teuthology.orchestra.run.trial187:> sudo yum clean all 2026-02-12T08:36:49.125 INFO:teuthology.orchestra.run.trial187.stdout:41 files removed 2026-02-12T08:36:49.139 DEBUG:teuthology.orchestra.run.trial187:> sudo yum -y install ceph-radosgw 2026-02-12T08:36:49.952 INFO:teuthology.orchestra.run.trial187.stdout:ceph packages for x86_64 140 kB/s | 94 kB 00:00 2026-02-12T08:36:50.917 INFO:teuthology.orchestra.run.trial187.stdout:ceph noarch packages 26 kB/s | 24 kB 00:00 2026-02-12T08:36:51.671 INFO:teuthology.orchestra.run.trial187.stdout:ceph source packages 2.8 kB/s | 2.1 kB 00:00 2026-02-12T08:36:52.516 INFO:teuthology.orchestra.run.trial187.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.9 MB 00:00 2026-02-12T08:36:55.812 INFO:teuthology.orchestra.run.trial187.stdout:CentOS Stream 9 - AppStream 9.7 MB/s | 27 MB 00:02 2026-02-12T08:37:06.708 INFO:teuthology.orchestra.run.trial187.stdout:CentOS Stream 9 - CRB 934 kB/s | 7.8 MB 00:08 2026-02-12T08:37:07.531 INFO:teuthology.orchestra.run.trial187.stdout:CentOS Stream 9 - Extras packages 195 kB/s | 20 kB 00:00 2026-02-12T08:37:08.098 INFO:teuthology.orchestra.run.trial187.stdout:Extra Packages for Enterprise Linux 41 MB/s | 20 MB 00:00 2026-02-12T08:37:11.299 INFO:teuthology.orchestra.run.trial187.stdout:lab-extras 1.7 MB/s | 50 kB 00:00 2026-02-12T08:37:12.597 INFO:teuthology.orchestra.run.trial187.stdout:Dependencies resolved. 2026-02-12T08:37:12.599 INFO:teuthology.orchestra.run.trial187.stdout:================================================================================ 2026-02-12T08:37:12.599 INFO:teuthology.orchestra.run.trial187.stdout: Package Arch Version Repository Size 2026-02-12T08:37:12.599 INFO:teuthology.orchestra.run.trial187.stdout:================================================================================ 2026-02-12T08:37:12.599 INFO:teuthology.orchestra.run.trial187.stdout:Installing: 2026-02-12T08:37:12.599 INFO:teuthology.orchestra.run.trial187.stdout: ceph-radosgw x86_64 2:20.3.0-5161.g2401a109.el9 ceph 11 M 2026-02-12T08:37:12.599 INFO:teuthology.orchestra.run.trial187.stdout:Upgrading: 2026-02-12T08:37:12.600 INFO:teuthology.orchestra.run.trial187.stdout: librados2 x86_64 2:20.3.0-5161.g2401a109.el9 ceph 3.3 M 2026-02-12T08:37:12.600 INFO:teuthology.orchestra.run.trial187.stdout: librbd1 x86_64 2:20.3.0-5161.g2401a109.el9 ceph 2.7 M 2026-02-12T08:37:12.600 INFO:teuthology.orchestra.run.trial187.stdout:Installing dependencies: 2026-02-12T08:37:12.600 INFO:teuthology.orchestra.run.trial187.stdout: boost-program-options x86_64 1.75.0-13.el9 appstream 104 k 2026-02-12T08:37:12.600 INFO:teuthology.orchestra.run.trial187.stdout: ceph-base x86_64 2:20.3.0-5161.g2401a109.el9 ceph 5.1 M 2026-02-12T08:37:12.600 INFO:teuthology.orchestra.run.trial187.stdout: ceph-common x86_64 2:20.3.0-5161.g2401a109.el9 ceph 21 M 2026-02-12T08:37:12.600 INFO:teuthology.orchestra.run.trial187.stdout: ceph-selinux x86_64 2:20.3.0-5161.g2401a109.el9 ceph 25 k 2026-02-12T08:37:12.600 INFO:teuthology.orchestra.run.trial187.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2026-02-12T08:37:12.600 INFO:teuthology.orchestra.run.trial187.stdout: libarrow x86_64 9.0.0-15.el9 epel 4.4 M 2026-02-12T08:37:12.600 INFO:teuthology.orchestra.run.trial187.stdout: libarrow-doc noarch 9.0.0-15.el9 epel 25 k 2026-02-12T08:37:12.600 INFO:teuthology.orchestra.run.trial187.stdout: libcephfs2 x86_64 2:20.3.0-5161.g2401a109.el9 ceph 860 k 2026-02-12T08:37:12.600 INFO:teuthology.orchestra.run.trial187.stdout: libnbd x86_64 1.20.3-4.el9 appstream 164 k 2026-02-12T08:37:12.601 INFO:teuthology.orchestra.run.trial187.stdout: liboath x86_64 2.6.12-1.el9 epel 49 k 2026-02-12T08:37:12.601 INFO:teuthology.orchestra.run.trial187.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2026-02-12T08:37:12.601 INFO:teuthology.orchestra.run.trial187.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2026-02-12T08:37:12.601 INFO:teuthology.orchestra.run.trial187.stdout: libradosstriper1 x86_64 2:20.3.0-5161.g2401a109.el9 ceph 263 k 2026-02-12T08:37:12.601 INFO:teuthology.orchestra.run.trial187.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2026-02-12T08:37:12.601 INFO:teuthology.orchestra.run.trial187.stdout: librgw2 x86_64 2:20.3.0-5161.g2401a109.el9 ceph 5.6 M 2026-02-12T08:37:12.601 INFO:teuthology.orchestra.run.trial187.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2026-02-12T08:37:12.601 INFO:teuthology.orchestra.run.trial187.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2026-02-12T08:37:12.601 INFO:teuthology.orchestra.run.trial187.stdout: mailcap noarch 2.1.49-5.el9 baseos 33 k 2026-02-12T08:37:12.601 INFO:teuthology.orchestra.run.trial187.stdout: parquet-libs x86_64 9.0.0-15.el9 epel 838 k 2026-02-12T08:37:12.601 INFO:teuthology.orchestra.run.trial187.stdout: pciutils x86_64 3.7.0-7.el9 baseos 93 k 2026-02-12T08:37:12.601 INFO:teuthology.orchestra.run.trial187.stdout: python3-ceph-argparse x86_64 2:20.3.0-5161.g2401a109.el9 ceph 45 k 2026-02-12T08:37:12.601 INFO:teuthology.orchestra.run.trial187.stdout: python3-ceph-common x86_64 2:20.3.0-5161.g2401a109.el9 ceph 184 k 2026-02-12T08:37:12.602 INFO:teuthology.orchestra.run.trial187.stdout: python3-cephfs x86_64 2:20.3.0-5161.g2401a109.el9 ceph 204 k 2026-02-12T08:37:12.602 INFO:teuthology.orchestra.run.trial187.stdout: python3-rados x86_64 2:20.3.0-5161.g2401a109.el9 ceph 323 k 2026-02-12T08:37:12.602 INFO:teuthology.orchestra.run.trial187.stdout: python3-rbd x86_64 2:20.3.0-5161.g2401a109.el9 ceph 305 k 2026-02-12T08:37:12.602 INFO:teuthology.orchestra.run.trial187.stdout: python3-rgw x86_64 2:20.3.0-5161.g2401a109.el9 ceph 100 k 2026-02-12T08:37:12.602 INFO:teuthology.orchestra.run.trial187.stdout: qatlib x86_64 25.08.0-2.el9 appstream 240 k 2026-02-12T08:37:12.602 INFO:teuthology.orchestra.run.trial187.stdout: qatzip-libs x86_64 1.3.1-1.el9 appstream 66 k 2026-02-12T08:37:12.602 INFO:teuthology.orchestra.run.trial187.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2026-02-12T08:37:12.602 INFO:teuthology.orchestra.run.trial187.stdout: thrift x86_64 0.15.0-4.el9 epel 1.6 M 2026-02-12T08:37:12.602 INFO:teuthology.orchestra.run.trial187.stdout:Installing weak dependencies: 2026-02-12T08:37:12.602 INFO:teuthology.orchestra.run.trial187.stdout: libcephfs-daemon x86_64 2:20.3.0-5161.g2401a109.el9 ceph 35 k 2026-02-12T08:37:12.602 INFO:teuthology.orchestra.run.trial187.stdout: libcephfs-proxy2 x86_64 2:20.3.0-5161.g2401a109.el9 ceph 24 k 2026-02-12T08:37:12.602 INFO:teuthology.orchestra.run.trial187.stdout: nvme-cli x86_64 2.13-1.el9 baseos 1.1 M 2026-02-12T08:37:12.602 INFO:teuthology.orchestra.run.trial187.stdout: qatlib-service x86_64 25.08.0-2.el9 appstream 37 k 2026-02-12T08:37:12.603 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:37:12.603 INFO:teuthology.orchestra.run.trial187.stdout:Transaction Summary 2026-02-12T08:37:12.603 INFO:teuthology.orchestra.run.trial187.stdout:================================================================================ 2026-02-12T08:37:12.603 INFO:teuthology.orchestra.run.trial187.stdout:Install 35 Packages 2026-02-12T08:37:12.603 INFO:teuthology.orchestra.run.trial187.stdout:Upgrade 2 Packages 2026-02-12T08:37:12.603 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:37:12.604 INFO:teuthology.orchestra.run.trial187.stdout:Total download size: 61 M 2026-02-12T08:37:12.604 INFO:teuthology.orchestra.run.trial187.stdout:Downloading Packages: 2026-02-12T08:37:20.055 INFO:teuthology.orchestra.run.trial187.stdout:(1/37): ceph-base-20.3.0-5161.g2401a109.el9.x86 723 kB/s | 5.1 MB 00:07 2026-02-12T08:37:20.125 INFO:teuthology.orchestra.run.trial187.stdout:(2/37): ceph-selinux-20.3.0-5161.g2401a109.el9. 357 kB/s | 25 kB 00:00 2026-02-12T08:37:20.197 INFO:teuthology.orchestra.run.trial187.stdout:(3/37): libcephfs-daemon-20.3.0-5161.g2401a109. 494 kB/s | 35 kB 00:00 2026-02-12T08:37:20.267 INFO:teuthology.orchestra.run.trial187.stdout:(4/37): libcephfs-proxy2-20.3.0-5161.g2401a109. 341 kB/s | 24 kB 00:00 2026-02-12T08:37:21.067 INFO:teuthology.orchestra.run.trial187.stdout:(5/37): libcephfs2-20.3.0-5161.g2401a109.el9.x8 1.0 MB/s | 860 kB 00:00 2026-02-12T08:37:21.338 INFO:teuthology.orchestra.run.trial187.stdout:(6/37): libradosstriper1-20.3.0-5161.g2401a109. 972 kB/s | 263 kB 00:00 2026-02-12T08:37:25.691 INFO:teuthology.orchestra.run.trial187.stdout:(7/37): ceph-radosgw-20.3.0-5161.g2401a109.el9. 890 kB/s | 11 MB 00:12 2026-02-12T08:37:25.759 INFO:teuthology.orchestra.run.trial187.stdout:(8/37): python3-ceph-argparse-20.3.0-5161.g2401 664 kB/s | 45 kB 00:00 2026-02-12T08:37:25.954 INFO:teuthology.orchestra.run.trial187.stdout:(9/37): python3-ceph-common-20.3.0-5161.g2401a1 947 kB/s | 184 kB 00:00 2026-02-12T08:37:26.157 INFO:teuthology.orchestra.run.trial187.stdout:(10/37): python3-cephfs-20.3.0-5161.g2401a109.e 1.0 MB/s | 204 kB 00:00 2026-02-12T08:37:26.421 INFO:teuthology.orchestra.run.trial187.stdout:(11/37): python3-rados-20.3.0-5161.g2401a109.el 1.2 MB/s | 323 kB 00:00 2026-02-12T08:37:26.694 INFO:teuthology.orchestra.run.trial187.stdout:(12/37): python3-rbd-20.3.0-5161.g2401a109.el9. 1.1 MB/s | 305 kB 00:00 2026-02-12T08:37:26.833 INFO:teuthology.orchestra.run.trial187.stdout:(13/37): python3-rgw-20.3.0-5161.g2401a109.el9. 720 kB/s | 100 kB 00:00 2026-02-12T08:37:27.228 INFO:teuthology.orchestra.run.trial187.stdout:(14/37): mailcap-2.1.49-5.el9.noarch.rpm 84 kB/s | 33 kB 00:00 2026-02-12T08:37:27.525 INFO:teuthology.orchestra.run.trial187.stdout:(15/37): librgw2-20.3.0-5161.g2401a109.el9.x86_ 932 kB/s | 5.6 MB 00:06 2026-02-12T08:37:28.062 INFO:teuthology.orchestra.run.trial187.stdout:(16/37): pciutils-3.7.0-7.el9.x86_64.rpm 173 kB/s | 93 kB 00:00 2026-02-12T08:37:28.293 INFO:teuthology.orchestra.run.trial187.stdout:[MIRROR] boost-program-options-1.75.0-13.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/boost-program-options-1.75.0-13.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-12T08:37:28.523 INFO:teuthology.orchestra.run.trial187.stdout:(17/37): nvme-cli-2.13-1.el9.x86_64.rpm 868 kB/s | 1.1 MB 00:01 2026-02-12T08:37:28.677 INFO:teuthology.orchestra.run.trial187.stdout:[MIRROR] libnbd-1.20.3-4.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/libnbd-1.20.3-4.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-12T08:37:28.933 INFO:teuthology.orchestra.run.trial187.stdout:[MIRROR] boost-program-options-1.75.0-13.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/boost-program-options-1.75.0-13.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-12T08:37:29.065 INFO:teuthology.orchestra.run.trial187.stdout:[MIRROR] libnbd-1.20.3-4.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/libnbd-1.20.3-4.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-12T08:37:29.286 INFO:teuthology.orchestra.run.trial187.stdout:[MIRROR] boost-program-options-1.75.0-13.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/boost-program-options-1.75.0-13.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-12T08:37:29.699 INFO:teuthology.orchestra.run.trial187.stdout:(18/37): libnbd-1.20.3-4.el9.x86_64.rpm 139 kB/s | 164 kB 00:01 2026-02-12T08:37:29.853 INFO:teuthology.orchestra.run.trial187.stdout:[MIRROR] libpmemobj-1.12.1-1.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/libpmemobj-1.12.1-1.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-12T08:37:29.907 INFO:teuthology.orchestra.run.trial187.stdout:(19/37): boost-program-options-1.75.0-13.el9.x8 56 kB/s | 104 kB 00:01 2026-02-12T08:37:30.156 INFO:teuthology.orchestra.run.trial187.stdout:(20/37): libpmemobj-1.12.1-1.el9.x86_64.rpm 350 kB/s | 160 kB 00:00 2026-02-12T08:37:30.295 INFO:teuthology.orchestra.run.trial187.stdout:(21/37): librabbitmq-0.11.0-7.el9.x86_64.rpm 116 kB/s | 45 kB 00:00 2026-02-12T08:37:30.752 INFO:teuthology.orchestra.run.trial187.stdout:(22/37): lttng-ust-2.12.0-6.el9.x86_64.rpm 641 kB/s | 292 kB 00:00 2026-02-12T08:37:31.131 INFO:teuthology.orchestra.run.trial187.stdout:(23/37): qatlib-25.08.0-2.el9.x86_64.rpm 631 kB/s | 240 kB 00:00 2026-02-12T08:37:31.284 INFO:teuthology.orchestra.run.trial187.stdout:(24/37): qatlib-service-25.08.0-2.el9.x86_64.rp 243 kB/s | 37 kB 00:00 2026-02-12T08:37:31.375 INFO:teuthology.orchestra.run.trial187.stdout:(25/37): librdkafka-1.6.1-102.el9.x86_64.rpm 544 kB/s | 662 kB 00:01 2026-02-12T08:37:31.437 INFO:teuthology.orchestra.run.trial187.stdout:(26/37): qatzip-libs-1.3.1-1.el9.x86_64.rpm 434 kB/s | 66 kB 00:00 2026-02-12T08:37:32.012 INFO:teuthology.orchestra.run.trial187.stdout:(27/37): gperftools-libs-2.9.1-3.el9.x86_64.rpm 483 kB/s | 308 kB 00:00 2026-02-12T08:37:32.060 INFO:teuthology.orchestra.run.trial187.stdout:(28/37): libarrow-doc-9.0.0-15.el9.noarch.rpm 512 kB/s | 25 kB 00:00 2026-02-12T08:37:32.151 INFO:teuthology.orchestra.run.trial187.stdout:(29/37): liboath-2.6.12-1.el9.x86_64.rpm 542 kB/s | 49 kB 00:00 2026-02-12T08:37:32.245 INFO:teuthology.orchestra.run.trial187.stdout:(30/37): libunwind-1.6.2-1.el9.x86_64.rpm 714 kB/s | 67 kB 00:00 2026-02-12T08:37:32.904 INFO:teuthology.orchestra.run.trial187.stdout:(31/37): parquet-libs-9.0.0-15.el9.x86_64.rpm 1.2 MB/s | 838 kB 00:00 2026-02-12T08:37:33.042 INFO:teuthology.orchestra.run.trial187.stdout:(32/37): re2-20211101-20.el9.x86_64.rpm 1.4 MB/s | 191 kB 00:00 2026-02-12T08:37:33.873 INFO:teuthology.orchestra.run.trial187.stdout:(33/37): thrift-0.15.0-4.el9.x86_64.rpm 1.9 MB/s | 1.6 MB 00:00 2026-02-12T08:37:34.324 INFO:teuthology.orchestra.run.trial187.stdout:(34/37): libarrow-9.0.0-15.el9.x86_64.rpm 1.5 MB/s | 4.4 MB 00:02 2026-02-12T08:37:35.062 INFO:teuthology.orchestra.run.trial187.stdout:(35/37): ceph-common-20.3.0-5161.g2401a109.el9. 958 kB/s | 21 MB 00:22 2026-02-12T08:37:38.162 INFO:teuthology.orchestra.run.trial187.stdout:(36/37): librbd1-20.3.0-5161.g2401a109.el9.x86_ 717 kB/s | 2.7 MB 00:03 2026-02-12T08:37:39.559 INFO:teuthology.orchestra.run.trial187.stdout:(37/37): librados2-20.3.0-5161.g2401a109.el9.x8 592 kB/s | 3.3 MB 00:05 2026-02-12T08:37:39.565 INFO:teuthology.orchestra.run.trial187.stdout:-------------------------------------------------------------------------------- 2026-02-12T08:37:39.565 INFO:teuthology.orchestra.run.trial187.stdout:Total 2.3 MB/s | 61 MB 00:26 2026-02-12T08:37:39.660 INFO:teuthology.orchestra.run.trial187.stdout:Running transaction check 2026-02-12T08:37:39.680 INFO:teuthology.orchestra.run.trial187.stdout:Transaction check succeeded. 2026-02-12T08:37:39.680 INFO:teuthology.orchestra.run.trial187.stdout:Running transaction test 2026-02-12T08:37:39.927 INFO:teuthology.orchestra.run.trial187.stdout:Transaction test succeeded. 2026-02-12T08:37:39.928 INFO:teuthology.orchestra.run.trial187.stdout:Running transaction 2026-02-12T08:37:40.225 INFO:teuthology.orchestra.run.trial187.stdout: Preparing : 1/1 2026-02-12T08:37:40.366 INFO:teuthology.orchestra.run.trial187.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/39 2026-02-12T08:37:40.370 INFO:teuthology.orchestra.run.trial187.stdout: Upgrading : librados2-2:20.3.0-5161.g2401a109.el9.x86_64 2/39 2026-02-12T08:37:40.420 INFO:teuthology.orchestra.run.trial187.stdout: Running scriptlet: librados2-2:20.3.0-5161.g2401a109.el9.x86_64 2/39 2026-02-12T08:37:40.421 INFO:teuthology.orchestra.run.trial187.stdout: Installing : libcephfs2-2:20.3.0-5161.g2401a109.el9.x86_64 3/39 2026-02-12T08:37:40.434 INFO:teuthology.orchestra.run.trial187.stdout: Running scriptlet: libcephfs2-2:20.3.0-5161.g2401a109.el9.x86_64 3/39 2026-02-12T08:37:40.436 INFO:teuthology.orchestra.run.trial187.stdout: Installing : libcephfs-daemon-2:20.3.0-5161.g2401a109.el9.x86_6 4/39 2026-02-12T08:37:40.437 INFO:teuthology.orchestra.run.trial187.stdout: Installing : libcephfs-proxy2-2:20.3.0-5161.g2401a109.el9.x86_6 5/39 2026-02-12T08:37:40.464 INFO:teuthology.orchestra.run.trial187.stdout: Running scriptlet: libcephfs-proxy2-2:20.3.0-5161.g2401a109.el9.x86_6 5/39 2026-02-12T08:37:40.467 INFO:teuthology.orchestra.run.trial187.stdout: Installing : python3-rados-2:20.3.0-5161.g2401a109.el9.x86_64 6/39 2026-02-12T08:37:40.476 INFO:teuthology.orchestra.run.trial187.stdout: Installing : liboath-2.6.12-1.el9.x86_64 7/39 2026-02-12T08:37:40.479 INFO:teuthology.orchestra.run.trial187.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/39 2026-02-12T08:37:40.482 INFO:teuthology.orchestra.run.trial187.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/39 2026-02-12T08:37:40.486 INFO:teuthology.orchestra.run.trial187.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/39 2026-02-12T08:37:40.498 INFO:teuthology.orchestra.run.trial187.stdout: Installing : python3-ceph-argparse-2:20.3.0-5161.g2401a109.el9. 11/39 2026-02-12T08:37:40.526 INFO:teuthology.orchestra.run.trial187.stdout: Installing : python3-cephfs-2:20.3.0-5161.g2401a109.el9.x86_64 12/39 2026-02-12T08:37:40.527 INFO:teuthology.orchestra.run.trial187.stdout: Installing : libradosstriper1-2:20.3.0-5161.g2401a109.el9.x86_6 13/39 2026-02-12T08:37:40.551 INFO:teuthology.orchestra.run.trial187.stdout: Running scriptlet: libradosstriper1-2:20.3.0-5161.g2401a109.el9.x86_6 13/39 2026-02-12T08:37:40.555 INFO:teuthology.orchestra.run.trial187.stdout: Installing : thrift-0.15.0-4.el9.x86_64 14/39 2026-02-12T08:37:40.589 INFO:teuthology.orchestra.run.trial187.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/39 2026-02-12T08:37:40.593 INFO:teuthology.orchestra.run.trial187.stdout: Installing : libarrow-9.0.0-15.el9.x86_64 16/39 2026-02-12T08:37:40.599 INFO:teuthology.orchestra.run.trial187.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/39 2026-02-12T08:37:40.602 INFO:teuthology.orchestra.run.trial187.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/39 2026-02-12T08:37:40.606 INFO:teuthology.orchestra.run.trial187.stdout: Installing : libarrow-doc-9.0.0-15.el9.noarch 19/39 2026-02-12T08:37:40.720 INFO:teuthology.orchestra.run.trial187.stdout: Installing : libnbd-1.20.3-4.el9.x86_64 20/39 2026-02-12T08:37:40.721 INFO:teuthology.orchestra.run.trial187.stdout: Upgrading : librbd1-2:20.3.0-5161.g2401a109.el9.x86_64 21/39 2026-02-12T08:37:40.748 INFO:teuthology.orchestra.run.trial187.stdout: Running scriptlet: librbd1-2:20.3.0-5161.g2401a109.el9.x86_64 21/39 2026-02-12T08:37:40.750 INFO:teuthology.orchestra.run.trial187.stdout: Installing : python3-rbd-2:20.3.0-5161.g2401a109.el9.x86_64 22/39 2026-02-12T08:37:40.756 INFO:teuthology.orchestra.run.trial187.stdout: Installing : boost-program-options-1.75.0-13.el9.x86_64 23/39 2026-02-12T08:37:41.006 INFO:teuthology.orchestra.run.trial187.stdout: Installing : parquet-libs-9.0.0-15.el9.x86_64 24/39 2026-02-12T08:37:41.007 INFO:teuthology.orchestra.run.trial187.stdout: Installing : librgw2-2:20.3.0-5161.g2401a109.el9.x86_64 25/39 2026-02-12T08:37:41.024 INFO:teuthology.orchestra.run.trial187.stdout: Running scriptlet: librgw2-2:20.3.0-5161.g2401a109.el9.x86_64 25/39 2026-02-12T08:37:41.028 INFO:teuthology.orchestra.run.trial187.stdout: Installing : python3-rgw-2:20.3.0-5161.g2401a109.el9.x86_64 26/39 2026-02-12T08:37:41.033 INFO:teuthology.orchestra.run.trial187.stdout: Installing : pciutils-3.7.0-7.el9.x86_64 27/39 2026-02-12T08:37:41.035 INFO:teuthology.orchestra.run.trial187.stdout: Installing : qatlib-25.08.0-2.el9.x86_64 28/39 2026-02-12T08:37:41.036 INFO:teuthology.orchestra.run.trial187.stdout: Installing : qatlib-service-25.08.0-2.el9.x86_64 29/39 2026-02-12T08:37:41.055 INFO:teuthology.orchestra.run.trial187.stdout: Running scriptlet: qatlib-service-25.08.0-2.el9.x86_64 29/39 2026-02-12T08:37:41.167 INFO:teuthology.orchestra.run.trial187.stdout: Installing : qatzip-libs-1.3.1-1.el9.x86_64 30/39 2026-02-12T08:37:41.169 INFO:teuthology.orchestra.run.trial187.stdout: Installing : nvme-cli-2.13-1.el9.x86_64 31/39 2026-02-12T08:37:41.450 INFO:teuthology.orchestra.run.trial187.stdout: Running scriptlet: nvme-cli-2.13-1.el9.x86_64 31/39 2026-02-12T08:37:41.450 INFO:teuthology.orchestra.run.trial187.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service → /usr/lib/systemd/system/nvmefc-boot-connections.service. 2026-02-12T08:37:41.450 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:37:41.493 INFO:teuthology.orchestra.run.trial187.stdout: Installing : mailcap-2.1.49-5.el9.noarch 32/39 2026-02-12T08:37:41.496 INFO:teuthology.orchestra.run.trial187.stdout: Installing : python3-ceph-common-2:20.3.0-5161.g2401a109.el9.x8 33/39 2026-02-12T08:37:42.434 INFO:teuthology.orchestra.run.trial187.stdout: Running scriptlet: ceph-common-2:20.3.0-5161.g2401a109.el9.x86_64 34/39 2026-02-12T08:37:42.437 INFO:teuthology.orchestra.run.trial187.stdout: Installing : ceph-common-2:20.3.0-5161.g2401a109.el9.x86_64 34/39 2026-02-12T08:37:42.698 INFO:teuthology.orchestra.run.trial187.stdout: Running scriptlet: ceph-common-2:20.3.0-5161.g2401a109.el9.x86_64 34/39 2026-02-12T08:37:42.701 INFO:teuthology.orchestra.run.trial187.stdout: Installing : ceph-base-2:20.3.0-5161.g2401a109.el9.x86_64 35/39 2026-02-12T08:37:42.739 INFO:teuthology.orchestra.run.trial187.stdout: Running scriptlet: ceph-base-2:20.3.0-5161.g2401a109.el9.x86_64 35/39 2026-02-12T08:37:42.740 INFO:teuthology.orchestra.run.trial187.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2026-02-12T08:37:42.740 INFO:teuthology.orchestra.run.trial187.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2026-02-12T08:37:42.740 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:37:42.743 INFO:teuthology.orchestra.run.trial187.stdout: Installing : ceph-selinux-2:20.3.0-5161.g2401a109.el9.x86_64 36/39 2026-02-12T08:37:47.250 INFO:teuthology.orchestra.run.trial187.stdout: Running scriptlet: ceph-selinux-2:20.3.0-5161.g2401a109.el9.x86_64 36/39 2026-02-12T08:37:47.250 INFO:teuthology.orchestra.run.trial187.stdout:skipping the directory /sys 2026-02-12T08:37:47.250 INFO:teuthology.orchestra.run.trial187.stdout:skipping the directory /proc 2026-02-12T08:37:47.250 INFO:teuthology.orchestra.run.trial187.stdout:skipping the directory /mnt 2026-02-12T08:37:47.250 INFO:teuthology.orchestra.run.trial187.stdout:skipping the directory /var/tmp 2026-02-12T08:37:47.250 INFO:teuthology.orchestra.run.trial187.stdout:skipping the directory /home 2026-02-12T08:37:47.251 INFO:teuthology.orchestra.run.trial187.stdout:skipping the directory /root 2026-02-12T08:37:47.251 INFO:teuthology.orchestra.run.trial187.stdout:skipping the directory /tmp 2026-02-12T08:37:47.251 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:37:47.740 INFO:teuthology.orchestra.run.trial187.stdout: Installing : ceph-radosgw-2:20.3.0-5161.g2401a109.el9.x86_64 37/39 2026-02-12T08:37:47.763 INFO:teuthology.orchestra.run.trial187.stdout: Running scriptlet: ceph-radosgw-2:20.3.0-5161.g2401a109.el9.x86_64 37/39 2026-02-12T08:37:47.763 INFO:teuthology.orchestra.run.trial187.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-12T08:37:47.763 INFO:teuthology.orchestra.run.trial187.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2026-02-12T08:37:47.763 INFO:teuthology.orchestra.run.trial187.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-12T08:37:47.763 INFO:teuthology.orchestra.run.trial187.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-12T08:37:47.764 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:37:47.765 INFO:teuthology.orchestra.run.trial187.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 38/39 2026-02-12T08:37:47.778 INFO:teuthology.orchestra.run.trial187.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 38/39 2026-02-12T08:37:47.778 INFO:teuthology.orchestra.run.trial187.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 39/39 2026-02-12T08:37:47.831 INFO:teuthology.orchestra.run.trial187.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 39/39 2026-02-12T08:37:47.832 INFO:teuthology.orchestra.run.trial187.stdout:Creating group 'qat' with GID 992. 2026-02-12T08:37:47.832 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:37:48.120 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : ceph-base-2:20.3.0-5161.g2401a109.el9.x86_64 1/39 2026-02-12T08:37:48.120 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : ceph-common-2:20.3.0-5161.g2401a109.el9.x86_64 2/39 2026-02-12T08:37:48.121 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : ceph-radosgw-2:20.3.0-5161.g2401a109.el9.x86_64 3/39 2026-02-12T08:37:48.121 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : ceph-selinux-2:20.3.0-5161.g2401a109.el9.x86_64 4/39 2026-02-12T08:37:48.121 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : libcephfs-daemon-2:20.3.0-5161.g2401a109.el9.x86_6 5/39 2026-02-12T08:37:48.121 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : libcephfs-proxy2-2:20.3.0-5161.g2401a109.el9.x86_6 6/39 2026-02-12T08:37:48.121 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : libcephfs2-2:20.3.0-5161.g2401a109.el9.x86_64 7/39 2026-02-12T08:37:48.121 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : libradosstriper1-2:20.3.0-5161.g2401a109.el9.x86_6 8/39 2026-02-12T08:37:48.121 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : librgw2-2:20.3.0-5161.g2401a109.el9.x86_64 9/39 2026-02-12T08:37:48.121 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : python3-ceph-argparse-2:20.3.0-5161.g2401a109.el9. 10/39 2026-02-12T08:37:48.121 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : python3-ceph-common-2:20.3.0-5161.g2401a109.el9.x8 11/39 2026-02-12T08:37:48.121 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : python3-cephfs-2:20.3.0-5161.g2401a109.el9.x86_64 12/39 2026-02-12T08:37:48.121 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : python3-rados-2:20.3.0-5161.g2401a109.el9.x86_64 13/39 2026-02-12T08:37:48.121 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : python3-rbd-2:20.3.0-5161.g2401a109.el9.x86_64 14/39 2026-02-12T08:37:48.122 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : python3-rgw-2:20.3.0-5161.g2401a109.el9.x86_64 15/39 2026-02-12T08:37:48.122 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : mailcap-2.1.49-5.el9.noarch 16/39 2026-02-12T08:37:48.122 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : nvme-cli-2.13-1.el9.x86_64 17/39 2026-02-12T08:37:48.122 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : pciutils-3.7.0-7.el9.x86_64 18/39 2026-02-12T08:37:48.122 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : boost-program-options-1.75.0-13.el9.x86_64 19/39 2026-02-12T08:37:48.122 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : libnbd-1.20.3-4.el9.x86_64 20/39 2026-02-12T08:37:48.122 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 21/39 2026-02-12T08:37:48.122 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 22/39 2026-02-12T08:37:48.122 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 23/39 2026-02-12T08:37:48.122 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 24/39 2026-02-12T08:37:48.122 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : qatlib-25.08.0-2.el9.x86_64 25/39 2026-02-12T08:37:48.122 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : qatlib-service-25.08.0-2.el9.x86_64 26/39 2026-02-12T08:37:48.123 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : qatzip-libs-1.3.1-1.el9.x86_64 27/39 2026-02-12T08:37:48.123 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 28/39 2026-02-12T08:37:48.123 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : libarrow-9.0.0-15.el9.x86_64 29/39 2026-02-12T08:37:48.123 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : libarrow-doc-9.0.0-15.el9.noarch 30/39 2026-02-12T08:37:48.123 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : liboath-2.6.12-1.el9.x86_64 31/39 2026-02-12T08:37:48.123 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 32/39 2026-02-12T08:37:48.123 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : parquet-libs-9.0.0-15.el9.x86_64 33/39 2026-02-12T08:37:48.123 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : re2-1:20211101-20.el9.x86_64 34/39 2026-02-12T08:37:48.123 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : thrift-0.15.0-4.el9.x86_64 35/39 2026-02-12T08:37:48.123 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : librados2-2:20.3.0-5161.g2401a109.el9.x86_64 36/39 2026-02-12T08:37:48.123 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 37/39 2026-02-12T08:37:48.123 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : librbd1-2:20.3.0-5161.g2401a109.el9.x86_64 38/39 2026-02-12T08:37:48.182 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 39/39 2026-02-12T08:37:48.182 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:37:48.182 INFO:teuthology.orchestra.run.trial187.stdout:Upgraded: 2026-02-12T08:37:48.182 INFO:teuthology.orchestra.run.trial187.stdout: librados2-2:20.3.0-5161.g2401a109.el9.x86_64 2026-02-12T08:37:48.182 INFO:teuthology.orchestra.run.trial187.stdout: librbd1-2:20.3.0-5161.g2401a109.el9.x86_64 2026-02-12T08:37:48.182 INFO:teuthology.orchestra.run.trial187.stdout:Installed: 2026-02-12T08:37:48.182 INFO:teuthology.orchestra.run.trial187.stdout: boost-program-options-1.75.0-13.el9.x86_64 2026-02-12T08:37:48.182 INFO:teuthology.orchestra.run.trial187.stdout: ceph-base-2:20.3.0-5161.g2401a109.el9.x86_64 2026-02-12T08:37:48.183 INFO:teuthology.orchestra.run.trial187.stdout: ceph-common-2:20.3.0-5161.g2401a109.el9.x86_64 2026-02-12T08:37:48.183 INFO:teuthology.orchestra.run.trial187.stdout: ceph-radosgw-2:20.3.0-5161.g2401a109.el9.x86_64 2026-02-12T08:37:48.183 INFO:teuthology.orchestra.run.trial187.stdout: ceph-selinux-2:20.3.0-5161.g2401a109.el9.x86_64 2026-02-12T08:37:48.183 INFO:teuthology.orchestra.run.trial187.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2026-02-12T08:37:48.183 INFO:teuthology.orchestra.run.trial187.stdout: libarrow-9.0.0-15.el9.x86_64 2026-02-12T08:37:48.183 INFO:teuthology.orchestra.run.trial187.stdout: libarrow-doc-9.0.0-15.el9.noarch 2026-02-12T08:37:48.183 INFO:teuthology.orchestra.run.trial187.stdout: libcephfs-daemon-2:20.3.0-5161.g2401a109.el9.x86_64 2026-02-12T08:37:48.183 INFO:teuthology.orchestra.run.trial187.stdout: libcephfs-proxy2-2:20.3.0-5161.g2401a109.el9.x86_64 2026-02-12T08:37:48.183 INFO:teuthology.orchestra.run.trial187.stdout: libcephfs2-2:20.3.0-5161.g2401a109.el9.x86_64 2026-02-12T08:37:48.183 INFO:teuthology.orchestra.run.trial187.stdout: libnbd-1.20.3-4.el9.x86_64 2026-02-12T08:37:48.183 INFO:teuthology.orchestra.run.trial187.stdout: liboath-2.6.12-1.el9.x86_64 2026-02-12T08:37:48.183 INFO:teuthology.orchestra.run.trial187.stdout: libpmemobj-1.12.1-1.el9.x86_64 2026-02-12T08:37:48.183 INFO:teuthology.orchestra.run.trial187.stdout: librabbitmq-0.11.0-7.el9.x86_64 2026-02-12T08:37:48.184 INFO:teuthology.orchestra.run.trial187.stdout: libradosstriper1-2:20.3.0-5161.g2401a109.el9.x86_64 2026-02-12T08:37:48.184 INFO:teuthology.orchestra.run.trial187.stdout: librdkafka-1.6.1-102.el9.x86_64 2026-02-12T08:37:48.184 INFO:teuthology.orchestra.run.trial187.stdout: librgw2-2:20.3.0-5161.g2401a109.el9.x86_64 2026-02-12T08:37:48.184 INFO:teuthology.orchestra.run.trial187.stdout: libunwind-1.6.2-1.el9.x86_64 2026-02-12T08:37:48.184 INFO:teuthology.orchestra.run.trial187.stdout: lttng-ust-2.12.0-6.el9.x86_64 2026-02-12T08:37:48.184 INFO:teuthology.orchestra.run.trial187.stdout: mailcap-2.1.49-5.el9.noarch 2026-02-12T08:37:48.184 INFO:teuthology.orchestra.run.trial187.stdout: nvme-cli-2.13-1.el9.x86_64 2026-02-12T08:37:48.184 INFO:teuthology.orchestra.run.trial187.stdout: parquet-libs-9.0.0-15.el9.x86_64 2026-02-12T08:37:48.184 INFO:teuthology.orchestra.run.trial187.stdout: pciutils-3.7.0-7.el9.x86_64 2026-02-12T08:37:48.184 INFO:teuthology.orchestra.run.trial187.stdout: python3-ceph-argparse-2:20.3.0-5161.g2401a109.el9.x86_64 2026-02-12T08:37:48.184 INFO:teuthology.orchestra.run.trial187.stdout: python3-ceph-common-2:20.3.0-5161.g2401a109.el9.x86_64 2026-02-12T08:37:48.184 INFO:teuthology.orchestra.run.trial187.stdout: python3-cephfs-2:20.3.0-5161.g2401a109.el9.x86_64 2026-02-12T08:37:48.185 INFO:teuthology.orchestra.run.trial187.stdout: python3-rados-2:20.3.0-5161.g2401a109.el9.x86_64 2026-02-12T08:37:48.185 INFO:teuthology.orchestra.run.trial187.stdout: python3-rbd-2:20.3.0-5161.g2401a109.el9.x86_64 2026-02-12T08:37:48.185 INFO:teuthology.orchestra.run.trial187.stdout: python3-rgw-2:20.3.0-5161.g2401a109.el9.x86_64 2026-02-12T08:37:48.185 INFO:teuthology.orchestra.run.trial187.stdout: qatlib-25.08.0-2.el9.x86_64 2026-02-12T08:37:48.185 INFO:teuthology.orchestra.run.trial187.stdout: qatlib-service-25.08.0-2.el9.x86_64 2026-02-12T08:37:48.185 INFO:teuthology.orchestra.run.trial187.stdout: qatzip-libs-1.3.1-1.el9.x86_64 2026-02-12T08:37:48.185 INFO:teuthology.orchestra.run.trial187.stdout: re2-1:20211101-20.el9.x86_64 2026-02-12T08:37:48.185 INFO:teuthology.orchestra.run.trial187.stdout: thrift-0.15.0-4.el9.x86_64 2026-02-12T08:37:48.185 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:37:48.185 INFO:teuthology.orchestra.run.trial187.stdout:Complete! 2026-02-12T08:37:48.222 DEBUG:teuthology.orchestra.run.trial187:> sudo yum -y install ceph-test 2026-02-12T08:37:48.523 INFO:teuthology.orchestra.run.trial187.stdout:Last metadata expiration check: 0:00:37 ago on Thu 12 Feb 2026 08:37:11 AM UTC. 2026-02-12T08:37:48.608 INFO:teuthology.orchestra.run.trial187.stdout:Dependencies resolved. 2026-02-12T08:37:48.609 INFO:teuthology.orchestra.run.trial187.stdout:================================================================================ 2026-02-12T08:37:48.609 INFO:teuthology.orchestra.run.trial187.stdout: Package Arch Version Repository Size 2026-02-12T08:37:48.609 INFO:teuthology.orchestra.run.trial187.stdout:================================================================================ 2026-02-12T08:37:48.609 INFO:teuthology.orchestra.run.trial187.stdout:Installing: 2026-02-12T08:37:48.609 INFO:teuthology.orchestra.run.trial187.stdout: ceph-test x86_64 2:20.3.0-5161.g2401a109.el9 ceph 126 M 2026-02-12T08:37:48.609 INFO:teuthology.orchestra.run.trial187.stdout:Installing dependencies: 2026-02-12T08:37:48.609 INFO:teuthology.orchestra.run.trial187.stdout: hwloc-libs x86_64 2.4.1-6.el9 baseos 2.1 M 2026-02-12T08:37:48.610 INFO:teuthology.orchestra.run.trial187.stdout: libasan8 x86_64 15.1.1-2.5.el9 appstream 523 k 2026-02-12T08:37:48.610 INFO:teuthology.orchestra.run.trial187.stdout: libatomic x86_64 11.5.0-14.el9 baseos 23 k 2026-02-12T08:37:48.610 INFO:teuthology.orchestra.run.trial187.stdout: libcephsqlite x86_64 2:20.3.0-5161.g2401a109.el9 ceph 153 k 2026-02-12T08:37:48.610 INFO:teuthology.orchestra.run.trial187.stdout: libubsan x86_64 11.5.0-14.el9 appstream 176 k 2026-02-12T08:37:48.610 INFO:teuthology.orchestra.run.trial187.stdout: libxslt x86_64 1.1.34-12.el9 appstream 233 k 2026-02-12T08:37:48.610 INFO:teuthology.orchestra.run.trial187.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2026-02-12T08:37:48.610 INFO:teuthology.orchestra.run.trial187.stdout: protobuf x86_64 3.14.0-17.el9 appstream 1.0 M 2026-02-12T08:37:48.610 INFO:teuthology.orchestra.run.trial187.stdout: socat x86_64 1.7.4.1-8.el9 appstream 303 k 2026-02-12T08:37:48.610 INFO:teuthology.orchestra.run.trial187.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2026-02-12T08:37:48.610 INFO:teuthology.orchestra.run.trial187.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2026-02-12T08:37:48.610 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:37:48.610 INFO:teuthology.orchestra.run.trial187.stdout:Transaction Summary 2026-02-12T08:37:48.611 INFO:teuthology.orchestra.run.trial187.stdout:================================================================================ 2026-02-12T08:37:48.611 INFO:teuthology.orchestra.run.trial187.stdout:Install 12 Packages 2026-02-12T08:37:48.611 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:37:48.611 INFO:teuthology.orchestra.run.trial187.stdout:Total download size: 131 M 2026-02-12T08:37:48.612 INFO:teuthology.orchestra.run.trial187.stdout:Installed size: 976 M 2026-02-12T08:37:48.612 INFO:teuthology.orchestra.run.trial187.stdout:Downloading Packages: 2026-02-12T08:37:49.416 INFO:teuthology.orchestra.run.trial187.stdout:(1/12): hwloc-libs-2.4.1-6.el9.x86_64.rpm 29 MB/s | 2.1 MB 00:00 2026-02-12T08:37:49.425 INFO:teuthology.orchestra.run.trial187.stdout:(2/12): libatomic-11.5.0-14.el9.x86_64.rpm 2.6 MB/s | 23 kB 00:00 2026-02-12T08:37:49.434 INFO:teuthology.orchestra.run.trial187.stdout:(3/12): lksctp-tools-1.0.19-2.el9.x86_64.rpm 9.5 MB/s | 94 kB 00:00 2026-02-12T08:37:49.798 INFO:teuthology.orchestra.run.trial187.stdout:(4/12): libcephsqlite-20.3.0-5161.g2401a109.el9 337 kB/s | 153 kB 00:00 2026-02-12T08:37:50.356 INFO:teuthology.orchestra.run.trial187.stdout:(5/12): libubsan-11.5.0-14.el9.x86_64.rpm 316 kB/s | 176 kB 00:00 2026-02-12T08:37:50.468 INFO:teuthology.orchestra.run.trial187.stdout:(6/12): libasan8-15.1.1-2.5.el9.x86_64.rpm 507 kB/s | 523 kB 00:01 2026-02-12T08:37:50.680 INFO:teuthology.orchestra.run.trial187.stdout:(7/12): libxslt-1.1.34-12.el9.x86_64.rpm 720 kB/s | 233 kB 00:00 2026-02-12T08:37:51.080 INFO:teuthology.orchestra.run.trial187.stdout:(8/12): socat-1.7.4.1-8.el9.x86_64.rpm 759 kB/s | 303 kB 00:00 2026-02-12T08:37:51.177 INFO:teuthology.orchestra.run.trial187.stdout:(9/12): xmlstarlet-1.6.1-20.el9.x86_64.rpm 659 kB/s | 64 kB 00:00 2026-02-12T08:37:51.213 INFO:teuthology.orchestra.run.trial187.stdout:(10/12): yaml-cpp-0.6.3-5.el9.x86_64.rpm 3.3 MB/s | 122 kB 00:00 2026-02-12T08:37:51.579 INFO:teuthology.orchestra.run.trial187.stdout:(11/12): protobuf-3.14.0-17.el9.x86_64.rpm 926 kB/s | 1.0 MB 00:01 2026-02-12T08:39:53.128 INFO:teuthology.orchestra.run.trial187.stdout:(12/12): ceph-test-20.3.0-5161.g2401a109.el9.x8 1.0 MB/s | 126 MB 02:03 2026-02-12T08:39:53.130 INFO:teuthology.orchestra.run.trial187.stdout:-------------------------------------------------------------------------------- 2026-02-12T08:39:53.130 INFO:teuthology.orchestra.run.trial187.stdout:Total 1.0 MB/s | 131 MB 02:04 2026-02-12T08:39:53.198 INFO:teuthology.orchestra.run.trial187.stdout:Running transaction check 2026-02-12T08:39:53.209 INFO:teuthology.orchestra.run.trial187.stdout:Transaction check succeeded. 2026-02-12T08:39:53.209 INFO:teuthology.orchestra.run.trial187.stdout:Running transaction test 2026-02-12T08:39:53.635 INFO:teuthology.orchestra.run.trial187.stdout:Transaction test succeeded. 2026-02-12T08:39:53.635 INFO:teuthology.orchestra.run.trial187.stdout:Running transaction 2026-02-12T08:39:54.089 INFO:teuthology.orchestra.run.trial187.stdout: Preparing : 1/1 2026-02-12T08:39:54.098 INFO:teuthology.orchestra.run.trial187.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/12 2026-02-12T08:39:54.106 INFO:teuthology.orchestra.run.trial187.stdout: Installing : socat-1.7.4.1-8.el9.x86_64 2/12 2026-02-12T08:39:54.111 INFO:teuthology.orchestra.run.trial187.stdout: Installing : protobuf-3.14.0-17.el9.x86_64 3/12 2026-02-12T08:39:54.116 INFO:teuthology.orchestra.run.trial187.stdout: Installing : libxslt-1.1.34-12.el9.x86_64 4/12 2026-02-12T08:39:54.118 INFO:teuthology.orchestra.run.trial187.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 5/12 2026-02-12T08:39:54.122 INFO:teuthology.orchestra.run.trial187.stdout: Installing : libubsan-11.5.0-14.el9.x86_64 6/12 2026-02-12T08:39:54.123 INFO:teuthology.orchestra.run.trial187.stdout: Installing : libasan8-15.1.1-2.5.el9.x86_64 7/12 2026-02-12T08:39:54.140 INFO:teuthology.orchestra.run.trial187.stdout: Running scriptlet: libasan8-15.1.1-2.5.el9.x86_64 7/12 2026-02-12T08:39:54.142 INFO:teuthology.orchestra.run.trial187.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 8/12 2026-02-12T08:39:54.152 INFO:teuthology.orchestra.run.trial187.stdout: Installing : libatomic-11.5.0-14.el9.x86_64 9/12 2026-02-12T08:39:54.161 INFO:teuthology.orchestra.run.trial187.stdout: Installing : hwloc-libs-2.4.1-6.el9.x86_64 10/12 2026-02-12T08:39:54.161 INFO:teuthology.orchestra.run.trial187.stdout: Installing : libcephsqlite-2:20.3.0-5161.g2401a109.el9.x86_64 11/12 2026-02-12T08:40:01.040 INFO:teuthology.orchestra.run.trial187.stdout: Running scriptlet: libcephsqlite-2:20.3.0-5161.g2401a109.el9.x86_64 11/12 2026-02-12T08:40:01.058 INFO:teuthology.orchestra.run.trial187.stdout: Installing : ceph-test-2:20.3.0-5161.g2401a109.el9.x86_64 12/12 2026-02-12T08:40:01.843 INFO:teuthology.orchestra.run.trial187.stdout: Running scriptlet: ceph-test-2:20.3.0-5161.g2401a109.el9.x86_64 12/12 2026-02-12T08:40:01.843 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : ceph-test-2:20.3.0-5161.g2401a109.el9.x86_64 1/12 2026-02-12T08:40:01.843 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : libcephsqlite-2:20.3.0-5161.g2401a109.el9.x86_64 2/12 2026-02-12T08:40:01.843 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : hwloc-libs-2.4.1-6.el9.x86_64 3/12 2026-02-12T08:40:01.843 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : libatomic-11.5.0-14.el9.x86_64 4/12 2026-02-12T08:40:01.843 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 5/12 2026-02-12T08:40:01.844 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : libasan8-15.1.1-2.5.el9.x86_64 6/12 2026-02-12T08:40:01.844 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : libubsan-11.5.0-14.el9.x86_64 7/12 2026-02-12T08:40:01.844 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : libxslt-1.1.34-12.el9.x86_64 8/12 2026-02-12T08:40:01.844 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : protobuf-3.14.0-17.el9.x86_64 9/12 2026-02-12T08:40:01.844 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : socat-1.7.4.1-8.el9.x86_64 10/12 2026-02-12T08:40:01.844 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 11/12 2026-02-12T08:40:01.898 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 12/12 2026-02-12T08:40:01.898 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:40:01.898 INFO:teuthology.orchestra.run.trial187.stdout:Installed: 2026-02-12T08:40:01.898 INFO:teuthology.orchestra.run.trial187.stdout: ceph-test-2:20.3.0-5161.g2401a109.el9.x86_64 2026-02-12T08:40:01.898 INFO:teuthology.orchestra.run.trial187.stdout: hwloc-libs-2.4.1-6.el9.x86_64 2026-02-12T08:40:01.898 INFO:teuthology.orchestra.run.trial187.stdout: libasan8-15.1.1-2.5.el9.x86_64 2026-02-12T08:40:01.898 INFO:teuthology.orchestra.run.trial187.stdout: libatomic-11.5.0-14.el9.x86_64 2026-02-12T08:40:01.899 INFO:teuthology.orchestra.run.trial187.stdout: libcephsqlite-2:20.3.0-5161.g2401a109.el9.x86_64 2026-02-12T08:40:01.899 INFO:teuthology.orchestra.run.trial187.stdout: libubsan-11.5.0-14.el9.x86_64 2026-02-12T08:40:01.899 INFO:teuthology.orchestra.run.trial187.stdout: libxslt-1.1.34-12.el9.x86_64 2026-02-12T08:40:01.899 INFO:teuthology.orchestra.run.trial187.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2026-02-12T08:40:01.899 INFO:teuthology.orchestra.run.trial187.stdout: protobuf-3.14.0-17.el9.x86_64 2026-02-12T08:40:01.899 INFO:teuthology.orchestra.run.trial187.stdout: socat-1.7.4.1-8.el9.x86_64 2026-02-12T08:40:01.899 INFO:teuthology.orchestra.run.trial187.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2026-02-12T08:40:01.899 INFO:teuthology.orchestra.run.trial187.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2026-02-12T08:40:01.899 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:40:01.899 INFO:teuthology.orchestra.run.trial187.stdout:Complete! 2026-02-12T08:40:01.939 DEBUG:teuthology.orchestra.run.trial187:> sudo yum -y install ceph 2026-02-12T08:40:02.242 INFO:teuthology.orchestra.run.trial187.stdout:Last metadata expiration check: 0:02:51 ago on Thu 12 Feb 2026 08:37:11 AM UTC. 2026-02-12T08:40:02.329 INFO:teuthology.orchestra.run.trial187.stdout:Dependencies resolved. 2026-02-12T08:40:02.332 INFO:teuthology.orchestra.run.trial187.stdout:======================================================================================= 2026-02-12T08:40:02.333 INFO:teuthology.orchestra.run.trial187.stdout: Package Arch Version Repository Size 2026-02-12T08:40:02.333 INFO:teuthology.orchestra.run.trial187.stdout:======================================================================================= 2026-02-12T08:40:02.333 INFO:teuthology.orchestra.run.trial187.stdout:Installing: 2026-02-12T08:40:02.333 INFO:teuthology.orchestra.run.trial187.stdout: ceph x86_64 2:20.3.0-5161.g2401a109.el9 ceph 6.5 k 2026-02-12T08:40:02.333 INFO:teuthology.orchestra.run.trial187.stdout:Installing dependencies: 2026-02-12T08:40:02.333 INFO:teuthology.orchestra.run.trial187.stdout: abseil-cpp x86_64 20211102.0-4.el9 epel 551 k 2026-02-12T08:40:02.333 INFO:teuthology.orchestra.run.trial187.stdout: ceph-grafana-dashboards noarch 2:20.3.0-5161.g2401a109.el9 ceph-noarch 57 k 2026-02-12T08:40:02.333 INFO:teuthology.orchestra.run.trial187.stdout: ceph-mds x86_64 2:20.3.0-5161.g2401a109.el9 ceph 2.0 M 2026-02-12T08:40:02.333 INFO:teuthology.orchestra.run.trial187.stdout: ceph-mgr x86_64 2:20.3.0-5161.g2401a109.el9 ceph 937 k 2026-02-12T08:40:02.333 INFO:teuthology.orchestra.run.trial187.stdout: ceph-mgr-modules-core noarch 2:20.3.0-5161.g2401a109.el9 ceph-noarch 297 k 2026-02-12T08:40:02.334 INFO:teuthology.orchestra.run.trial187.stdout: ceph-mon x86_64 2:20.3.0-5161.g2401a109.el9 ceph 4.8 M 2026-02-12T08:40:02.334 INFO:teuthology.orchestra.run.trial187.stdout: ceph-osd x86_64 2:20.3.0-5161.g2401a109.el9 ceph 163 k 2026-02-12T08:40:02.334 INFO:teuthology.orchestra.run.trial187.stdout: ceph-osd-classic x86_64 2:20.3.0-5161.g2401a109.el9 ceph 15 M 2026-02-12T08:40:02.334 INFO:teuthology.orchestra.run.trial187.stdout: ceph-osd-crimson x86_64 2:20.3.0-5161.g2401a109.el9 ceph 87 M 2026-02-12T08:40:02.334 INFO:teuthology.orchestra.run.trial187.stdout: ceph-prometheus-alerts noarch 2:20.3.0-5161.g2401a109.el9 ceph-noarch 17 k 2026-02-12T08:40:02.334 INFO:teuthology.orchestra.run.trial187.stdout: cephadm noarch 2:20.3.0-5161.g2401a109.el9 ceph-noarch 1.0 M 2026-02-12T08:40:02.334 INFO:teuthology.orchestra.run.trial187.stdout: cryptsetup x86_64 2.8.1-2.el9 baseos 350 k 2026-02-12T08:40:02.334 INFO:teuthology.orchestra.run.trial187.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2026-02-12T08:40:02.334 INFO:teuthology.orchestra.run.trial187.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2026-02-12T08:40:02.334 INFO:teuthology.orchestra.run.trial187.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2026-02-12T08:40:02.334 INFO:teuthology.orchestra.run.trial187.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2026-02-12T08:40:02.335 INFO:teuthology.orchestra.run.trial187.stdout: ledmon-libs x86_64 1.1.0-3.el9 baseos 40 k 2026-02-12T08:40:02.335 INFO:teuthology.orchestra.run.trial187.stdout: libconfig x86_64 1.7.2-9.el9 baseos 72 k 2026-02-12T08:40:02.335 INFO:teuthology.orchestra.run.trial187.stdout: libgfortran x86_64 11.5.0-14.el9 baseos 794 k 2026-02-12T08:40:02.335 INFO:teuthology.orchestra.run.trial187.stdout: libquadmath x86_64 11.5.0-14.el9 baseos 184 k 2026-02-12T08:40:02.335 INFO:teuthology.orchestra.run.trial187.stdout: libstoragemgmt x86_64 1.10.1-1.el9 appstream 246 k 2026-02-12T08:40:02.335 INFO:teuthology.orchestra.run.trial187.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2026-02-12T08:40:02.335 INFO:teuthology.orchestra.run.trial187.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2026-02-12T08:40:02.335 INFO:teuthology.orchestra.run.trial187.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2026-02-12T08:40:02.335 INFO:teuthology.orchestra.run.trial187.stdout: openblas x86_64 0.3.29-1.el9 appstream 42 k 2026-02-12T08:40:02.335 INFO:teuthology.orchestra.run.trial187.stdout: openblas-openmp x86_64 0.3.29-1.el9 appstream 5.3 M 2026-02-12T08:40:02.335 INFO:teuthology.orchestra.run.trial187.stdout: protobuf-compiler x86_64 3.14.0-17.el9 crb 862 k 2026-02-12T08:40:02.336 INFO:teuthology.orchestra.run.trial187.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2026-02-12T08:40:02.336 INFO:teuthology.orchestra.run.trial187.stdout: python3-autocommand noarch 2.2.2-8.el9 epel 29 k 2026-02-12T08:40:02.336 INFO:teuthology.orchestra.run.trial187.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2026-02-12T08:40:02.336 INFO:teuthology.orchestra.run.trial187.stdout: python3-backports-tarfile noarch 1.2.0-1.el9 epel 60 k 2026-02-12T08:40:02.336 INFO:teuthology.orchestra.run.trial187.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2026-02-12T08:40:02.336 INFO:teuthology.orchestra.run.trial187.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2026-02-12T08:40:02.336 INFO:teuthology.orchestra.run.trial187.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2026-02-12T08:40:02.336 INFO:teuthology.orchestra.run.trial187.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2026-02-12T08:40:02.336 INFO:teuthology.orchestra.run.trial187.stdout: python3-cheroot noarch 10.0.1-4.el9 epel 173 k 2026-02-12T08:40:02.336 INFO:teuthology.orchestra.run.trial187.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2026-02-12T08:40:02.336 INFO:teuthology.orchestra.run.trial187.stdout: python3-cryptography x86_64 36.0.1-5.el9 baseos 1.2 M 2026-02-12T08:40:02.336 INFO:teuthology.orchestra.run.trial187.stdout: python3-devel x86_64 3.9.25-2.el9 appstream 244 k 2026-02-12T08:40:02.337 INFO:teuthology.orchestra.run.trial187.stdout: python3-google-auth noarch 1:2.45.0-1.el9 epel 254 k 2026-02-12T08:40:02.337 INFO:teuthology.orchestra.run.trial187.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2026-02-12T08:40:02.337 INFO:teuthology.orchestra.run.trial187.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2026-02-12T08:40:02.337 INFO:teuthology.orchestra.run.trial187.stdout: python3-isodate noarch 0.6.1-3.el9 epel 56 k 2026-02-12T08:40:02.337 INFO:teuthology.orchestra.run.trial187.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2026-02-12T08:40:02.337 INFO:teuthology.orchestra.run.trial187.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2026-02-12T08:40:02.337 INFO:teuthology.orchestra.run.trial187.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2026-02-12T08:40:02.337 INFO:teuthology.orchestra.run.trial187.stdout: python3-jaraco-context noarch 6.0.1-3.el9 epel 20 k 2026-02-12T08:40:02.337 INFO:teuthology.orchestra.run.trial187.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2026-02-12T08:40:02.337 INFO:teuthology.orchestra.run.trial187.stdout: python3-jaraco-text noarch 4.0.0-2.el9 epel 26 k 2026-02-12T08:40:02.337 INFO:teuthology.orchestra.run.trial187.stdout: python3-jinja2 noarch 2.11.3-8.el9 appstream 249 k 2026-02-12T08:40:02.337 INFO:teuthology.orchestra.run.trial187.stdout: python3-jmespath noarch 1.0.1-1.el9 appstream 48 k 2026-02-12T08:40:02.338 INFO:teuthology.orchestra.run.trial187.stdout: python3-kubernetes noarch 1:26.1.0-3.el9 epel 1.0 M 2026-02-12T08:40:02.338 INFO:teuthology.orchestra.run.trial187.stdout: python3-libstoragemgmt x86_64 1.10.1-1.el9 appstream 177 k 2026-02-12T08:40:02.338 INFO:teuthology.orchestra.run.trial187.stdout: python3-lxml x86_64 4.6.5-3.el9 appstream 1.2 M 2026-02-12T08:40:02.338 INFO:teuthology.orchestra.run.trial187.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2026-02-12T08:40:02.338 INFO:teuthology.orchestra.run.trial187.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2026-02-12T08:40:02.338 INFO:teuthology.orchestra.run.trial187.stdout: python3-msgpack x86_64 1.0.3-2.el9 epel 86 k 2026-02-12T08:40:02.338 INFO:teuthology.orchestra.run.trial187.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2026-02-12T08:40:02.338 INFO:teuthology.orchestra.run.trial187.stdout: python3-numpy x86_64 1:1.23.5-2.el9 appstream 6.1 M 2026-02-12T08:40:02.338 INFO:teuthology.orchestra.run.trial187.stdout: python3-numpy-f2py x86_64 1:1.23.5-2.el9 appstream 442 k 2026-02-12T08:40:02.338 INFO:teuthology.orchestra.run.trial187.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2026-02-12T08:40:02.338 INFO:teuthology.orchestra.run.trial187.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2026-02-12T08:40:02.338 INFO:teuthology.orchestra.run.trial187.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2026-02-12T08:40:02.338 INFO:teuthology.orchestra.run.trial187.stdout: python3-protobuf noarch 3.14.0-17.el9 appstream 267 k 2026-02-12T08:40:02.339 INFO:teuthology.orchestra.run.trial187.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2026-02-12T08:40:02.339 INFO:teuthology.orchestra.run.trial187.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2026-02-12T08:40:02.339 INFO:teuthology.orchestra.run.trial187.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2026-02-12T08:40:02.339 INFO:teuthology.orchestra.run.trial187.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2026-02-12T08:40:02.339 INFO:teuthology.orchestra.run.trial187.stdout: python3-pyparsing noarch 2.4.7-9.el9 baseos 150 k 2026-02-12T08:40:02.339 INFO:teuthology.orchestra.run.trial187.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2026-02-12T08:40:02.339 INFO:teuthology.orchestra.run.trial187.stdout: python3-requests noarch 2.25.1-10.el9 baseos 126 k 2026-02-12T08:40:02.339 INFO:teuthology.orchestra.run.trial187.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2026-02-12T08:40:02.339 INFO:teuthology.orchestra.run.trial187.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2026-02-12T08:40:02.339 INFO:teuthology.orchestra.run.trial187.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2026-02-12T08:40:02.339 INFO:teuthology.orchestra.run.trial187.stdout: python3-scipy x86_64 1.9.3-2.el9 appstream 19 M 2026-02-12T08:40:02.339 INFO:teuthology.orchestra.run.trial187.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2026-02-12T08:40:02.340 INFO:teuthology.orchestra.run.trial187.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2026-02-12T08:40:02.340 INFO:teuthology.orchestra.run.trial187.stdout: python3-typing-extensions noarch 4.15.0-1.el9 epel 86 k 2026-02-12T08:40:02.340 INFO:teuthology.orchestra.run.trial187.stdout: python3-urllib3 noarch 1.26.5-7.el9 baseos 218 k 2026-02-12T08:40:02.340 INFO:teuthology.orchestra.run.trial187.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2026-02-12T08:40:02.340 INFO:teuthology.orchestra.run.trial187.stdout: python3-xmlsec x86_64 1.3.13-1.el9 epel 48 k 2026-02-12T08:40:02.340 INFO:teuthology.orchestra.run.trial187.stdout: python3-xmltodict noarch 0.12.0-15.el9 epel 22 k 2026-02-12T08:40:02.340 INFO:teuthology.orchestra.run.trial187.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2026-02-12T08:40:02.340 INFO:teuthology.orchestra.run.trial187.stdout: unzip x86_64 6.0-59.el9 baseos 182 k 2026-02-12T08:40:02.340 INFO:teuthology.orchestra.run.trial187.stdout: xmlsec1 x86_64 1.2.29-13.el9 appstream 189 k 2026-02-12T08:40:02.340 INFO:teuthology.orchestra.run.trial187.stdout: xmlsec1-openssl x86_64 1.2.29-13.el9 appstream 90 k 2026-02-12T08:40:02.340 INFO:teuthology.orchestra.run.trial187.stdout: zip x86_64 3.0-35.el9 baseos 266 k 2026-02-12T08:40:02.340 INFO:teuthology.orchestra.run.trial187.stdout:Installing weak dependencies: 2026-02-12T08:40:02.340 INFO:teuthology.orchestra.run.trial187.stdout: ceph-mgr-cephadm noarch 2:20.3.0-5161.g2401a109.el9 ceph-noarch 183 k 2026-02-12T08:40:02.341 INFO:teuthology.orchestra.run.trial187.stdout: ceph-mgr-dashboard noarch 2:20.3.0-5161.g2401a109.el9 ceph-noarch 15 M 2026-02-12T08:40:02.341 INFO:teuthology.orchestra.run.trial187.stdout: ceph-mgr-diskprediction-local noarch 2:20.3.0-5161.g2401a109.el9 ceph-noarch 7.4 M 2026-02-12T08:40:02.341 INFO:teuthology.orchestra.run.trial187.stdout: ceph-mgr-k8sevents noarch 2:20.3.0-5161.g2401a109.el9 ceph-noarch 22 k 2026-02-12T08:40:02.341 INFO:teuthology.orchestra.run.trial187.stdout: ceph-mgr-rook noarch 2:20.3.0-5161.g2401a109.el9 ceph-noarch 50 k 2026-02-12T08:40:02.341 INFO:teuthology.orchestra.run.trial187.stdout: ceph-volume noarch 2:20.3.0-5161.g2401a109.el9 ceph-noarch 299 k 2026-02-12T08:40:02.341 INFO:teuthology.orchestra.run.trial187.stdout: python3-influxdb noarch 5.3.1-1.el9 epel 139 k 2026-02-12T08:40:02.341 INFO:teuthology.orchestra.run.trial187.stdout: python3-saml noarch 1.16.0-1.el9 epel 125 k 2026-02-12T08:40:02.341 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:40:02.341 INFO:teuthology.orchestra.run.trial187.stdout:Transaction Summary 2026-02-12T08:40:02.341 INFO:teuthology.orchestra.run.trial187.stdout:======================================================================================= 2026-02-12T08:40:02.341 INFO:teuthology.orchestra.run.trial187.stdout:Install 96 Packages 2026-02-12T08:40:02.342 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:40:02.342 INFO:teuthology.orchestra.run.trial187.stdout:Total download size: 189 M 2026-02-12T08:40:02.342 INFO:teuthology.orchestra.run.trial187.stdout:Installed size: 1.4 G 2026-02-12T08:40:02.342 INFO:teuthology.orchestra.run.trial187.stdout:Downloading Packages: 2026-02-12T08:40:03.912 INFO:teuthology.orchestra.run.trial187.stdout:(1/96): ceph-20.3.0-5161.g2401a109.el9.x86_64.r 23 kB/s | 6.5 kB 00:00 2026-02-12T08:40:04.579 INFO:teuthology.orchestra.run.trial187.stdout:(2/96): ceph-mds-20.3.0-5161.g2401a109.el9.x86_ 2.1 MB/s | 2.0 MB 00:00 2026-02-12T08:40:04.641 INFO:teuthology.orchestra.run.trial187.stdout:(3/96): ceph-osd-20.3.0-5161.g2401a109.el9.x86_ 2.6 MB/s | 163 kB 00:00 2026-02-12T08:40:05.315 INFO:teuthology.orchestra.run.trial187.stdout:(4/96): ceph-mon-20.3.0-5161.g2401a109.el9.x86_ 3.5 MB/s | 4.8 MB 00:01 2026-02-12T08:40:05.886 INFO:teuthology.orchestra.run.trial187.stdout:(5/96): ceph-mgr-20.3.0-5161.g2401a109.el9.x86_ 415 kB/s | 937 kB 00:02 2026-02-12T08:40:06.020 INFO:teuthology.orchestra.run.trial187.stdout:(6/96): ceph-grafana-dashboards-20.3.0-5161.g24 428 kB/s | 57 kB 00:00 2026-02-12T08:40:06.417 INFO:teuthology.orchestra.run.trial187.stdout:(7/96): ceph-mgr-cephadm-20.3.0-5161.g2401a109. 462 kB/s | 183 kB 00:00 2026-02-12T08:40:07.174 INFO:teuthology.orchestra.run.trial187.stdout:(8/96): ceph-osd-classic-20.3.0-5161.g2401a109. 5.8 MB/s | 15 MB 00:02 2026-02-12T08:40:08.966 INFO:teuthology.orchestra.run.trial187.stdout:(9/96): ceph-mgr-diskprediction-local-20.3.0-51 4.1 MB/s | 7.4 MB 00:01 2026-02-12T08:40:09.028 INFO:teuthology.orchestra.run.trial187.stdout:(10/96): ceph-mgr-k8sevents-20.3.0-5161.g2401a1 358 kB/s | 22 kB 00:00 2026-02-12T08:40:09.165 INFO:teuthology.orchestra.run.trial187.stdout:(11/96): ceph-mgr-modules-core-20.3.0-5161.g240 2.1 MB/s | 297 kB 00:00 2026-02-12T08:40:09.235 INFO:teuthology.orchestra.run.trial187.stdout:(12/96): ceph-mgr-rook-20.3.0-5161.g2401a109.el 717 kB/s | 50 kB 00:00 2026-02-12T08:40:09.306 INFO:teuthology.orchestra.run.trial187.stdout:(13/96): ceph-prometheus-alerts-20.3.0-5161.g24 246 kB/s | 17 kB 00:00 2026-02-12T08:40:09.444 INFO:teuthology.orchestra.run.trial187.stdout:(14/96): ceph-volume-20.3.0-5161.g2401a109.el9. 2.1 MB/s | 299 kB 00:00 2026-02-12T08:40:09.923 INFO:teuthology.orchestra.run.trial187.stdout:(15/96): cephadm-20.3.0-5161.g2401a109.el9.noar 2.1 MB/s | 1.0 MB 00:00 2026-02-12T08:40:11.337 INFO:teuthology.orchestra.run.trial187.stdout:(16/96): cryptsetup-2.8.1-2.el9.x86_64.rpm 248 kB/s | 350 kB 00:01 2026-02-12T08:40:11.509 INFO:teuthology.orchestra.run.trial187.stdout:(17/96): ledmon-libs-1.1.0-3.el9.x86_64.rpm 235 kB/s | 40 kB 00:00 2026-02-12T08:40:11.756 INFO:teuthology.orchestra.run.trial187.stdout:(18/96): libconfig-1.7.2-9.el9.x86_64.rpm 291 kB/s | 72 kB 00:00 2026-02-12T08:40:12.913 INFO:teuthology.orchestra.run.trial187.stdout:(19/96): libgfortran-11.5.0-14.el9.x86_64.rpm 687 kB/s | 794 kB 00:01 2026-02-12T08:40:13.161 INFO:teuthology.orchestra.run.trial187.stdout:(20/96): libquadmath-11.5.0-14.el9.x86_64.rpm 744 kB/s | 184 kB 00:00 2026-02-12T08:40:13.485 INFO:teuthology.orchestra.run.trial187.stdout:(21/96): python3-cffi-1.14.5-5.el9.x86_64.rpm 781 kB/s | 253 kB 00:00 2026-02-12T08:40:14.947 INFO:teuthology.orchestra.run.trial187.stdout:(22/96): python3-cryptography-36.0.1-5.el9.x86_ 873 kB/s | 1.2 MB 00:01 2026-02-12T08:40:15.119 INFO:teuthology.orchestra.run.trial187.stdout:(23/96): python3-ply-3.11-14.el9.noarch.rpm 617 kB/s | 106 kB 00:00 2026-02-12T08:40:15.367 INFO:teuthology.orchestra.run.trial187.stdout:(24/96): python3-pycparser-2.20-6.el9.noarch.rp 544 kB/s | 135 kB 00:00 2026-02-12T08:40:15.616 INFO:teuthology.orchestra.run.trial187.stdout:(25/96): python3-pyparsing-2.4.7-9.el9.noarch.r 606 kB/s | 150 kB 00:00 2026-02-12T08:40:15.789 INFO:teuthology.orchestra.run.trial187.stdout:(26/96): python3-requests-2.25.1-10.el9.noarch. 732 kB/s | 126 kB 00:00 2026-02-12T08:40:16.113 INFO:teuthology.orchestra.run.trial187.stdout:(27/96): python3-urllib3-1.26.5-7.el9.noarch.rp 673 kB/s | 218 kB 00:00 2026-02-12T08:40:16.361 INFO:teuthology.orchestra.run.trial187.stdout:(28/96): unzip-6.0-59.el9.x86_64.rpm 733 kB/s | 182 kB 00:00 2026-02-12T08:40:16.685 INFO:teuthology.orchestra.run.trial187.stdout:(29/96): zip-3.0-35.el9.x86_64.rpm 820 kB/s | 266 kB 00:00 2026-02-12T08:40:16.706 INFO:teuthology.orchestra.run.trial187.stdout:(30/96): flexiblas-3.0.4-8.el9.x86_64.rpm 1.6 MB/s | 33 kB 00:00 2026-02-12T08:40:16.732 INFO:teuthology.orchestra.run.trial187.stdout:(31/96): flexiblas-netlib-3.0.4-8.el9.x86_64.rp 114 MB/s | 3.0 MB 00:00 2026-02-12T08:40:16.736 INFO:teuthology.orchestra.run.trial187.stdout:(32/96): flexiblas-openblas-openmp-3.0.4-8.el9. 4.4 MB/s | 18 kB 00:00 2026-02-12T08:40:16.742 INFO:teuthology.orchestra.run.trial187.stdout:(33/96): libstoragemgmt-1.10.1-1.el9.x86_64.rpm 41 MB/s | 246 kB 00:00 2026-02-12T08:40:16.748 INFO:teuthology.orchestra.run.trial187.stdout:(34/96): lua-5.4.4-4.el9.x86_64.rpm 32 MB/s | 188 kB 00:00 2026-02-12T08:40:16.752 INFO:teuthology.orchestra.run.trial187.stdout:(35/96): openblas-0.3.29-1.el9.x86_64.rpm 9.6 MB/s | 42 kB 00:00 2026-02-12T08:40:16.773 INFO:teuthology.orchestra.run.trial187.stdout:(36/96): openblas-openmp-0.3.29-1.el9.x86_64.rp 264 MB/s | 5.3 MB 00:00 2026-02-12T08:40:16.798 INFO:teuthology.orchestra.run.trial187.stdout:(37/96): python3-babel-2.9.1-2.el9.noarch.rpm 241 MB/s | 6.0 MB 00:00 2026-02-12T08:40:16.805 INFO:teuthology.orchestra.run.trial187.stdout:(38/96): python3-devel-3.9.25-2.el9.x86_64.rpm 33 MB/s | 244 kB 00:00 2026-02-12T08:40:16.812 INFO:teuthology.orchestra.run.trial187.stdout:(39/96): python3-jinja2-2.11.3-8.el9.noarch.rpm 36 MB/s | 249 kB 00:00 2026-02-12T08:40:16.817 INFO:teuthology.orchestra.run.trial187.stdout:(40/96): python3-jmespath-1.0.1-1.el9.noarch.rp 9.6 MB/s | 48 kB 00:00 2026-02-12T08:40:16.822 INFO:teuthology.orchestra.run.trial187.stdout:(41/96): python3-libstoragemgmt-1.10.1-1.el9.x8 30 MB/s | 177 kB 00:00 2026-02-12T08:40:16.832 INFO:teuthology.orchestra.run.trial187.stdout:(42/96): python3-lxml-4.6.5-3.el9.x86_64.rpm 130 MB/s | 1.2 MB 00:00 2026-02-12T08:40:16.837 INFO:teuthology.orchestra.run.trial187.stdout:(43/96): python3-markupsafe-1.1.1-12.el9.x86_64 7.5 MB/s | 35 kB 00:00 2026-02-12T08:40:16.860 INFO:teuthology.orchestra.run.trial187.stdout:(44/96): python3-numpy-1.23.5-2.el9.x86_64.rpm 264 MB/s | 6.1 MB 00:00 2026-02-12T08:40:16.868 INFO:teuthology.orchestra.run.trial187.stdout:(45/96): python3-numpy-f2py-1.23.5-2.el9.x86_64 58 MB/s | 442 kB 00:00 2026-02-12T08:40:16.873 INFO:teuthology.orchestra.run.trial187.stdout:(46/96): python3-packaging-20.9-5.el9.noarch.rp 14 MB/s | 77 kB 00:00 2026-02-12T08:40:16.879 INFO:teuthology.orchestra.run.trial187.stdout:(47/96): python3-protobuf-3.14.0-17.el9.noarch. 46 MB/s | 267 kB 00:00 2026-02-12T08:40:16.885 INFO:teuthology.orchestra.run.trial187.stdout:(48/96): python3-pyasn1-0.4.8-6.el9.noarch.rpm 25 MB/s | 159 kB 00:00 2026-02-12T08:40:16.892 INFO:teuthology.orchestra.run.trial187.stdout:(49/96): python3-pyasn1-modules-0.4.8-6.el9.noa 40 MB/s | 279 kB 00:00 2026-02-12T08:40:16.897 INFO:teuthology.orchestra.run.trial187.stdout:(50/96): python3-requests-oauthlib-1.3.0-12.el9 11 MB/s | 54 kB 00:00 2026-02-12T08:40:16.959 INFO:teuthology.orchestra.run.trial187.stdout:(51/96): python3-scipy-1.9.3-2.el9.x86_64.rpm 309 MB/s | 19 MB 00:00 2026-02-12T08:40:16.965 INFO:teuthology.orchestra.run.trial187.stdout:(52/96): python3-toml-0.10.2-6.el9.noarch.rpm 7.6 MB/s | 42 kB 00:00 2026-02-12T08:40:16.970 INFO:teuthology.orchestra.run.trial187.stdout:(53/96): xmlsec1-1.2.29-13.el9.x86_64.rpm 34 MB/s | 189 kB 00:00 2026-02-12T08:40:16.975 INFO:teuthology.orchestra.run.trial187.stdout:(54/96): xmlsec1-openssl-1.2.29-13.el9.x86_64.r 18 MB/s | 90 kB 00:00 2026-02-12T08:40:17.215 INFO:teuthology.orchestra.run.trial187.stdout:(55/96): lua-devel-5.4.4-4.el9.x86_64.rpm 93 kB/s | 22 kB 00:00 2026-02-12T08:40:18.277 INFO:teuthology.orchestra.run.trial187.stdout:(56/96): protobuf-compiler-3.14.0-17.el9.x86_64 812 kB/s | 862 kB 00:01 2026-02-12T08:40:18.418 INFO:teuthology.orchestra.run.trial187.stdout:(57/96): ceph-mgr-dashboard-20.3.0-5161.g2401a1 1.2 MB/s | 15 MB 00:12 2026-02-12T08:40:18.543 INFO:teuthology.orchestra.run.trial187.stdout:(58/96): grpc-data-1.46.7-10.el9.noarch.rpm 155 kB/s | 19 kB 00:00 2026-02-12T08:40:18.553 INFO:teuthology.orchestra.run.trial187.stdout:(59/96): abseil-cpp-20211102.0-4.el9.x86_64.rpm 2.0 MB/s | 551 kB 00:00 2026-02-12T08:40:18.604 INFO:teuthology.orchestra.run.trial187.stdout:(60/96): python3-asyncssh-2.13.2-5.el9.noarch.r 10 MB/s | 548 kB 00:00 2026-02-12T08:40:18.634 INFO:teuthology.orchestra.run.trial187.stdout:(61/96): python3-autocommand-2.2.2-8.el9.noarch 968 kB/s | 29 kB 00:00 2026-02-12T08:40:18.647 INFO:teuthology.orchestra.run.trial187.stdout:(62/96): luarocks-3.9.2-5.el9.noarch.rpm 1.4 MB/s | 151 kB 00:00 2026-02-12T08:40:18.666 INFO:teuthology.orchestra.run.trial187.stdout:(63/96): python3-backports-tarfile-1.2.0-1.el9. 1.9 MB/s | 60 kB 00:00 2026-02-12T08:40:18.682 INFO:teuthology.orchestra.run.trial187.stdout:(64/96): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.2 MB/s | 43 kB 00:00 2026-02-12T08:40:18.696 INFO:teuthology.orchestra.run.trial187.stdout:(65/96): python3-cachetools-4.2.4-1.el9.noarch. 1.0 MB/s | 32 kB 00:00 2026-02-12T08:40:18.713 INFO:teuthology.orchestra.run.trial187.stdout:(66/96): python3-certifi-2023.05.07-4.el9.noarc 451 kB/s | 14 kB 00:00 2026-02-12T08:40:18.732 INFO:teuthology.orchestra.run.trial187.stdout:(67/96): python3-cheroot-10.0.1-4.el9.noarch.rp 4.8 MB/s | 173 kB 00:00 2026-02-12T08:40:18.770 INFO:teuthology.orchestra.run.trial187.stdout:(68/96): python3-google-auth-2.45.0-1.el9.noarc 6.6 MB/s | 254 kB 00:00 2026-02-12T08:40:18.786 INFO:teuthology.orchestra.run.trial187.stdout:(69/96): python3-cherrypy-18.6.1-2.el9.noarch.r 4.8 MB/s | 358 kB 00:00 2026-02-12T08:40:18.823 INFO:teuthology.orchestra.run.trial187.stdout:(70/96): python3-grpcio-tools-1.46.7-10.el9.x86 3.9 MB/s | 144 kB 00:00 2026-02-12T08:40:18.858 INFO:teuthology.orchestra.run.trial187.stdout:(71/96): python3-influxdb-5.3.1-1.el9.noarch.rp 3.9 MB/s | 139 kB 00:00 2026-02-12T08:40:18.862 INFO:teuthology.orchestra.run.trial187.stdout:(72/96): python3-grpcio-1.46.7-10.el9.x86_64.rp 22 MB/s | 2.0 MB 00:00 2026-02-12T08:40:18.889 INFO:teuthology.orchestra.run.trial187.stdout:(73/96): python3-isodate-0.6.1-3.el9.noarch.rpm 1.7 MB/s | 56 kB 00:00 2026-02-12T08:40:18.893 INFO:teuthology.orchestra.run.trial187.stdout:(74/96): python3-jaraco-8.2.1-3.el9.noarch.rpm 355 kB/s | 11 kB 00:00 2026-02-12T08:40:18.920 INFO:teuthology.orchestra.run.trial187.stdout:(75/96): python3-jaraco-classes-3.2.1-5.el9.noa 579 kB/s | 18 kB 00:00 2026-02-12T08:40:18.923 INFO:teuthology.orchestra.run.trial187.stdout:(76/96): python3-jaraco-collections-3.0.0-8.el9 763 kB/s | 23 kB 00:00 2026-02-12T08:40:18.950 INFO:teuthology.orchestra.run.trial187.stdout:(77/96): python3-jaraco-context-6.0.1-3.el9.noa 644 kB/s | 20 kB 00:00 2026-02-12T08:40:18.953 INFO:teuthology.orchestra.run.trial187.stdout:(78/96): python3-jaraco-functools-3.5.0-2.el9.n 642 kB/s | 19 kB 00:00 2026-02-12T08:40:18.981 INFO:teuthology.orchestra.run.trial187.stdout:(79/96): python3-jaraco-text-4.0.0-2.el9.noarch 866 kB/s | 26 kB 00:00 2026-02-12T08:40:18.995 INFO:teuthology.orchestra.run.trial187.stdout:(80/96): python3-kubernetes-26.1.0-3.el9.noarch 25 MB/s | 1.0 MB 00:00 2026-02-12T08:40:19.014 INFO:teuthology.orchestra.run.trial187.stdout:(81/96): python3-more-itertools-8.12.0-2.el9.no 2.4 MB/s | 79 kB 00:00 2026-02-12T08:40:19.026 INFO:teuthology.orchestra.run.trial187.stdout:(82/96): python3-msgpack-1.0.3-2.el9.x86_64.rpm 2.7 MB/s | 86 kB 00:00 2026-02-12T08:40:19.045 INFO:teuthology.orchestra.run.trial187.stdout:(83/96): python3-natsort-7.1.1-5.el9.noarch.rpm 1.8 MB/s | 58 kB 00:00 2026-02-12T08:40:19.056 INFO:teuthology.orchestra.run.trial187.stdout:(84/96): python3-portend-3.1.0-2.el9.noarch.rpm 544 kB/s | 16 kB 00:00 2026-02-12T08:40:19.079 INFO:teuthology.orchestra.run.trial187.stdout:(85/96): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.6 MB/s | 90 kB 00:00 2026-02-12T08:40:19.086 INFO:teuthology.orchestra.run.trial187.stdout:(86/96): python3-repoze-lru-0.7-16.el9.noarch.r 1.0 MB/s | 31 kB 00:00 2026-02-12T08:40:19.117 INFO:teuthology.orchestra.run.trial187.stdout:(87/96): python3-routes-2.5.1-5.el9.noarch.rpm 4.9 MB/s | 188 kB 00:00 2026-02-12T08:40:19.117 INFO:teuthology.orchestra.run.trial187.stdout:(88/96): python3-rsa-4.9-2.el9.noarch.rpm 1.9 MB/s | 59 kB 00:00 2026-02-12T08:40:19.147 INFO:teuthology.orchestra.run.trial187.stdout:(89/96): python3-tempora-5.0.0-2.el9.noarch.rpm 1.2 MB/s | 36 kB 00:00 2026-02-12T08:40:19.152 INFO:teuthology.orchestra.run.trial187.stdout:(90/96): python3-saml-1.16.0-1.el9.noarch.rpm 3.5 MB/s | 125 kB 00:00 2026-02-12T08:40:19.178 INFO:teuthology.orchestra.run.trial187.stdout:(91/96): python3-typing-extensions-4.15.0-1.el9 2.7 MB/s | 86 kB 00:00 2026-02-12T08:40:19.185 INFO:teuthology.orchestra.run.trial187.stdout:(92/96): python3-websocket-client-1.2.3-2.el9.n 2.6 MB/s | 90 kB 00:00 2026-02-12T08:40:19.208 INFO:teuthology.orchestra.run.trial187.stdout:(93/96): python3-xmlsec-1.3.13-1.el9.x86_64.rpm 1.6 MB/s | 48 kB 00:00 2026-02-12T08:40:19.215 INFO:teuthology.orchestra.run.trial187.stdout:(94/96): python3-xmltodict-0.12.0-15.el9.noarch 728 kB/s | 22 kB 00:00 2026-02-12T08:40:19.240 INFO:teuthology.orchestra.run.trial187.stdout:(95/96): python3-zc-lockfile-2.0-10.el9.noarch. 634 kB/s | 20 kB 00:00 2026-02-12T08:40:47.720 INFO:teuthology.orchestra.run.trial187.stdout:(96/96): ceph-osd-crimson-20.3.0-5161.g2401a109 2.0 MB/s | 87 MB 00:42 2026-02-12T08:40:47.723 INFO:teuthology.orchestra.run.trial187.stdout:-------------------------------------------------------------------------------- 2026-02-12T08:40:47.723 INFO:teuthology.orchestra.run.trial187.stdout:Total 4.2 MB/s | 189 MB 00:45 2026-02-12T08:40:48.101 INFO:teuthology.orchestra.run.trial187.stdout:Running transaction check 2026-02-12T08:40:48.141 INFO:teuthology.orchestra.run.trial187.stdout:Transaction check succeeded. 2026-02-12T08:40:48.141 INFO:teuthology.orchestra.run.trial187.stdout:Running transaction test 2026-02-12T08:40:48.835 INFO:teuthology.orchestra.run.trial187.stdout:Transaction test succeeded. 2026-02-12T08:40:48.835 INFO:teuthology.orchestra.run.trial187.stdout:Running transaction 2026-02-12T08:40:49.556 INFO:teuthology.orchestra.run.trial187.stdout: Preparing : 1/1 2026-02-12T08:40:49.558 INFO:teuthology.orchestra.run.trial187.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/96 2026-02-12T08:40:49.561 INFO:teuthology.orchestra.run.trial187.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/96 2026-02-12T08:40:49.583 INFO:teuthology.orchestra.run.trial187.stdout: Installing : xmlsec1-1.2.29-13.el9.x86_64 3/96 2026-02-12T08:40:49.616 INFO:teuthology.orchestra.run.trial187.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 4/96 2026-02-12T08:40:49.623 INFO:teuthology.orchestra.run.trial187.stdout: Installing : python3-lxml-4.6.5-3.el9.x86_64 5/96 2026-02-12T08:40:49.625 INFO:teuthology.orchestra.run.trial187.stdout: Installing : lua-5.4.4-4.el9.x86_64 6/96 2026-02-12T08:40:49.630 INFO:teuthology.orchestra.run.trial187.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 7/96 2026-02-12T08:40:49.653 INFO:teuthology.orchestra.run.trial187.stdout: Installing : unzip-6.0-59.el9.x86_64 8/96 2026-02-12T08:40:49.666 INFO:teuthology.orchestra.run.trial187.stdout: Installing : python3-urllib3-1.26.5-7.el9.noarch 9/96 2026-02-12T08:40:49.669 INFO:teuthology.orchestra.run.trial187.stdout: Installing : python3-requests-2.25.1-10.el9.noarch 10/96 2026-02-12T08:40:49.675 INFO:teuthology.orchestra.run.trial187.stdout: Installing : libquadmath-11.5.0-14.el9.x86_64 11/96 2026-02-12T08:40:49.677 INFO:teuthology.orchestra.run.trial187.stdout: Installing : libgfortran-11.5.0-14.el9.x86_64 12/96 2026-02-12T08:40:49.688 INFO:teuthology.orchestra.run.trial187.stdout: Installing : ledmon-libs-1.1.0-3.el9.x86_64 13/96 2026-02-12T08:40:49.694 INFO:teuthology.orchestra.run.trial187.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 14/96 2026-02-12T08:40:49.718 INFO:teuthology.orchestra.run.trial187.stdout: Installing : zip-3.0-35.el9.x86_64 15/96 2026-02-12T08:40:49.722 INFO:teuthology.orchestra.run.trial187.stdout: Installing : luarocks-3.9.2-5.el9.noarch 16/96 2026-02-12T08:40:49.772 INFO:teuthology.orchestra.run.trial187.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 17/96 2026-02-12T08:40:49.785 INFO:teuthology.orchestra.run.trial187.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 18/96 2026-02-12T08:40:49.787 INFO:teuthology.orchestra.run.trial187.stdout: Installing : python3-rsa-4.9-2.el9.noarch 19/96 2026-02-12T08:40:49.792 INFO:teuthology.orchestra.run.trial187.stdout: Installing : xmlsec1-openssl-1.2.29-13.el9.x86_64 20/96 2026-02-12T08:40:49.797 INFO:teuthology.orchestra.run.trial187.stdout: Installing : python3-xmlsec-1.3.13-1.el9.x86_64 21/96 2026-02-12T08:40:49.802 INFO:teuthology.orchestra.run.trial187.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 22/96 2026-02-12T08:40:49.805 INFO:teuthology.orchestra.run.trial187.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 23/96 2026-02-12T08:40:49.819 INFO:teuthology.orchestra.run.trial187.stdout: Installing : python3-xmltodict-0.12.0-15.el9.noarch 24/96 2026-02-12T08:40:49.824 INFO:teuthology.orchestra.run.trial187.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 25/96 2026-02-12T08:40:49.829 INFO:teuthology.orchestra.run.trial187.stdout: Installing : python3-typing-extensions-4.15.0-1.el9.noarch 26/96 2026-02-12T08:40:49.841 INFO:teuthology.orchestra.run.trial187.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 27/96 2026-02-12T08:40:49.850 INFO:teuthology.orchestra.run.trial187.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 28/96 2026-02-12T08:40:49.856 INFO:teuthology.orchestra.run.trial187.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 29/96 2026-02-12T08:40:49.876 INFO:teuthology.orchestra.run.trial187.stdout: Installing : python3-msgpack-1.0.3-2.el9.x86_64 30/96 2026-02-12T08:40:49.887 INFO:teuthology.orchestra.run.trial187.stdout: Installing : python3-influxdb-5.3.1-1.el9.noarch 31/96 2026-02-12T08:40:49.902 INFO:teuthology.orchestra.run.trial187.stdout: Installing : python3-isodate-0.6.1-3.el9.noarch 32/96 2026-02-12T08:40:49.906 INFO:teuthology.orchestra.run.trial187.stdout: Installing : python3-saml-1.16.0-1.el9.noarch 33/96 2026-02-12T08:40:49.914 INFO:teuthology.orchestra.run.trial187.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 34/96 2026-02-12T08:40:49.954 INFO:teuthology.orchestra.run.trial187.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 35/96 2026-02-12T08:40:50.274 INFO:teuthology.orchestra.run.trial187.stdout: Installing : python3-google-auth-1:2.45.0-1.el9.noarch 36/96 2026-02-12T08:40:50.284 INFO:teuthology.orchestra.run.trial187.stdout: Installing : python3-kubernetes-1:26.1.0-3.el9.noarch 37/96 2026-02-12T08:40:50.288 INFO:teuthology.orchestra.run.trial187.stdout: Installing : python3-backports-tarfile-1.2.0-1.el9.noarch 38/96 2026-02-12T08:40:50.294 INFO:teuthology.orchestra.run.trial187.stdout: Installing : python3-jaraco-context-6.0.1-3.el9.noarch 39/96 2026-02-12T08:40:50.296 INFO:teuthology.orchestra.run.trial187.stdout: Installing : python3-autocommand-2.2.2-8.el9.noarch 40/96 2026-02-12T08:40:50.323 INFO:teuthology.orchestra.run.trial187.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 41/96 2026-02-12T08:40:50.365 INFO:teuthology.orchestra.run.trial187.stdout: Installing : abseil-cpp-20211102.0-4.el9.x86_64 42/96 2026-02-12T08:40:50.374 INFO:teuthology.orchestra.run.trial187.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 43/96 2026-02-12T08:40:50.379 INFO:teuthology.orchestra.run.trial187.stdout: Installing : protobuf-compiler-3.14.0-17.el9.x86_64 44/96 2026-02-12T08:40:50.383 INFO:teuthology.orchestra.run.trial187.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 45/96 2026-02-12T08:40:50.389 INFO:teuthology.orchestra.run.trial187.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 46/96 2026-02-12T08:40:50.392 INFO:teuthology.orchestra.run.trial187.stdout: Installing : python3-jaraco-text-4.0.0-2.el9.noarch 47/96 2026-02-12T08:40:50.399 INFO:teuthology.orchestra.run.trial187.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 48/96 2026-02-12T08:40:50.402 INFO:teuthology.orchestra.run.trial187.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 49/96 2026-02-12T08:40:50.430 INFO:teuthology.orchestra.run.trial187.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 50/96 2026-02-12T08:40:50.440 INFO:teuthology.orchestra.run.trial187.stdout: Installing : python3-protobuf-3.14.0-17.el9.noarch 51/96 2026-02-12T08:40:50.447 INFO:teuthology.orchestra.run.trial187.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 52/96 2026-02-12T08:40:50.454 INFO:teuthology.orchestra.run.trial187.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 53/96 2026-02-12T08:40:50.488 INFO:teuthology.orchestra.run.trial187.stdout: Installing : python3-jmespath-1.0.1-1.el9.noarch 54/96 2026-02-12T08:40:50.709 INFO:teuthology.orchestra.run.trial187.stdout: Installing : python3-devel-3.9.25-2.el9.x86_64 55/96 2026-02-12T08:40:50.733 INFO:teuthology.orchestra.run.trial187.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 56/96 2026-02-12T08:40:50.737 INFO:teuthology.orchestra.run.trial187.stdout: Installing : python3-jinja2-2.11.3-8.el9.noarch 57/96 2026-02-12T08:40:50.794 INFO:teuthology.orchestra.run.trial187.stdout: Installing : openblas-0.3.29-1.el9.x86_64 58/96 2026-02-12T08:40:50.796 INFO:teuthology.orchestra.run.trial187.stdout: Installing : openblas-openmp-0.3.29-1.el9.x86_64 59/96 2026-02-12T08:40:50.817 INFO:teuthology.orchestra.run.trial187.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 60/96 2026-02-12T08:40:51.137 INFO:teuthology.orchestra.run.trial187.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 61/96 2026-02-12T08:40:51.209 INFO:teuthology.orchestra.run.trial187.stdout: Installing : python3-numpy-1:1.23.5-2.el9.x86_64 62/96 2026-02-12T08:40:51.869 INFO:teuthology.orchestra.run.trial187.stdout: Installing : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 63/96 2026-02-12T08:40:51.886 INFO:teuthology.orchestra.run.trial187.stdout: Installing : python3-scipy-1.9.3-2.el9.x86_64 64/96 2026-02-12T08:40:51.895 INFO:teuthology.orchestra.run.trial187.stdout: Installing : python3-pyparsing-2.4.7-9.el9.noarch 65/96 2026-02-12T08:40:51.902 INFO:teuthology.orchestra.run.trial187.stdout: Installing : python3-packaging-20.9-5.el9.noarch 66/96 2026-02-12T08:40:51.915 INFO:teuthology.orchestra.run.trial187.stdout: Installing : python3-ply-3.11-14.el9.noarch 67/96 2026-02-12T08:40:51.931 INFO:teuthology.orchestra.run.trial187.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 68/96 2026-02-12T08:40:52.005 INFO:teuthology.orchestra.run.trial187.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 69/96 2026-02-12T08:40:52.013 INFO:teuthology.orchestra.run.trial187.stdout: Installing : python3-cryptography-36.0.1-5.el9.x86_64 70/96 2026-02-12T08:40:52.036 INFO:teuthology.orchestra.run.trial187.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 71/96 2026-02-12T08:40:52.066 INFO:teuthology.orchestra.run.trial187.stdout: Installing : python3-cheroot-10.0.1-4.el9.noarch 72/96 2026-02-12T08:40:52.115 INFO:teuthology.orchestra.run.trial187.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 73/96 2026-02-12T08:40:52.121 INFO:teuthology.orchestra.run.trial187.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 74/96 2026-02-12T08:40:52.125 INFO:teuthology.orchestra.run.trial187.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 75/96 2026-02-12T08:40:52.126 INFO:teuthology.orchestra.run.trial187.stdout: Installing : libconfig-1.7.2-9.el9.x86_64 76/96 2026-02-12T08:40:52.142 INFO:teuthology.orchestra.run.trial187.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-12T08:40:52.143 INFO:teuthology.orchestra.run.trial187.stdout:Creating group 'libstoragemgmt' with GID 991. 2026-02-12T08:40:52.143 INFO:teuthology.orchestra.run.trial187.stdout:Creating user 'libstoragemgmt' (daemon account for libstoragemgmt) with UID 991 and GID 991. 2026-02-12T08:40:52.143 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:40:52.150 INFO:teuthology.orchestra.run.trial187.stdout: Installing : libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-12T08:40:52.173 INFO:teuthology.orchestra.run.trial187.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-12T08:40:52.174 INFO:teuthology.orchestra.run.trial187.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/libstoragemgmt.service → /usr/lib/systemd/system/libstoragemgmt.service. 2026-02-12T08:40:52.174 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:40:52.207 INFO:teuthology.orchestra.run.trial187.stdout: Installing : python3-libstoragemgmt-1.10.1-1.el9.x86_64 78/96 2026-02-12T08:40:52.804 INFO:teuthology.orchestra.run.trial187.stdout: Installing : cryptsetup-2.8.1-2.el9.x86_64 79/96 2026-02-12T08:40:58.209 INFO:teuthology.orchestra.run.trial187.stdout: Installing : ceph-osd-classic-2:20.3.0-5161.g2401a109.el9.x86_6 80/96 2026-02-12T08:40:58.272 INFO:teuthology.orchestra.run.trial187.stdout: Installing : ceph-osd-crimson-2:20.3.0-5161.g2401a109.el9.x86_6 81/96 2026-02-12T08:40:58.274 INFO:teuthology.orchestra.run.trial187.stdout: Installing : ceph-volume-2:20.3.0-5161.g2401a109.el9.noarch 82/96 2026-02-12T08:40:58.288 INFO:teuthology.orchestra.run.trial187.stdout: Running scriptlet: ceph-volume-2:20.3.0-5161.g2401a109.el9.noarch 82/96 2026-02-12T08:40:58.288 INFO:teuthology.orchestra.run.trial187.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-12T08:40:58.288 INFO:teuthology.orchestra.run.trial187.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2026-02-12T08:40:58.288 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:40:58.298 INFO:teuthology.orchestra.run.trial187.stdout: Installing : ceph-osd-2:20.3.0-5161.g2401a109.el9.x86_64 83/96 2026-02-12T08:40:58.320 INFO:teuthology.orchestra.run.trial187.stdout: Running scriptlet: ceph-osd-2:20.3.0-5161.g2401a109.el9.x86_64 83/96 2026-02-12T08:40:58.320 INFO:teuthology.orchestra.run.trial187.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-12T08:40:58.320 INFO:teuthology.orchestra.run.trial187.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2026-02-12T08:40:58.321 INFO:teuthology.orchestra.run.trial187.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-12T08:40:58.321 INFO:teuthology.orchestra.run.trial187.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-12T08:40:58.321 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:40:58.381 INFO:teuthology.orchestra.run.trial187.stdout: Running scriptlet: cephadm-2:20.3.0-5161.g2401a109.el9.noarch 84/96 2026-02-12T08:40:58.383 INFO:teuthology.orchestra.run.trial187.stdout: Installing : cephadm-2:20.3.0-5161.g2401a109.el9.noarch 84/96 2026-02-12T08:40:58.392 INFO:teuthology.orchestra.run.trial187.stdout: Installing : ceph-prometheus-alerts-2:20.3.0-5161.g2401a109.el9 85/96 2026-02-12T08:40:58.416 INFO:teuthology.orchestra.run.trial187.stdout: Installing : ceph-grafana-dashboards-2:20.3.0-5161.g2401a109.el 86/96 2026-02-12T08:40:58.417 INFO:teuthology.orchestra.run.trial187.stdout: Installing : ceph-mgr-cephadm-2:20.3.0-5161.g2401a109.el9.noarc 87/96 2026-02-12T08:41:00.003 INFO:teuthology.orchestra.run.trial187.stdout: Running scriptlet: ceph-mgr-cephadm-2:20.3.0-5161.g2401a109.el9.noarc 87/96 2026-02-12T08:41:00.010 INFO:teuthology.orchestra.run.trial187.stdout: Installing : ceph-mgr-dashboard-2:20.3.0-5161.g2401a109.el9.noa 88/96 2026-02-12T08:41:00.471 INFO:teuthology.orchestra.run.trial187.stdout: Running scriptlet: ceph-mgr-dashboard-2:20.3.0-5161.g2401a109.el9.noa 88/96 2026-02-12T08:41:00.472 INFO:teuthology.orchestra.run.trial187.stdout: Installing : ceph-mgr-diskprediction-local-2:20.3.0-5161.g2401a 89/96 2026-02-12T08:41:00.484 INFO:teuthology.orchestra.run.trial187.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:20.3.0-5161.g2401a 89/96 2026-02-12T08:41:00.485 INFO:teuthology.orchestra.run.trial187.stdout: Installing : ceph-mgr-k8sevents-2:20.3.0-5161.g2401a109.el9.noa 90/96 2026-02-12T08:41:00.547 INFO:teuthology.orchestra.run.trial187.stdout: Running scriptlet: ceph-mgr-k8sevents-2:20.3.0-5161.g2401a109.el9.noa 90/96 2026-02-12T08:41:00.588 INFO:teuthology.orchestra.run.trial187.stdout: Installing : ceph-mgr-modules-core-2:20.3.0-5161.g2401a109.el9. 91/96 2026-02-12T08:41:00.590 INFO:teuthology.orchestra.run.trial187.stdout: Installing : ceph-mgr-2:20.3.0-5161.g2401a109.el9.x86_64 92/96 2026-02-12T08:41:00.611 INFO:teuthology.orchestra.run.trial187.stdout: Running scriptlet: ceph-mgr-2:20.3.0-5161.g2401a109.el9.x86_64 92/96 2026-02-12T08:41:00.611 INFO:teuthology.orchestra.run.trial187.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-12T08:41:00.611 INFO:teuthology.orchestra.run.trial187.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2026-02-12T08:41:00.611 INFO:teuthology.orchestra.run.trial187.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-12T08:41:00.611 INFO:teuthology.orchestra.run.trial187.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-12T08:41:00.611 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:41:00.623 INFO:teuthology.orchestra.run.trial187.stdout: Installing : ceph-mgr-rook-2:20.3.0-5161.g2401a109.el9.noarch 93/96 2026-02-12T08:41:00.842 INFO:teuthology.orchestra.run.trial187.stdout: Running scriptlet: ceph-mgr-rook-2:20.3.0-5161.g2401a109.el9.noarch 93/96 2026-02-12T08:41:00.843 INFO:teuthology.orchestra.run.trial187.stdout: Installing : ceph-mon-2:20.3.0-5161.g2401a109.el9.x86_64 94/96 2026-02-12T08:41:00.862 INFO:teuthology.orchestra.run.trial187.stdout: Running scriptlet: ceph-mon-2:20.3.0-5161.g2401a109.el9.x86_64 94/96 2026-02-12T08:41:00.863 INFO:teuthology.orchestra.run.trial187.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-12T08:41:00.863 INFO:teuthology.orchestra.run.trial187.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2026-02-12T08:41:00.863 INFO:teuthology.orchestra.run.trial187.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-12T08:41:00.863 INFO:teuthology.orchestra.run.trial187.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-12T08:41:00.863 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:41:00.947 INFO:teuthology.orchestra.run.trial187.stdout: Installing : ceph-mds-2:20.3.0-5161.g2401a109.el9.x86_64 95/96 2026-02-12T08:41:00.967 INFO:teuthology.orchestra.run.trial187.stdout: Running scriptlet: ceph-mds-2:20.3.0-5161.g2401a109.el9.x86_64 95/96 2026-02-12T08:41:00.967 INFO:teuthology.orchestra.run.trial187.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-12T08:41:00.967 INFO:teuthology.orchestra.run.trial187.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2026-02-12T08:41:00.968 INFO:teuthology.orchestra.run.trial187.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-12T08:41:00.968 INFO:teuthology.orchestra.run.trial187.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-12T08:41:00.968 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:41:00.968 INFO:teuthology.orchestra.run.trial187.stdout: Installing : ceph-2:20.3.0-5161.g2401a109.el9.x86_64 96/96 2026-02-12T08:41:00.975 INFO:teuthology.orchestra.run.trial187.stdout: Running scriptlet: ceph-osd-classic-2:20.3.0-5161.g2401a109.el9.x86_6 96/96 2026-02-12T08:41:00.998 INFO:teuthology.orchestra.run.trial187.stdout: Running scriptlet: ceph-osd-crimson-2:20.3.0-5161.g2401a109.el9.x86_6 96/96 2026-02-12T08:41:02.298 INFO:teuthology.orchestra.run.trial187.stdout: Running scriptlet: ceph-2:20.3.0-5161.g2401a109.el9.x86_64 96/96 2026-02-12T08:41:02.298 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : ceph-2:20.3.0-5161.g2401a109.el9.x86_64 1/96 2026-02-12T08:41:02.298 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : ceph-mds-2:20.3.0-5161.g2401a109.el9.x86_64 2/96 2026-02-12T08:41:02.298 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : ceph-mgr-2:20.3.0-5161.g2401a109.el9.x86_64 3/96 2026-02-12T08:41:02.298 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : ceph-mon-2:20.3.0-5161.g2401a109.el9.x86_64 4/96 2026-02-12T08:41:02.298 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : ceph-osd-2:20.3.0-5161.g2401a109.el9.x86_64 5/96 2026-02-12T08:41:02.298 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : ceph-osd-classic-2:20.3.0-5161.g2401a109.el9.x86_6 6/96 2026-02-12T08:41:02.298 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : ceph-osd-crimson-2:20.3.0-5161.g2401a109.el9.x86_6 7/96 2026-02-12T08:41:02.298 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : ceph-grafana-dashboards-2:20.3.0-5161.g2401a109.el 8/96 2026-02-12T08:41:02.298 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : ceph-mgr-cephadm-2:20.3.0-5161.g2401a109.el9.noarc 9/96 2026-02-12T08:41:02.298 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : ceph-mgr-dashboard-2:20.3.0-5161.g2401a109.el9.noa 10/96 2026-02-12T08:41:02.299 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : ceph-mgr-diskprediction-local-2:20.3.0-5161.g2401a 11/96 2026-02-12T08:41:02.299 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : ceph-mgr-k8sevents-2:20.3.0-5161.g2401a109.el9.noa 12/96 2026-02-12T08:41:02.299 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : ceph-mgr-modules-core-2:20.3.0-5161.g2401a109.el9. 13/96 2026-02-12T08:41:02.299 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : ceph-mgr-rook-2:20.3.0-5161.g2401a109.el9.noarch 14/96 2026-02-12T08:41:02.299 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : ceph-prometheus-alerts-2:20.3.0-5161.g2401a109.el9 15/96 2026-02-12T08:41:02.299 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : ceph-volume-2:20.3.0-5161.g2401a109.el9.noarch 16/96 2026-02-12T08:41:02.299 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : cephadm-2:20.3.0-5161.g2401a109.el9.noarch 17/96 2026-02-12T08:41:02.299 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : cryptsetup-2.8.1-2.el9.x86_64 18/96 2026-02-12T08:41:02.299 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : ledmon-libs-1.1.0-3.el9.x86_64 19/96 2026-02-12T08:41:02.299 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : libconfig-1.7.2-9.el9.x86_64 20/96 2026-02-12T08:41:02.299 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : libgfortran-11.5.0-14.el9.x86_64 21/96 2026-02-12T08:41:02.299 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : libquadmath-11.5.0-14.el9.x86_64 22/96 2026-02-12T08:41:02.299 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 23/96 2026-02-12T08:41:02.299 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : python3-cryptography-36.0.1-5.el9.x86_64 24/96 2026-02-12T08:41:02.299 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : python3-ply-3.11-14.el9.noarch 25/96 2026-02-12T08:41:02.299 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 26/96 2026-02-12T08:41:02.301 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : python3-pyparsing-2.4.7-9.el9.noarch 27/96 2026-02-12T08:41:02.301 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : python3-requests-2.25.1-10.el9.noarch 28/96 2026-02-12T08:41:02.301 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : python3-urllib3-1.26.5-7.el9.noarch 29/96 2026-02-12T08:41:02.301 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : unzip-6.0-59.el9.x86_64 30/96 2026-02-12T08:41:02.301 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : zip-3.0-35.el9.x86_64 31/96 2026-02-12T08:41:02.301 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 32/96 2026-02-12T08:41:02.301 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 33/96 2026-02-12T08:41:02.301 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 34/96 2026-02-12T08:41:02.301 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : libstoragemgmt-1.10.1-1.el9.x86_64 35/96 2026-02-12T08:41:02.301 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : lua-5.4.4-4.el9.x86_64 36/96 2026-02-12T08:41:02.301 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : openblas-0.3.29-1.el9.x86_64 37/96 2026-02-12T08:41:02.301 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : openblas-openmp-0.3.29-1.el9.x86_64 38/96 2026-02-12T08:41:02.301 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 39/96 2026-02-12T08:41:02.301 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : python3-devel-3.9.25-2.el9.x86_64 40/96 2026-02-12T08:41:02.301 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : python3-jinja2-2.11.3-8.el9.noarch 41/96 2026-02-12T08:41:02.301 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : python3-jmespath-1.0.1-1.el9.noarch 42/96 2026-02-12T08:41:02.301 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : python3-libstoragemgmt-1.10.1-1.el9.x86_64 43/96 2026-02-12T08:41:02.301 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : python3-lxml-4.6.5-3.el9.x86_64 44/96 2026-02-12T08:41:02.301 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 45/96 2026-02-12T08:41:02.301 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : python3-numpy-1:1.23.5-2.el9.x86_64 46/96 2026-02-12T08:41:02.301 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 47/96 2026-02-12T08:41:02.301 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 48/96 2026-02-12T08:41:02.301 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : python3-protobuf-3.14.0-17.el9.noarch 49/96 2026-02-12T08:41:02.301 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 50/96 2026-02-12T08:41:02.301 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 51/96 2026-02-12T08:41:02.301 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 52/96 2026-02-12T08:41:02.301 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : python3-scipy-1.9.3-2.el9.x86_64 53/96 2026-02-12T08:41:02.301 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 54/96 2026-02-12T08:41:02.301 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : xmlsec1-1.2.29-13.el9.x86_64 55/96 2026-02-12T08:41:02.301 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : xmlsec1-openssl-1.2.29-13.el9.x86_64 56/96 2026-02-12T08:41:02.301 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 57/96 2026-02-12T08:41:02.301 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : protobuf-compiler-3.14.0-17.el9.x86_64 58/96 2026-02-12T08:41:02.301 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : abseil-cpp-20211102.0-4.el9.x86_64 59/96 2026-02-12T08:41:02.301 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 60/96 2026-02-12T08:41:02.301 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 61/96 2026-02-12T08:41:02.301 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 62/96 2026-02-12T08:41:02.301 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : python3-autocommand-2.2.2-8.el9.noarch 63/96 2026-02-12T08:41:02.301 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : python3-backports-tarfile-1.2.0-1.el9.noarch 64/96 2026-02-12T08:41:02.301 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 65/96 2026-02-12T08:41:02.301 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 66/96 2026-02-12T08:41:02.302 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 67/96 2026-02-12T08:41:02.302 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : python3-cheroot-10.0.1-4.el9.noarch 68/96 2026-02-12T08:41:02.302 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 69/96 2026-02-12T08:41:02.302 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : python3-google-auth-1:2.45.0-1.el9.noarch 70/96 2026-02-12T08:41:02.302 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 71/96 2026-02-12T08:41:02.302 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 72/96 2026-02-12T08:41:02.302 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : python3-influxdb-5.3.1-1.el9.noarch 73/96 2026-02-12T08:41:02.302 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : python3-isodate-0.6.1-3.el9.noarch 74/96 2026-02-12T08:41:02.302 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 75/96 2026-02-12T08:41:02.302 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 76/96 2026-02-12T08:41:02.302 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 77/96 2026-02-12T08:41:02.302 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : python3-jaraco-context-6.0.1-3.el9.noarch 78/96 2026-02-12T08:41:02.302 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 79/96 2026-02-12T08:41:02.302 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : python3-jaraco-text-4.0.0-2.el9.noarch 80/96 2026-02-12T08:41:02.302 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : python3-kubernetes-1:26.1.0-3.el9.noarch 81/96 2026-02-12T08:41:02.302 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 82/96 2026-02-12T08:41:02.302 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : python3-msgpack-1.0.3-2.el9.x86_64 83/96 2026-02-12T08:41:02.302 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 84/96 2026-02-12T08:41:02.302 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 85/96 2026-02-12T08:41:02.302 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/96 2026-02-12T08:41:02.302 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 87/96 2026-02-12T08:41:02.302 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 88/96 2026-02-12T08:41:02.302 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 89/96 2026-02-12T08:41:02.302 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : python3-saml-1.16.0-1.el9.noarch 90/96 2026-02-12T08:41:02.302 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 91/96 2026-02-12T08:41:02.302 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : python3-typing-extensions-4.15.0-1.el9.noarch 92/96 2026-02-12T08:41:02.302 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 93/96 2026-02-12T08:41:02.302 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : python3-xmlsec-1.3.13-1.el9.x86_64 94/96 2026-02-12T08:41:02.302 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : python3-xmltodict-0.12.0-15.el9.noarch 95/96 2026-02-12T08:41:02.368 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 96/96 2026-02-12T08:41:02.368 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:41:02.368 INFO:teuthology.orchestra.run.trial187.stdout:Installed: 2026-02-12T08:41:02.368 INFO:teuthology.orchestra.run.trial187.stdout: abseil-cpp-20211102.0-4.el9.x86_64 2026-02-12T08:41:02.368 INFO:teuthology.orchestra.run.trial187.stdout: ceph-2:20.3.0-5161.g2401a109.el9.x86_64 2026-02-12T08:41:02.368 INFO:teuthology.orchestra.run.trial187.stdout: ceph-grafana-dashboards-2:20.3.0-5161.g2401a109.el9.noarch 2026-02-12T08:41:02.368 INFO:teuthology.orchestra.run.trial187.stdout: ceph-mds-2:20.3.0-5161.g2401a109.el9.x86_64 2026-02-12T08:41:02.368 INFO:teuthology.orchestra.run.trial187.stdout: ceph-mgr-2:20.3.0-5161.g2401a109.el9.x86_64 2026-02-12T08:41:02.368 INFO:teuthology.orchestra.run.trial187.stdout: ceph-mgr-cephadm-2:20.3.0-5161.g2401a109.el9.noarch 2026-02-12T08:41:02.368 INFO:teuthology.orchestra.run.trial187.stdout: ceph-mgr-dashboard-2:20.3.0-5161.g2401a109.el9.noarch 2026-02-12T08:41:02.368 INFO:teuthology.orchestra.run.trial187.stdout: ceph-mgr-diskprediction-local-2:20.3.0-5161.g2401a109.el9.noarch 2026-02-12T08:41:02.369 INFO:teuthology.orchestra.run.trial187.stdout: ceph-mgr-k8sevents-2:20.3.0-5161.g2401a109.el9.noarch 2026-02-12T08:41:02.369 INFO:teuthology.orchestra.run.trial187.stdout: ceph-mgr-modules-core-2:20.3.0-5161.g2401a109.el9.noarch 2026-02-12T08:41:02.369 INFO:teuthology.orchestra.run.trial187.stdout: ceph-mgr-rook-2:20.3.0-5161.g2401a109.el9.noarch 2026-02-12T08:41:02.369 INFO:teuthology.orchestra.run.trial187.stdout: ceph-mon-2:20.3.0-5161.g2401a109.el9.x86_64 2026-02-12T08:41:02.369 INFO:teuthology.orchestra.run.trial187.stdout: ceph-osd-2:20.3.0-5161.g2401a109.el9.x86_64 2026-02-12T08:41:02.369 INFO:teuthology.orchestra.run.trial187.stdout: ceph-osd-classic-2:20.3.0-5161.g2401a109.el9.x86_64 2026-02-12T08:41:02.369 INFO:teuthology.orchestra.run.trial187.stdout: ceph-osd-crimson-2:20.3.0-5161.g2401a109.el9.x86_64 2026-02-12T08:41:02.369 INFO:teuthology.orchestra.run.trial187.stdout: ceph-prometheus-alerts-2:20.3.0-5161.g2401a109.el9.noarch 2026-02-12T08:41:02.369 INFO:teuthology.orchestra.run.trial187.stdout: ceph-volume-2:20.3.0-5161.g2401a109.el9.noarch 2026-02-12T08:41:02.369 INFO:teuthology.orchestra.run.trial187.stdout: cephadm-2:20.3.0-5161.g2401a109.el9.noarch 2026-02-12T08:41:02.369 INFO:teuthology.orchestra.run.trial187.stdout: cryptsetup-2.8.1-2.el9.x86_64 2026-02-12T08:41:02.369 INFO:teuthology.orchestra.run.trial187.stdout: flexiblas-3.0.4-8.el9.x86_64 2026-02-12T08:41:02.369 INFO:teuthology.orchestra.run.trial187.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2026-02-12T08:41:02.369 INFO:teuthology.orchestra.run.trial187.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2026-02-12T08:41:02.369 INFO:teuthology.orchestra.run.trial187.stdout: grpc-data-1.46.7-10.el9.noarch 2026-02-12T08:41:02.369 INFO:teuthology.orchestra.run.trial187.stdout: ledmon-libs-1.1.0-3.el9.x86_64 2026-02-12T08:41:02.369 INFO:teuthology.orchestra.run.trial187.stdout: libconfig-1.7.2-9.el9.x86_64 2026-02-12T08:41:02.369 INFO:teuthology.orchestra.run.trial187.stdout: libgfortran-11.5.0-14.el9.x86_64 2026-02-12T08:41:02.369 INFO:teuthology.orchestra.run.trial187.stdout: libquadmath-11.5.0-14.el9.x86_64 2026-02-12T08:41:02.369 INFO:teuthology.orchestra.run.trial187.stdout: libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-12T08:41:02.369 INFO:teuthology.orchestra.run.trial187.stdout: lua-5.4.4-4.el9.x86_64 2026-02-12T08:41:02.369 INFO:teuthology.orchestra.run.trial187.stdout: lua-devel-5.4.4-4.el9.x86_64 2026-02-12T08:41:02.369 INFO:teuthology.orchestra.run.trial187.stdout: luarocks-3.9.2-5.el9.noarch 2026-02-12T08:41:02.369 INFO:teuthology.orchestra.run.trial187.stdout: openblas-0.3.29-1.el9.x86_64 2026-02-12T08:41:02.369 INFO:teuthology.orchestra.run.trial187.stdout: openblas-openmp-0.3.29-1.el9.x86_64 2026-02-12T08:41:02.369 INFO:teuthology.orchestra.run.trial187.stdout: protobuf-compiler-3.14.0-17.el9.x86_64 2026-02-12T08:41:02.369 INFO:teuthology.orchestra.run.trial187.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2026-02-12T08:41:02.369 INFO:teuthology.orchestra.run.trial187.stdout: python3-autocommand-2.2.2-8.el9.noarch 2026-02-12T08:41:02.369 INFO:teuthology.orchestra.run.trial187.stdout: python3-babel-2.9.1-2.el9.noarch 2026-02-12T08:41:02.369 INFO:teuthology.orchestra.run.trial187.stdout: python3-backports-tarfile-1.2.0-1.el9.noarch 2026-02-12T08:41:02.369 INFO:teuthology.orchestra.run.trial187.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2026-02-12T08:41:02.369 INFO:teuthology.orchestra.run.trial187.stdout: python3-cachetools-4.2.4-1.el9.noarch 2026-02-12T08:41:02.369 INFO:teuthology.orchestra.run.trial187.stdout: python3-certifi-2023.05.07-4.el9.noarch 2026-02-12T08:41:02.369 INFO:teuthology.orchestra.run.trial187.stdout: python3-cffi-1.14.5-5.el9.x86_64 2026-02-12T08:41:02.369 INFO:teuthology.orchestra.run.trial187.stdout: python3-cheroot-10.0.1-4.el9.noarch 2026-02-12T08:41:02.369 INFO:teuthology.orchestra.run.trial187.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2026-02-12T08:41:02.369 INFO:teuthology.orchestra.run.trial187.stdout: python3-cryptography-36.0.1-5.el9.x86_64 2026-02-12T08:41:02.369 INFO:teuthology.orchestra.run.trial187.stdout: python3-devel-3.9.25-2.el9.x86_64 2026-02-12T08:41:02.369 INFO:teuthology.orchestra.run.trial187.stdout: python3-google-auth-1:2.45.0-1.el9.noarch 2026-02-12T08:41:02.369 INFO:teuthology.orchestra.run.trial187.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2026-02-12T08:41:02.369 INFO:teuthology.orchestra.run.trial187.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2026-02-12T08:41:02.369 INFO:teuthology.orchestra.run.trial187.stdout: python3-influxdb-5.3.1-1.el9.noarch 2026-02-12T08:41:02.369 INFO:teuthology.orchestra.run.trial187.stdout: python3-isodate-0.6.1-3.el9.noarch 2026-02-12T08:41:02.369 INFO:teuthology.orchestra.run.trial187.stdout: python3-jaraco-8.2.1-3.el9.noarch 2026-02-12T08:41:02.370 INFO:teuthology.orchestra.run.trial187.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2026-02-12T08:41:02.370 INFO:teuthology.orchestra.run.trial187.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2026-02-12T08:41:02.370 INFO:teuthology.orchestra.run.trial187.stdout: python3-jaraco-context-6.0.1-3.el9.noarch 2026-02-12T08:41:02.370 INFO:teuthology.orchestra.run.trial187.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2026-02-12T08:41:02.370 INFO:teuthology.orchestra.run.trial187.stdout: python3-jaraco-text-4.0.0-2.el9.noarch 2026-02-12T08:41:02.370 INFO:teuthology.orchestra.run.trial187.stdout: python3-jinja2-2.11.3-8.el9.noarch 2026-02-12T08:41:02.370 INFO:teuthology.orchestra.run.trial187.stdout: python3-jmespath-1.0.1-1.el9.noarch 2026-02-12T08:41:02.370 INFO:teuthology.orchestra.run.trial187.stdout: python3-kubernetes-1:26.1.0-3.el9.noarch 2026-02-12T08:41:02.370 INFO:teuthology.orchestra.run.trial187.stdout: python3-libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-12T08:41:02.370 INFO:teuthology.orchestra.run.trial187.stdout: python3-lxml-4.6.5-3.el9.x86_64 2026-02-12T08:41:02.370 INFO:teuthology.orchestra.run.trial187.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2026-02-12T08:41:02.370 INFO:teuthology.orchestra.run.trial187.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2026-02-12T08:41:02.370 INFO:teuthology.orchestra.run.trial187.stdout: python3-msgpack-1.0.3-2.el9.x86_64 2026-02-12T08:41:02.370 INFO:teuthology.orchestra.run.trial187.stdout: python3-natsort-7.1.1-5.el9.noarch 2026-02-12T08:41:02.370 INFO:teuthology.orchestra.run.trial187.stdout: python3-numpy-1:1.23.5-2.el9.x86_64 2026-02-12T08:41:02.370 INFO:teuthology.orchestra.run.trial187.stdout: python3-numpy-f2py-1:1.23.5-2.el9.x86_64 2026-02-12T08:41:02.370 INFO:teuthology.orchestra.run.trial187.stdout: python3-packaging-20.9-5.el9.noarch 2026-02-12T08:41:02.370 INFO:teuthology.orchestra.run.trial187.stdout: python3-ply-3.11-14.el9.noarch 2026-02-12T08:41:02.370 INFO:teuthology.orchestra.run.trial187.stdout: python3-portend-3.1.0-2.el9.noarch 2026-02-12T08:41:02.370 INFO:teuthology.orchestra.run.trial187.stdout: python3-protobuf-3.14.0-17.el9.noarch 2026-02-12T08:41:02.370 INFO:teuthology.orchestra.run.trial187.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2026-02-12T08:41:02.370 INFO:teuthology.orchestra.run.trial187.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2026-02-12T08:41:02.370 INFO:teuthology.orchestra.run.trial187.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2026-02-12T08:41:02.370 INFO:teuthology.orchestra.run.trial187.stdout: python3-pycparser-2.20-6.el9.noarch 2026-02-12T08:41:02.370 INFO:teuthology.orchestra.run.trial187.stdout: python3-pyparsing-2.4.7-9.el9.noarch 2026-02-12T08:41:02.370 INFO:teuthology.orchestra.run.trial187.stdout: python3-repoze-lru-0.7-16.el9.noarch 2026-02-12T08:41:02.370 INFO:teuthology.orchestra.run.trial187.stdout: python3-requests-2.25.1-10.el9.noarch 2026-02-12T08:41:02.370 INFO:teuthology.orchestra.run.trial187.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2026-02-12T08:41:02.370 INFO:teuthology.orchestra.run.trial187.stdout: python3-routes-2.5.1-5.el9.noarch 2026-02-12T08:41:02.370 INFO:teuthology.orchestra.run.trial187.stdout: python3-rsa-4.9-2.el9.noarch 2026-02-12T08:41:02.370 INFO:teuthology.orchestra.run.trial187.stdout: python3-saml-1.16.0-1.el9.noarch 2026-02-12T08:41:02.370 INFO:teuthology.orchestra.run.trial187.stdout: python3-scipy-1.9.3-2.el9.x86_64 2026-02-12T08:41:02.370 INFO:teuthology.orchestra.run.trial187.stdout: python3-tempora-5.0.0-2.el9.noarch 2026-02-12T08:41:02.370 INFO:teuthology.orchestra.run.trial187.stdout: python3-toml-0.10.2-6.el9.noarch 2026-02-12T08:41:02.370 INFO:teuthology.orchestra.run.trial187.stdout: python3-typing-extensions-4.15.0-1.el9.noarch 2026-02-12T08:41:02.370 INFO:teuthology.orchestra.run.trial187.stdout: python3-urllib3-1.26.5-7.el9.noarch 2026-02-12T08:41:02.370 INFO:teuthology.orchestra.run.trial187.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2026-02-12T08:41:02.370 INFO:teuthology.orchestra.run.trial187.stdout: python3-xmlsec-1.3.13-1.el9.x86_64 2026-02-12T08:41:02.370 INFO:teuthology.orchestra.run.trial187.stdout: python3-xmltodict-0.12.0-15.el9.noarch 2026-02-12T08:41:02.370 INFO:teuthology.orchestra.run.trial187.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2026-02-12T08:41:02.370 INFO:teuthology.orchestra.run.trial187.stdout: unzip-6.0-59.el9.x86_64 2026-02-12T08:41:02.370 INFO:teuthology.orchestra.run.trial187.stdout: xmlsec1-1.2.29-13.el9.x86_64 2026-02-12T08:41:02.370 INFO:teuthology.orchestra.run.trial187.stdout: xmlsec1-openssl-1.2.29-13.el9.x86_64 2026-02-12T08:41:02.370 INFO:teuthology.orchestra.run.trial187.stdout: zip-3.0-35.el9.x86_64 2026-02-12T08:41:02.370 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:41:02.370 INFO:teuthology.orchestra.run.trial187.stdout:Complete! 2026-02-12T08:41:02.434 DEBUG:teuthology.orchestra.run.trial187:> sudo yum -y install ceph-base 2026-02-12T08:41:02.739 INFO:teuthology.orchestra.run.trial187.stdout:Last metadata expiration check: 0:03:51 ago on Thu 12 Feb 2026 08:37:11 AM UTC. 2026-02-12T08:41:02.806 INFO:teuthology.orchestra.run.trial187.stdout:Package ceph-base-2:20.3.0-5161.g2401a109.el9.x86_64 is already installed. 2026-02-12T08:41:02.825 INFO:teuthology.orchestra.run.trial187.stdout:Dependencies resolved. 2026-02-12T08:41:02.826 INFO:teuthology.orchestra.run.trial187.stdout:Nothing to do. 2026-02-12T08:41:02.826 INFO:teuthology.orchestra.run.trial187.stdout:Complete! 2026-02-12T08:41:02.847 DEBUG:teuthology.orchestra.run.trial187:> sudo yum -y install cephadm 2026-02-12T08:41:03.154 INFO:teuthology.orchestra.run.trial187.stdout:Last metadata expiration check: 0:03:52 ago on Thu 12 Feb 2026 08:37:11 AM UTC. 2026-02-12T08:41:03.221 INFO:teuthology.orchestra.run.trial187.stdout:Package cephadm-2:20.3.0-5161.g2401a109.el9.noarch is already installed. 2026-02-12T08:41:03.240 INFO:teuthology.orchestra.run.trial187.stdout:Dependencies resolved. 2026-02-12T08:41:03.240 INFO:teuthology.orchestra.run.trial187.stdout:Nothing to do. 2026-02-12T08:41:03.241 INFO:teuthology.orchestra.run.trial187.stdout:Complete! 2026-02-12T08:41:03.262 DEBUG:teuthology.orchestra.run.trial187:> sudo yum -y install ceph-immutable-object-cache 2026-02-12T08:41:03.569 INFO:teuthology.orchestra.run.trial187.stdout:Last metadata expiration check: 0:03:52 ago on Thu 12 Feb 2026 08:37:11 AM UTC. 2026-02-12T08:41:03.655 INFO:teuthology.orchestra.run.trial187.stdout:Dependencies resolved. 2026-02-12T08:41:03.656 INFO:teuthology.orchestra.run.trial187.stdout:================================================================================ 2026-02-12T08:41:03.656 INFO:teuthology.orchestra.run.trial187.stdout: Package Arch Version Repo Size 2026-02-12T08:41:03.656 INFO:teuthology.orchestra.run.trial187.stdout:================================================================================ 2026-02-12T08:41:03.657 INFO:teuthology.orchestra.run.trial187.stdout:Installing: 2026-02-12T08:41:03.657 INFO:teuthology.orchestra.run.trial187.stdout: ceph-immutable-object-cache x86_64 2:20.3.0-5161.g2401a109.el9 ceph 149 k 2026-02-12T08:41:03.657 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:41:03.657 INFO:teuthology.orchestra.run.trial187.stdout:Transaction Summary 2026-02-12T08:41:03.657 INFO:teuthology.orchestra.run.trial187.stdout:================================================================================ 2026-02-12T08:41:03.657 INFO:teuthology.orchestra.run.trial187.stdout:Install 1 Package 2026-02-12T08:41:03.657 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:41:03.657 INFO:teuthology.orchestra.run.trial187.stdout:Total download size: 149 k 2026-02-12T08:41:03.657 INFO:teuthology.orchestra.run.trial187.stdout:Installed size: 455 k 2026-02-12T08:41:03.657 INFO:teuthology.orchestra.run.trial187.stdout:Downloading Packages: 2026-02-12T08:41:04.144 INFO:teuthology.orchestra.run.trial187.stdout:ceph-immutable-object-cache-20.3.0-5161.g2401a1 306 kB/s | 149 kB 00:00 2026-02-12T08:41:04.144 INFO:teuthology.orchestra.run.trial187.stdout:-------------------------------------------------------------------------------- 2026-02-12T08:41:04.145 INFO:teuthology.orchestra.run.trial187.stdout:Total 306 kB/s | 149 kB 00:00 2026-02-12T08:41:04.145 INFO:teuthology.orchestra.run.trial187.stdout:Running transaction check 2026-02-12T08:41:04.149 INFO:teuthology.orchestra.run.trial187.stdout:Transaction check succeeded. 2026-02-12T08:41:04.149 INFO:teuthology.orchestra.run.trial187.stdout:Running transaction test 2026-02-12T08:41:04.181 INFO:teuthology.orchestra.run.trial187.stdout:Transaction test succeeded. 2026-02-12T08:41:04.181 INFO:teuthology.orchestra.run.trial187.stdout:Running transaction 2026-02-12T08:41:04.236 INFO:teuthology.orchestra.run.trial187.stdout: Preparing : 1/1 2026-02-12T08:41:04.238 INFO:teuthology.orchestra.run.trial187.stdout: Installing : ceph-immutable-object-cache-2:20.3.0-5161.g2401a109. 1/1 2026-02-12T08:41:04.261 INFO:teuthology.orchestra.run.trial187.stdout: Running scriptlet: ceph-immutable-object-cache-2:20.3.0-5161.g2401a109. 1/1 2026-02-12T08:41:04.261 INFO:teuthology.orchestra.run.trial187.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-12T08:41:04.262 INFO:teuthology.orchestra.run.trial187.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2026-02-12T08:41:04.262 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:41:04.590 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : ceph-immutable-object-cache-2:20.3.0-5161.g2401a109. 1/1 2026-02-12T08:41:04.591 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:41:04.591 INFO:teuthology.orchestra.run.trial187.stdout:Installed: 2026-02-12T08:41:04.591 INFO:teuthology.orchestra.run.trial187.stdout: ceph-immutable-object-cache-2:20.3.0-5161.g2401a109.el9.x86_64 2026-02-12T08:41:04.591 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:41:04.591 INFO:teuthology.orchestra.run.trial187.stdout:Complete! 2026-02-12T08:41:04.619 DEBUG:teuthology.orchestra.run.trial187:> sudo yum -y install ceph-mgr 2026-02-12T08:41:04.930 INFO:teuthology.orchestra.run.trial187.stdout:Last metadata expiration check: 0:03:53 ago on Thu 12 Feb 2026 08:37:11 AM UTC. 2026-02-12T08:41:04.997 INFO:teuthology.orchestra.run.trial187.stdout:Package ceph-mgr-2:20.3.0-5161.g2401a109.el9.x86_64 is already installed. 2026-02-12T08:41:05.015 INFO:teuthology.orchestra.run.trial187.stdout:Dependencies resolved. 2026-02-12T08:41:05.016 INFO:teuthology.orchestra.run.trial187.stdout:Nothing to do. 2026-02-12T08:41:05.016 INFO:teuthology.orchestra.run.trial187.stdout:Complete! 2026-02-12T08:41:05.038 DEBUG:teuthology.orchestra.run.trial187:> sudo yum -y install ceph-mgr-dashboard 2026-02-12T08:41:05.342 INFO:teuthology.orchestra.run.trial187.stdout:Last metadata expiration check: 0:03:54 ago on Thu 12 Feb 2026 08:37:11 AM UTC. 2026-02-12T08:41:05.409 INFO:teuthology.orchestra.run.trial187.stdout:Package ceph-mgr-dashboard-2:20.3.0-5161.g2401a109.el9.noarch is already installed. 2026-02-12T08:41:05.428 INFO:teuthology.orchestra.run.trial187.stdout:Dependencies resolved. 2026-02-12T08:41:05.429 INFO:teuthology.orchestra.run.trial187.stdout:Nothing to do. 2026-02-12T08:41:05.429 INFO:teuthology.orchestra.run.trial187.stdout:Complete! 2026-02-12T08:41:05.450 DEBUG:teuthology.orchestra.run.trial187:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-12T08:41:05.753 INFO:teuthology.orchestra.run.trial187.stdout:Last metadata expiration check: 0:03:54 ago on Thu 12 Feb 2026 08:37:11 AM UTC. 2026-02-12T08:41:05.820 INFO:teuthology.orchestra.run.trial187.stdout:Package ceph-mgr-diskprediction-local-2:20.3.0-5161.g2401a109.el9.noarch is already installed. 2026-02-12T08:41:05.838 INFO:teuthology.orchestra.run.trial187.stdout:Dependencies resolved. 2026-02-12T08:41:05.839 INFO:teuthology.orchestra.run.trial187.stdout:Nothing to do. 2026-02-12T08:41:05.839 INFO:teuthology.orchestra.run.trial187.stdout:Complete! 2026-02-12T08:41:05.861 DEBUG:teuthology.orchestra.run.trial187:> sudo yum -y install ceph-mgr-rook 2026-02-12T08:41:06.162 INFO:teuthology.orchestra.run.trial187.stdout:Last metadata expiration check: 0:03:55 ago on Thu 12 Feb 2026 08:37:11 AM UTC. 2026-02-12T08:41:06.229 INFO:teuthology.orchestra.run.trial187.stdout:Package ceph-mgr-rook-2:20.3.0-5161.g2401a109.el9.noarch is already installed. 2026-02-12T08:41:06.247 INFO:teuthology.orchestra.run.trial187.stdout:Dependencies resolved. 2026-02-12T08:41:06.248 INFO:teuthology.orchestra.run.trial187.stdout:Nothing to do. 2026-02-12T08:41:06.248 INFO:teuthology.orchestra.run.trial187.stdout:Complete! 2026-02-12T08:41:06.269 DEBUG:teuthology.orchestra.run.trial187:> sudo yum -y install ceph-mgr-cephadm 2026-02-12T08:41:06.574 INFO:teuthology.orchestra.run.trial187.stdout:Last metadata expiration check: 0:03:55 ago on Thu 12 Feb 2026 08:37:11 AM UTC. 2026-02-12T08:41:06.639 INFO:teuthology.orchestra.run.trial187.stdout:Package ceph-mgr-cephadm-2:20.3.0-5161.g2401a109.el9.noarch is already installed. 2026-02-12T08:41:06.658 INFO:teuthology.orchestra.run.trial187.stdout:Dependencies resolved. 2026-02-12T08:41:06.659 INFO:teuthology.orchestra.run.trial187.stdout:Nothing to do. 2026-02-12T08:41:06.659 INFO:teuthology.orchestra.run.trial187.stdout:Complete! 2026-02-12T08:41:06.680 DEBUG:teuthology.orchestra.run.trial187:> sudo yum -y install ceph-osd 2026-02-12T08:41:06.978 INFO:teuthology.orchestra.run.trial187.stdout:Last metadata expiration check: 0:03:55 ago on Thu 12 Feb 2026 08:37:11 AM UTC. 2026-02-12T08:41:07.045 INFO:teuthology.orchestra.run.trial187.stdout:Package ceph-osd-2:20.3.0-5161.g2401a109.el9.x86_64 is already installed. 2026-02-12T08:41:07.063 INFO:teuthology.orchestra.run.trial187.stdout:Dependencies resolved. 2026-02-12T08:41:07.064 INFO:teuthology.orchestra.run.trial187.stdout:Nothing to do. 2026-02-12T08:41:07.064 INFO:teuthology.orchestra.run.trial187.stdout:Complete! 2026-02-12T08:41:07.085 DEBUG:teuthology.orchestra.run.trial187:> sudo yum -y install ceph-osd-classic 2026-02-12T08:41:07.382 INFO:teuthology.orchestra.run.trial187.stdout:Last metadata expiration check: 0:03:56 ago on Thu 12 Feb 2026 08:37:11 AM UTC. 2026-02-12T08:41:07.449 INFO:teuthology.orchestra.run.trial187.stdout:Package ceph-osd-classic-2:20.3.0-5161.g2401a109.el9.x86_64 is already installed. 2026-02-12T08:41:07.468 INFO:teuthology.orchestra.run.trial187.stdout:Dependencies resolved. 2026-02-12T08:41:07.469 INFO:teuthology.orchestra.run.trial187.stdout:Nothing to do. 2026-02-12T08:41:07.469 INFO:teuthology.orchestra.run.trial187.stdout:Complete! 2026-02-12T08:41:07.490 DEBUG:teuthology.orchestra.run.trial187:> sudo yum -y install ceph-fuse 2026-02-12T08:41:07.790 INFO:teuthology.orchestra.run.trial187.stdout:Last metadata expiration check: 0:03:56 ago on Thu 12 Feb 2026 08:37:11 AM UTC. 2026-02-12T08:41:07.876 INFO:teuthology.orchestra.run.trial187.stdout:Dependencies resolved. 2026-02-12T08:41:07.876 INFO:teuthology.orchestra.run.trial187.stdout:================================================================================ 2026-02-12T08:41:07.876 INFO:teuthology.orchestra.run.trial187.stdout: Package Arch Version Repo Size 2026-02-12T08:41:07.876 INFO:teuthology.orchestra.run.trial187.stdout:================================================================================ 2026-02-12T08:41:07.876 INFO:teuthology.orchestra.run.trial187.stdout:Installing: 2026-02-12T08:41:07.876 INFO:teuthology.orchestra.run.trial187.stdout: ceph-fuse x86_64 2:20.3.0-5161.g2401a109.el9 ceph 914 k 2026-02-12T08:41:07.876 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:41:07.876 INFO:teuthology.orchestra.run.trial187.stdout:Transaction Summary 2026-02-12T08:41:07.877 INFO:teuthology.orchestra.run.trial187.stdout:================================================================================ 2026-02-12T08:41:07.877 INFO:teuthology.orchestra.run.trial187.stdout:Install 1 Package 2026-02-12T08:41:07.877 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:41:07.877 INFO:teuthology.orchestra.run.trial187.stdout:Total download size: 914 k 2026-02-12T08:41:07.877 INFO:teuthology.orchestra.run.trial187.stdout:Installed size: 2.9 M 2026-02-12T08:41:07.877 INFO:teuthology.orchestra.run.trial187.stdout:Downloading Packages: 2026-02-12T08:41:08.759 INFO:teuthology.orchestra.run.trial187.stdout:ceph-fuse-20.3.0-5161.g2401a109.el9.x86_64.rpm 1.0 MB/s | 914 kB 00:00 2026-02-12T08:41:08.759 INFO:teuthology.orchestra.run.trial187.stdout:-------------------------------------------------------------------------------- 2026-02-12T08:41:08.759 INFO:teuthology.orchestra.run.trial187.stdout:Total 1.0 MB/s | 914 kB 00:00 2026-02-12T08:41:08.759 INFO:teuthology.orchestra.run.trial187.stdout:Running transaction check 2026-02-12T08:41:08.764 INFO:teuthology.orchestra.run.trial187.stdout:Transaction check succeeded. 2026-02-12T08:41:08.764 INFO:teuthology.orchestra.run.trial187.stdout:Running transaction test 2026-02-12T08:41:08.798 INFO:teuthology.orchestra.run.trial187.stdout:Transaction test succeeded. 2026-02-12T08:41:08.798 INFO:teuthology.orchestra.run.trial187.stdout:Running transaction 2026-02-12T08:41:08.889 INFO:teuthology.orchestra.run.trial187.stdout: Preparing : 1/1 2026-02-12T08:41:08.906 INFO:teuthology.orchestra.run.trial187.stdout: Installing : ceph-fuse-2:20.3.0-5161.g2401a109.el9.x86_64 1/1 2026-02-12T08:41:09.119 INFO:teuthology.orchestra.run.trial187.stdout: Running scriptlet: ceph-fuse-2:20.3.0-5161.g2401a109.el9.x86_64 1/1 2026-02-12T08:41:09.176 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : ceph-fuse-2:20.3.0-5161.g2401a109.el9.x86_64 1/1 2026-02-12T08:41:09.176 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:41:09.176 INFO:teuthology.orchestra.run.trial187.stdout:Installed: 2026-02-12T08:41:09.176 INFO:teuthology.orchestra.run.trial187.stdout: ceph-fuse-2:20.3.0-5161.g2401a109.el9.x86_64 2026-02-12T08:41:09.176 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:41:09.176 INFO:teuthology.orchestra.run.trial187.stdout:Complete! 2026-02-12T08:41:09.205 DEBUG:teuthology.orchestra.run.trial187:> sudo yum -y install ceph-volume 2026-02-12T08:41:09.510 INFO:teuthology.orchestra.run.trial187.stdout:Last metadata expiration check: 0:03:58 ago on Thu 12 Feb 2026 08:37:11 AM UTC. 2026-02-12T08:41:09.575 INFO:teuthology.orchestra.run.trial187.stdout:Package ceph-volume-2:20.3.0-5161.g2401a109.el9.noarch is already installed. 2026-02-12T08:41:09.592 INFO:teuthology.orchestra.run.trial187.stdout:Dependencies resolved. 2026-02-12T08:41:09.592 INFO:teuthology.orchestra.run.trial187.stdout:Nothing to do. 2026-02-12T08:41:09.593 INFO:teuthology.orchestra.run.trial187.stdout:Complete! 2026-02-12T08:41:09.614 DEBUG:teuthology.orchestra.run.trial187:> sudo yum -y install librados-devel 2026-02-12T08:41:09.919 INFO:teuthology.orchestra.run.trial187.stdout:Last metadata expiration check: 0:03:58 ago on Thu 12 Feb 2026 08:37:11 AM UTC. 2026-02-12T08:41:10.003 INFO:teuthology.orchestra.run.trial187.stdout:Dependencies resolved. 2026-02-12T08:41:10.003 INFO:teuthology.orchestra.run.trial187.stdout:================================================================================ 2026-02-12T08:41:10.004 INFO:teuthology.orchestra.run.trial187.stdout: Package Arch Version Repo Size 2026-02-12T08:41:10.004 INFO:teuthology.orchestra.run.trial187.stdout:================================================================================ 2026-02-12T08:41:10.004 INFO:teuthology.orchestra.run.trial187.stdout:Installing: 2026-02-12T08:41:10.004 INFO:teuthology.orchestra.run.trial187.stdout: librados-devel x86_64 2:20.3.0-5161.g2401a109.el9 ceph 124 k 2026-02-12T08:41:10.004 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:41:10.004 INFO:teuthology.orchestra.run.trial187.stdout:Transaction Summary 2026-02-12T08:41:10.004 INFO:teuthology.orchestra.run.trial187.stdout:================================================================================ 2026-02-12T08:41:10.004 INFO:teuthology.orchestra.run.trial187.stdout:Install 1 Package 2026-02-12T08:41:10.004 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:41:10.004 INFO:teuthology.orchestra.run.trial187.stdout:Total download size: 124 k 2026-02-12T08:41:10.005 INFO:teuthology.orchestra.run.trial187.stdout:Installed size: 434 k 2026-02-12T08:41:10.005 INFO:teuthology.orchestra.run.trial187.stdout:Downloading Packages: 2026-02-12T08:41:10.486 INFO:teuthology.orchestra.run.trial187.stdout:librados-devel-20.3.0-5161.g2401a109.el9.x86_64 256 kB/s | 124 kB 00:00 2026-02-12T08:41:10.486 INFO:teuthology.orchestra.run.trial187.stdout:-------------------------------------------------------------------------------- 2026-02-12T08:41:10.486 INFO:teuthology.orchestra.run.trial187.stdout:Total 256 kB/s | 124 kB 00:00 2026-02-12T08:41:10.486 INFO:teuthology.orchestra.run.trial187.stdout:Running transaction check 2026-02-12T08:41:10.489 INFO:teuthology.orchestra.run.trial187.stdout:Transaction check succeeded. 2026-02-12T08:41:10.489 INFO:teuthology.orchestra.run.trial187.stdout:Running transaction test 2026-02-12T08:41:10.522 INFO:teuthology.orchestra.run.trial187.stdout:Transaction test succeeded. 2026-02-12T08:41:10.522 INFO:teuthology.orchestra.run.trial187.stdout:Running transaction 2026-02-12T08:41:10.576 INFO:teuthology.orchestra.run.trial187.stdout: Preparing : 1/1 2026-02-12T08:41:10.593 INFO:teuthology.orchestra.run.trial187.stdout: Installing : librados-devel-2:20.3.0-5161.g2401a109.el9.x86_64 1/1 2026-02-12T08:41:10.676 INFO:teuthology.orchestra.run.trial187.stdout: Running scriptlet: librados-devel-2:20.3.0-5161.g2401a109.el9.x86_64 1/1 2026-02-12T08:41:10.733 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : librados-devel-2:20.3.0-5161.g2401a109.el9.x86_64 1/1 2026-02-12T08:41:10.733 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:41:10.734 INFO:teuthology.orchestra.run.trial187.stdout:Installed: 2026-02-12T08:41:10.734 INFO:teuthology.orchestra.run.trial187.stdout: librados-devel-2:20.3.0-5161.g2401a109.el9.x86_64 2026-02-12T08:41:10.734 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:41:10.734 INFO:teuthology.orchestra.run.trial187.stdout:Complete! 2026-02-12T08:41:10.760 DEBUG:teuthology.orchestra.run.trial187:> sudo yum -y install libcephfs2 2026-02-12T08:41:11.062 INFO:teuthology.orchestra.run.trial187.stdout:Last metadata expiration check: 0:04:00 ago on Thu 12 Feb 2026 08:37:11 AM UTC. 2026-02-12T08:41:11.128 INFO:teuthology.orchestra.run.trial187.stdout:Package libcephfs2-2:20.3.0-5161.g2401a109.el9.x86_64 is already installed. 2026-02-12T08:41:11.145 INFO:teuthology.orchestra.run.trial187.stdout:Dependencies resolved. 2026-02-12T08:41:11.146 INFO:teuthology.orchestra.run.trial187.stdout:Nothing to do. 2026-02-12T08:41:11.146 INFO:teuthology.orchestra.run.trial187.stdout:Complete! 2026-02-12T08:41:11.167 DEBUG:teuthology.orchestra.run.trial187:> sudo yum -y install libcephfs-devel 2026-02-12T08:41:11.471 INFO:teuthology.orchestra.run.trial187.stdout:Last metadata expiration check: 0:04:00 ago on Thu 12 Feb 2026 08:37:11 AM UTC. 2026-02-12T08:41:11.557 INFO:teuthology.orchestra.run.trial187.stdout:Dependencies resolved. 2026-02-12T08:41:11.557 INFO:teuthology.orchestra.run.trial187.stdout:================================================================================ 2026-02-12T08:41:11.557 INFO:teuthology.orchestra.run.trial187.stdout: Package Arch Version Repo Size 2026-02-12T08:41:11.557 INFO:teuthology.orchestra.run.trial187.stdout:================================================================================ 2026-02-12T08:41:11.557 INFO:teuthology.orchestra.run.trial187.stdout:Installing: 2026-02-12T08:41:11.557 INFO:teuthology.orchestra.run.trial187.stdout: libcephfs-devel x86_64 2:20.3.0-5161.g2401a109.el9 ceph 37 k 2026-02-12T08:41:11.557 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:41:11.557 INFO:teuthology.orchestra.run.trial187.stdout:Transaction Summary 2026-02-12T08:41:11.557 INFO:teuthology.orchestra.run.trial187.stdout:================================================================================ 2026-02-12T08:41:11.557 INFO:teuthology.orchestra.run.trial187.stdout:Install 1 Package 2026-02-12T08:41:11.557 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:41:11.557 INFO:teuthology.orchestra.run.trial187.stdout:Total download size: 37 k 2026-02-12T08:41:11.557 INFO:teuthology.orchestra.run.trial187.stdout:Installed size: 171 k 2026-02-12T08:41:11.557 INFO:teuthology.orchestra.run.trial187.stdout:Downloading Packages: 2026-02-12T08:41:11.911 INFO:teuthology.orchestra.run.trial187.stdout:libcephfs-devel-20.3.0-5161.g2401a109.el9.x86_6 106 kB/s | 37 kB 00:00 2026-02-12T08:41:11.912 INFO:teuthology.orchestra.run.trial187.stdout:-------------------------------------------------------------------------------- 2026-02-12T08:41:11.913 INFO:teuthology.orchestra.run.trial187.stdout:Total 105 kB/s | 37 kB 00:00 2026-02-12T08:41:11.913 INFO:teuthology.orchestra.run.trial187.stdout:Running transaction check 2026-02-12T08:41:11.915 INFO:teuthology.orchestra.run.trial187.stdout:Transaction check succeeded. 2026-02-12T08:41:11.915 INFO:teuthology.orchestra.run.trial187.stdout:Running transaction test 2026-02-12T08:41:11.918 INFO:teuthology.orchestra.run.trial187.stdout:Transaction test succeeded. 2026-02-12T08:41:11.918 INFO:teuthology.orchestra.run.trial187.stdout:Running transaction 2026-02-12T08:41:11.940 INFO:teuthology.orchestra.run.trial187.stdout: Preparing : 1/1 2026-02-12T08:41:11.957 INFO:teuthology.orchestra.run.trial187.stdout: Installing : libcephfs-devel-2:20.3.0-5161.g2401a109.el9.x86_64 1/1 2026-02-12T08:41:12.011 INFO:teuthology.orchestra.run.trial187.stdout: Running scriptlet: libcephfs-devel-2:20.3.0-5161.g2401a109.el9.x86_64 1/1 2026-02-12T08:41:12.067 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : libcephfs-devel-2:20.3.0-5161.g2401a109.el9.x86_64 1/1 2026-02-12T08:41:12.067 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:41:12.067 INFO:teuthology.orchestra.run.trial187.stdout:Installed: 2026-02-12T08:41:12.067 INFO:teuthology.orchestra.run.trial187.stdout: libcephfs-devel-2:20.3.0-5161.g2401a109.el9.x86_64 2026-02-12T08:41:12.067 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:41:12.067 INFO:teuthology.orchestra.run.trial187.stdout:Complete! 2026-02-12T08:41:12.094 DEBUG:teuthology.orchestra.run.trial187:> sudo yum -y install librados2 2026-02-12T08:41:12.401 INFO:teuthology.orchestra.run.trial187.stdout:Last metadata expiration check: 0:04:01 ago on Thu 12 Feb 2026 08:37:11 AM UTC. 2026-02-12T08:41:12.467 INFO:teuthology.orchestra.run.trial187.stdout:Package librados2-2:20.3.0-5161.g2401a109.el9.x86_64 is already installed. 2026-02-12T08:41:12.484 INFO:teuthology.orchestra.run.trial187.stdout:Dependencies resolved. 2026-02-12T08:41:12.485 INFO:teuthology.orchestra.run.trial187.stdout:Nothing to do. 2026-02-12T08:41:12.485 INFO:teuthology.orchestra.run.trial187.stdout:Complete! 2026-02-12T08:41:12.506 DEBUG:teuthology.orchestra.run.trial187:> sudo yum -y install librbd1 2026-02-12T08:41:12.805 INFO:teuthology.orchestra.run.trial187.stdout:Last metadata expiration check: 0:04:01 ago on Thu 12 Feb 2026 08:37:11 AM UTC. 2026-02-12T08:41:12.871 INFO:teuthology.orchestra.run.trial187.stdout:Package librbd1-2:20.3.0-5161.g2401a109.el9.x86_64 is already installed. 2026-02-12T08:41:12.888 INFO:teuthology.orchestra.run.trial187.stdout:Dependencies resolved. 2026-02-12T08:41:12.889 INFO:teuthology.orchestra.run.trial187.stdout:Nothing to do. 2026-02-12T08:41:12.889 INFO:teuthology.orchestra.run.trial187.stdout:Complete! 2026-02-12T08:41:12.910 DEBUG:teuthology.orchestra.run.trial187:> sudo yum -y install python3-rados 2026-02-12T08:41:13.213 INFO:teuthology.orchestra.run.trial187.stdout:Last metadata expiration check: 0:04:02 ago on Thu 12 Feb 2026 08:37:11 AM UTC. 2026-02-12T08:41:13.280 INFO:teuthology.orchestra.run.trial187.stdout:Package python3-rados-2:20.3.0-5161.g2401a109.el9.x86_64 is already installed. 2026-02-12T08:41:13.297 INFO:teuthology.orchestra.run.trial187.stdout:Dependencies resolved. 2026-02-12T08:41:13.298 INFO:teuthology.orchestra.run.trial187.stdout:Nothing to do. 2026-02-12T08:41:13.298 INFO:teuthology.orchestra.run.trial187.stdout:Complete! 2026-02-12T08:41:13.319 DEBUG:teuthology.orchestra.run.trial187:> sudo yum -y install python3-rgw 2026-02-12T08:41:13.624 INFO:teuthology.orchestra.run.trial187.stdout:Last metadata expiration check: 0:04:02 ago on Thu 12 Feb 2026 08:37:11 AM UTC. 2026-02-12T08:41:13.691 INFO:teuthology.orchestra.run.trial187.stdout:Package python3-rgw-2:20.3.0-5161.g2401a109.el9.x86_64 is already installed. 2026-02-12T08:41:13.708 INFO:teuthology.orchestra.run.trial187.stdout:Dependencies resolved. 2026-02-12T08:41:13.709 INFO:teuthology.orchestra.run.trial187.stdout:Nothing to do. 2026-02-12T08:41:13.709 INFO:teuthology.orchestra.run.trial187.stdout:Complete! 2026-02-12T08:41:13.731 DEBUG:teuthology.orchestra.run.trial187:> sudo yum -y install python3-cephfs 2026-02-12T08:41:14.034 INFO:teuthology.orchestra.run.trial187.stdout:Last metadata expiration check: 0:04:03 ago on Thu 12 Feb 2026 08:37:11 AM UTC. 2026-02-12T08:41:14.100 INFO:teuthology.orchestra.run.trial187.stdout:Package python3-cephfs-2:20.3.0-5161.g2401a109.el9.x86_64 is already installed. 2026-02-12T08:41:14.118 INFO:teuthology.orchestra.run.trial187.stdout:Dependencies resolved. 2026-02-12T08:41:14.119 INFO:teuthology.orchestra.run.trial187.stdout:Nothing to do. 2026-02-12T08:41:14.119 INFO:teuthology.orchestra.run.trial187.stdout:Complete! 2026-02-12T08:41:14.140 DEBUG:teuthology.orchestra.run.trial187:> sudo yum -y install python3-rbd 2026-02-12T08:41:14.447 INFO:teuthology.orchestra.run.trial187.stdout:Last metadata expiration check: 0:04:03 ago on Thu 12 Feb 2026 08:37:11 AM UTC. 2026-02-12T08:41:14.513 INFO:teuthology.orchestra.run.trial187.stdout:Package python3-rbd-2:20.3.0-5161.g2401a109.el9.x86_64 is already installed. 2026-02-12T08:41:14.532 INFO:teuthology.orchestra.run.trial187.stdout:Dependencies resolved. 2026-02-12T08:41:14.533 INFO:teuthology.orchestra.run.trial187.stdout:Nothing to do. 2026-02-12T08:41:14.533 INFO:teuthology.orchestra.run.trial187.stdout:Complete! 2026-02-12T08:41:14.555 DEBUG:teuthology.orchestra.run.trial187:> sudo yum -y install rbd-fuse 2026-02-12T08:41:14.864 INFO:teuthology.orchestra.run.trial187.stdout:Last metadata expiration check: 0:04:03 ago on Thu 12 Feb 2026 08:37:11 AM UTC. 2026-02-12T08:41:14.950 INFO:teuthology.orchestra.run.trial187.stdout:Dependencies resolved. 2026-02-12T08:41:14.950 INFO:teuthology.orchestra.run.trial187.stdout:================================================================================ 2026-02-12T08:41:14.950 INFO:teuthology.orchestra.run.trial187.stdout: Package Architecture Version Repository Size 2026-02-12T08:41:14.950 INFO:teuthology.orchestra.run.trial187.stdout:================================================================================ 2026-02-12T08:41:14.951 INFO:teuthology.orchestra.run.trial187.stdout:Installing: 2026-02-12T08:41:14.951 INFO:teuthology.orchestra.run.trial187.stdout: rbd-fuse x86_64 2:20.3.0-5161.g2401a109.el9 ceph 87 k 2026-02-12T08:41:14.951 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:41:14.951 INFO:teuthology.orchestra.run.trial187.stdout:Transaction Summary 2026-02-12T08:41:14.951 INFO:teuthology.orchestra.run.trial187.stdout:================================================================================ 2026-02-12T08:41:14.951 INFO:teuthology.orchestra.run.trial187.stdout:Install 1 Package 2026-02-12T08:41:14.951 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:41:14.951 INFO:teuthology.orchestra.run.trial187.stdout:Total download size: 87 k 2026-02-12T08:41:14.951 INFO:teuthology.orchestra.run.trial187.stdout:Installed size: 236 k 2026-02-12T08:41:14.951 INFO:teuthology.orchestra.run.trial187.stdout:Downloading Packages: 2026-02-12T08:41:15.347 INFO:teuthology.orchestra.run.trial187.stdout:rbd-fuse-20.3.0-5161.g2401a109.el9.x86_64.rpm 219 kB/s | 87 kB 00:00 2026-02-12T08:41:15.347 INFO:teuthology.orchestra.run.trial187.stdout:-------------------------------------------------------------------------------- 2026-02-12T08:41:15.347 INFO:teuthology.orchestra.run.trial187.stdout:Total 218 kB/s | 87 kB 00:00 2026-02-12T08:41:15.347 INFO:teuthology.orchestra.run.trial187.stdout:Running transaction check 2026-02-12T08:41:15.351 INFO:teuthology.orchestra.run.trial187.stdout:Transaction check succeeded. 2026-02-12T08:41:15.351 INFO:teuthology.orchestra.run.trial187.stdout:Running transaction test 2026-02-12T08:41:15.383 INFO:teuthology.orchestra.run.trial187.stdout:Transaction test succeeded. 2026-02-12T08:41:15.383 INFO:teuthology.orchestra.run.trial187.stdout:Running transaction 2026-02-12T08:41:15.435 INFO:teuthology.orchestra.run.trial187.stdout: Preparing : 1/1 2026-02-12T08:41:15.452 INFO:teuthology.orchestra.run.trial187.stdout: Installing : rbd-fuse-2:20.3.0-5161.g2401a109.el9.x86_64 1/1 2026-02-12T08:41:15.533 INFO:teuthology.orchestra.run.trial187.stdout: Running scriptlet: rbd-fuse-2:20.3.0-5161.g2401a109.el9.x86_64 1/1 2026-02-12T08:41:15.589 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : rbd-fuse-2:20.3.0-5161.g2401a109.el9.x86_64 1/1 2026-02-12T08:41:15.590 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:41:15.590 INFO:teuthology.orchestra.run.trial187.stdout:Installed: 2026-02-12T08:41:15.590 INFO:teuthology.orchestra.run.trial187.stdout: rbd-fuse-2:20.3.0-5161.g2401a109.el9.x86_64 2026-02-12T08:41:15.590 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:41:15.590 INFO:teuthology.orchestra.run.trial187.stdout:Complete! 2026-02-12T08:41:15.619 DEBUG:teuthology.orchestra.run.trial187:> sudo yum -y install rbd-mirror 2026-02-12T08:41:15.930 INFO:teuthology.orchestra.run.trial187.stdout:Last metadata expiration check: 0:04:04 ago on Thu 12 Feb 2026 08:37:11 AM UTC. 2026-02-12T08:41:16.014 INFO:teuthology.orchestra.run.trial187.stdout:Dependencies resolved. 2026-02-12T08:41:16.015 INFO:teuthology.orchestra.run.trial187.stdout:================================================================================ 2026-02-12T08:41:16.015 INFO:teuthology.orchestra.run.trial187.stdout: Package Arch Version Repo Size 2026-02-12T08:41:16.015 INFO:teuthology.orchestra.run.trial187.stdout:================================================================================ 2026-02-12T08:41:16.015 INFO:teuthology.orchestra.run.trial187.stdout:Installing: 2026-02-12T08:41:16.015 INFO:teuthology.orchestra.run.trial187.stdout: rbd-mirror x86_64 2:20.3.0-5161.g2401a109.el9 ceph 2.6 M 2026-02-12T08:41:16.015 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:41:16.016 INFO:teuthology.orchestra.run.trial187.stdout:Transaction Summary 2026-02-12T08:41:16.016 INFO:teuthology.orchestra.run.trial187.stdout:================================================================================ 2026-02-12T08:41:16.016 INFO:teuthology.orchestra.run.trial187.stdout:Install 1 Package 2026-02-12T08:41:16.016 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:41:16.016 INFO:teuthology.orchestra.run.trial187.stdout:Total download size: 2.6 M 2026-02-12T08:41:16.016 INFO:teuthology.orchestra.run.trial187.stdout:Installed size: 11 M 2026-02-12T08:41:16.016 INFO:teuthology.orchestra.run.trial187.stdout:Downloading Packages: 2026-02-12T08:41:16.943 INFO:teuthology.orchestra.run.trial187.stdout:rbd-mirror-20.3.0-5161.g2401a109.el9.x86_64.rpm 2.8 MB/s | 2.6 MB 00:00 2026-02-12T08:41:16.944 INFO:teuthology.orchestra.run.trial187.stdout:-------------------------------------------------------------------------------- 2026-02-12T08:41:16.944 INFO:teuthology.orchestra.run.trial187.stdout:Total 2.8 MB/s | 2.6 MB 00:00 2026-02-12T08:41:16.944 INFO:teuthology.orchestra.run.trial187.stdout:Running transaction check 2026-02-12T08:41:16.949 INFO:teuthology.orchestra.run.trial187.stdout:Transaction check succeeded. 2026-02-12T08:41:16.949 INFO:teuthology.orchestra.run.trial187.stdout:Running transaction test 2026-02-12T08:41:16.988 INFO:teuthology.orchestra.run.trial187.stdout:Transaction test succeeded. 2026-02-12T08:41:16.988 INFO:teuthology.orchestra.run.trial187.stdout:Running transaction 2026-02-12T08:41:17.150 INFO:teuthology.orchestra.run.trial187.stdout: Preparing : 1/1 2026-02-12T08:41:17.152 INFO:teuthology.orchestra.run.trial187.stdout: Installing : rbd-mirror-2:20.3.0-5161.g2401a109.el9.x86_64 1/1 2026-02-12T08:41:17.175 INFO:teuthology.orchestra.run.trial187.stdout: Running scriptlet: rbd-mirror-2:20.3.0-5161.g2401a109.el9.x86_64 1/1 2026-02-12T08:41:17.175 INFO:teuthology.orchestra.run.trial187.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-12T08:41:17.175 INFO:teuthology.orchestra.run.trial187.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2026-02-12T08:41:17.176 INFO:teuthology.orchestra.run.trial187.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-12T08:41:17.176 INFO:teuthology.orchestra.run.trial187.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-12T08:41:17.176 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:41:17.498 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : rbd-mirror-2:20.3.0-5161.g2401a109.el9.x86_64 1/1 2026-02-12T08:41:17.498 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:41:17.499 INFO:teuthology.orchestra.run.trial187.stdout:Installed: 2026-02-12T08:41:17.499 INFO:teuthology.orchestra.run.trial187.stdout: rbd-mirror-2:20.3.0-5161.g2401a109.el9.x86_64 2026-02-12T08:41:17.499 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:41:17.499 INFO:teuthology.orchestra.run.trial187.stdout:Complete! 2026-02-12T08:41:17.528 DEBUG:teuthology.orchestra.run.trial187:> sudo yum -y install rbd-nbd 2026-02-12T08:41:17.837 INFO:teuthology.orchestra.run.trial187.stdout:Last metadata expiration check: 0:04:06 ago on Thu 12 Feb 2026 08:37:11 AM UTC. 2026-02-12T08:41:17.923 INFO:teuthology.orchestra.run.trial187.stdout:Dependencies resolved. 2026-02-12T08:41:17.923 INFO:teuthology.orchestra.run.trial187.stdout:================================================================================ 2026-02-12T08:41:17.923 INFO:teuthology.orchestra.run.trial187.stdout: Package Architecture Version Repository Size 2026-02-12T08:41:17.923 INFO:teuthology.orchestra.run.trial187.stdout:================================================================================ 2026-02-12T08:41:17.923 INFO:teuthology.orchestra.run.trial187.stdout:Installing: 2026-02-12T08:41:17.923 INFO:teuthology.orchestra.run.trial187.stdout: rbd-nbd x86_64 2:20.3.0-5161.g2401a109.el9 ceph 166 k 2026-02-12T08:41:17.924 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:41:17.924 INFO:teuthology.orchestra.run.trial187.stdout:Transaction Summary 2026-02-12T08:41:17.924 INFO:teuthology.orchestra.run.trial187.stdout:================================================================================ 2026-02-12T08:41:17.924 INFO:teuthology.orchestra.run.trial187.stdout:Install 1 Package 2026-02-12T08:41:17.924 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:41:17.924 INFO:teuthology.orchestra.run.trial187.stdout:Total download size: 166 k 2026-02-12T08:41:17.924 INFO:teuthology.orchestra.run.trial187.stdout:Installed size: 496 k 2026-02-12T08:41:17.924 INFO:teuthology.orchestra.run.trial187.stdout:Downloading Packages: 2026-02-12T08:41:18.395 INFO:teuthology.orchestra.run.trial187.stdout:rbd-nbd-20.3.0-5161.g2401a109.el9.x86_64.rpm 353 kB/s | 166 kB 00:00 2026-02-12T08:41:18.395 INFO:teuthology.orchestra.run.trial187.stdout:-------------------------------------------------------------------------------- 2026-02-12T08:41:18.396 INFO:teuthology.orchestra.run.trial187.stdout:Total 352 kB/s | 166 kB 00:00 2026-02-12T08:41:18.396 INFO:teuthology.orchestra.run.trial187.stdout:Running transaction check 2026-02-12T08:41:18.400 INFO:teuthology.orchestra.run.trial187.stdout:Transaction check succeeded. 2026-02-12T08:41:18.400 INFO:teuthology.orchestra.run.trial187.stdout:Running transaction test 2026-02-12T08:41:18.432 INFO:teuthology.orchestra.run.trial187.stdout:Transaction test succeeded. 2026-02-12T08:41:18.433 INFO:teuthology.orchestra.run.trial187.stdout:Running transaction 2026-02-12T08:41:18.488 INFO:teuthology.orchestra.run.trial187.stdout: Preparing : 1/1 2026-02-12T08:41:18.505 INFO:teuthology.orchestra.run.trial187.stdout: Installing : rbd-nbd-2:20.3.0-5161.g2401a109.el9.x86_64 1/1 2026-02-12T08:41:18.589 INFO:teuthology.orchestra.run.trial187.stdout: Running scriptlet: rbd-nbd-2:20.3.0-5161.g2401a109.el9.x86_64 1/1 2026-02-12T08:41:18.646 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : rbd-nbd-2:20.3.0-5161.g2401a109.el9.x86_64 1/1 2026-02-12T08:41:18.646 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:41:18.647 INFO:teuthology.orchestra.run.trial187.stdout:Installed: 2026-02-12T08:41:18.647 INFO:teuthology.orchestra.run.trial187.stdout: rbd-nbd-2:20.3.0-5161.g2401a109.el9.x86_64 2026-02-12T08:41:18.647 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:41:18.647 INFO:teuthology.orchestra.run.trial187.stdout:Complete! 2026-02-12T08:41:18.675 DEBUG:teuthology.orchestra.run.trial187:> sudo yum -y install ceph-osd-crimson 2026-02-12T08:41:18.992 INFO:teuthology.orchestra.run.trial187.stdout:Last metadata expiration check: 0:04:07 ago on Thu 12 Feb 2026 08:37:11 AM UTC. 2026-02-12T08:41:19.057 INFO:teuthology.orchestra.run.trial187.stdout:Package ceph-osd-crimson-2:20.3.0-5161.g2401a109.el9.x86_64 is already installed. 2026-02-12T08:41:19.074 INFO:teuthology.orchestra.run.trial187.stdout:Dependencies resolved. 2026-02-12T08:41:19.075 INFO:teuthology.orchestra.run.trial187.stdout:Nothing to do. 2026-02-12T08:41:19.075 INFO:teuthology.orchestra.run.trial187.stdout:Complete! 2026-02-12T08:41:19.098 DEBUG:teuthology.orchestra.run.trial187:> sudo yum -y install ceph-osd-crimson-debuginfo 2026-02-12T08:41:19.402 INFO:teuthology.orchestra.run.trial187.stdout:Last metadata expiration check: 0:04:08 ago on Thu 12 Feb 2026 08:37:11 AM UTC. 2026-02-12T08:41:19.488 INFO:teuthology.orchestra.run.trial187.stdout:Dependencies resolved. 2026-02-12T08:41:19.488 INFO:teuthology.orchestra.run.trial187.stdout:================================================================================ 2026-02-12T08:41:19.488 INFO:teuthology.orchestra.run.trial187.stdout: Package Arch Version Repo Size 2026-02-12T08:41:19.488 INFO:teuthology.orchestra.run.trial187.stdout:================================================================================ 2026-02-12T08:41:19.489 INFO:teuthology.orchestra.run.trial187.stdout:Installing: 2026-02-12T08:41:19.489 INFO:teuthology.orchestra.run.trial187.stdout: ceph-osd-crimson-debuginfo x86_64 2:20.3.0-5161.g2401a109.el9 ceph 429 M 2026-02-12T08:41:19.489 INFO:teuthology.orchestra.run.trial187.stdout:Installing weak dependencies: 2026-02-12T08:41:19.489 INFO:teuthology.orchestra.run.trial187.stdout: ceph-debugsource x86_64 2:20.3.0-5161.g2401a109.el9 ceph 12 M 2026-02-12T08:41:19.489 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:41:19.489 INFO:teuthology.orchestra.run.trial187.stdout:Transaction Summary 2026-02-12T08:41:19.489 INFO:teuthology.orchestra.run.trial187.stdout:================================================================================ 2026-02-12T08:41:19.489 INFO:teuthology.orchestra.run.trial187.stdout:Install 2 Packages 2026-02-12T08:41:19.489 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:41:19.489 INFO:teuthology.orchestra.run.trial187.stdout:Total download size: 441 M 2026-02-12T08:41:19.490 INFO:teuthology.orchestra.run.trial187.stdout:Installed size: 2.7 G 2026-02-12T08:41:19.490 INFO:teuthology.orchestra.run.trial187.stdout:Downloading Packages: 2026-02-12T08:41:29.665 INFO:teuthology.orchestra.run.trial187.stdout:(1/2): ceph-debugsource-20.3.0-5161.g2401a109.e 1.2 MB/s | 12 MB 00:10 2026-02-12T08:46:52.276 INFO:teuthology.orchestra.run.trial187.stdout:(2/2): ceph-osd-crimson-debuginfo-20.3.0-5161.g 1.3 MB/s | 429 MB 05:32 2026-02-12T08:46:52.277 INFO:teuthology.orchestra.run.trial187.stdout:-------------------------------------------------------------------------------- 2026-02-12T08:46:52.278 INFO:teuthology.orchestra.run.trial187.stdout:Total 1.3 MB/s | 441 MB 05:32 2026-02-12T08:46:52.278 INFO:teuthology.orchestra.run.trial187.stdout:Running transaction check 2026-02-12T08:46:52.283 INFO:teuthology.orchestra.run.trial187.stdout:Transaction check succeeded. 2026-02-12T08:46:52.283 INFO:teuthology.orchestra.run.trial187.stdout:Running transaction test 2026-02-12T08:46:53.645 INFO:teuthology.orchestra.run.trial187.stdout:Transaction test succeeded. 2026-02-12T08:46:53.645 INFO:teuthology.orchestra.run.trial187.stdout:Running transaction 2026-02-12T08:46:57.237 INFO:teuthology.orchestra.run.trial187.stdout: Preparing : 1/1 2026-02-12T08:47:19.274 INFO:teuthology.orchestra.run.trial187.stdout: Installing : ceph-debugsource-2:20.3.0-5161.g2401a109.el9.x86_64 1/2 2026-02-12T08:47:19.293 INFO:teuthology.orchestra.run.trial187.stdout: Installing : ceph-osd-crimson-debuginfo-2:20.3.0-5161.g2401a109.e 2/2 2026-02-12T08:47:21.118 INFO:teuthology.orchestra.run.trial187.stdout: Running scriptlet: ceph-osd-crimson-debuginfo-2:20.3.0-5161.g2401a109.e 2/2 2026-02-12T08:47:21.119 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : ceph-debugsource-2:20.3.0-5161.g2401a109.el9.x86_64 1/2 2026-02-12T08:47:21.176 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : ceph-osd-crimson-debuginfo-2:20.3.0-5161.g2401a109.e 2/2 2026-02-12T08:47:21.176 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:47:21.176 INFO:teuthology.orchestra.run.trial187.stdout:Installed: 2026-02-12T08:47:21.176 INFO:teuthology.orchestra.run.trial187.stdout: ceph-debugsource-2:20.3.0-5161.g2401a109.el9.x86_64 2026-02-12T08:47:21.176 INFO:teuthology.orchestra.run.trial187.stdout: ceph-osd-crimson-debuginfo-2:20.3.0-5161.g2401a109.el9.x86_64 2026-02-12T08:47:21.177 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:47:21.177 INFO:teuthology.orchestra.run.trial187.stdout:Complete! 2026-02-12T08:47:21.254 DEBUG:teuthology.parallel:result is None 2026-02-12T08:47:21.259 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=2401a109e2ac85dfa20fc57328da663aa754c875 2026-02-12T08:47:21.447 DEBUG:teuthology.orchestra.run.trial187:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-12T08:47:21.469 INFO:teuthology.orchestra.run.trial187.stdout:20.3.0-5161.g2401a109.el9 2026-02-12T08:47:21.470 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5161.g2401a109.el9 2026-02-12T08:47:21.470 INFO:teuthology.task.install:The correct ceph version 20.3.0-5161.g2401a109 is installed. 2026-02-12T08:47:21.472 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2026-02-12T08:47:21.473 DEBUG:teuthology.orchestra.run.trial187:> set -ex 2026-02-12T08:47:21.473 DEBUG:teuthology.orchestra.run.trial187:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-12T08:47:21.535 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2026-02-12T08:47:21.535 DEBUG:teuthology.orchestra.run.trial187:> set -ex 2026-02-12T08:47:21.535 DEBUG:teuthology.orchestra.run.trial187:> sudo dd of=/usr/bin/daemon-helper 2026-02-12T08:47:21.599 DEBUG:teuthology.orchestra.run.trial187:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-12T08:47:21.663 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2026-02-12T08:47:21.664 DEBUG:teuthology.orchestra.run.trial187:> set -ex 2026-02-12T08:47:21.664 DEBUG:teuthology.orchestra.run.trial187:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-12T08:47:21.727 DEBUG:teuthology.orchestra.run.trial187:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-12T08:47:21.791 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2026-02-12T08:47:21.791 DEBUG:teuthology.orchestra.run.trial187:> set -ex 2026-02-12T08:47:21.792 DEBUG:teuthology.orchestra.run.trial187:> sudo dd of=/usr/bin/stdin-killer 2026-02-12T08:47:21.854 DEBUG:teuthology.orchestra.run.trial187:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-12T08:47:21.919 INFO:teuthology.run_tasks:Running task pexec... 2026-02-12T08:47:21.940 INFO:teuthology.task.pexec:Executing custom commands... 2026-02-12T08:47:21.941 DEBUG:teuthology.orchestra.run.trial187:> TESTDIR=/home/ubuntu/cephtest bash -s 2026-02-12T08:47:21.961 INFO:teuthology.task.pexec:Running commands on host ubuntu@trial187.front.sepia.ceph.com 2026-02-12T08:47:21.962 INFO:teuthology.task.pexec:sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson 2026-02-12T08:47:22.012 DEBUG:teuthology.parallel:result is None 2026-02-12T08:47:22.013 INFO:teuthology.run_tasks:Running task install... 2026-02-12T08:47:22.027 DEBUG:teuthology.task.install:project ceph 2026-02-12T08:47:22.027 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'debug', 'sha1': '2401a109e2ac85dfa20fc57328da663aa754c875'}} 2026-02-12T08:47:22.027 DEBUG:teuthology.task.install:config {'flavor': 'debug', 'sha1': '2401a109e2ac85dfa20fc57328da663aa754c875'} 2026-02-12T08:47:22.028 INFO:teuthology.task.install:Using flavor: debug 2026-02-12T08:47:22.037 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-osd', 'ceph-osd-classic', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-osd', 'ceph-osd-classic', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-12T08:47:22.037 INFO:teuthology.task.install:extra packages: [] 2026-02-12T08:47:22.037 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'debug', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '2401a109e2ac85dfa20fc57328da663aa754c875', 'tag': None, 'wait_for_package': False} 2026-02-12T08:47:22.038 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=2401a109e2ac85dfa20fc57328da663aa754c875 2026-02-12T08:47:22.199 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-matanb-crimson-66488-revert/2401a109e2ac85dfa20fc57328da663aa754c875/centos/9/flavors/debug/ 2026-02-12T08:47:22.199 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5161.g2401a109 2026-02-12T08:47:22.605 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-matanb-crimson-66488-revert/2401a109e2ac85dfa20fc57328da663aa754c875/centos/9/flavors/debug/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-matanb-crimson-66488-revert/2401a109e2ac85dfa20fc57328da663aa754c875/centos/9/flavors/debug/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-matanb-crimson-66488-revert/2401a109e2ac85dfa20fc57328da663aa754c875/centos/9/flavors/debug/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-12T08:47:22.605 DEBUG:teuthology.orchestra.run.trial187:> set -ex 2026-02-12T08:47:22.605 DEBUG:teuthology.orchestra.run.trial187:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-12T08:47:22.630 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2026-02-12T08:47:22.631 DEBUG:teuthology.orchestra.run.trial187:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/2401a109e2ac85dfa20fc57328da663aa754c875/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-12T08:47:22.696 DEBUG:teuthology.orchestra.run.trial187:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-12T08:47:22.760 DEBUG:teuthology.orchestra.run.trial187:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-12T08:47:22.815 INFO:teuthology.orchestra.run.trial187.stdout:check_obsoletes = 1 2026-02-12T08:47:22.825 DEBUG:teuthology.orchestra.run.trial187:> sudo yum clean all 2026-02-12T08:47:22.985 INFO:teuthology.orchestra.run.trial187.stdout:56 files removed 2026-02-12T08:47:23.000 DEBUG:teuthology.orchestra.run.trial187:> sudo yum -y install ceph-radosgw 2026-02-12T08:47:24.001 INFO:teuthology.orchestra.run.trial187.stdout:ceph packages for x86_64 110 kB/s | 94 kB 00:00 2026-02-12T08:47:24.652 INFO:teuthology.orchestra.run.trial187.stdout:ceph noarch packages 39 kB/s | 24 kB 00:00 2026-02-12T08:47:25.226 INFO:teuthology.orchestra.run.trial187.stdout:ceph source packages 3.7 kB/s | 2.1 kB 00:00 2026-02-12T08:47:26.581 INFO:teuthology.orchestra.run.trial187.stdout:CentOS Stream 9 - BaseOS 6.7 MB/s | 8.9 MB 00:01 2026-02-12T08:47:29.827 INFO:teuthology.orchestra.run.trial187.stdout:CentOS Stream 9 - AppStream 9.9 MB/s | 27 MB 00:02 2026-02-12T08:47:34.062 INFO:teuthology.orchestra.run.trial187.stdout:CentOS Stream 9 - CRB 4.3 MB/s | 7.8 MB 00:01 2026-02-12T08:47:34.950 INFO:teuthology.orchestra.run.trial187.stdout:CentOS Stream 9 - Extras packages 120 kB/s | 20 kB 00:00 2026-02-12T08:47:35.676 INFO:teuthology.orchestra.run.trial187.stdout:Extra Packages for Enterprise Linux 31 MB/s | 20 MB 00:00 2026-02-12T08:47:38.910 INFO:teuthology.orchestra.run.trial187.stdout:lab-extras 1.6 MB/s | 50 kB 00:00 2026-02-12T08:47:40.159 INFO:teuthology.orchestra.run.trial187.stdout:Package ceph-radosgw-2:20.3.0-5161.g2401a109.el9.x86_64 is already installed. 2026-02-12T08:47:40.177 INFO:teuthology.orchestra.run.trial187.stdout:Dependencies resolved. 2026-02-12T08:47:40.178 INFO:teuthology.orchestra.run.trial187.stdout:Nothing to do. 2026-02-12T08:47:40.178 INFO:teuthology.orchestra.run.trial187.stdout:Complete! 2026-02-12T08:47:40.202 DEBUG:teuthology.orchestra.run.trial187:> sudo yum -y install ceph-test 2026-02-12T08:47:40.501 INFO:teuthology.orchestra.run.trial187.stdout:Last metadata expiration check: 0:00:02 ago on Thu 12 Feb 2026 08:47:38 AM UTC. 2026-02-12T08:47:40.567 INFO:teuthology.orchestra.run.trial187.stdout:Package ceph-test-2:20.3.0-5161.g2401a109.el9.x86_64 is already installed. 2026-02-12T08:47:40.584 INFO:teuthology.orchestra.run.trial187.stdout:Dependencies resolved. 2026-02-12T08:47:40.585 INFO:teuthology.orchestra.run.trial187.stdout:Nothing to do. 2026-02-12T08:47:40.585 INFO:teuthology.orchestra.run.trial187.stdout:Complete! 2026-02-12T08:47:40.606 DEBUG:teuthology.orchestra.run.trial187:> sudo yum -y install ceph 2026-02-12T08:47:40.910 INFO:teuthology.orchestra.run.trial187.stdout:Last metadata expiration check: 0:00:02 ago on Thu 12 Feb 2026 08:47:38 AM UTC. 2026-02-12T08:47:40.977 INFO:teuthology.orchestra.run.trial187.stdout:Package ceph-2:20.3.0-5161.g2401a109.el9.x86_64 is already installed. 2026-02-12T08:47:40.995 INFO:teuthology.orchestra.run.trial187.stdout:Dependencies resolved. 2026-02-12T08:47:40.996 INFO:teuthology.orchestra.run.trial187.stdout:Nothing to do. 2026-02-12T08:47:40.996 INFO:teuthology.orchestra.run.trial187.stdout:Complete! 2026-02-12T08:47:41.018 DEBUG:teuthology.orchestra.run.trial187:> sudo yum -y install ceph-base 2026-02-12T08:47:41.321 INFO:teuthology.orchestra.run.trial187.stdout:Last metadata expiration check: 0:00:03 ago on Thu 12 Feb 2026 08:47:38 AM UTC. 2026-02-12T08:47:41.387 INFO:teuthology.orchestra.run.trial187.stdout:Package ceph-base-2:20.3.0-5161.g2401a109.el9.x86_64 is already installed. 2026-02-12T08:47:41.405 INFO:teuthology.orchestra.run.trial187.stdout:Dependencies resolved. 2026-02-12T08:47:41.406 INFO:teuthology.orchestra.run.trial187.stdout:Nothing to do. 2026-02-12T08:47:41.406 INFO:teuthology.orchestra.run.trial187.stdout:Complete! 2026-02-12T08:47:41.427 DEBUG:teuthology.orchestra.run.trial187:> sudo yum -y install cephadm 2026-02-12T08:47:41.732 INFO:teuthology.orchestra.run.trial187.stdout:Last metadata expiration check: 0:00:03 ago on Thu 12 Feb 2026 08:47:38 AM UTC. 2026-02-12T08:47:41.799 INFO:teuthology.orchestra.run.trial187.stdout:Package cephadm-2:20.3.0-5161.g2401a109.el9.noarch is already installed. 2026-02-12T08:47:41.816 INFO:teuthology.orchestra.run.trial187.stdout:Dependencies resolved. 2026-02-12T08:47:41.817 INFO:teuthology.orchestra.run.trial187.stdout:Nothing to do. 2026-02-12T08:47:41.817 INFO:teuthology.orchestra.run.trial187.stdout:Complete! 2026-02-12T08:47:41.839 DEBUG:teuthology.orchestra.run.trial187:> sudo yum -y install ceph-immutable-object-cache 2026-02-12T08:47:42.150 INFO:teuthology.orchestra.run.trial187.stdout:Last metadata expiration check: 0:00:04 ago on Thu 12 Feb 2026 08:47:38 AM UTC. 2026-02-12T08:47:42.217 INFO:teuthology.orchestra.run.trial187.stdout:Package ceph-immutable-object-cache-2:20.3.0-5161.g2401a109.el9.x86_64 is already installed. 2026-02-12T08:47:42.234 INFO:teuthology.orchestra.run.trial187.stdout:Dependencies resolved. 2026-02-12T08:47:42.235 INFO:teuthology.orchestra.run.trial187.stdout:Nothing to do. 2026-02-12T08:47:42.235 INFO:teuthology.orchestra.run.trial187.stdout:Complete! 2026-02-12T08:47:42.258 DEBUG:teuthology.orchestra.run.trial187:> sudo yum -y install ceph-mgr 2026-02-12T08:47:42.562 INFO:teuthology.orchestra.run.trial187.stdout:Last metadata expiration check: 0:00:04 ago on Thu 12 Feb 2026 08:47:38 AM UTC. 2026-02-12T08:47:42.629 INFO:teuthology.orchestra.run.trial187.stdout:Package ceph-mgr-2:20.3.0-5161.g2401a109.el9.x86_64 is already installed. 2026-02-12T08:47:42.646 INFO:teuthology.orchestra.run.trial187.stdout:Dependencies resolved. 2026-02-12T08:47:42.647 INFO:teuthology.orchestra.run.trial187.stdout:Nothing to do. 2026-02-12T08:47:42.647 INFO:teuthology.orchestra.run.trial187.stdout:Complete! 2026-02-12T08:47:42.669 DEBUG:teuthology.orchestra.run.trial187:> sudo yum -y install ceph-mgr-dashboard 2026-02-12T08:47:42.974 INFO:teuthology.orchestra.run.trial187.stdout:Last metadata expiration check: 0:00:04 ago on Thu 12 Feb 2026 08:47:38 AM UTC. 2026-02-12T08:47:43.039 INFO:teuthology.orchestra.run.trial187.stdout:Package ceph-mgr-dashboard-2:20.3.0-5161.g2401a109.el9.noarch is already installed. 2026-02-12T08:47:43.057 INFO:teuthology.orchestra.run.trial187.stdout:Dependencies resolved. 2026-02-12T08:47:43.058 INFO:teuthology.orchestra.run.trial187.stdout:Nothing to do. 2026-02-12T08:47:43.058 INFO:teuthology.orchestra.run.trial187.stdout:Complete! 2026-02-12T08:47:43.080 DEBUG:teuthology.orchestra.run.trial187:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-12T08:47:43.383 INFO:teuthology.orchestra.run.trial187.stdout:Last metadata expiration check: 0:00:05 ago on Thu 12 Feb 2026 08:47:38 AM UTC. 2026-02-12T08:47:43.449 INFO:teuthology.orchestra.run.trial187.stdout:Package ceph-mgr-diskprediction-local-2:20.3.0-5161.g2401a109.el9.noarch is already installed. 2026-02-12T08:47:43.467 INFO:teuthology.orchestra.run.trial187.stdout:Dependencies resolved. 2026-02-12T08:47:43.467 INFO:teuthology.orchestra.run.trial187.stdout:Nothing to do. 2026-02-12T08:47:43.468 INFO:teuthology.orchestra.run.trial187.stdout:Complete! 2026-02-12T08:47:43.489 DEBUG:teuthology.orchestra.run.trial187:> sudo yum -y install ceph-mgr-rook 2026-02-12T08:47:43.788 INFO:teuthology.orchestra.run.trial187.stdout:Last metadata expiration check: 0:00:05 ago on Thu 12 Feb 2026 08:47:38 AM UTC. 2026-02-12T08:47:43.853 INFO:teuthology.orchestra.run.trial187.stdout:Package ceph-mgr-rook-2:20.3.0-5161.g2401a109.el9.noarch is already installed. 2026-02-12T08:47:43.870 INFO:teuthology.orchestra.run.trial187.stdout:Dependencies resolved. 2026-02-12T08:47:43.871 INFO:teuthology.orchestra.run.trial187.stdout:Nothing to do. 2026-02-12T08:47:43.871 INFO:teuthology.orchestra.run.trial187.stdout:Complete! 2026-02-12T08:47:43.892 DEBUG:teuthology.orchestra.run.trial187:> sudo yum -y install ceph-mgr-cephadm 2026-02-12T08:47:44.191 INFO:teuthology.orchestra.run.trial187.stdout:Last metadata expiration check: 0:00:06 ago on Thu 12 Feb 2026 08:47:38 AM UTC. 2026-02-12T08:47:44.259 INFO:teuthology.orchestra.run.trial187.stdout:Package ceph-mgr-cephadm-2:20.3.0-5161.g2401a109.el9.noarch is already installed. 2026-02-12T08:47:44.276 INFO:teuthology.orchestra.run.trial187.stdout:Dependencies resolved. 2026-02-12T08:47:44.277 INFO:teuthology.orchestra.run.trial187.stdout:Nothing to do. 2026-02-12T08:47:44.277 INFO:teuthology.orchestra.run.trial187.stdout:Complete! 2026-02-12T08:47:44.299 DEBUG:teuthology.orchestra.run.trial187:> sudo yum -y install ceph-osd 2026-02-12T08:47:44.599 INFO:teuthology.orchestra.run.trial187.stdout:Last metadata expiration check: 0:00:06 ago on Thu 12 Feb 2026 08:47:38 AM UTC. 2026-02-12T08:47:44.666 INFO:teuthology.orchestra.run.trial187.stdout:Package ceph-osd-2:20.3.0-5161.g2401a109.el9.x86_64 is already installed. 2026-02-12T08:47:44.684 INFO:teuthology.orchestra.run.trial187.stdout:Dependencies resolved. 2026-02-12T08:47:44.685 INFO:teuthology.orchestra.run.trial187.stdout:Nothing to do. 2026-02-12T08:47:44.685 INFO:teuthology.orchestra.run.trial187.stdout:Complete! 2026-02-12T08:47:44.706 DEBUG:teuthology.orchestra.run.trial187:> sudo yum -y install ceph-osd-classic 2026-02-12T08:47:45.008 INFO:teuthology.orchestra.run.trial187.stdout:Last metadata expiration check: 0:00:07 ago on Thu 12 Feb 2026 08:47:38 AM UTC. 2026-02-12T08:47:45.074 INFO:teuthology.orchestra.run.trial187.stdout:Package ceph-osd-classic-2:20.3.0-5161.g2401a109.el9.x86_64 is already installed. 2026-02-12T08:47:45.092 INFO:teuthology.orchestra.run.trial187.stdout:Dependencies resolved. 2026-02-12T08:47:45.092 INFO:teuthology.orchestra.run.trial187.stdout:Nothing to do. 2026-02-12T08:47:45.093 INFO:teuthology.orchestra.run.trial187.stdout:Complete! 2026-02-12T08:47:45.114 DEBUG:teuthology.orchestra.run.trial187:> sudo yum -y install ceph-fuse 2026-02-12T08:47:45.417 INFO:teuthology.orchestra.run.trial187.stdout:Last metadata expiration check: 0:00:07 ago on Thu 12 Feb 2026 08:47:38 AM UTC. 2026-02-12T08:47:45.486 INFO:teuthology.orchestra.run.trial187.stdout:Package ceph-fuse-2:20.3.0-5161.g2401a109.el9.x86_64 is already installed. 2026-02-12T08:47:45.504 INFO:teuthology.orchestra.run.trial187.stdout:Dependencies resolved. 2026-02-12T08:47:45.504 INFO:teuthology.orchestra.run.trial187.stdout:Nothing to do. 2026-02-12T08:47:45.505 INFO:teuthology.orchestra.run.trial187.stdout:Complete! 2026-02-12T08:47:45.527 DEBUG:teuthology.orchestra.run.trial187:> sudo yum -y install ceph-volume 2026-02-12T08:47:45.828 INFO:teuthology.orchestra.run.trial187.stdout:Last metadata expiration check: 0:00:07 ago on Thu 12 Feb 2026 08:47:38 AM UTC. 2026-02-12T08:47:45.892 INFO:teuthology.orchestra.run.trial187.stdout:Package ceph-volume-2:20.3.0-5161.g2401a109.el9.noarch is already installed. 2026-02-12T08:47:45.909 INFO:teuthology.orchestra.run.trial187.stdout:Dependencies resolved. 2026-02-12T08:47:45.910 INFO:teuthology.orchestra.run.trial187.stdout:Nothing to do. 2026-02-12T08:47:45.910 INFO:teuthology.orchestra.run.trial187.stdout:Complete! 2026-02-12T08:47:45.932 DEBUG:teuthology.orchestra.run.trial187:> sudo yum -y install librados-devel 2026-02-12T08:47:46.232 INFO:teuthology.orchestra.run.trial187.stdout:Last metadata expiration check: 0:00:08 ago on Thu 12 Feb 2026 08:47:38 AM UTC. 2026-02-12T08:47:46.298 INFO:teuthology.orchestra.run.trial187.stdout:Package librados-devel-2:20.3.0-5161.g2401a109.el9.x86_64 is already installed. 2026-02-12T08:47:46.316 INFO:teuthology.orchestra.run.trial187.stdout:Dependencies resolved. 2026-02-12T08:47:46.317 INFO:teuthology.orchestra.run.trial187.stdout:Nothing to do. 2026-02-12T08:47:46.317 INFO:teuthology.orchestra.run.trial187.stdout:Complete! 2026-02-12T08:47:46.339 DEBUG:teuthology.orchestra.run.trial187:> sudo yum -y install libcephfs2 2026-02-12T08:47:46.641 INFO:teuthology.orchestra.run.trial187.stdout:Last metadata expiration check: 0:00:08 ago on Thu 12 Feb 2026 08:47:38 AM UTC. 2026-02-12T08:47:46.706 INFO:teuthology.orchestra.run.trial187.stdout:Package libcephfs2-2:20.3.0-5161.g2401a109.el9.x86_64 is already installed. 2026-02-12T08:47:46.724 INFO:teuthology.orchestra.run.trial187.stdout:Dependencies resolved. 2026-02-12T08:47:46.724 INFO:teuthology.orchestra.run.trial187.stdout:Nothing to do. 2026-02-12T08:47:46.724 INFO:teuthology.orchestra.run.trial187.stdout:Complete! 2026-02-12T08:47:46.747 DEBUG:teuthology.orchestra.run.trial187:> sudo yum -y install libcephfs-devel 2026-02-12T08:47:47.052 INFO:teuthology.orchestra.run.trial187.stdout:Last metadata expiration check: 0:00:09 ago on Thu 12 Feb 2026 08:47:38 AM UTC. 2026-02-12T08:47:47.117 INFO:teuthology.orchestra.run.trial187.stdout:Package libcephfs-devel-2:20.3.0-5161.g2401a109.el9.x86_64 is already installed. 2026-02-12T08:47:47.135 INFO:teuthology.orchestra.run.trial187.stdout:Dependencies resolved. 2026-02-12T08:47:47.136 INFO:teuthology.orchestra.run.trial187.stdout:Nothing to do. 2026-02-12T08:47:47.136 INFO:teuthology.orchestra.run.trial187.stdout:Complete! 2026-02-12T08:47:47.158 DEBUG:teuthology.orchestra.run.trial187:> sudo yum -y install librados2 2026-02-12T08:47:47.463 INFO:teuthology.orchestra.run.trial187.stdout:Last metadata expiration check: 0:00:09 ago on Thu 12 Feb 2026 08:47:38 AM UTC. 2026-02-12T08:47:47.530 INFO:teuthology.orchestra.run.trial187.stdout:Package librados2-2:20.3.0-5161.g2401a109.el9.x86_64 is already installed. 2026-02-12T08:47:47.548 INFO:teuthology.orchestra.run.trial187.stdout:Dependencies resolved. 2026-02-12T08:47:47.549 INFO:teuthology.orchestra.run.trial187.stdout:Nothing to do. 2026-02-12T08:47:47.549 INFO:teuthology.orchestra.run.trial187.stdout:Complete! 2026-02-12T08:47:47.570 DEBUG:teuthology.orchestra.run.trial187:> sudo yum -y install librbd1 2026-02-12T08:47:47.873 INFO:teuthology.orchestra.run.trial187.stdout:Last metadata expiration check: 0:00:09 ago on Thu 12 Feb 2026 08:47:38 AM UTC. 2026-02-12T08:47:47.940 INFO:teuthology.orchestra.run.trial187.stdout:Package librbd1-2:20.3.0-5161.g2401a109.el9.x86_64 is already installed. 2026-02-12T08:47:47.957 INFO:teuthology.orchestra.run.trial187.stdout:Dependencies resolved. 2026-02-12T08:47:47.958 INFO:teuthology.orchestra.run.trial187.stdout:Nothing to do. 2026-02-12T08:47:47.958 INFO:teuthology.orchestra.run.trial187.stdout:Complete! 2026-02-12T08:47:47.980 DEBUG:teuthology.orchestra.run.trial187:> sudo yum -y install python3-rados 2026-02-12T08:47:48.286 INFO:teuthology.orchestra.run.trial187.stdout:Last metadata expiration check: 0:00:10 ago on Thu 12 Feb 2026 08:47:38 AM UTC. 2026-02-12T08:47:48.350 INFO:teuthology.orchestra.run.trial187.stdout:Package python3-rados-2:20.3.0-5161.g2401a109.el9.x86_64 is already installed. 2026-02-12T08:47:48.368 INFO:teuthology.orchestra.run.trial187.stdout:Dependencies resolved. 2026-02-12T08:47:48.368 INFO:teuthology.orchestra.run.trial187.stdout:Nothing to do. 2026-02-12T08:47:48.369 INFO:teuthology.orchestra.run.trial187.stdout:Complete! 2026-02-12T08:47:48.391 DEBUG:teuthology.orchestra.run.trial187:> sudo yum -y install python3-rgw 2026-02-12T08:47:48.695 INFO:teuthology.orchestra.run.trial187.stdout:Last metadata expiration check: 0:00:10 ago on Thu 12 Feb 2026 08:47:38 AM UTC. 2026-02-12T08:47:48.761 INFO:teuthology.orchestra.run.trial187.stdout:Package python3-rgw-2:20.3.0-5161.g2401a109.el9.x86_64 is already installed. 2026-02-12T08:47:48.778 INFO:teuthology.orchestra.run.trial187.stdout:Dependencies resolved. 2026-02-12T08:47:48.779 INFO:teuthology.orchestra.run.trial187.stdout:Nothing to do. 2026-02-12T08:47:48.779 INFO:teuthology.orchestra.run.trial187.stdout:Complete! 2026-02-12T08:47:48.801 DEBUG:teuthology.orchestra.run.trial187:> sudo yum -y install python3-cephfs 2026-02-12T08:47:49.107 INFO:teuthology.orchestra.run.trial187.stdout:Last metadata expiration check: 0:00:11 ago on Thu 12 Feb 2026 08:47:38 AM UTC. 2026-02-12T08:47:49.173 INFO:teuthology.orchestra.run.trial187.stdout:Package python3-cephfs-2:20.3.0-5161.g2401a109.el9.x86_64 is already installed. 2026-02-12T08:47:49.190 INFO:teuthology.orchestra.run.trial187.stdout:Dependencies resolved. 2026-02-12T08:47:49.191 INFO:teuthology.orchestra.run.trial187.stdout:Nothing to do. 2026-02-12T08:47:49.191 INFO:teuthology.orchestra.run.trial187.stdout:Complete! 2026-02-12T08:47:49.213 DEBUG:teuthology.orchestra.run.trial187:> sudo yum -y install python3-rbd 2026-02-12T08:47:49.520 INFO:teuthology.orchestra.run.trial187.stdout:Last metadata expiration check: 0:00:11 ago on Thu 12 Feb 2026 08:47:38 AM UTC. 2026-02-12T08:47:49.586 INFO:teuthology.orchestra.run.trial187.stdout:Package python3-rbd-2:20.3.0-5161.g2401a109.el9.x86_64 is already installed. 2026-02-12T08:47:49.604 INFO:teuthology.orchestra.run.trial187.stdout:Dependencies resolved. 2026-02-12T08:47:49.605 INFO:teuthology.orchestra.run.trial187.stdout:Nothing to do. 2026-02-12T08:47:49.605 INFO:teuthology.orchestra.run.trial187.stdout:Complete! 2026-02-12T08:47:49.627 DEBUG:teuthology.orchestra.run.trial187:> sudo yum -y install rbd-fuse 2026-02-12T08:47:49.932 INFO:teuthology.orchestra.run.trial187.stdout:Last metadata expiration check: 0:00:11 ago on Thu 12 Feb 2026 08:47:38 AM UTC. 2026-02-12T08:47:49.997 INFO:teuthology.orchestra.run.trial187.stdout:Package rbd-fuse-2:20.3.0-5161.g2401a109.el9.x86_64 is already installed. 2026-02-12T08:47:50.014 INFO:teuthology.orchestra.run.trial187.stdout:Dependencies resolved. 2026-02-12T08:47:50.015 INFO:teuthology.orchestra.run.trial187.stdout:Nothing to do. 2026-02-12T08:47:50.015 INFO:teuthology.orchestra.run.trial187.stdout:Complete! 2026-02-12T08:47:50.037 DEBUG:teuthology.orchestra.run.trial187:> sudo yum -y install rbd-mirror 2026-02-12T08:47:50.337 INFO:teuthology.orchestra.run.trial187.stdout:Last metadata expiration check: 0:00:12 ago on Thu 12 Feb 2026 08:47:38 AM UTC. 2026-02-12T08:47:50.404 INFO:teuthology.orchestra.run.trial187.stdout:Package rbd-mirror-2:20.3.0-5161.g2401a109.el9.x86_64 is already installed. 2026-02-12T08:47:50.421 INFO:teuthology.orchestra.run.trial187.stdout:Dependencies resolved. 2026-02-12T08:47:50.422 INFO:teuthology.orchestra.run.trial187.stdout:Nothing to do. 2026-02-12T08:47:50.422 INFO:teuthology.orchestra.run.trial187.stdout:Complete! 2026-02-12T08:47:50.444 DEBUG:teuthology.orchestra.run.trial187:> sudo yum -y install rbd-nbd 2026-02-12T08:47:50.750 INFO:teuthology.orchestra.run.trial187.stdout:Last metadata expiration check: 0:00:12 ago on Thu 12 Feb 2026 08:47:38 AM UTC. 2026-02-12T08:47:50.817 INFO:teuthology.orchestra.run.trial187.stdout:Package rbd-nbd-2:20.3.0-5161.g2401a109.el9.x86_64 is already installed. 2026-02-12T08:47:50.834 INFO:teuthology.orchestra.run.trial187.stdout:Dependencies resolved. 2026-02-12T08:47:50.835 INFO:teuthology.orchestra.run.trial187.stdout:Nothing to do. 2026-02-12T08:47:50.835 INFO:teuthology.orchestra.run.trial187.stdout:Complete! 2026-02-12T08:47:50.860 DEBUG:teuthology.parallel:result is None 2026-02-12T08:47:50.861 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=2401a109e2ac85dfa20fc57328da663aa754c875 2026-02-12T08:47:51.028 DEBUG:teuthology.orchestra.run.trial187:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-12T08:47:51.049 INFO:teuthology.orchestra.run.trial187.stdout:20.3.0-5161.g2401a109.el9 2026-02-12T08:47:51.049 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5161.g2401a109.el9 2026-02-12T08:47:51.049 INFO:teuthology.task.install:The correct ceph version 20.3.0-5161.g2401a109 is installed. 2026-02-12T08:47:51.051 INFO:teuthology.task.install.util:Utilities already shipped, skip it... 2026-02-12T08:47:51.051 INFO:teuthology.run_tasks:Running task ceph... 2026-02-12T08:47:51.150 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2026-02-12T08:47:51.150 DEBUG:teuthology.orchestra.run.trial187:> sudo chmod 777 /var/log/ceph 2026-02-12T08:47:51.175 INFO:tasks.ceph:Disabling ceph logrotate... 2026-02-12T08:47:51.175 DEBUG:teuthology.orchestra.run.trial187:> sudo rm -f -- /etc/logrotate.d/ceph 2026-02-12T08:47:51.240 INFO:tasks.ceph:Creating extra log directories... 2026-02-12T08:47:51.240 DEBUG:teuthology.orchestra.run.trial187:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2026-02-12T08:47:51.314 INFO:tasks.ceph:Creating ceph cluster ceph... 2026-02-12T08:47:51.315 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore max concurrent transactions': 8}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2026-02-12T08:47:51.315 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/mnt/teuthology/matan-2026-02-09_15:50:17-crimson-rados-main-distro-debug-trial/41545', 'branch': 'main', 'description': 'crimson-rados/perf/{0-crimson_install clusters/{crimson-fixed-1} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/segmented$/{crimson_seastore_segmented} perf_ignorelist settings/optimized workloads/radosbench_4M_seq_read}', 'email': None, 'first_in_suite': False, 'flavor': 'debug', 'job_id': '41545', 'kernel': {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'trial', 'name': 'matan-2026-02-09_15:50:17-crimson-rados-main-distro-debug-trial', 'no_nested_subset': False, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd': {'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore max concurrent transactions': 8}}, 'crimson_compat': True, 'flavor': 'debug', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH'], 'sha1': '2401a109e2ac85dfa20fc57328da663aa754c875'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'global': {'osd crush chooseleaf type': 0, 'osd pool default pg num': 128, 'osd pool default pgp num': 128, 'osd pool default size': 2}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'debug', 'sha1': '2401a109e2ac85dfa20fc57328da663aa754c875'}}, 'workunit': {'branch': 'main', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '6bf51b64d7733c58c2fe4a45746d4b9e65a38d1c'}}, 'owner': 'scheduled_matan@soko04.front.sepia.ceph.com', 'priority': 100, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 9353, 'sha1': '2401a109e2ac85dfa20fc57328da663aa754c875', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph_6bf51b64d7733c58c2fe4a45746d4b9e65a38d1c/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph.git', 'suite_sha1': '6bf51b64d7733c58c2fe4a45746d4b9e65a38d1c', 'targets': {'trial187.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGUQF4Dh4CZ7JoVPAoUH6MZNB4MuhPX5CLuhWSf5k0gS8waXJhCD8tcGvw5PQReoIDiSI5u+TF2iKA/e4SIG90k='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'flavor': 'debug', 'sha1': '2401a109e2ac85dfa20fc57328da663aa754c875'}}, {'pexec': {'all': ['sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson']}}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore max concurrent transactions': 8}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}}, 'wait-for-scrub': False, 'crimson_compat': True, 'flavor': 'debug', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH'], 'sha1': '2401a109e2ac85dfa20fc57328da663aa754c875', 'cluster': 'ceph'}}, {'ssh_keys': None}, {'cbt': {'benchmarks': {'radosbench': {'concurrent_ops': 4, 'concurrent_procs': 2, 'op_size': [4194304], 'pool_monitoring_list': ['collectl'], 'pool_profile': 'replicated', 'run_monitoring_list': ['collectl'], 'time': 60, 'write_only': False}}, 'cluster': {'iterations': 1, 'osds_per_node': 3, 'pool_profiles': {'replicated': {'pg_size': 256, 'pgp_size': 256, 'replication': 'replicated'}}, 'user': 'ubuntu'}, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'c433f1062990a0488dc29a553589bc609a460691', 'timestamp': '2026-02-09_15:50:17', 'tube': 'trial', 'user': 'matan', 'verbose': False, 'worker_log': '/home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.2161924'} 2026-02-12T08:47:51.316 DEBUG:teuthology.orchestra.run.trial187:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2026-02-12T08:47:51.362 DEBUG:teuthology.orchestra.run.trial187:> sudo install -d -m0777 -- /var/run/ceph 2026-02-12T08:47:51.427 DEBUG:teuthology.orchestra.run.trial187:> set -ex 2026-02-12T08:47:51.427 DEBUG:teuthology.orchestra.run.trial187:> dd if=/scratch_devs of=/dev/stdout 2026-02-12T08:47:51.485 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2026-02-12T08:47:51.485 DEBUG:teuthology.orchestra.run.trial187:> stat /dev/vg_nvme/lv_1 2026-02-12T08:47:51.541 INFO:teuthology.orchestra.run.trial187.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2026-02-12T08:47:51.541 INFO:teuthology.orchestra.run.trial187.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-12T08:47:51.541 INFO:teuthology.orchestra.run.trial187.stdout:Device: 6h/6d Inode: 1297 Links: 1 2026-02-12T08:47:51.541 INFO:teuthology.orchestra.run.trial187.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-12T08:47:51.541 INFO:teuthology.orchestra.run.trial187.stdout:Context: system_u:object_r:device_t:s0 2026-02-12T08:47:51.541 INFO:teuthology.orchestra.run.trial187.stdout:Access: 2026-02-12 08:41:17.316425380 +0000 2026-02-12T08:47:51.542 INFO:teuthology.orchestra.run.trial187.stdout:Modify: 2026-02-12 08:37:41.474082919 +0000 2026-02-12T08:47:51.542 INFO:teuthology.orchestra.run.trial187.stdout:Change: 2026-02-12 08:37:41.474082919 +0000 2026-02-12T08:47:51.542 INFO:teuthology.orchestra.run.trial187.stdout: Birth: 2026-02-12 08:37:41.474082919 +0000 2026-02-12T08:47:51.542 DEBUG:teuthology.orchestra.run.trial187:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2026-02-12T08:47:51.604 INFO:teuthology.orchestra.run.trial187.stderr:1+0 records in 2026-02-12T08:47:51.604 INFO:teuthology.orchestra.run.trial187.stderr:1+0 records out 2026-02-12T08:47:51.604 INFO:teuthology.orchestra.run.trial187.stderr:512 bytes copied, 0.000147987 s, 3.5 MB/s 2026-02-12T08:47:51.606 DEBUG:teuthology.orchestra.run.trial187:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2026-02-12T08:47:51.664 DEBUG:teuthology.orchestra.run.trial187:> stat /dev/vg_nvme/lv_2 2026-02-12T08:47:51.720 INFO:teuthology.orchestra.run.trial187.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2026-02-12T08:47:51.720 INFO:teuthology.orchestra.run.trial187.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-12T08:47:51.720 INFO:teuthology.orchestra.run.trial187.stdout:Device: 6h/6d Inode: 1301 Links: 1 2026-02-12T08:47:51.720 INFO:teuthology.orchestra.run.trial187.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-12T08:47:51.720 INFO:teuthology.orchestra.run.trial187.stdout:Context: system_u:object_r:device_t:s0 2026-02-12T08:47:51.721 INFO:teuthology.orchestra.run.trial187.stdout:Access: 2026-02-12 08:41:17.316425380 +0000 2026-02-12T08:47:51.721 INFO:teuthology.orchestra.run.trial187.stdout:Modify: 2026-02-12 08:37:41.474082919 +0000 2026-02-12T08:47:51.721 INFO:teuthology.orchestra.run.trial187.stdout:Change: 2026-02-12 08:37:41.474082919 +0000 2026-02-12T08:47:51.721 INFO:teuthology.orchestra.run.trial187.stdout: Birth: 2026-02-12 08:37:41.474082919 +0000 2026-02-12T08:47:51.721 DEBUG:teuthology.orchestra.run.trial187:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2026-02-12T08:47:51.782 INFO:teuthology.orchestra.run.trial187.stderr:1+0 records in 2026-02-12T08:47:51.782 INFO:teuthology.orchestra.run.trial187.stderr:1+0 records out 2026-02-12T08:47:51.782 INFO:teuthology.orchestra.run.trial187.stderr:512 bytes copied, 0.000145051 s, 3.5 MB/s 2026-02-12T08:47:51.783 DEBUG:teuthology.orchestra.run.trial187:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2026-02-12T08:47:51.841 DEBUG:teuthology.orchestra.run.trial187:> stat /dev/vg_nvme/lv_3 2026-02-12T08:47:51.897 INFO:teuthology.orchestra.run.trial187.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2026-02-12T08:47:51.897 INFO:teuthology.orchestra.run.trial187.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-12T08:47:51.897 INFO:teuthology.orchestra.run.trial187.stdout:Device: 6h/6d Inode: 1315 Links: 1 2026-02-12T08:47:51.897 INFO:teuthology.orchestra.run.trial187.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-12T08:47:51.897 INFO:teuthology.orchestra.run.trial187.stdout:Context: system_u:object_r:device_t:s0 2026-02-12T08:47:51.898 INFO:teuthology.orchestra.run.trial187.stdout:Access: 2026-02-12 08:41:17.317425369 +0000 2026-02-12T08:47:51.898 INFO:teuthology.orchestra.run.trial187.stdout:Modify: 2026-02-12 08:37:41.475082902 +0000 2026-02-12T08:47:51.898 INFO:teuthology.orchestra.run.trial187.stdout:Change: 2026-02-12 08:37:41.475082902 +0000 2026-02-12T08:47:51.898 INFO:teuthology.orchestra.run.trial187.stdout: Birth: 2026-02-12 08:37:41.475082902 +0000 2026-02-12T08:47:51.898 DEBUG:teuthology.orchestra.run.trial187:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2026-02-12T08:47:51.959 INFO:teuthology.orchestra.run.trial187.stderr:1+0 records in 2026-02-12T08:47:51.959 INFO:teuthology.orchestra.run.trial187.stderr:1+0 records out 2026-02-12T08:47:51.959 INFO:teuthology.orchestra.run.trial187.stderr:512 bytes copied, 0.000145723 s, 3.5 MB/s 2026-02-12T08:47:51.960 DEBUG:teuthology.orchestra.run.trial187:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2026-02-12T08:47:52.018 DEBUG:teuthology.orchestra.run.trial187:> stat /dev/vg_nvme/lv_4 2026-02-12T08:47:52.074 INFO:teuthology.orchestra.run.trial187.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2026-02-12T08:47:52.074 INFO:teuthology.orchestra.run.trial187.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-12T08:47:52.075 INFO:teuthology.orchestra.run.trial187.stdout:Device: 6h/6d Inode: 1304 Links: 1 2026-02-12T08:47:52.075 INFO:teuthology.orchestra.run.trial187.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-12T08:47:52.075 INFO:teuthology.orchestra.run.trial187.stdout:Context: system_u:object_r:device_t:s0 2026-02-12T08:47:52.075 INFO:teuthology.orchestra.run.trial187.stdout:Access: 2026-02-12 08:41:17.317425369 +0000 2026-02-12T08:47:52.075 INFO:teuthology.orchestra.run.trial187.stdout:Modify: 2026-02-12 08:37:41.474082919 +0000 2026-02-12T08:47:52.075 INFO:teuthology.orchestra.run.trial187.stdout:Change: 2026-02-12 08:37:41.474082919 +0000 2026-02-12T08:47:52.075 INFO:teuthology.orchestra.run.trial187.stdout: Birth: 2026-02-12 08:37:41.474082919 +0000 2026-02-12T08:47:52.075 DEBUG:teuthology.orchestra.run.trial187:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2026-02-12T08:47:52.138 INFO:teuthology.orchestra.run.trial187.stderr:1+0 records in 2026-02-12T08:47:52.138 INFO:teuthology.orchestra.run.trial187.stderr:1+0 records out 2026-02-12T08:47:52.138 INFO:teuthology.orchestra.run.trial187.stderr:512 bytes copied, 0.000131937 s, 3.9 MB/s 2026-02-12T08:47:52.140 DEBUG:teuthology.orchestra.run.trial187:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2026-02-12T08:47:52.198 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-12T08:47:52.198 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@trial187.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2026-02-12T08:47:52.198 INFO:tasks.ceph:Generating config... 2026-02-12T08:47:52.201 INFO:tasks.ceph:[osd] debug monc = 20 2026-02-12T08:47:52.201 INFO:tasks.ceph:[osd] crimson cpu num = 4 2026-02-12T08:47:52.201 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2026-02-12T08:47:52.201 INFO:tasks.ceph:[osd] debug filestore = 0/0 2026-02-12T08:47:52.201 INFO:tasks.ceph:[osd] debug journal = 0/0 2026-02-12T08:47:52.202 INFO:tasks.ceph:[osd] debug ms = 0/0 2026-02-12T08:47:52.202 INFO:tasks.ceph:[osd] debug osd = 0/0 2026-02-12T08:47:52.202 INFO:tasks.ceph:[osd] debug seastore = 20 2026-02-12T08:47:52.202 INFO:tasks.ceph:[osd] debug seastore backref = 20 2026-02-12T08:47:52.202 INFO:tasks.ceph:[osd] debug seastore cache = 20 2026-02-12T08:47:52.202 INFO:tasks.ceph:[osd] debug seastore cleaner = 20 2026-02-12T08:47:52.202 INFO:tasks.ceph:[osd] debug seastore device = 20 2026-02-12T08:47:52.202 INFO:tasks.ceph:[osd] debug seastore epm = 20 2026-02-12T08:47:52.202 INFO:tasks.ceph:[osd] debug seastore fixedkv tree = 20 2026-02-12T08:47:52.202 INFO:tasks.ceph:[osd] debug seastore journal = 20 2026-02-12T08:47:52.202 INFO:tasks.ceph:[osd] debug seastore lba = 20 2026-02-12T08:47:52.202 INFO:tasks.ceph:[osd] debug seastore odata = 20 2026-02-12T08:47:52.202 INFO:tasks.ceph:[osd] debug seastore omap = 20 2026-02-12T08:47:52.203 INFO:tasks.ceph:[osd] debug seastore onode = 20 2026-02-12T08:47:52.203 INFO:tasks.ceph:[osd] debug seastore t = 20 2026-02-12T08:47:52.203 INFO:tasks.ceph:[osd] debug seastore tm = 20 2026-02-12T08:47:52.203 INFO:tasks.ceph:[osd] osd objectstore = seastore 2026-02-12T08:47:52.203 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2026-02-12T08:47:52.203 INFO:tasks.ceph:[osd] seastore cachepin size pershard = 64M 2026-02-12T08:47:52.203 INFO:tasks.ceph:[osd] seastore max concurrent transactions = 8 2026-02-12T08:47:52.203 INFO:tasks.ceph:[global] auth client required = none 2026-02-12T08:47:52.203 INFO:tasks.ceph:[global] auth cluster required = none 2026-02-12T08:47:52.203 INFO:tasks.ceph:[global] auth service required = none 2026-02-12T08:47:52.203 INFO:tasks.ceph:[global] auth supported = none 2026-02-12T08:47:52.203 INFO:tasks.ceph:[global] debug asok = 0/0 2026-02-12T08:47:52.204 INFO:tasks.ceph:[global] debug auth = 0/0 2026-02-12T08:47:52.204 INFO:tasks.ceph:[global] debug bdev = 0/0 2026-02-12T08:47:52.204 INFO:tasks.ceph:[global] debug bluefs = 0/0 2026-02-12T08:47:52.204 INFO:tasks.ceph:[global] debug bluestore = 0/0 2026-02-12T08:47:52.204 INFO:tasks.ceph:[global] debug buffer = 0/0 2026-02-12T08:47:52.204 INFO:tasks.ceph:[global] debug civetweb = 0/0 2026-02-12T08:47:52.204 INFO:tasks.ceph:[global] debug client = 0/0 2026-02-12T08:47:52.204 INFO:tasks.ceph:[global] debug compressor = 0/0 2026-02-12T08:47:52.204 INFO:tasks.ceph:[global] debug context = 0/0 2026-02-12T08:47:52.204 INFO:tasks.ceph:[global] debug crush = 0/0 2026-02-12T08:47:52.204 INFO:tasks.ceph:[global] debug crypto = 0/0 2026-02-12T08:47:52.204 INFO:tasks.ceph:[global] debug dpdk = 0/0 2026-02-12T08:47:52.204 INFO:tasks.ceph:[global] debug eventtrace = 0/0 2026-02-12T08:47:52.205 INFO:tasks.ceph:[global] debug filer = 0/0 2026-02-12T08:47:52.205 INFO:tasks.ceph:[global] debug filestore = 0/0 2026-02-12T08:47:52.205 INFO:tasks.ceph:[global] debug finisher = 0/0 2026-02-12T08:47:52.205 INFO:tasks.ceph:[global] debug fuse = 0/0 2026-02-12T08:47:52.205 INFO:tasks.ceph:[global] debug heartbeatmap = 0/0 2026-02-12T08:47:52.205 INFO:tasks.ceph:[global] debug javaclient = 0/0 2026-02-12T08:47:52.205 INFO:tasks.ceph:[global] debug journal = 0/0 2026-02-12T08:47:52.205 INFO:tasks.ceph:[global] debug journaler = 0/0 2026-02-12T08:47:52.205 INFO:tasks.ceph:[global] debug kstore = 0/0 2026-02-12T08:47:52.205 INFO:tasks.ceph:[global] debug leveldb = 0/0 2026-02-12T08:47:52.205 INFO:tasks.ceph:[global] debug lockdep = 0/0 2026-02-12T08:47:52.205 INFO:tasks.ceph:[global] debug mds = 0/0 2026-02-12T08:47:52.205 INFO:tasks.ceph:[global] debug mds balancer = 0/0 2026-02-12T08:47:52.205 INFO:tasks.ceph:[global] debug mds locker = 0/0 2026-02-12T08:47:52.206 INFO:tasks.ceph:[global] debug mds log = 0/0 2026-02-12T08:47:52.206 INFO:tasks.ceph:[global] debug mds log expire = 0/0 2026-02-12T08:47:52.206 INFO:tasks.ceph:[global] debug mds migrator = 0/0 2026-02-12T08:47:52.206 INFO:tasks.ceph:[global] debug memdb = 0/0 2026-02-12T08:47:52.206 INFO:tasks.ceph:[global] debug mgr = 0/0 2026-02-12T08:47:52.206 INFO:tasks.ceph:[global] debug mgrc = 0/0 2026-02-12T08:47:52.206 INFO:tasks.ceph:[global] debug mon = 0/0 2026-02-12T08:47:52.206 INFO:tasks.ceph:[global] debug monc = 0/0 2026-02-12T08:47:52.206 INFO:tasks.ceph:[global] debug ms = 0/0 2026-02-12T08:47:52.206 INFO:tasks.ceph:[global] debug objclass = 0/0 2026-02-12T08:47:52.206 INFO:tasks.ceph:[global] debug objectcacher = 0/0 2026-02-12T08:47:52.206 INFO:tasks.ceph:[global] debug objecter = 0/0 2026-02-12T08:47:52.206 INFO:tasks.ceph:[global] debug optracker = 0/0 2026-02-12T08:47:52.207 INFO:tasks.ceph:[global] debug osd = 0/0 2026-02-12T08:47:52.207 INFO:tasks.ceph:[global] debug paxos = 0/0 2026-02-12T08:47:52.207 INFO:tasks.ceph:[global] debug perfcounter = 0/0 2026-02-12T08:47:52.207 INFO:tasks.ceph:[global] debug rados = 0/0 2026-02-12T08:47:52.207 INFO:tasks.ceph:[global] debug rbd = 0/0 2026-02-12T08:47:52.207 INFO:tasks.ceph:[global] debug rbd mirror = 0/0 2026-02-12T08:47:52.207 INFO:tasks.ceph:[global] debug rbd replay = 0/0 2026-02-12T08:47:52.207 INFO:tasks.ceph:[global] debug refs = 0/0 2026-02-12T08:47:52.207 INFO:tasks.ceph:[global] debug rgw = 0/0 2026-02-12T08:47:52.207 INFO:tasks.ceph:[global] debug rgw lifecycle = 0/0 2026-02-12T08:47:52.207 INFO:tasks.ceph:[global] debug rgw notification = 0/0 2026-02-12T08:47:52.207 INFO:tasks.ceph:[global] debug rgw sync = 0/0 2026-02-12T08:47:52.207 INFO:tasks.ceph:[global] debug rocksdb = 0/0 2026-02-12T08:47:52.208 INFO:tasks.ceph:[global] debug striper = 0/0 2026-02-12T08:47:52.208 INFO:tasks.ceph:[global] debug throttle = 0/0 2026-02-12T08:47:52.208 INFO:tasks.ceph:[global] debug timer = 0/0 2026-02-12T08:47:52.208 INFO:tasks.ceph:[global] debug tp = 0/0 2026-02-12T08:47:52.208 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2026-02-12T08:47:52.208 INFO:tasks.ceph:[global] ms client mode = crc 2026-02-12T08:47:52.208 INFO:tasks.ceph:[global] ms cluster mode = crc 2026-02-12T08:47:52.208 INFO:tasks.ceph:[global] ms mon client mode = crc 2026-02-12T08:47:52.208 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2026-02-12T08:47:52.208 INFO:tasks.ceph:[global] ms mon service mode = crc 2026-02-12T08:47:52.208 INFO:tasks.ceph:[global] ms service mode = crc 2026-02-12T08:47:52.208 INFO:tasks.ceph:[mgr] debug mgr = 20 2026-02-12T08:47:52.208 INFO:tasks.ceph:[mgr] debug ms = 1 2026-02-12T08:47:52.209 INFO:tasks.ceph:[mon] debug mon = 0/0 2026-02-12T08:47:52.209 INFO:tasks.ceph:[mon] debug ms = 0/0 2026-02-12T08:47:52.209 INFO:tasks.ceph:[mon] debug paxos = 0/0 2026-02-12T08:47:52.209 INFO:tasks.ceph:[mon] osd pool default crimson = True 2026-02-12T08:47:52.209 INFO:tasks.ceph:Setting up mon.a... 2026-02-12T08:47:52.209 DEBUG:teuthology.orchestra.run.trial187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2026-02-12T08:47:52.278 INFO:teuthology.orchestra.run.trial187.stdout:creating /etc/ceph/ceph.keyring 2026-02-12T08:47:52.281 DEBUG:teuthology.orchestra.run.trial187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2026-02-12T08:47:52.367 DEBUG:teuthology.orchestra.run.trial187:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-12T08:47:52.392 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '10.20.193.187')] 2026-02-12T08:47:52.393 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'none', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '10.20.193.187', 'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': 'true', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore max concurrent transactions': 8}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': '0/0', 'debug mon': '0/0', 'debug paxos': '0/0', 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}} 2026-02-12T08:47:52.395 DEBUG:teuthology.orchestra.run.trial187:> set -ex 2026-02-12T08:47:52.395 DEBUG:teuthology.orchestra.run.trial187:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2026-02-12T08:47:52.449 DEBUG:teuthology.orchestra.run.trial187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 10.20.193.187 --print /home/ubuntu/cephtest/ceph.monmap 2026-02-12T08:47:52.531 INFO:teuthology.orchestra.run.trial187.stderr:2026-02-12T08:47:52.527+0000 7f9330c96f40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T08:47:52.531 INFO:teuthology.orchestra.run.trial187.stderr:2026-02-12T08:47:52.527+0000 7f9330c96f40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T08:47:52.531 INFO:teuthology.orchestra.run.trial187.stderr:2026-02-12T08:47:52.527+0000 7f9330c96f40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T08:47:52.532 INFO:teuthology.orchestra.run.trial187.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2026-02-12T08:47:52.532 INFO:teuthology.orchestra.run.trial187.stdout:monmaptool: generated fsid 7b7dfe4c-4997-4a67-b28c-45558d8d7840 2026-02-12T08:47:52.532 INFO:teuthology.orchestra.run.trial187.stdout:setting min_mon_release = tentacle 2026-02-12T08:47:52.532 INFO:teuthology.orchestra.run.trial187.stdout:epoch 0 2026-02-12T08:47:52.533 INFO:teuthology.orchestra.run.trial187.stdout:fsid 7b7dfe4c-4997-4a67-b28c-45558d8d7840 2026-02-12T08:47:52.533 INFO:teuthology.orchestra.run.trial187.stdout:last_changed 2026-02-12T08:47:52.528981+0000 2026-02-12T08:47:52.533 INFO:teuthology.orchestra.run.trial187.stdout:created 2026-02-12T08:47:52.528981+0000 2026-02-12T08:47:52.533 INFO:teuthology.orchestra.run.trial187.stdout:min_mon_release 20 (tentacle) 2026-02-12T08:47:52.533 INFO:teuthology.orchestra.run.trial187.stdout:election_strategy: 1 2026-02-12T08:47:52.533 INFO:teuthology.orchestra.run.trial187.stdout:0: [v2:10.20.193.187:3300/0,v1:10.20.193.187:6789/0] mon.a 2026-02-12T08:47:52.533 INFO:teuthology.orchestra.run.trial187.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2026-02-12T08:47:52.534 DEBUG:teuthology.orchestra.run.trial187:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2026-02-12T08:47:52.590 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 7b7dfe4c-4997-4a67-b28c-45558d8d7840... 2026-02-12T08:47:52.592 DEBUG:teuthology.orchestra.run.trial187:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2026-02-12T08:47:52.670 INFO:teuthology.orchestra.run.trial187.stdout:[global] 2026-02-12T08:47:52.670 INFO:teuthology.orchestra.run.trial187.stdout: chdir = "" 2026-02-12T08:47:52.671 INFO:teuthology.orchestra.run.trial187.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2026-02-12T08:47:52.671 INFO:teuthology.orchestra.run.trial187.stdout: auth supported = none 2026-02-12T08:47:52.671 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:47:52.671 INFO:teuthology.orchestra.run.trial187.stdout: filestore xattr use omap = true 2026-02-12T08:47:52.671 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:47:52.671 INFO:teuthology.orchestra.run.trial187.stdout: mon clock drift allowed = 1.000 2026-02-12T08:47:52.671 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:47:52.671 INFO:teuthology.orchestra.run.trial187.stdout: osd crush chooseleaf type = 0 2026-02-12T08:47:52.671 INFO:teuthology.orchestra.run.trial187.stdout: auth debug = true 2026-02-12T08:47:52.671 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:47:52.671 INFO:teuthology.orchestra.run.trial187.stdout: ms die on old message = true 2026-02-12T08:47:52.672 INFO:teuthology.orchestra.run.trial187.stdout: ms die on bug = true 2026-02-12T08:47:52.672 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:47:52.672 INFO:teuthology.orchestra.run.trial187.stdout: mon max pg per osd = 10000 # >= luminous 2026-02-12T08:47:52.672 INFO:teuthology.orchestra.run.trial187.stdout: mon pg warn max object skew = 0 2026-02-12T08:47:52.672 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:47:52.672 INFO:teuthology.orchestra.run.trial187.stdout: # disable pg_autoscaler by default for new pools 2026-02-12T08:47:52.672 INFO:teuthology.orchestra.run.trial187.stdout: osd_pool_default_pg_autoscale_mode = off 2026-02-12T08:47:52.672 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:47:52.672 INFO:teuthology.orchestra.run.trial187.stdout: osd pool default size = 2 2026-02-12T08:47:52.672 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:47:52.672 INFO:teuthology.orchestra.run.trial187.stdout: mon osd allow primary affinity = true 2026-02-12T08:47:52.672 INFO:teuthology.orchestra.run.trial187.stdout: mon osd allow pg remap = true 2026-02-12T08:47:52.673 INFO:teuthology.orchestra.run.trial187.stdout: mon warn on legacy crush tunables = false 2026-02-12T08:47:52.673 INFO:teuthology.orchestra.run.trial187.stdout: mon warn on crush straw calc version zero = false 2026-02-12T08:47:52.673 INFO:teuthology.orchestra.run.trial187.stdout: mon warn on no sortbitwise = false 2026-02-12T08:47:52.673 INFO:teuthology.orchestra.run.trial187.stdout: mon warn on osd down out interval zero = false 2026-02-12T08:47:52.673 INFO:teuthology.orchestra.run.trial187.stdout: mon warn on too few osds = false 2026-02-12T08:47:52.673 INFO:teuthology.orchestra.run.trial187.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2026-02-12T08:47:52.673 INFO:teuthology.orchestra.run.trial187.stdout: mon_warn_on_pool_no_redundancy = false 2026-02-12T08:47:52.673 INFO:teuthology.orchestra.run.trial187.stdout: mon_allow_pool_size_one = true 2026-02-12T08:47:52.673 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:47:52.673 INFO:teuthology.orchestra.run.trial187.stdout: osd pool default erasure code profile = plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2026-02-12T08:47:52.673 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:47:52.673 INFO:teuthology.orchestra.run.trial187.stdout: osd default data pool replay window = 5 2026-02-12T08:47:52.674 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:47:52.674 INFO:teuthology.orchestra.run.trial187.stdout: mon allow pool delete = true 2026-02-12T08:47:52.674 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:47:52.674 INFO:teuthology.orchestra.run.trial187.stdout: mon cluster log file level = debug 2026-02-12T08:47:52.674 INFO:teuthology.orchestra.run.trial187.stdout: debug asserts on shutdown = true 2026-02-12T08:47:52.674 INFO:teuthology.orchestra.run.trial187.stdout: mon health detail to clog = false 2026-02-12T08:47:52.674 INFO:teuthology.orchestra.run.trial187.stdout: mon host = 10.20.193.187 2026-02-12T08:47:52.674 INFO:teuthology.orchestra.run.trial187.stdout: auth client required = none 2026-02-12T08:47:52.674 INFO:teuthology.orchestra.run.trial187.stdout: auth cluster required = none 2026-02-12T08:47:52.674 INFO:teuthology.orchestra.run.trial187.stdout: auth service required = none 2026-02-12T08:47:52.674 INFO:teuthology.orchestra.run.trial187.stdout: debug asok = 0/0 2026-02-12T08:47:52.674 INFO:teuthology.orchestra.run.trial187.stdout: debug auth = 0/0 2026-02-12T08:47:52.674 INFO:teuthology.orchestra.run.trial187.stdout: debug bdev = 0/0 2026-02-12T08:47:52.675 INFO:teuthology.orchestra.run.trial187.stdout: debug bluefs = 0/0 2026-02-12T08:47:52.675 INFO:teuthology.orchestra.run.trial187.stdout: debug bluestore = 0/0 2026-02-12T08:47:52.675 INFO:teuthology.orchestra.run.trial187.stdout: debug buffer = 0/0 2026-02-12T08:47:52.675 INFO:teuthology.orchestra.run.trial187.stdout: debug civetweb = 0/0 2026-02-12T08:47:52.675 INFO:teuthology.orchestra.run.trial187.stdout: debug client = 0/0 2026-02-12T08:47:52.675 INFO:teuthology.orchestra.run.trial187.stdout: debug compressor = 0/0 2026-02-12T08:47:52.675 INFO:teuthology.orchestra.run.trial187.stdout: debug context = 0/0 2026-02-12T08:47:52.675 INFO:teuthology.orchestra.run.trial187.stdout: debug crush = 0/0 2026-02-12T08:47:52.675 INFO:teuthology.orchestra.run.trial187.stdout: debug crypto = 0/0 2026-02-12T08:47:52.675 INFO:teuthology.orchestra.run.trial187.stdout: debug dpdk = 0/0 2026-02-12T08:47:52.675 INFO:teuthology.orchestra.run.trial187.stdout: debug eventtrace = 0/0 2026-02-12T08:47:52.675 INFO:teuthology.orchestra.run.trial187.stdout: debug filer = 0/0 2026-02-12T08:47:52.675 INFO:teuthology.orchestra.run.trial187.stdout: debug filestore = 0/0 2026-02-12T08:47:52.676 INFO:teuthology.orchestra.run.trial187.stdout: debug finisher = 0/0 2026-02-12T08:47:52.676 INFO:teuthology.orchestra.run.trial187.stdout: debug fuse = 0/0 2026-02-12T08:47:52.676 INFO:teuthology.orchestra.run.trial187.stdout: debug heartbeatmap = 0/0 2026-02-12T08:47:52.676 INFO:teuthology.orchestra.run.trial187.stdout: debug javaclient = 0/0 2026-02-12T08:47:52.676 INFO:teuthology.orchestra.run.trial187.stdout: debug journal = 0/0 2026-02-12T08:47:52.676 INFO:teuthology.orchestra.run.trial187.stdout: debug journaler = 0/0 2026-02-12T08:47:52.676 INFO:teuthology.orchestra.run.trial187.stdout: debug kstore = 0/0 2026-02-12T08:47:52.676 INFO:teuthology.orchestra.run.trial187.stdout: debug leveldb = 0/0 2026-02-12T08:47:52.676 INFO:teuthology.orchestra.run.trial187.stdout: debug lockdep = 0/0 2026-02-12T08:47:52.676 INFO:teuthology.orchestra.run.trial187.stdout: debug mds = 0/0 2026-02-12T08:47:52.676 INFO:teuthology.orchestra.run.trial187.stdout: debug mds balancer = 0/0 2026-02-12T08:47:52.676 INFO:teuthology.orchestra.run.trial187.stdout: debug mds locker = 0/0 2026-02-12T08:47:52.676 INFO:teuthology.orchestra.run.trial187.stdout: debug mds log = 0/0 2026-02-12T08:47:52.676 INFO:teuthology.orchestra.run.trial187.stdout: debug mds log expire = 0/0 2026-02-12T08:47:52.677 INFO:teuthology.orchestra.run.trial187.stdout: debug mds migrator = 0/0 2026-02-12T08:47:52.677 INFO:teuthology.orchestra.run.trial187.stdout: debug memdb = 0/0 2026-02-12T08:47:52.677 INFO:teuthology.orchestra.run.trial187.stdout: debug mgr = 0/0 2026-02-12T08:47:52.677 INFO:teuthology.orchestra.run.trial187.stdout: debug mgrc = 0/0 2026-02-12T08:47:52.677 INFO:teuthology.orchestra.run.trial187.stdout: debug mon = 0/0 2026-02-12T08:47:52.677 INFO:teuthology.orchestra.run.trial187.stdout: debug monc = 0/0 2026-02-12T08:47:52.677 INFO:teuthology.orchestra.run.trial187.stdout: debug ms = 0/0 2026-02-12T08:47:52.677 INFO:teuthology.orchestra.run.trial187.stdout: debug objclass = 0/0 2026-02-12T08:47:52.677 INFO:teuthology.orchestra.run.trial187.stdout: debug objectcacher = 0/0 2026-02-12T08:47:52.677 INFO:teuthology.orchestra.run.trial187.stdout: debug objecter = 0/0 2026-02-12T08:47:52.677 INFO:teuthology.orchestra.run.trial187.stdout: debug optracker = 0/0 2026-02-12T08:47:52.677 INFO:teuthology.orchestra.run.trial187.stdout: debug osd = 0/0 2026-02-12T08:47:52.677 INFO:teuthology.orchestra.run.trial187.stdout: debug paxos = 0/0 2026-02-12T08:47:52.678 INFO:teuthology.orchestra.run.trial187.stdout: debug perfcounter = 0/0 2026-02-12T08:47:52.678 INFO:teuthology.orchestra.run.trial187.stdout: debug rados = 0/0 2026-02-12T08:47:52.678 INFO:teuthology.orchestra.run.trial187.stdout: debug rbd = 0/0 2026-02-12T08:47:52.678 INFO:teuthology.orchestra.run.trial187.stdout: debug rbd mirror = 0/0 2026-02-12T08:47:52.678 INFO:teuthology.orchestra.run.trial187.stdout: debug rbd replay = 0/0 2026-02-12T08:47:52.678 INFO:teuthology.orchestra.run.trial187.stdout: debug refs = 0/0 2026-02-12T08:47:52.678 INFO:teuthology.orchestra.run.trial187.stdout: debug rgw = 0/0 2026-02-12T08:47:52.678 INFO:teuthology.orchestra.run.trial187.stdout: debug rgw lifecycle = 0/0 2026-02-12T08:47:52.678 INFO:teuthology.orchestra.run.trial187.stdout: debug rgw notification = 0/0 2026-02-12T08:47:52.678 INFO:teuthology.orchestra.run.trial187.stdout: debug rgw sync = 0/0 2026-02-12T08:47:52.678 INFO:teuthology.orchestra.run.trial187.stdout: debug rocksdb = 0/0 2026-02-12T08:47:52.678 INFO:teuthology.orchestra.run.trial187.stdout: debug striper = 0/0 2026-02-12T08:47:52.678 INFO:teuthology.orchestra.run.trial187.stdout: debug throttle = 0/0 2026-02-12T08:47:52.678 INFO:teuthology.orchestra.run.trial187.stdout: debug timer = 0/0 2026-02-12T08:47:52.679 INFO:teuthology.orchestra.run.trial187.stdout: debug tp = 0/0 2026-02-12T08:47:52.679 INFO:teuthology.orchestra.run.trial187.stdout: enable experimental unrecoverable data corrupting features = crimson 2026-02-12T08:47:52.679 INFO:teuthology.orchestra.run.trial187.stdout: ms client mode = crc 2026-02-12T08:47:52.679 INFO:teuthology.orchestra.run.trial187.stdout: ms cluster mode = crc 2026-02-12T08:47:52.679 INFO:teuthology.orchestra.run.trial187.stdout: ms mon client mode = crc 2026-02-12T08:47:52.679 INFO:teuthology.orchestra.run.trial187.stdout: ms mon cluster mode = crc 2026-02-12T08:47:52.679 INFO:teuthology.orchestra.run.trial187.stdout: ms mon service mode = crc 2026-02-12T08:47:52.679 INFO:teuthology.orchestra.run.trial187.stdout: ms service mode = crc 2026-02-12T08:47:52.679 INFO:teuthology.orchestra.run.trial187.stdout: fsid = 7b7dfe4c-4997-4a67-b28c-45558d8d7840 2026-02-12T08:47:52.679 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:47:52.679 INFO:teuthology.orchestra.run.trial187.stdout:[osd] 2026-02-12T08:47:52.679 INFO:teuthology.orchestra.run.trial187.stdout: osd journal size = 100 2026-02-12T08:47:52.679 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:47:52.680 INFO:teuthology.orchestra.run.trial187.stdout: osd scrub load threshold = 5.0 2026-02-12T08:47:52.680 INFO:teuthology.orchestra.run.trial187.stdout: osd scrub max interval = 600 2026-02-12T08:47:52.680 INFO:teuthology.orchestra.run.trial187.stdout: osd mclock profile = high_recovery_ops 2026-02-12T08:47:52.680 INFO:teuthology.orchestra.run.trial187.stdout: osd mclock skip benchmark = true 2026-02-12T08:47:52.680 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:47:52.680 INFO:teuthology.orchestra.run.trial187.stdout: osd recover clone overlap = true 2026-02-12T08:47:52.680 INFO:teuthology.orchestra.run.trial187.stdout: osd recovery max chunk = 1048576 2026-02-12T08:47:52.680 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:47:52.680 INFO:teuthology.orchestra.run.trial187.stdout: osd debug shutdown = true 2026-02-12T08:47:52.680 INFO:teuthology.orchestra.run.trial187.stdout: osd debug op order = true 2026-02-12T08:47:52.680 INFO:teuthology.orchestra.run.trial187.stdout: osd debug verify stray on activate = true 2026-02-12T08:47:52.680 INFO:teuthology.orchestra.run.trial187.stdout: osd debug trim objects = true 2026-02-12T08:47:52.680 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:47:52.681 INFO:teuthology.orchestra.run.trial187.stdout: osd open classes on start = true 2026-02-12T08:47:52.681 INFO:teuthology.orchestra.run.trial187.stdout: osd debug pg log writeout = true 2026-02-12T08:47:52.681 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:47:52.681 INFO:teuthology.orchestra.run.trial187.stdout: osd deep scrub update digest min age = 30 2026-02-12T08:47:52.681 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:47:52.681 INFO:teuthology.orchestra.run.trial187.stdout: osd map max advance = 10 2026-02-12T08:47:52.681 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:47:52.681 INFO:teuthology.orchestra.run.trial187.stdout: journal zero on create = true 2026-02-12T08:47:52.681 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:47:52.681 INFO:teuthology.orchestra.run.trial187.stdout: filestore ondisk finisher threads = 3 2026-02-12T08:47:52.681 INFO:teuthology.orchestra.run.trial187.stdout: filestore apply finisher threads = 3 2026-02-12T08:47:52.681 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:47:52.681 INFO:teuthology.orchestra.run.trial187.stdout: bdev debug aio = true 2026-02-12T08:47:52.682 INFO:teuthology.orchestra.run.trial187.stdout: osd debug misdirected ops = true 2026-02-12T08:47:52.682 INFO:teuthology.orchestra.run.trial187.stdout: debug monc = 20 2026-02-12T08:47:52.682 INFO:teuthology.orchestra.run.trial187.stdout: crimson cpu num = 4 2026-02-12T08:47:52.682 INFO:teuthology.orchestra.run.trial187.stdout: crimson osd obc lru size = 10 2026-02-12T08:47:52.682 INFO:teuthology.orchestra.run.trial187.stdout: debug filestore = 0/0 2026-02-12T08:47:52.682 INFO:teuthology.orchestra.run.trial187.stdout: debug journal = 0/0 2026-02-12T08:47:52.682 INFO:teuthology.orchestra.run.trial187.stdout: debug ms = 0/0 2026-02-12T08:47:52.682 INFO:teuthology.orchestra.run.trial187.stdout: debug osd = 0/0 2026-02-12T08:47:52.682 INFO:teuthology.orchestra.run.trial187.stdout: debug seastore = 20 2026-02-12T08:47:52.682 INFO:teuthology.orchestra.run.trial187.stdout: debug seastore backref = 20 2026-02-12T08:47:52.682 INFO:teuthology.orchestra.run.trial187.stdout: debug seastore cache = 20 2026-02-12T08:47:52.682 INFO:teuthology.orchestra.run.trial187.stdout: debug seastore cleaner = 20 2026-02-12T08:47:52.682 INFO:teuthology.orchestra.run.trial187.stdout: debug seastore device = 20 2026-02-12T08:47:52.683 INFO:teuthology.orchestra.run.trial187.stdout: debug seastore epm = 20 2026-02-12T08:47:52.683 INFO:teuthology.orchestra.run.trial187.stdout: debug seastore fixedkv tree = 20 2026-02-12T08:47:52.683 INFO:teuthology.orchestra.run.trial187.stdout: debug seastore journal = 20 2026-02-12T08:47:52.683 INFO:teuthology.orchestra.run.trial187.stdout: debug seastore lba = 20 2026-02-12T08:47:52.683 INFO:teuthology.orchestra.run.trial187.stdout: debug seastore odata = 20 2026-02-12T08:47:52.683 INFO:teuthology.orchestra.run.trial187.stdout: debug seastore omap = 20 2026-02-12T08:47:52.683 INFO:teuthology.orchestra.run.trial187.stdout: debug seastore onode = 20 2026-02-12T08:47:52.683 INFO:teuthology.orchestra.run.trial187.stdout: debug seastore t = 20 2026-02-12T08:47:52.683 INFO:teuthology.orchestra.run.trial187.stdout: debug seastore tm = 20 2026-02-12T08:47:52.683 INFO:teuthology.orchestra.run.trial187.stdout: osd objectstore = seastore 2026-02-12T08:47:52.683 INFO:teuthology.orchestra.run.trial187.stdout: osd shutdown pgref assert = True 2026-02-12T08:47:52.683 INFO:teuthology.orchestra.run.trial187.stdout: seastore cachepin size pershard = 64M 2026-02-12T08:47:52.683 INFO:teuthology.orchestra.run.trial187.stdout: seastore max concurrent transactions = 8 2026-02-12T08:47:52.683 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:47:52.684 INFO:teuthology.orchestra.run.trial187.stdout:[mgr] 2026-02-12T08:47:52.684 INFO:teuthology.orchestra.run.trial187.stdout: debug ms = 1 2026-02-12T08:47:52.684 INFO:teuthology.orchestra.run.trial187.stdout: debug mgr = 20 2026-02-12T08:47:52.684 INFO:teuthology.orchestra.run.trial187.stdout: debug mon = 20 2026-02-12T08:47:52.684 INFO:teuthology.orchestra.run.trial187.stdout: debug auth = 20 2026-02-12T08:47:52.684 INFO:teuthology.orchestra.run.trial187.stdout: mon reweight min pgs per osd = 4 2026-02-12T08:47:52.684 INFO:teuthology.orchestra.run.trial187.stdout: mon reweight min bytes per osd = 10 2026-02-12T08:47:52.684 INFO:teuthology.orchestra.run.trial187.stdout: mgr/telemetry/nag = false 2026-02-12T08:47:52.684 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:47:52.684 INFO:teuthology.orchestra.run.trial187.stdout:[mon] 2026-02-12T08:47:52.684 INFO:teuthology.orchestra.run.trial187.stdout: debug ms = 0/0 2026-02-12T08:47:52.684 INFO:teuthology.orchestra.run.trial187.stdout: debug mon = 0/0 2026-02-12T08:47:52.685 INFO:teuthology.orchestra.run.trial187.stdout: debug paxos = 0/0 2026-02-12T08:47:52.685 INFO:teuthology.orchestra.run.trial187.stdout: debug auth = 20 2026-02-12T08:47:52.685 INFO:teuthology.orchestra.run.trial187.stdout: mon data avail warn = 5 2026-02-12T08:47:52.685 INFO:teuthology.orchestra.run.trial187.stdout: mon mgr mkfs grace = 240 2026-02-12T08:47:52.685 INFO:teuthology.orchestra.run.trial187.stdout: mon reweight min pgs per osd = 4 2026-02-12T08:47:52.685 INFO:teuthology.orchestra.run.trial187.stdout: mon osd reporter subtree level = osd 2026-02-12T08:47:52.685 INFO:teuthology.orchestra.run.trial187.stdout: mon osd prime pg temp = true 2026-02-12T08:47:52.685 INFO:teuthology.orchestra.run.trial187.stdout: mon reweight min bytes per osd = 10 2026-02-12T08:47:52.685 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:47:52.685 INFO:teuthology.orchestra.run.trial187.stdout: # rotate auth tickets quickly to exercise renewal paths 2026-02-12T08:47:52.685 INFO:teuthology.orchestra.run.trial187.stdout: auth mon ticket ttl = 660 # 11m 2026-02-12T08:47:52.685 INFO:teuthology.orchestra.run.trial187.stdout: auth service ticket ttl = 240 # 4m 2026-02-12T08:47:52.685 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:47:52.685 INFO:teuthology.orchestra.run.trial187.stdout: # don't complain about insecure global_id in the test suite 2026-02-12T08:47:52.686 INFO:teuthology.orchestra.run.trial187.stdout: mon_warn_on_insecure_global_id_reclaim = false 2026-02-12T08:47:52.686 INFO:teuthology.orchestra.run.trial187.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2026-02-12T08:47:52.686 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:47:52.686 INFO:teuthology.orchestra.run.trial187.stdout: # 1m isn't quite enough 2026-02-12T08:47:52.686 INFO:teuthology.orchestra.run.trial187.stdout: mon_down_mkfs_grace = 2m 2026-02-12T08:47:52.686 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:47:52.686 INFO:teuthology.orchestra.run.trial187.stdout: mon_warn_on_filestore_osds = false 2026-02-12T08:47:52.686 INFO:teuthology.orchestra.run.trial187.stdout: osd pool default crimson = True 2026-02-12T08:47:52.686 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:47:52.686 INFO:teuthology.orchestra.run.trial187.stdout:[client] 2026-02-12T08:47:52.686 INFO:teuthology.orchestra.run.trial187.stdout: rgw cache enabled = true 2026-02-12T08:47:52.686 INFO:teuthology.orchestra.run.trial187.stdout: rgw enable ops log = true 2026-02-12T08:47:52.686 INFO:teuthology.orchestra.run.trial187.stdout: rgw enable usage log = true 2026-02-12T08:47:52.687 INFO:teuthology.orchestra.run.trial187.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2026-02-12T08:47:52.687 INFO:teuthology.orchestra.run.trial187.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2026-02-12T08:47:52.687 INFO:teuthology.orchestra.run.trial187.stdout:[mon.a] 2026-02-12T08:47:52.688 INFO:tasks.ceph:Creating admin key on mon.a... 2026-02-12T08:47:52.688 DEBUG:teuthology.orchestra.run.trial187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2026-02-12T08:47:52.777 INFO:tasks.ceph:Copying monmap to all nodes... 2026-02-12T08:47:52.778 DEBUG:teuthology.orchestra.run.trial187:> set -ex 2026-02-12T08:47:52.778 DEBUG:teuthology.orchestra.run.trial187:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2026-02-12T08:47:52.794 DEBUG:teuthology.orchestra.run.trial187:> set -ex 2026-02-12T08:47:52.795 DEBUG:teuthology.orchestra.run.trial187:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2026-02-12T08:47:52.853 INFO:tasks.ceph:Sending monmap to node ubuntu@trial187.front.sepia.ceph.com 2026-02-12T08:47:52.853 DEBUG:teuthology.orchestra.run.trial187:> set -ex 2026-02-12T08:47:52.853 DEBUG:teuthology.orchestra.run.trial187:> sudo dd of=/etc/ceph/ceph.keyring 2026-02-12T08:47:52.853 DEBUG:teuthology.orchestra.run.trial187:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-12T08:47:52.926 DEBUG:teuthology.orchestra.run.trial187:> set -ex 2026-02-12T08:47:52.926 DEBUG:teuthology.orchestra.run.trial187:> dd of=/home/ubuntu/cephtest/ceph.monmap 2026-02-12T08:47:52.983 INFO:tasks.ceph:Setting up mon nodes... 2026-02-12T08:47:52.983 INFO:tasks.ceph:Setting up mgr nodes... 2026-02-12T08:47:52.983 DEBUG:teuthology.orchestra.run.trial187:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2026-02-12T08:47:53.074 INFO:teuthology.orchestra.run.trial187.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2026-02-12T08:47:53.077 INFO:tasks.ceph:Setting up mds nodes... 2026-02-12T08:47:53.077 INFO:tasks.ceph_client:Setting up client nodes... 2026-02-12T08:47:53.077 DEBUG:teuthology.orchestra.run.trial187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2026-02-12T08:47:53.119 INFO:teuthology.orchestra.run.trial187.stdout:creating /etc/ceph/ceph.client.0.keyring 2026-02-12T08:47:53.130 INFO:tasks.ceph:Running mkfs on osd nodes... 2026-02-12T08:47:53.131 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@trial187.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2026-02-12T08:47:53.131 DEBUG:teuthology.orchestra.run.trial187:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2026-02-12T08:47:53.196 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-12T08:47:53.196 INFO:tasks.ceph:role: osd.0 2026-02-12T08:47:53.196 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@trial187.front.sepia.ceph.com 2026-02-12T08:47:53.196 DEBUG:teuthology.orchestra.run.trial187:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2026-02-12T08:47:53.260 INFO:teuthology.orchestra.run.trial187.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=45875200 blks 2026-02-12T08:47:53.260 INFO:teuthology.orchestra.run.trial187.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-12T08:47:53.261 INFO:teuthology.orchestra.run.trial187.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-12T08:47:53.261 INFO:teuthology.orchestra.run.trial187.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-12T08:47:53.261 INFO:teuthology.orchestra.run.trial187.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-12T08:47:53.261 INFO:teuthology.orchestra.run.trial187.stdout: = sunit=0 swidth=0 blks 2026-02-12T08:47:53.261 INFO:teuthology.orchestra.run.trial187.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-12T08:47:53.261 INFO:teuthology.orchestra.run.trial187.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-12T08:47:53.261 INFO:teuthology.orchestra.run.trial187.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-12T08:47:53.261 INFO:teuthology.orchestra.run.trial187.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-12T08:47:53.841 INFO:teuthology.orchestra.run.trial187.stdout:Discarding blocks...Done. 2026-02-12T08:47:53.843 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@trial187.front.sepia.ceph.com -o noatime 2026-02-12T08:47:53.843 DEBUG:teuthology.orchestra.run.trial187:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2026-02-12T08:47:53.911 DEBUG:teuthology.orchestra.run.trial187:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2026-02-12T08:47:53.979 DEBUG:teuthology.orchestra.run.trial187:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2026-02-12T08:47:54.043 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-12T08:47:54.043 INFO:tasks.ceph:role: osd.1 2026-02-12T08:47:54.043 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@trial187.front.sepia.ceph.com 2026-02-12T08:47:54.044 DEBUG:teuthology.orchestra.run.trial187:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2026-02-12T08:47:54.108 INFO:teuthology.orchestra.run.trial187.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=45875200 blks 2026-02-12T08:47:54.108 INFO:teuthology.orchestra.run.trial187.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-12T08:47:54.108 INFO:teuthology.orchestra.run.trial187.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-12T08:47:54.108 INFO:teuthology.orchestra.run.trial187.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-12T08:47:54.109 INFO:teuthology.orchestra.run.trial187.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-12T08:47:54.109 INFO:teuthology.orchestra.run.trial187.stdout: = sunit=0 swidth=0 blks 2026-02-12T08:47:54.109 INFO:teuthology.orchestra.run.trial187.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-12T08:47:54.109 INFO:teuthology.orchestra.run.trial187.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-12T08:47:54.109 INFO:teuthology.orchestra.run.trial187.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-12T08:47:54.109 INFO:teuthology.orchestra.run.trial187.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-12T08:47:54.667 INFO:teuthology.orchestra.run.trial187.stdout:Discarding blocks...Done. 2026-02-12T08:47:54.669 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@trial187.front.sepia.ceph.com -o noatime 2026-02-12T08:47:54.669 DEBUG:teuthology.orchestra.run.trial187:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2026-02-12T08:47:54.736 DEBUG:teuthology.orchestra.run.trial187:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2026-02-12T08:47:54.809 DEBUG:teuthology.orchestra.run.trial187:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2026-02-12T08:47:54.872 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-12T08:47:54.872 INFO:tasks.ceph:role: osd.2 2026-02-12T08:47:54.872 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@trial187.front.sepia.ceph.com 2026-02-12T08:47:54.872 DEBUG:teuthology.orchestra.run.trial187:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2026-02-12T08:47:54.938 INFO:teuthology.orchestra.run.trial187.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=45875200 blks 2026-02-12T08:47:54.939 INFO:teuthology.orchestra.run.trial187.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-12T08:47:54.939 INFO:teuthology.orchestra.run.trial187.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-12T08:47:54.939 INFO:teuthology.orchestra.run.trial187.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-12T08:47:54.939 INFO:teuthology.orchestra.run.trial187.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-12T08:47:54.939 INFO:teuthology.orchestra.run.trial187.stdout: = sunit=0 swidth=0 blks 2026-02-12T08:47:54.939 INFO:teuthology.orchestra.run.trial187.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-12T08:47:54.939 INFO:teuthology.orchestra.run.trial187.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-12T08:47:54.939 INFO:teuthology.orchestra.run.trial187.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-12T08:47:54.939 INFO:teuthology.orchestra.run.trial187.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-12T08:47:55.502 INFO:teuthology.orchestra.run.trial187.stdout:Discarding blocks...Done. 2026-02-12T08:47:55.504 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@trial187.front.sepia.ceph.com -o noatime 2026-02-12T08:47:55.504 DEBUG:teuthology.orchestra.run.trial187:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2026-02-12T08:47:55.572 DEBUG:teuthology.orchestra.run.trial187:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2026-02-12T08:47:55.643 DEBUG:teuthology.orchestra.run.trial187:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-12T08:47:55.793 INFO:teuthology.orchestra.run.trial187.stderr:INFO  osd - OSD::main: parsing early config 2026-02-12T08:47:55.847 INFO:teuthology.orchestra.run.trial187.stderr:INFO 2026-02-12 08:47:55,844 seastar - Reactor backend: linux-aio 2026-02-12T08:47:55.857 INFO:teuthology.orchestra.run.trial187.stderr:INFO 2026-02-12 08:47:55,855 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-12T08:47:55.857 INFO:teuthology.orchestra.run.trial187.stderr:INFO 2026-02-12 08:47:55,855 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-12T08:47:55.859 INFO:teuthology.orchestra.run.trial187.stderr:==21566==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-12T08:47:55.920 INFO:teuthology.orchestra.run.trial187.stderr:DEBUG 2026-02-12 08:47:55,917 [shard 0:main] seastar - reactor::drain 2026-02-12T08:47:58.044 INFO:teuthology.orchestra.run.trial187.stderr:----------------------------------------------------- 2026-02-12T08:47:58.044 INFO:teuthology.orchestra.run.trial187.stderr:Suppressions used: 2026-02-12T08:47:58.044 INFO:teuthology.orchestra.run.trial187.stderr: count bytes template 2026-02-12T08:47:58.044 INFO:teuthology.orchestra.run.trial187.stderr: 1 8 MallocExtension::Register 2026-02-12T08:47:58.044 INFO:teuthology.orchestra.run.trial187.stderr:----------------------------------------------------- 2026-02-12T08:47:58.045 INFO:teuthology.orchestra.run.trial187.stderr: 2026-02-12T08:47:58.064 INFO:teuthology.orchestra.run.trial187.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-12T08:47:58.065 INFO:teuthology.orchestra.run.trial187.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-12T08:47:58.109 INFO:teuthology.orchestra.run.trial187.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-12T08:47:58.109 INFO:teuthology.orchestra.run.trial187.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-12T08:47:58.116 INFO:teuthology.orchestra.run.trial187.stderr:INFO 2026-02-12 08:47:58,114 seastar - Reactor backend: linux-aio 2026-02-12T08:47:58.127 INFO:teuthology.orchestra.run.trial187.stderr:INFO 2026-02-12 08:47:58,124 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-12T08:47:58.127 INFO:teuthology.orchestra.run.trial187.stderr:INFO 2026-02-12 08:47:58,124 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-12T08:47:58.131 INFO:teuthology.orchestra.run.trial187.stderr:==21564==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-12T08:47:58.131 INFO:teuthology.orchestra.run.trial187.stderr:INFO 2026-02-12 08:47:58,128 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-12T08:47:58.163 INFO:teuthology.orchestra.run.trial187.stderr:Reactor stalled for 32 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e14eec 0x9e151d5 0x9e75b73 0x9e76366 0x9e765d4 0x9e7671c 0x9e76bdb /lib64/libc.so.6+0x3fc2f 0xb756b4 0xbb8eaa 0xe05b40 0x15435b7 0x356db1f 0x356dd1a 0x356fbb1 0x33d66cd 0x34552bf 0x467863c 0x46a2793 0x46a3c59 0x46a3cde 0x46a3de0 0x46a3ed8 0x46a40e3 0x46a42c9 0x46a4344 0x46a43bf 0x46a443a 0x46a4da7 0x46a5148 0x46a5373 0x46a55da 0x46a58a5 0x46a5ac3 0x46a5b7b 0x46a5c33 0x46a5cec 0x46a5da4 0x46a5e6a 0x46a5f4b 0x310ca3c 0x310cb42 0x310cbfa 0x310ccb2 0x310cd6a 0x3141145 0x31a0915 0x31a0cd6 0x31a0e18 0x9ecfa3b 0x9e23d10 0x9efd4ab 0x9f65c8e 0x9f674ac 0x9be50b1 0x9be58ec 0xeced0d /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb46b04 2026-02-12T08:47:58.273 INFO:teuthology.orchestra.run.trial187.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e14eec 0x9e151d5 0x9e75b73 0x9e76366 0x9e765d4 0x9e7671c 0x9e76bdb /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x16b36a /lib64/libasan.so.8+0xb0581 /lib64/libasan.so.8+0xb1477 0xb69317 0x1090438 0xf2dc9c 0x9c3d9d0 0x9c3dccd 0x9c3df40 0x9c3fc67 0x9c401b4 0x9c403b3 0x9d256d5 0x9d25e30 0x9d300be 0x9d34537 0x9cf4b05 0x9cf3b05 0x7bd73a3 0x7be0f12 0x7a3027b 0x7a6d86f 0x710f04d 0x6f906da 0x6f944db 0x6f9a085 0x7408a8a 0x742b492 0x727f2b9 0x727f3b1 0x7351548 0x7351601 0x73517b8 0x742b513 0x727f42e 0x727f4b3 0x72ab251 0x741391a 0x741399f 0x7413a7d 0x7296bd6 0x73a49b2 0x741fcb0 0x741fd2b 0x741fda6 0x74e35b8 0x75d8e38 0x9ecfa3b 0x9e23d10 0x9efd4ab 0x9f65c8e 0x9f8071e 0x9f80d9a 0x26e4fbb 0xa58eca0 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T08:47:58.273 INFO:teuthology.orchestra.run.trial187.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e14eec 0x9e151d5 0x9e75b73 0x9e76366 0x9e765d4 0x9e7671c 0x9e76bdb /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x129747 /lib64/libasan.so.8+0xc6631 /lib64/libasan.so.8+0x151b7 0x316a7d9 0x9c3de81 0x9c3e1f6 0x9c3e7b8 0x9c46c5f 0x9c4733f 0x9d2fe8c 0x9d34537 0x9cf4b05 0x9cf3b05 0x7bd73a3 0x7be0f12 0x7a3027b 0x7a6d86f 0x710f04d 0x6f906da 0x6f944db 0x6f9a085 0x7408a8a 0x742b492 0x727f2b9 0x727f3b1 0x7351548 0x7351601 0x73517b8 0x742b513 0x727f42e 0x727f4b3 0x72ab251 0x741391a 0x741399f 0x7413a7d 0x7296bd6 0x73a49b2 0x741fcb0 0x741fd2b 0x741fda6 0x74e35b8 0x75d8e38 0x9ecfa3b 0x9e23d10 0x9efd4ab 0x9f65c8e 0x9f8071e 0x9f80d9a 0x26e4fbb 0xa58eca0 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T08:47:58.273 INFO:teuthology.orchestra.run.trial187.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e14eec 0x9e151d5 0x9e75b73 0x9e76366 0x9e765d4 0x9e7671c 0x9e76bdb /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc6639 /lib64/libasan.so.8+0x15e57 0x9c3db8e 0x9c3df40 0x9c3fc67 0x9c4d55c 0x9c4da13 0x9d252fc 0x9d258ea 0x9d25e30 0x9d300be 0x9d34537 0x9cf4b05 0x9cf3b05 0x7bd73a3 0x7be0f12 0x7a3027b 0x7a6d86f 0x710f04d 0x6f906da 0x6f944db 0x6f9a085 0x7408a8a 0x742b492 0x727f2b9 0x727f3b1 0x7351548 0x7351601 0x73517b8 0x742b513 0x727f42e 0x727f4b3 0x72ab251 0x741391a 0x741399f 0x7413a7d 0x7296bd6 0x73a49b2 0x741fcb0 0x741fd2b 0x741fda6 0x74e35b8 0x75d8e38 0x9ecfa3b 0x9e23d10 0x9efd4ab 0x9f65c8e 0x9f8071e 0x9f80d9a 0x26e4fbb 0xa58eca0 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T08:47:58.305 INFO:teuthology.orchestra.run.trial187.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e14eec 0x9e151d5 0x9e75b73 0x9e76366 0x9e765d4 0x9e7671c 0x9e76bdb /lib64/libc.so.6+0x3fc2f 0x9d354c5 0x9d356d7 0x9d358a0 0x9d35933 0x9d36778 0x788188f 0x7bd6cd5 0x7be0f12 0x7a3027b 0x7a6d86f 0x710f04d 0x6f914c6 0x754759e 0x75476f2 0x9ecfa3b 0x9e23d10 0x9efd4ab 0x9f65c8e 0x9f674ac 0x9be50b1 0x9be58ec 0xeced0d /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb46b04 2026-02-12T08:47:58.382 INFO:teuthology.orchestra.run.trial187.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e14eec 0x9e151d5 0x9e75b73 0x9e76366 0x9e765d4 0x9e7671c 0x9e76bdb /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc661c /lib64/libasan.so.8+0x151b7 0x2a5129f 0x7d7086b 0x7d8ce10 0x7e2ea95 0x7dae38f 0x7864ca5 0x786599a 0x799acdc 0x7823dae 0x789ee50 0x78e75e3 0x78e785c 0x7988a34 0x7988b93 0x799ae3f 0x7855926 0x79776a3 0x7977732 0x79777d6 0x73d42f8 0x73f4757 0x73f93f8 0x753a894 0x9ecfa3b 0x9e23d10 0x9efd4ab 0x9f65c8e 0x9f674ac 0x9be50b1 0x9be58ec 0xeced0d /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb46b04 2026-02-12T08:47:58.412 INFO:teuthology.orchestra.run.trial187.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e14eec 0x9e151d5 0x9e75b73 0x9e76366 0x9e765d4 0x9e7671c 0x9e76bdb /lib64/libc.so.6+0x3fc2f 0x2a5122e 0x7d7086b 0x7d8ce10 0x7e2ea95 0x7dae38f 0x7864ca5 0x786599a 0x799acdc 0x7823dae 0x789ee50 0x78e75e3 0x78e785c 0x7988a34 0x7988b93 0x799ae3f 0x7855926 0x79776a3 0x7977732 0x79777d6 0x73d42f8 0x73f4757 0x73f93f8 0x753a894 0x9ecfa3b 0x9e23d10 0x9efd4ab 0x9f65c8e 0x9f8071e 0x9f80d9a 0x26e4fbb 0xa58eca0 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T08:47:58.413 INFO:teuthology.orchestra.run.trial187.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e14eec 0x9e151d5 0x9e75b73 0x9e76366 0x9e765d4 0x9e7671c 0x9e76bdb /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x129858 /lib64/libasan.so.8+0xc6631 /lib64/libasan.so.8+0x151b7 0x2a5129f 0x7d7086b 0x7d8ce10 0x7e2ea95 0x7dae38f 0x7864ca5 0x786599a 0x799acdc 0x7823dae 0x789ee50 0x78e75e3 0x78e785c 0x7988a34 0x7988b93 0x799ae3f 0x7855926 0x79776a3 0x7977732 0x79777d6 0x73d42f8 0x73f4757 0x73f93f8 0x753a894 0x9ecfa3b 0x9e23d10 0x9efd4ab 0x9f65c8e 0x9f8071e 0x9f80d9a 0x26e4fbb 0xa58eca0 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T08:47:58.416 INFO:teuthology.orchestra.run.trial187.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e14eec 0x9e151d5 0x9e75b73 0x9e76366 0x9e765d4 0x9e7671c 0x9e76bdb /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x8cc0c /lib64/libasan.so.8+0x1006bc /lib64/libasan.so.8+0xc6619 /lib64/libasan.so.8+0x151b7 0x2a5129f 0x7d7086b 0x7d8ce10 0x7e2ea95 0x7dae38f 0x7864ca5 0x786599a 0x799acdc 0x7823dae 0x789ee50 0x78e75e3 0x78e785c 0x7988a34 0x7988b93 0x799ae3f 0x7855926 0x79776a3 0x7977732 0x79777d6 0x73d42f8 0x73f4757 0x73f93f8 0x753a894 0x9ecfa3b 0x9e23d10 0x9efd4ab 0x9f65c8e 0x9f8071e 0x9f80d9a 0x26e4fbb 0xa58eca0 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T08:47:58.544 INFO:teuthology.orchestra.run.trial187.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e14eec 0x9e151d5 0x9e75b73 0x9e76366 0x9e765d4 0x9e7671c 0x9e76bdb /lib64/libc.so.6+0x3fc2f /lib64/libstdc++.so.6+0xc6243 0x3169341 0x9c3dc3d 0x9c3df40 0x9c3e1f6 0x9c3eca5 0x9c3f1a3 0x9c3f7d3 0x9c3f93d 0x9d1fcad 0x9d32ab0 0x9cf4b05 0x9cf3b05 0x7bd73a3 0x7be0f12 0x7a3027b 0x7a6d86f 0x710f04d 0x6f336f2 0x6f35a5a 0x7473988 0x729685f 0x7296961 0x7296a59 0x7350714 0x73507cd 0x7350980 0x7350ab8 0x7350b33 0x7350bb8 0x7350d13 0x7413397 0x7473a11 0x727dcf4 0x7296bd6 0x73a49b2 0x741fcb0 0x741fd2b 0x741fda6 0x74e35b8 0x75d8e38 0x9ecfa3b 0x9e23d10 0x9efd4ab 0x9f65c8e 0x9f8071e 0x9f80d9a 0x26e4fbb 0xa58eca0 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T08:47:58.545 INFO:teuthology.orchestra.run.trial187.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e14eec 0x9e151d5 0x9e75b73 0x9e76366 0x9e765d4 0x9e7671c 0x9e76bdb /lib64/libc.so.6+0x3fc2f /lib64/ld-linux-x86-64.so.2+0x16ed3 /lib64/libasan.so.8+0xc6611 /lib64/libasan.so.8+0x157e7 0x35e89ab 0x35e8b2b 0x36049a8 0x9c4fecb 0x9c4ff8d 0x9d0fd7b 0x9d0ff10 0x9d102a2 0x9d11889 0x9d12ba4 0x9d12fc4 0x9d2f84f 0x9d34537 0x9cf4b05 0x9cf3b05 0x7bd73a3 0x7be0f12 0x7a3027b 0x7a6d86f 0x710f04d 0x6f336f2 0x6f35a5a 0x7473988 0x729685f 0x7296961 0x7296a59 0x7350714 0x73507cd 0x7350980 0x7350ab8 0x7350b33 0x7350bb8 0x7350d13 0x7413397 0x7473a11 0x727dcf4 0x7296bd6 0x73a49b2 0x741fcb0 0x741fd2b 0x741fda6 0x74e35b8 0x75d8e38 0x9ecfa3b 0x9e23d10 0x9efd4ab 0x9f65c8e 0x9f8071e 0x9f80d9a 0x26e4fbb 0xa58eca0 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T08:47:58.545 INFO:teuthology.orchestra.run.trial187.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e14eec 0x9e151d5 0x9e75b73 0x9e76366 0x9e765d4 0x9e7671c 0x9e76bdb /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x5093 /lib64/libasan.so.8+0x1006bc /lib64/libasan.so.8+0xc6619 /lib64/libasan.so.8+0x151b7 0x36048ed 0x9c4fecb 0x9c4ff8d 0x9d0fd7b 0x9d0ff10 0x9d102a2 0x9d11889 0x9d12ba4 0x9d12fc4 0x9d2f84f 0x9d34537 0x9cf4b05 0x9cf3b05 0x7bd73a3 0x7be0f12 0x7a3027b 0x7a6d86f 0x710f04d 0x6f336f2 0x6f35a5a 0x7473988 0x729685f 0x7296961 0x7296a59 0x7350714 0x73507cd 0x7350980 0x7350ab8 0x7350b33 0x7350bb8 0x7350d13 0x7413397 0x7473a11 0x727dcf4 0x7296bd6 0x73a49b2 0x741fcb0 0x741fd2b 0x741fda6 0x74e35b8 0x75d8e38 0x9ecfa3b 0x9e23d10 0x9efd4ab 0x9f65c8e 0x9f8071e 0x9f80d9a 0x26e4fbb 0xa58eca0 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T08:47:58.609 INFO:teuthology.orchestra.run.trial187.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e14eec 0x9e151d5 0x9e75b73 0x9e76366 0x9e765d4 0x9e7671c 0x9e76bdb /lib64/libc.so.6+0x3fc2f 0x29fc44a 0x2a51240 0x7d7086b 0x7d8ce10 0x7e2ea95 0x7dae38f 0x7864ca5 0x786599a 0x799acdc 0x7823dae 0x789ee50 0x78e75e3 0x78e785c 0x7988a34 0x7988b93 0x799ae3f 0x7855926 0x79776a3 0x7977732 0x79777d6 0x73d42f8 0x73f4757 0x73f93f8 0x753a894 0x9ecfa3b 0x9e23d10 0x9efd4ab 0x9f65c8e 0x9f8071e 0x9f80d9a 0x26e4fbb 0xa58eca0 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T08:47:58.627 INFO:teuthology.orchestra.run.trial187.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e14eec 0x9e151d5 0x9e75b73 0x9e76366 0x9e765d4 0x9e7671c 0x9e76bdb /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1525f 0x2a5129f 0x7d7086b 0x7d8ce10 0x7e2ea95 0x7dae38f 0x7864ca5 0x786599a 0x799acdc 0x7823dae 0x789ee50 0x78e75e3 0x78e785c 0x7988a34 0x7988b93 0x799ae3f 0x7855926 0x79776a3 0x7977732 0x79777d6 0x73d42f8 0x73f4757 0x73f93f8 0x753a894 0x9ecfa3b 0x9e23d10 0x9efd4ab 0x9f65c8e 0x9f674ac 0x9be50b1 0x9be58ec 0xeced0d /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb46b04 2026-02-12T08:47:58.642 INFO:teuthology.orchestra.run.trial187.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e14eec 0x9e151d5 0x9e75b73 0x9e76366 0x9e765d4 0x9e7671c 0x9e76bdb /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1294d6 /lib64/libasan.so.8+0x1297ad /lib64/libasan.so.8+0xc6631 /lib64/libasan.so.8+0x151b7 0x2a5129f 0x7d7086b 0x7d8ce10 0x7e2ea95 0x7dae38f 0x7864ca5 0x786599a 0x799acdc 0x7823dae 0x789ee50 0x78e75e3 0x78e785c 0x7988a34 0x7988b93 0x799ae3f 0x7855926 0x79776a3 0x7977732 0x79777d6 0x73d42f8 0x73f4757 0x73f93f8 0x753a894 0x9ecfa3b 0x9e23d10 0x9efd4ab 0x9f65c8e 0x9f8071e 0x9f80d9a 0x26e4fbb 0xa58eca0 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T08:47:58.642 INFO:teuthology.orchestra.run.trial187.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e14eec 0x9e151d5 0x9e75b73 0x9e76366 0x9e765d4 0x9e7671c 0x9e76bdb /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc661c /lib64/libasan.so.8+0x151b7 0x2a5129f 0x7d7086b 0x7d8ce10 0x7e2ea95 0x7dae38f 0x7864ca5 0x786599a 0x799acdc 0x7823dae 0x789ee50 0x78e75e3 0x78e785c 0x7988a34 0x7988b93 0x799ae3f 0x7855926 0x79776a3 0x7977732 0x79777d6 0x73d42f8 0x73f4757 0x73f93f8 0x753a894 0x9ecfa3b 0x9e23d10 0x9efd4ab 0x9f65c8e 0x9f8071e 0x9f80d9a 0x26e4fbb 0xa58eca0 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T08:48:00.840 INFO:teuthology.orchestra.run.trial187.stderr:----------------------------------------------------- 2026-02-12T08:48:00.840 INFO:teuthology.orchestra.run.trial187.stderr:Suppressions used: 2026-02-12T08:48:00.840 INFO:teuthology.orchestra.run.trial187.stderr: count bytes template 2026-02-12T08:48:00.840 INFO:teuthology.orchestra.run.trial187.stderr: 1 8 MallocExtension::Register 2026-02-12T08:48:00.841 INFO:teuthology.orchestra.run.trial187.stderr:----------------------------------------------------- 2026-02-12T08:48:00.841 INFO:teuthology.orchestra.run.trial187.stderr: 2026-02-12T08:48:00.875 DEBUG:teuthology.orchestra.run.trial187:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2026-02-12T08:48:00.901 DEBUG:teuthology.orchestra.run.trial187:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-12T08:48:01.054 INFO:teuthology.orchestra.run.trial187.stderr:INFO  osd - OSD::main: parsing early config 2026-02-12T08:48:01.108 INFO:teuthology.orchestra.run.trial187.stderr:INFO 2026-02-12 08:48:01,106 seastar - Reactor backend: linux-aio 2026-02-12T08:48:01.123 INFO:teuthology.orchestra.run.trial187.stderr:INFO 2026-02-12 08:48:01,121 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-12T08:48:01.123 INFO:teuthology.orchestra.run.trial187.stderr:INFO 2026-02-12 08:48:01,121 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-12T08:48:01.126 INFO:teuthology.orchestra.run.trial187.stderr:==21634==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-12T08:48:01.186 INFO:teuthology.orchestra.run.trial187.stderr:DEBUG 2026-02-12 08:48:01,184 [shard 0:main] seastar - reactor::drain 2026-02-12T08:48:03.309 INFO:teuthology.orchestra.run.trial187.stderr:----------------------------------------------------- 2026-02-12T08:48:03.309 INFO:teuthology.orchestra.run.trial187.stderr:Suppressions used: 2026-02-12T08:48:03.309 INFO:teuthology.orchestra.run.trial187.stderr: count bytes template 2026-02-12T08:48:03.310 INFO:teuthology.orchestra.run.trial187.stderr: 1 8 MallocExtension::Register 2026-02-12T08:48:03.310 INFO:teuthology.orchestra.run.trial187.stderr:----------------------------------------------------- 2026-02-12T08:48:03.310 INFO:teuthology.orchestra.run.trial187.stderr: 2026-02-12T08:48:03.329 INFO:teuthology.orchestra.run.trial187.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-12T08:48:03.330 INFO:teuthology.orchestra.run.trial187.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-12T08:48:03.374 INFO:teuthology.orchestra.run.trial187.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-12T08:48:03.374 INFO:teuthology.orchestra.run.trial187.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-12T08:48:03.381 INFO:teuthology.orchestra.run.trial187.stderr:INFO 2026-02-12 08:48:03,378 seastar - Reactor backend: linux-aio 2026-02-12T08:48:03.393 INFO:teuthology.orchestra.run.trial187.stderr:INFO 2026-02-12 08:48:03,390 [shard 3:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-12T08:48:03.393 INFO:teuthology.orchestra.run.trial187.stderr:INFO 2026-02-12 08:48:03,390 [shard 3:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-12T08:48:03.396 INFO:teuthology.orchestra.run.trial187.stderr:==21632==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-12T08:48:03.397 INFO:teuthology.orchestra.run.trial187.stderr:INFO 2026-02-12 08:48:03,394 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-12T08:48:03.538 INFO:teuthology.orchestra.run.trial187.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e14eec 0x9e151d5 0x9e75b73 0x9e76366 0x9e765d4 0x9e7671c 0x9e76bdb /lib64/libc.so.6+0x3fc2f 0x35e8a3a 0x35e8b2b 0x36049a8 0x9c4fecb 0x9c4ffae 0x9d0fd7b 0x9d0ff10 0x9d102a2 0x9d11889 0x9d12ba4 0x9d12fc4 0x9d2f84f 0x9d34537 0x9cf4b05 0x9cf3b05 0x7bd73a3 0x7be0f12 0x7a3027b 0x7a6d86f 0x710f04d 0x6f906da 0x6f944db 0x6f9a085 0x7408a8a 0x742b492 0x727f2b9 0x727f3b1 0x7351548 0x7351601 0x73517b8 0x742b513 0x727f42e 0x727f4b3 0x72ab251 0x741391a 0x741399f 0x7413a7d 0x7296bd6 0x73a49b2 0x741fcb0 0x741fd2b 0x741fda6 0x74e35b8 0x75d8e38 0x9ecfa3b 0x9e23d10 0x9efd4ab 0x9f65c8e 0x9f8071e 0x9f80d9a 0x26e4fbb 0xa58eca0 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T08:48:03.538 INFO:teuthology.orchestra.run.trial187.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e14eec 0x9e151d5 0x9e75b73 0x9e76366 0x9e765d4 0x9e7671c 0x9e76bdb /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xa35e /lib64/libasan.so.8+0xc468 /lib64/libasan.so.8+0x8eea /lib64/libasan.so.8+0xedc65 0xe1d15b 0x78816dd 0x7bd6cd5 0x7be0f12 0x7a3027b 0x7a6d86f 0x710f04d 0x6f906da 0x6f944db 0x6f9a085 0x7408a8a 0x742b492 0x727f2b9 0x727f3b1 0x7351548 0x7351601 0x73517b8 0x742b513 0x727f42e 0x727f4b3 0x72ab251 0x741391a 0x741399f 0x7413a7d 0x7296bd6 0x73a49b2 0x741fcb0 0x741fd2b 0x741fda6 0x74e35b8 0x75d8e38 0x9ecfa3b 0x9e23d10 0x9efd4ab 0x9f65c8e 0x9f8071e 0x9f80d9a 0x26e4fbb 0xa58eca0 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T08:48:03.538 INFO:teuthology.orchestra.run.trial187.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e14eec 0x9e151d5 0x9e75b73 0x9e76366 0x9e765d4 0x9e7671c 0x9e76bdb /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc3cc /lib64/libasan.so.8+0x91c1 /lib64/libasan.so.8+0xee7fc 0x9c967d5 0x9c96894 0x9c96b3c 0x9d2512b 0x9d258ea 0x9d25e30 0x9d300be 0x9d34537 0x9cf4b05 0x9cf3b05 0x7bd73a3 0x7be0f12 0x7a3027b 0x7a6d86f 0x710f04d 0x6f906da 0x6f944db 0x6f9a085 0x7408a8a 0x742b492 0x727f2b9 0x727f3b1 0x7351548 0x7351601 0x73517b8 0x742b513 0x727f42e 0x727f4b3 0x72ab251 0x741391a 0x741399f 0x7413a7d 0x7296bd6 0x73a49b2 0x741fcb0 0x741fd2b 0x741fda6 0x74e35b8 0x75d8e38 0x9ecfa3b 0x9e23d10 0x9efd4ab 0x9f65c8e 0x9f8071e 0x9f80d9a 0x26e4fbb 0xa58eca0 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T08:48:03.570 INFO:teuthology.orchestra.run.trial187.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e14eec 0x9e151d5 0x9e75b73 0x9e76366 0x9e765d4 0x9e7671c 0x9e76bdb /lib64/libc.so.6+0x3fc2f 0x35fac8d 0x4b12e5b 0x7e2d732 0x7e50ef6 0x8f4a6cf 0x8f4d03d 0x7869e31 0x78a0834 0x6f918a5 0x754759e 0x75476f2 0x9ecfa3b 0x9e23d10 0x9efd4ab 0x9f65c8e 0x9f674ac 0x9be50b1 0x9be58ec 0xeced0d /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb46b04 2026-02-12T08:48:03.636 INFO:teuthology.orchestra.run.trial187.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e14eec 0x9e151d5 0x9e75b73 0x9e76366 0x9e765d4 0x9e7671c 0x9e76bdb /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x129758 /lib64/libasan.so.8+0xc6631 /lib64/libasan.so.8+0x151b7 0x2a5129f 0x7d7086b 0x7d8ce10 0x7e2ea95 0x7dae38f 0x7864ca5 0x786599a 0x799acdc 0x7823dae 0x789ee50 0x78e75e3 0x78e785c 0x7988a34 0x7988b93 0x799ae3f 0x7855926 0x79776a3 0x7977732 0x79777d6 0x73d42f8 0x73f4757 0x73f93f8 0x753a894 0x9ecfa3b 0x9e23d10 0x9efd4ab 0x9f65c8e 0x9f674ac 0x9be50b1 0x9be58ec 0xeced0d /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb46b04 2026-02-12T08:48:03.677 INFO:teuthology.orchestra.run.trial187.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e14eec 0x9e151d5 0x9e75b73 0x9e76366 0x9e765d4 0x9e7671c 0x9e76bdb /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x8cc25 /lib64/libasan.so.8+0x1006bc /lib64/libasan.so.8+0xc6619 /lib64/libasan.so.8+0x151b7 0x2a5129f 0x7d7086b 0x7d8ce10 0x7e2ea95 0x7dae38f 0x7864ca5 0x786599a 0x799acdc 0x7823dae 0x789ee50 0x78e75e3 0x78e785c 0x7988a34 0x7988b93 0x799ae3f 0x7855926 0x79776a3 0x7977732 0x79777d6 0x73d42f8 0x73f4757 0x73f93f8 0x753a894 0x9ecfa3b 0x9e23d10 0x9efd4ab 0x9f65c8e 0x9f8071e 0x9f80d9a 0x26e4fbb 0xa58eca0 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T08:48:03.678 INFO:teuthology.orchestra.run.trial187.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e14eec 0x9e151d5 0x9e75b73 0x9e76366 0x9e765d4 0x9e7671c 0x9e76bdb /lib64/libc.so.6+0x3fc2f 0x2a51280 0x7d7086b 0x7d8ce10 0x7e2ea95 0x7dae38f 0x7864ca5 0x786599a 0x799acdc 0x7823dae 0x789ee50 0x78e75e3 0x78e785c 0x7988a34 0x7988b93 0x799ae3f 0x7855926 0x79776a3 0x7977732 0x79777d6 0x73d42f8 0x73f4757 0x73f93f8 0x753a894 0x9ecfa3b 0x9e23d10 0x9efd4ab 0x9f65c8e 0x9f8071e 0x9f80d9a 0x26e4fbb 0xa58eca0 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T08:48:03.678 INFO:teuthology.orchestra.run.trial187.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e14eec 0x9e151d5 0x9e75b73 0x9e76366 0x9e765d4 0x9e7671c 0x9e76bdb /lib64/libc.so.6+0x3fc2f 0x29fc3fc 0x2a51240 0x7d7086b 0x7d8ce10 0x7e2ea95 0x7dae38f 0x7864ca5 0x786599a 0x799acdc 0x7823dae 0x789ee50 0x78e75e3 0x78e785c 0x7988a34 0x7988b93 0x799ae3f 0x7855926 0x79776a3 0x7977732 0x79777d6 0x73d42f8 0x73f4757 0x73f93f8 0x753a894 0x9ecfa3b 0x9e23d10 0x9efd4ab 0x9f65c8e 0x9f8071e 0x9f80d9a 0x26e4fbb 0xa58eca0 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T08:48:03.804 INFO:teuthology.orchestra.run.trial187.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e14eec 0x9e151d5 0x9e75b73 0x9e76366 0x9e765d4 0x9e7671c 0x9e76bdb /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xb0708 /lib64/libasan.so.8+0xb1477 0xb69317 0x1090438 0xf2dc9c 0x9c4222a 0x9c424d7 0x9d2f9c9 0x9d34537 0x9cf4b05 0x9cf3b05 0x7bd73a3 0x7be0f12 0x7a3027b 0x7a6d86f 0x710f04d 0x6f336f2 0x6f35a5a 0x7473988 0x729685f 0x7296961 0x7296a59 0x7350714 0x73507cd 0x7350980 0x7350ab8 0x7350b33 0x7350bb8 0x7350d13 0x7413397 0x7473a11 0x727dcf4 0x7296bd6 0x73a49b2 0x741fcb0 0x741fd2b 0x741fda6 0x74e35b8 0x75d8e38 0x9ecfa3b 0x9e23d10 0x9efd4ab 0x9f65c8e 0x9f8071e 0x9f80d9a 0x26e4fbb 0xa58eca0 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T08:48:03.804 INFO:teuthology.orchestra.run.trial187.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e14eec 0x9e151d5 0x9e75b73 0x9e76366 0x9e765d4 0x9e7671c 0x9e76bdb /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xfa83f /lib64/libasan.so.8+0xb13e7 0xa5c3d96 0x9d354f0 0x9d356d7 0x9d358a0 0x9d35933 0x9d36778 0x788188f 0x7bd6cd5 0x7be0f12 0x7a3027b 0x7a6d86f 0x710f04d 0x6f336f2 0x6f35a5a 0x7473988 0x729685f 0x7296961 0x7296a59 0x7350714 0x73507cd 0x7350980 0x7350ab8 0x7350b33 0x7350bb8 0x7350d13 0x7413397 0x7473a11 0x727dcf4 0x7296bd6 0x73a49b2 0x741fcb0 0x741fd2b 0x741fda6 0x74e35b8 0x75d8e38 0x9ecfa3b 0x9e23d10 0x9efd4ab 0x9f65c8e 0x9f8071e 0x9f80d9a 0x26e4fbb 0xa58eca0 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T08:48:03.805 INFO:teuthology.orchestra.run.trial187.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e14eec 0x9e151d5 0x9e75b73 0x9e76366 0x9e765d4 0x9e7671c 0x9e76bdb /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x11916f /lib64/libasan.so.8+0xc5f4 /lib64/libasan.so.8+0x91c1 /lib64/libasan.so.8+0xee7fc 0x9b1228b 0x9b8fa26 0x9b8fb3c 0x9b8fc18 0x9b8fe94 0x9b9043b 0x9b9083f 0x9b90afc 0x9d32705 0x9cf4b05 0x9cf3b05 0x7bd73a3 0x7be0f12 0x7a3027b 0x7a6d86f 0x710f04d 0x6f336f2 0x6f35a5a 0x7473988 0x729685f 0x7296961 0x7296a59 0x7350714 0x73507cd 0x7350980 0x7350ab8 0x7350b33 0x7350bb8 0x7350d13 0x7413397 0x7473a11 0x727dcf4 0x7296bd6 0x73a49b2 0x741fcb0 0x741fd2b 0x741fda6 0x74e35b8 0x75d8e38 0x9ecfa3b 0x9e23d10 0x9efd4ab 0x9f65c8e 0x9f8071e 0x9f80d9a 0x26e4fbb 0xa58eca0 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T08:48:03.854 INFO:teuthology.orchestra.run.trial187.stderr:Reactor stalled for 32 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e14eec 0x9e151d5 0x9e75b73 0x9e76366 0x9e765d4 0x9e7671c 0x9e76bdb /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc65f2 /lib64/libasan.so.8+0x151b7 0x2a5129f 0x7d7086b 0x7d8ce10 0x7e2ea95 0x7dae38f 0x7864ca5 0x786599a 0x799acdc 0x7823dae 0x789ee50 0x78e75e3 0x78e785c 0x7988a34 0x7988b93 0x799ae3f 0x7855926 0x79776a3 0x7977732 0x79777d6 0x73d42f8 0x73f4757 0x73f93f8 0x753a894 0x9ecfa3b 0x9e23d10 0x9efd4ab 0x9f65c8e 0x9f674ac 0x9be50b1 0x9be58ec 0xeced0d /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb46b04 2026-02-12T08:48:03.902 INFO:teuthology.orchestra.run.trial187.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e14eec 0x9e151d5 0x9e75b73 0x9e76366 0x9e765d4 0x9e7671c 0x9e76bdb /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xfcb23 /lib64/libasan.so.8+0xc65f2 /lib64/libasan.so.8+0x151b7 0x2a5129f 0x7d7086b 0x7d8ce10 0x7e2ea95 0x7dae38f 0x7864ca5 0x786599a 0x799acdc 0x7823dae 0x789ee50 0x78e75e3 0x78e785c 0x7988a34 0x7988b93 0x799ae3f 0x7855926 0x79776a3 0x7977732 0x79777d6 0x73d42f8 0x73f4757 0x73f93f8 0x753a894 0x9ecfa3b 0x9e23d10 0x9efd4ab 0x9f65c8e 0x9f8071e 0x9f80d9a 0x26e4fbb 0xa58eca0 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T08:48:03.902 INFO:teuthology.orchestra.run.trial187.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e14eec 0x9e151d5 0x9e75b73 0x9e76366 0x9e765d4 0x9e7671c 0x9e76bdb /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1006bf /lib64/libasan.so.8+0xc6619 /lib64/libasan.so.8+0x151b7 0x2a5129f 0x7d7086b 0x7d8ce10 0x7e2ea95 0x7dae38f 0x7864ca5 0x786599a 0x799acdc 0x7823dae 0x789ee50 0x78e75e3 0x78e785c 0x7988a34 0x7988b93 0x799ae3f 0x7855926 0x79776a3 0x7977732 0x79777d6 0x73d42f8 0x73f4757 0x73f93f8 0x753a894 0x9ecfa3b 0x9e23d10 0x9efd4ab 0x9f65c8e 0x9f8071e 0x9f80d9a 0x26e4fbb 0xa58eca0 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T08:48:03.902 INFO:teuthology.orchestra.run.trial187.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e14eec 0x9e151d5 0x9e75b73 0x9e76366 0x9e765d4 0x9e7671c 0x9e76bdb /lib64/libc.so.6+0x3fc2f 0x2a50295 0x7d70884 0x7d8ce10 0x7e2ea95 0x7dae38f 0x7864ca5 0x786599a 0x799acdc 0x7823dae 0x789ee50 0x78e75e3 0x78e785c 0x7988a34 0x7988b93 0x799ae3f 0x7855926 0x79776a3 0x7977732 0x79777d6 0x73d42f8 0x73f4757 0x73f93f8 0x753a894 0x9ecfa3b 0x9e23d10 0x9efd4ab 0x9f65c8e 0x9f8071e 0x9f80d9a 0x26e4fbb 0xa58eca0 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T08:48:06.106 INFO:teuthology.orchestra.run.trial187.stderr:----------------------------------------------------- 2026-02-12T08:48:06.107 INFO:teuthology.orchestra.run.trial187.stderr:Suppressions used: 2026-02-12T08:48:06.107 INFO:teuthology.orchestra.run.trial187.stderr: count bytes template 2026-02-12T08:48:06.107 INFO:teuthology.orchestra.run.trial187.stderr: 1 8 MallocExtension::Register 2026-02-12T08:48:06.107 INFO:teuthology.orchestra.run.trial187.stderr:----------------------------------------------------- 2026-02-12T08:48:06.107 INFO:teuthology.orchestra.run.trial187.stderr: 2026-02-12T08:48:06.141 DEBUG:teuthology.orchestra.run.trial187:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2026-02-12T08:48:06.166 DEBUG:teuthology.orchestra.run.trial187:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-12T08:48:06.320 INFO:teuthology.orchestra.run.trial187.stderr:INFO  osd - OSD::main: parsing early config 2026-02-12T08:48:06.375 INFO:teuthology.orchestra.run.trial187.stderr:INFO 2026-02-12 08:48:06,373 seastar - Reactor backend: linux-aio 2026-02-12T08:48:06.393 INFO:teuthology.orchestra.run.trial187.stderr:INFO 2026-02-12 08:48:06,391 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-12T08:48:06.393 INFO:teuthology.orchestra.run.trial187.stderr:INFO 2026-02-12 08:48:06,391 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-12T08:48:06.396 INFO:teuthology.orchestra.run.trial187.stderr:==21698==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-12T08:48:06.428 INFO:teuthology.orchestra.run.trial187.stderr:Reactor stalled for 32 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e14eec 0x9e151d5 0x9e75b73 0x9e76366 0x9e765d4 0x9e7671c 0x9e76bdb /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc6622 /lib64/libasan.so.8+0x151b7 0x2ac7b30 0x350f2bd 0x350f4df 0x350f73c 0x350f8c9 0x350f9ce 0x3579fd6 0x35805be 0x3580804 0x35823db 0x3582545 0x35829a0 0x3582cd1 0x3583046 0x35c8d7d 0x35c9537 0x35c9839 0x35c99ca 0x35c9c73 0x35c9f7a 0x35ca0c0 0x3579fd6 0x35805be 0x3580804 0x3580958 0x3580a06 0x3580cd0 0x3580ed3 0x3581154 0x3581316 0x3581435 0x3579fd6 0x357a382 0x357a5c8 0x357a69d 0x35833fe 0x35836f2 0x358395b 0x35932f5 0x35cbf8b 0x35cc2ea 0x35cc662 0x35ccb00 0x35ccf3f 0x3594a20 0x3595a57 0x3595d0e 0x35961ad 0x35962c9 0x359657d 0x359690f 0x3596c42 0x3596f9f 0x35971d7 0x3597812 0x3597b42 0x3597ea8 0x3598355 0x3598794 0x35989c0 0x3598dcf 0x3599086 0x35991d7 0x35994a8 0x3599833 0x3599b66 0x359a059 0x359a43b 0x359a5c7 0x359a9ff 0x359ad6b 0x359b0d0 0x359b3f3 0x359b4d5 0x359b763 0x359b93a 0x359bba1 0x35cef59 0x35cf6a9 0x35cfb95 0x35cff0f 0x35d018e 0x35d0243 0x3578e62 0x35797c5 0x3579dff 0x3578e62 0x3579271 0x3579528 2026-02-12T08:48:06.455 INFO:teuthology.orchestra.run.trial187.stderr:DEBUG 2026-02-12 08:48:06,453 [shard 0:main] seastar - reactor::drain 2026-02-12T08:48:08.598 INFO:teuthology.orchestra.run.trial187.stderr:----------------------------------------------------- 2026-02-12T08:48:08.598 INFO:teuthology.orchestra.run.trial187.stderr:Suppressions used: 2026-02-12T08:48:08.598 INFO:teuthology.orchestra.run.trial187.stderr: count bytes template 2026-02-12T08:48:08.598 INFO:teuthology.orchestra.run.trial187.stderr: 1 8 MallocExtension::Register 2026-02-12T08:48:08.598 INFO:teuthology.orchestra.run.trial187.stderr:----------------------------------------------------- 2026-02-12T08:48:08.598 INFO:teuthology.orchestra.run.trial187.stderr: 2026-02-12T08:48:08.617 INFO:teuthology.orchestra.run.trial187.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-12T08:48:08.617 INFO:teuthology.orchestra.run.trial187.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-12T08:48:08.659 INFO:teuthology.orchestra.run.trial187.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-12T08:48:08.659 INFO:teuthology.orchestra.run.trial187.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-12T08:48:08.666 INFO:teuthology.orchestra.run.trial187.stderr:INFO 2026-02-12 08:48:08,664 seastar - Reactor backend: linux-aio 2026-02-12T08:48:08.680 INFO:teuthology.orchestra.run.trial187.stderr:INFO 2026-02-12 08:48:08,678 [shard 2:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-12T08:48:08.681 INFO:teuthology.orchestra.run.trial187.stderr:INFO 2026-02-12 08:48:08,678 [shard 2:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-12T08:48:08.684 INFO:teuthology.orchestra.run.trial187.stderr:==21696==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-12T08:48:08.684 INFO:teuthology.orchestra.run.trial187.stderr:INFO 2026-02-12 08:48:08,682 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-12T08:48:08.831 INFO:teuthology.orchestra.run.trial187.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e14eec 0x9e151d5 0x9e75b73 0x9e76366 0x9e765d4 0x9e7671c 0x9e76bdb /lib64/libc.so.6+0x3fc2f 0xe0b2f2 0xe0b5ab 0x7bd7158 0x7be0f12 0x7a3027b 0x7a6d86f 0x710f04d 0x6f906da 0x6f944db 0x6f9a085 0x7408a8a 0x742b492 0x727f2b9 0x727f3b1 0x7351548 0x7351601 0x73517b8 0x742b513 0x727f42e 0x727f4b3 0x72ab251 0x741391a 0x741399f 0x7413a7d 0x7296bd6 0x73a49b2 0x741fcb0 0x741fd2b 0x741fda6 0x74e35b8 0x75d8e38 0x9ecfa3b 0x9e23d10 0x9efd4ab 0x9f65c8e 0x9f8071e 0x9f80d9a 0x26e4fbb 0xa58eca0 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T08:48:08.831 INFO:teuthology.orchestra.run.trial187.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e14eec 0x9e151d5 0x9e75b73 0x9e76366 0x9e765d4 0x9e7671c 0x9e76bdb /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1297e4 /lib64/libasan.so.8+0xc6631 /lib64/libasan.so.8+0x151b7 0x36048ed 0x9c4fecb 0x9c4ff8d 0x9d0fd7b 0x9d0ff10 0x9d102a2 0x9d11889 0x9d12ba4 0x9d12fc4 0x9d2f84f 0x9d34537 0x9cf4b05 0x9cf3b05 0x7bd73a3 0x7be0f12 0x7a3027b 0x7a6d86f 0x710f04d 0x6f906da 0x6f944db 0x6f9a085 0x7408a8a 0x742b492 0x727f2b9 0x727f3b1 0x7351548 0x7351601 0x73517b8 0x742b513 0x727f42e 0x727f4b3 0x72ab251 0x741391a 0x741399f 0x7413a7d 0x7296bd6 0x73a49b2 0x741fcb0 0x741fd2b 0x741fda6 0x74e35b8 0x75d8e38 0x9ecfa3b 0x9e23d10 0x9efd4ab 0x9f65c8e 0x9f8071e 0x9f80d9a 0x26e4fbb 0xa58eca0 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T08:48:08.831 INFO:teuthology.orchestra.run.trial187.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e14eec 0x9e151d5 0x9e75b73 0x9e76366 0x9e765d4 0x9e7671c 0x9e76bdb /lib64/libc.so.6+0x3fc2f 0x36032c8 0x3605c35 0x9d10267 0x9d11889 0x9d12ba4 0x9d12fc4 0x9d2f84f 0x9d34537 0x9cf4b05 0x9cf3b05 0x7bd73a3 0x7be0f12 0x7a3027b 0x7a6d86f 0x710f04d 0x6f906da 0x6f944db 0x6f9a085 0x7408a8a 0x742b492 0x727f2b9 0x727f3b1 0x7351548 0x7351601 0x73517b8 0x742b513 0x727f42e 0x727f4b3 0x72ab251 0x741391a 0x741399f 0x7413a7d 0x7296bd6 0x73a49b2 0x741fcb0 0x741fd2b 0x741fda6 0x74e35b8 0x75d8e38 0x9ecfa3b 0x9e23d10 0x9efd4ab 0x9f65c8e 0x9f8071e 0x9f80d9a 0x26e4fbb 0xa58eca0 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T08:48:08.863 INFO:teuthology.orchestra.run.trial187.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e14eec 0x9e151d5 0x9e75b73 0x9e76366 0x9e765d4 0x9e7671c 0x9e76bdb /lib64/libc.so.6+0x3fc2f 0xc95915 0xca6f17 0x9c86b00 0x9d300b3 0x9d34537 0x9cf4b05 0x9cf3b05 0x7bd81a4 0x7be0f12 0x7a3027b 0x7a6d86f 0x710f04d 0x6f914c6 0x754759e 0x75476f2 0x9ecfa3b 0x9e23d10 0x9efd4ab 0x9f65c8e 0x9f674ac 0x9be50b1 0x9be58ec 0xeced0d /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb46b04 2026-02-12T08:48:08.934 INFO:teuthology.orchestra.run.trial187.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e14eec 0x9e151d5 0x9e75b73 0x9e76366 0x9e765d4 0x9e7671c 0x9e76bdb /lib64/libc.so.6+0x3fc2f 0x2a5122e 0x7d7086b 0x7d8ce10 0x7e2ea95 0x7dae38f 0x7864ca5 0x786599a 0x799acdc 0x7823dae 0x789ee50 0x78e75e3 0x78e785c 0x7988a34 0x7988b93 0x799ae3f 0x7855926 0x79776a3 0x7977732 0x79777d6 0x73d42f8 0x73f4757 0x73f93f8 0x753a894 0x9ecfa3b 0x9e23d10 0x9efd4ab 0x9f65c8e 0x9f674ac 0x9be50b1 0x9be58ec 0xeced0d /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb46b04 2026-02-12T08:48:08.969 INFO:teuthology.orchestra.run.trial187.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e14eec 0x9e151d5 0x9e75b73 0x9e76366 0x9e765d4 0x9e7671c 0x9e76bdb /lib64/libc.so.6+0x3fc2f 0x2a50222 0x7d70884 0x7d8ce10 0x7e2ea95 0x7dae38f 0x7864ca5 0x786599a 0x799acdc 0x7823dae 0x789ee50 0x78e75e3 0x78e785c 0x7988a34 0x7988b93 0x799ae3f 0x7855926 0x79776a3 0x7977732 0x79777d6 0x73d42f8 0x73f4757 0x73f93f8 0x753a894 0x9ecfa3b 0x9e23d10 0x9efd4ab 0x9f65c8e 0x9f8071e 0x9f80d9a 0x26e4fbb 0xa58eca0 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T08:48:08.969 INFO:teuthology.orchestra.run.trial187.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e14eec 0x9e151d5 0x9e75b73 0x9e76366 0x9e765d4 0x9e7671c 0x9e76bdb /lib64/libc.so.6+0x3fc2f 0x2a50173 0x7d70884 0x7d8ce10 0x7e2ea95 0x7dae38f 0x7864ca5 0x786599a 0x799acdc 0x7823dae 0x789ee50 0x78e75e3 0x78e785c 0x7988a34 0x7988b93 0x799ae3f 0x7855926 0x79776a3 0x7977732 0x79777d6 0x73d42f8 0x73f4757 0x73f93f8 0x753a894 0x9ecfa3b 0x9e23d10 0x9efd4ab 0x9f65c8e 0x9f8071e 0x9f80d9a 0x26e4fbb 0xa58eca0 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T08:48:08.970 INFO:teuthology.orchestra.run.trial187.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e14eec 0x9e151d5 0x9e75b73 0x9e76366 0x9e765d4 0x9e7671c 0x9e76bdb /lib64/libc.so.6+0x3fc2f 0x7d708a6 0x7d8ce10 0x7e2ea95 0x7dae38f 0x7864ca5 0x786599a 0x799acdc 0x7823dae 0x789ee50 0x78e75e3 0x78e785c 0x7988a34 0x7988b93 0x799ae3f 0x7855926 0x79776a3 0x7977732 0x79777d6 0x73d42f8 0x73f4757 0x73f93f8 0x753a894 0x9ecfa3b 0x9e23d10 0x9efd4ab 0x9f65c8e 0x9f8071e 0x9f80d9a 0x26e4fbb 0xa58eca0 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T08:48:09.099 INFO:teuthology.orchestra.run.trial187.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e14eec 0x9e151d5 0x9e75b73 0x9e76366 0x9e765d4 0x9e7671c 0x9e76bdb /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1297e4 /lib64/libasan.so.8+0xc6631 /lib64/libasan.so.8+0x151b7 0x1124fe8 0xf2dc91 0x9c4222a 0x9c42b69 0x9d2eb39 0x9d2fa59 0x9d34537 0x9cf4b05 0x9cf3b05 0x7bd73a3 0x7be0f12 0x7a3027b 0x7a6d86f 0x710f04d 0x6f336f2 0x6f35a5a 0x7473988 0x729685f 0x7296961 0x7296a59 0x7350714 0x73507cd 0x7350980 0x7350ab8 0x7350b33 0x7350bb8 0x7350d13 0x7413397 0x7473a11 0x727dcf4 0x7296bd6 0x73a49b2 0x741fcb0 0x741fd2b 0x741fda6 0x74e35b8 0x75d8e38 0x9ecfa3b 0x9e23d10 0x9efd4ab 0x9f65c8e 0x9f8071e 0x9f80d9a 0x26e4fbb 0xa58eca0 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T08:48:09.099 INFO:teuthology.orchestra.run.trial187.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e14eec 0x9e151d5 0x9e75b73 0x9e76366 0x9e765d4 0x9e7671c 0x9e76bdb /lib64/libc.so.6+0x3fc2f 0x1124f0a 0xf2dc91 0x9c3d9d0 0x9c3dccd 0x9c3df40 0x9c3fc67 0x9d25bc1 0x9d25e30 0x9d300be 0x9d34537 0x9cf4b05 0x9cf3b05 0x7bd73a3 0x7be0f12 0x7a3027b 0x7a6d86f 0x710f04d 0x6f336f2 0x6f35a5a 0x7473988 0x729685f 0x7296961 0x7296a59 0x7350714 0x73507cd 0x7350980 0x7350ab8 0x7350b33 0x7350bb8 0x7350d13 0x7413397 0x7473a11 0x727dcf4 0x7296bd6 0x73a49b2 0x741fcb0 0x741fd2b 0x741fda6 0x74e35b8 0x75d8e38 0x9ecfa3b 0x9e23d10 0x9efd4ab 0x9f65c8e 0x9f8071e 0x9f80d9a 0x26e4fbb 0xa58eca0 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T08:48:09.099 INFO:teuthology.orchestra.run.trial187.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e14eec 0x9e151d5 0x9e75b73 0x9e76366 0x9e765d4 0x9e7671c 0x9e76bdb /lib64/libc.so.6+0x3fc2f 0x9d10130 0x9d102a2 0x9d11889 0x9d12ba4 0x9d12fc4 0x9d2f84f 0x9d34537 0x9cf4b05 0x9cf3b05 0x7bd73a3 0x7be0f12 0x7a3027b 0x7a6d86f 0x710f04d 0x6f336f2 0x6f35a5a 0x7473988 0x729685f 0x7296961 0x7296a59 0x7350714 0x73507cd 0x7350980 0x7350ab8 0x7350b33 0x7350bb8 0x7350d13 0x7413397 0x7473a11 0x727dcf4 0x7296bd6 0x73a49b2 0x741fcb0 0x741fd2b 0x741fda6 0x74e35b8 0x75d8e38 0x9ecfa3b 0x9e23d10 0x9efd4ab 0x9f65c8e 0x9f8071e 0x9f80d9a 0x26e4fbb 0xa58eca0 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T08:48:09.181 INFO:teuthology.orchestra.run.trial187.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e14eec 0x9e151d5 0x9e75b73 0x9e76366 0x9e765d4 0x9e7671c 0x9e76bdb /lib64/libc.so.6+0x3fc2f 0x2a5116c 0x7d7086b 0x7d8ce10 0x7e2ea95 0x7dae38f 0x7864ca5 0x786599a 0x799acdc 0x7823dae 0x789ee50 0x78e75e3 0x78e785c 0x7988a34 0x7988b93 0x799ae3f 0x7855926 0x79776a3 0x7977732 0x79777d6 0x73d42f8 0x73f4757 0x73f93f8 0x753a894 0x9ecfa3b 0x9e23d10 0x9efd4ab 0x9f65c8e 0x9f674ac 0x9be50b1 0x9be58ec 0xeced0d /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb46b04 2026-02-12T08:48:09.196 INFO:teuthology.orchestra.run.trial187.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e14eec 0x9e151d5 0x9e75b73 0x9e76366 0x9e765d4 0x9e7671c 0x9e76bdb /lib64/libc.so.6+0x3fc2f 0x29fc443 0x2a51240 0x7d7086b 0x7d8ce10 0x7e2ea95 0x7dae38f 0x7864ca5 0x786599a 0x799acdc 0x7823dae 0x789ee50 0x78e75e3 0x78e785c 0x7988a34 0x7988b93 0x799ae3f 0x7855926 0x79776a3 0x7977732 0x79777d6 0x73d42f8 0x73f4757 0x73f93f8 0x753a894 0x9ecfa3b 0x9e23d10 0x9efd4ab 0x9f65c8e 0x9f8071e 0x9f80d9a 0x26e4fbb 0xa58eca0 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T08:48:09.196 INFO:teuthology.orchestra.run.trial187.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e14eec 0x9e151d5 0x9e75b73 0x9e76366 0x9e765d4 0x9e7671c 0x9e76bdb /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc6639 /lib64/libasan.so.8+0x151b7 0x2a5129f 0x7d7086b 0x7d8ce10 0x7e2ea95 0x7dae38f 0x7864ca5 0x786599a 0x799acdc 0x7823dae 0x789ee50 0x78e75e3 0x78e785c 0x7988a34 0x7988b93 0x799ae3f 0x7855926 0x79776a3 0x7977732 0x79777d6 0x73d42f8 0x73f4757 0x73f93f8 0x753a894 0x9ecfa3b 0x9e23d10 0x9efd4ab 0x9f65c8e 0x9f8071e 0x9f80d9a 0x26e4fbb 0xa58eca0 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T08:48:09.197 INFO:teuthology.orchestra.run.trial187.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e14eec 0x9e151d5 0x9e75b73 0x9e76366 0x9e765d4 0x9e7671c 0x9e76bdb /lib64/libc.so.6+0x3fc2f 0x2a51099 0x7d7086b 0x7d8ce10 0x7e2ea95 0x7dae38f 0x7864ca5 0x786599a 0x799acdc 0x7823dae 0x789ee50 0x78e75e3 0x78e785c 0x7988a34 0x7988b93 0x799ae3f 0x7855926 0x79776a3 0x7977732 0x79777d6 0x73d42f8 0x73f4757 0x73f93f8 0x753a894 0x9ecfa3b 0x9e23d10 0x9efd4ab 0x9f65c8e 0x9f8071e 0x9f80d9a 0x26e4fbb 0xa58eca0 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T08:48:11.419 INFO:teuthology.orchestra.run.trial187.stderr:----------------------------------------------------- 2026-02-12T08:48:11.419 INFO:teuthology.orchestra.run.trial187.stderr:Suppressions used: 2026-02-12T08:48:11.419 INFO:teuthology.orchestra.run.trial187.stderr: count bytes template 2026-02-12T08:48:11.420 INFO:teuthology.orchestra.run.trial187.stderr: 1 8 MallocExtension::Register 2026-02-12T08:48:11.420 INFO:teuthology.orchestra.run.trial187.stderr:----------------------------------------------------- 2026-02-12T08:48:11.420 INFO:teuthology.orchestra.run.trial187.stderr: 2026-02-12T08:48:11.455 DEBUG:teuthology.orchestra.run.trial187:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2026-02-12T08:48:11.481 INFO:tasks.ceph:Reading keys from all nodes... 2026-02-12T08:48:11.482 DEBUG:teuthology.orchestra.run.trial187:> set -ex 2026-02-12T08:48:11.482 DEBUG:teuthology.orchestra.run.trial187:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2026-02-12T08:48:11.547 DEBUG:teuthology.orchestra.run.trial187:> set -ex 2026-02-12T08:48:11.547 DEBUG:teuthology.orchestra.run.trial187:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2026-02-12T08:48:11.611 DEBUG:teuthology.orchestra.run.trial187:> set -ex 2026-02-12T08:48:11.611 DEBUG:teuthology.orchestra.run.trial187:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2026-02-12T08:48:11.677 DEBUG:teuthology.orchestra.run.trial187:> set -ex 2026-02-12T08:48:11.677 DEBUG:teuthology.orchestra.run.trial187:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2026-02-12T08:48:11.740 DEBUG:teuthology.orchestra.run.trial187:> set -ex 2026-02-12T08:48:11.741 DEBUG:teuthology.orchestra.run.trial187:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2026-02-12T08:48:11.797 INFO:tasks.ceph:Adding keys to all mons... 2026-02-12T08:48:11.797 DEBUG:teuthology.orchestra.run.trial187:> sudo tee -a /etc/ceph/ceph.keyring 2026-02-12T08:48:11.860 INFO:teuthology.orchestra.run.trial187.stdout:[mgr.x] 2026-02-12T08:48:11.860 INFO:teuthology.orchestra.run.trial187.stdout: key = AQC5k41p525GBBAAGuJyAelPkBCHLQZaLhAKew== 2026-02-12T08:48:11.860 INFO:teuthology.orchestra.run.trial187.stdout:[osd.0] 2026-02-12T08:48:11.860 INFO:teuthology.orchestra.run.trial187.stdout: key = AQC+k41pnfCaCxAAgP7I8iH8Gg0bDaDhr8gB/w== 2026-02-12T08:48:11.861 INFO:teuthology.orchestra.run.trial187.stdout:[osd.1] 2026-02-12T08:48:11.861 INFO:teuthology.orchestra.run.trial187.stdout: key = AQDDk41pwHdmGxAAcrqj1G8ry02l8t7VqLbHJw== 2026-02-12T08:48:11.861 INFO:teuthology.orchestra.run.trial187.stdout:[osd.2] 2026-02-12T08:48:11.861 INFO:teuthology.orchestra.run.trial187.stdout: key = AQDIk41pF3bOLBAAOZaNHjBsLl0+Zt0dKzzkaA== 2026-02-12T08:48:11.861 INFO:teuthology.orchestra.run.trial187.stdout:[client.0] 2026-02-12T08:48:11.861 INFO:teuthology.orchestra.run.trial187.stdout: key = AQC5k41ppJb8BhAAAwGFdhSfu8yE4EnSvtxTkQ== 2026-02-12T08:48:11.862 DEBUG:teuthology.orchestra.run.trial187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2026-02-12T08:48:11.951 DEBUG:teuthology.orchestra.run.trial187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-12T08:48:11.999 DEBUG:teuthology.orchestra.run.trial187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-12T08:48:12.048 DEBUG:teuthology.orchestra.run.trial187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-12T08:48:12.096 DEBUG:teuthology.orchestra.run.trial187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-12T08:48:12.147 INFO:tasks.ceph:Running mkfs on mon nodes... 2026-02-12T08:48:12.147 DEBUG:teuthology.orchestra.run.trial187:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2026-02-12T08:48:12.173 DEBUG:teuthology.orchestra.run.trial187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2026-02-12T08:48:12.258 INFO:teuthology.orchestra.run.trial187.stderr:2026-02-12T08:48:12.255+0000 7f50132b9c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T08:48:12.259 INFO:teuthology.orchestra.run.trial187.stderr:2026-02-12T08:48:12.255+0000 7f50132b9c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T08:48:12.265 DEBUG:teuthology.orchestra.run.trial187:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2026-02-12T08:48:12.290 DEBUG:teuthology.orchestra.run.trial187:> rm -- /home/ubuntu/cephtest/ceph.monmap 2026-02-12T08:48:12.345 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2026-02-12T08:48:12.346 INFO:tasks.ceph.mon.a:Restarting daemon 2026-02-12T08:48:12.346 DEBUG:teuthology.orchestra.run.trial187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2026-02-12T08:48:12.388 INFO:tasks.ceph.mon.a:Started 2026-02-12T08:48:12.388 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2026-02-12T08:48:12.388 INFO:tasks.ceph.mgr.x:Restarting daemon 2026-02-12T08:48:12.389 DEBUG:teuthology.orchestra.run.trial187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2026-02-12T08:48:12.391 INFO:tasks.ceph.mgr.x:Started 2026-02-12T08:48:12.391 DEBUG:tasks.ceph:set 0 configs 2026-02-12T08:48:12.391 DEBUG:teuthology.orchestra.run.trial187:> sudo ceph --cluster ceph config dump 2026-02-12T08:48:12.453 INFO:tasks.ceph.mgr.x.trial187.stderr:2026-02-12T08:48:12.449+0000 7f6edfdc0fc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T08:48:12.454 INFO:tasks.ceph.mon.a.trial187.stderr:2026-02-12T08:48:12.450+0000 7f07ceeddc00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T08:48:12.454 INFO:tasks.ceph.mon.a.trial187.stderr:2026-02-12T08:48:12.450+0000 7f07ceeddc00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T08:48:12.454 INFO:tasks.ceph.mon.a.trial187.stderr:2026-02-12T08:48:12.450+0000 7f07ceeddc00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T08:48:12.467 INFO:teuthology.orchestra.run.trial187.stderr:2026-02-12T08:48:12.462+0000 7fadadd83640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T08:48:12.469 INFO:teuthology.orchestra.run.trial187.stderr:2026-02-12T08:48:12.465+0000 7fadadd83640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T08:48:12.500 INFO:teuthology.orchestra.run.trial187.stdout:WHO MASK LEVEL OPTION VALUE RO 2026-02-12T08:48:12.508 INFO:tasks.ceph:Setting crush tunables to default 2026-02-12T08:48:12.508 DEBUG:teuthology.orchestra.run.trial187:> sudo ceph --cluster ceph osd crush tunables default 2026-02-12T08:48:12.581 INFO:teuthology.orchestra.run.trial187.stderr:2026-02-12T08:48:12.574+0000 7fb6c5acb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T08:48:12.582 INFO:teuthology.orchestra.run.trial187.stderr:2026-02-12T08:48:12.578+0000 7fb6c5acb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T08:48:12.615 INFO:teuthology.orchestra.run.trial187.stderr:adjusted tunables profile to default 2026-02-12T08:48:12.623 INFO:tasks.ceph:check_enable_crimson: True 2026-02-12T08:48:12.623 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2026-02-12T08:48:12.624 DEBUG:teuthology.orchestra.run.trial187:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2026-02-12T08:48:12.656 INFO:tasks.ceph.mgr.x.trial187.stderr:2026-02-12T08:48:12.653+0000 7f6edfdc0fc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T08:48:12.657 INFO:tasks.ceph.mgr.x.trial187.stderr:2026-02-12T08:48:12.653+0000 7f6edfdc0fc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T08:48:12.696 INFO:teuthology.orchestra.run.trial187.stderr:2026-02-12T08:48:12.689+0000 7fef1010b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T08:48:12.697 INFO:teuthology.orchestra.run.trial187.stderr:2026-02-12T08:48:12.693+0000 7fef1010b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T08:48:12.738 INFO:tasks.ceph.mon.a.trial187.stderr:2026-02-12T08:48:12.734+0000 7f07c65f1640 -1 WARNING: experimental feature 'crimson' is enabled 2026-02-12T08:48:12.738 INFO:tasks.ceph.mon.a.trial187.stderr:Please be aware that this feature is experimental, untested, 2026-02-12T08:48:12.738 INFO:tasks.ceph.mon.a.trial187.stderr:unsupported, and may result in data corruption, data loss, 2026-02-12T08:48:12.738 INFO:tasks.ceph.mon.a.trial187.stderr:and/or irreparable damage to your cluster. Do not use 2026-02-12T08:48:12.738 INFO:tasks.ceph.mon.a.trial187.stderr:feature with important data. 2026-02-12T08:48:12.738 INFO:tasks.ceph.mon.a.trial187.stderr: 2026-02-12T08:48:13.496 INFO:tasks.ceph.mgr.x.trial187.stderr:/usr/lib64/python3.9/site-packages/scipy/__init__.py:73: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2026-02-12T08:48:13.496 INFO:tasks.ceph.mgr.x.trial187.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2026-02-12T08:48:13.496 INFO:tasks.ceph.mgr.x.trial187.stderr: from numpy import show_config as show_numpy_config 2026-02-12T08:48:13.626 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2026-02-12T08:48:13.626 DEBUG:teuthology.orchestra.run.trial187:> set -ex 2026-02-12T08:48:13.626 DEBUG:teuthology.orchestra.run.trial187:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2026-02-12T08:48:13.691 DEBUG:teuthology.orchestra.run.trial187:> set -ex 2026-02-12T08:48:13.691 DEBUG:teuthology.orchestra.run.trial187:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2026-02-12T08:48:13.754 DEBUG:teuthology.orchestra.run.trial187:> set -ex 2026-02-12T08:48:13.754 DEBUG:teuthology.orchestra.run.trial187:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2026-02-12T08:48:13.816 DEBUG:teuthology.orchestra.run.trial187:> sudo ceph --cluster ceph osd new 47e6ade1-ddda-4d98-b3f2-d5299ec8c7e9 0 2026-02-12T08:48:13.932 INFO:teuthology.orchestra.run.trial187.stderr:2026-02-12T08:48:13.925+0000 7eff71011640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T08:48:13.933 INFO:teuthology.orchestra.run.trial187.stderr:2026-02-12T08:48:13.930+0000 7eff71011640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T08:48:13.970 INFO:teuthology.orchestra.run.trial187.stdout:0 2026-02-12T08:48:13.978 DEBUG:teuthology.orchestra.run.trial187:> sudo ceph --cluster ceph osd new 11ebae76-9d8b-4a71-bc2e-69d4dc7d6e9f 1 2026-02-12T08:48:14.052 INFO:teuthology.orchestra.run.trial187.stderr:2026-02-12T08:48:14.045+0000 7fbf79c46640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T08:48:14.053 INFO:teuthology.orchestra.run.trial187.stderr:2026-02-12T08:48:14.050+0000 7fbf79c46640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T08:48:14.090 INFO:teuthology.orchestra.run.trial187.stdout:1 2026-02-12T08:48:14.098 DEBUG:teuthology.orchestra.run.trial187:> sudo ceph --cluster ceph osd new b8bf0d6d-c4c2-4ca1-a416-5e2fc282efa3 2 2026-02-12T08:48:14.173 INFO:teuthology.orchestra.run.trial187.stderr:2026-02-12T08:48:14.166+0000 7faefff0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T08:48:14.174 INFO:teuthology.orchestra.run.trial187.stderr:2026-02-12T08:48:14.170+0000 7faefff0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T08:48:14.212 INFO:teuthology.orchestra.run.trial187.stdout:2 2026-02-12T08:48:14.225 DEBUG:teuthology.orchestra.run.trial187:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2026-02-12T08:48:14.252 INFO:tasks.ceph.osd.0:Restarting daemon 2026-02-12T08:48:14.253 DEBUG:teuthology.orchestra.run.trial187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2026-02-12T08:48:14.295 INFO:tasks.ceph.osd.0:Started 2026-02-12T08:48:14.295 DEBUG:teuthology.orchestra.run.trial187:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2026-02-12T08:48:14.323 INFO:tasks.ceph.osd.1:Restarting daemon 2026-02-12T08:48:14.324 DEBUG:teuthology.orchestra.run.trial187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2026-02-12T08:48:14.366 INFO:tasks.ceph.osd.1:Started 2026-02-12T08:48:14.366 DEBUG:teuthology.orchestra.run.trial187:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2026-02-12T08:48:14.395 INFO:tasks.ceph.osd.2:Restarting daemon 2026-02-12T08:48:14.395 DEBUG:teuthology.orchestra.run.trial187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2026-02-12T08:48:14.438 INFO:tasks.ceph.osd.2:Started 2026-02-12T08:48:14.438 DEBUG:teuthology.orchestra.run.trial187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-12T08:48:14.516 INFO:teuthology.orchestra.run.trial187.stderr:2026-02-12T08:48:14.511+0000 7f121765a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T08:48:14.518 INFO:teuthology.orchestra.run.trial187.stderr:2026-02-12T08:48:14.514+0000 7f121765a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T08:48:14.550 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:48:14.551 INFO:teuthology.orchestra.run.trial187.stdout:{"epoch":6,"fsid":"7b7dfe4c-4997-4a67-b28c-45558d8d7840","created":"2026-02-12T08:48:12.461759+0000","modified":"2026-02-12T08:48:14.209663+0000","last_up_change":"0.000000","last_in_change":"2026-02-12T08:48:14.209663+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"47e6ade1-ddda-4d98-b3f2-d5299ec8c7e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"11ebae76-9d8b-4a71-bc2e-69d4dc7d6e9f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b8bf0d6d-c4c2-4ca1-a416-5e2fc282efa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-12T08:48:14.560 INFO:tasks.ceph.ceph_manager.ceph:[] 2026-02-12T08:48:14.560 INFO:tasks.ceph:Waiting for OSDs to come up 2026-02-12T08:48:14.862 DEBUG:teuthology.orchestra.run.trial187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-12T08:48:14.933 INFO:teuthology.misc.health.trial187.stderr:2026-02-12T08:48:14.927+0000 7f9961411640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T08:48:14.936 INFO:teuthology.misc.health.trial187.stderr:2026-02-12T08:48:14.932+0000 7f9961411640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T08:48:14.969 INFO:teuthology.misc.health.trial187.stdout: 2026-02-12T08:48:14.969 INFO:teuthology.misc.health.trial187.stdout:{"epoch":6,"fsid":"7b7dfe4c-4997-4a67-b28c-45558d8d7840","created":"2026-02-12T08:48:12.461759+0000","modified":"2026-02-12T08:48:14.209663+0000","last_up_change":"0.000000","last_in_change":"2026-02-12T08:48:14.209663+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"47e6ade1-ddda-4d98-b3f2-d5299ec8c7e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"11ebae76-9d8b-4a71-bc2e-69d4dc7d6e9f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b8bf0d6d-c4c2-4ca1-a416-5e2fc282efa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-12T08:48:14.977 DEBUG:teuthology.misc:0 of 3 OSDs are up 2026-02-12T08:48:15.922 INFO:tasks.ceph.mgr.x.trial187.stderr:2026-02-12T08:48:15.918+0000 7f6dfad54640 -1 mgr.server handle_report got status from non-daemon mon.a 2026-02-12T08:48:21.279 DEBUG:teuthology.orchestra.run.trial187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-12T08:48:21.364 INFO:teuthology.misc.health.trial187.stderr:2026-02-12T08:48:21.357+0000 7efdbdfa4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T08:48:21.366 INFO:teuthology.misc.health.trial187.stderr:2026-02-12T08:48:21.362+0000 7efdbdfa4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T08:48:21.496 INFO:teuthology.misc.health.trial187.stdout: 2026-02-12T08:48:21.497 INFO:teuthology.misc.health.trial187.stdout:{"epoch":7,"fsid":"7b7dfe4c-4997-4a67-b28c-45558d8d7840","created":"2026-02-12T08:48:12.461759+0000","modified":"2026-02-12T08:48:20.676545+0000","last_up_change":"0.000000","last_in_change":"2026-02-12T08:48:14.209663+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"47e6ade1-ddda-4d98-b3f2-d5299ec8c7e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"11ebae76-9d8b-4a71-bc2e-69d4dc7d6e9f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b8bf0d6d-c4c2-4ca1-a416-5e2fc282efa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-12T08:48:21.514 DEBUG:teuthology.misc:0 of 3 OSDs are up 2026-02-12T08:48:27.817 DEBUG:teuthology.orchestra.run.trial187:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-12T08:48:27.887 INFO:teuthology.misc.health.trial187.stderr:2026-02-12T08:48:27.882+0000 7f6067ddc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T08:48:27.888 INFO:teuthology.misc.health.trial187.stderr:2026-02-12T08:48:27.885+0000 7f6067ddc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T08:48:27.998 INFO:teuthology.misc.health.trial187.stdout: 2026-02-12T08:48:27.998 INFO:teuthology.misc.health.trial187.stdout:{"epoch":13,"fsid":"7b7dfe4c-4997-4a67-b28c-45558d8d7840","created":"2026-02-12T08:48:12.461759+0000","modified":"2026-02-12T08:48:27.688757+0000","last_up_change":"2026-02-12T08:48:23.678422+0000","last_in_change":"2026-02-12T08:48:14.209663+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-12T08:48:24.928999+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"47e6ade1-ddda-4d98-b3f2-d5299ec8c7e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.187:6807","nonce":2609858288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.187:6802","nonce":2609858288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.187:6812","nonce":2609858288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.187:6813","nonce":2609858288}]},"public_addr":"10.20.193.187:6807/2609858288","cluster_addr":"10.20.193.187:6802/2609858288","heartbeat_back_addr":"10.20.193.187:6812/2609858288","heartbeat_front_addr":"10.20.193.187:6813/2609858288","state":["exists","up"]},{"osd":1,"uuid":"11ebae76-9d8b-4a71-bc2e-69d4dc7d6e9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.187:6806","nonce":1640918308}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.187:6805","nonce":1640918308}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.187:6810","nonce":1640918308}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.187:6811","nonce":1640918308}]},"public_addr":"10.20.193.187:6806/1640918308","cluster_addr":"10.20.193.187:6805/1640918308","heartbeat_back_addr":"10.20.193.187:6810/1640918308","heartbeat_front_addr":"10.20.193.187:6811/1640918308","state":["exists","up"]},{"osd":2,"uuid":"b8bf0d6d-c4c2-4ca1-a416-5e2fc282efa3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.187:6804","nonce":1581523883}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.187:6803","nonce":1581523883}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.187:6809","nonce":1581523883}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.187:6808","nonce":1581523883}]},"public_addr":"10.20.193.187:6804/1581523883","cluster_addr":"10.20.193.187:6803/1581523883","heartbeat_back_addr":"10.20.193.187:6809/1581523883","heartbeat_front_addr":"10.20.193.187:6808/1581523883","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-12T08:48:28.006 DEBUG:teuthology.misc:3 of 3 OSDs are up 2026-02-12T08:48:28.006 INFO:tasks.ceph:Creating RBD pool 2026-02-12T08:48:28.006 DEBUG:teuthology.orchestra.run.trial187:> sudo ceph --cluster ceph osd pool create rbd 8 2026-02-12T08:48:28.080 INFO:teuthology.orchestra.run.trial187.stderr:2026-02-12T08:48:28.074+0000 7fb013ecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T08:48:28.081 INFO:teuthology.orchestra.run.trial187.stderr:2026-02-12T08:48:28.078+0000 7fb013ecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T08:48:28.697 INFO:teuthology.orchestra.run.trial187.stderr:pool 'rbd' created 2026-02-12T08:48:28.710 DEBUG:teuthology.orchestra.run.trial187:> rbd --cluster ceph pool init rbd 2026-02-12T08:48:28.752 INFO:teuthology.orchestra.run.trial187.stderr:2026-02-12T08:48:28.749+0000 7fe1f0e45240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T08:48:28.752 INFO:teuthology.orchestra.run.trial187.stderr:2026-02-12T08:48:28.749+0000 7fe1f0e45240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T08:48:28.753 INFO:teuthology.orchestra.run.trial187.stderr:2026-02-12T08:48:28.751+0000 7fe1f0e45240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T08:48:31.735 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2026-02-12T08:48:31.736 DEBUG:teuthology.orchestra.run.trial187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2026-02-12T08:48:31.736 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2026-02-12T08:48:31.853 INFO:teuthology.orchestra.run.trial187.stderr:2026-02-12T08:48:31.848+0000 7f01d565a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T08:48:31.854 INFO:teuthology.orchestra.run.trial187.stderr:2026-02-12T08:48:31.851+0000 7f01d565a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T08:48:31.964 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:48:31.974 INFO:teuthology.orchestra.run.trial187.stdout:[{"version":1,"timestamp":"0.000000","name":"","changes":[]}] 2026-02-12T08:48:31.974 INFO:tasks.ceph_manager:config epoch is 1 2026-02-12T08:48:31.974 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2026-02-12T08:48:31.974 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2026-02-12T08:48:31.975 DEBUG:teuthology.orchestra.run.trial187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2026-02-12T08:48:32.050 INFO:teuthology.orchestra.run.trial187.stderr:2026-02-12T08:48:32.046+0000 7ff3265ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T08:48:32.052 INFO:teuthology.orchestra.run.trial187.stderr:2026-02-12T08:48:32.049+0000 7ff3265ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T08:48:32.193 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:48:32.201 INFO:teuthology.orchestra.run.trial187.stdout:{"epoch":5,"flags":0,"active_gid":4103,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.187:6800","nonce":1671341843},{"type":"v1","addr":"10.20.193.187:6801","nonce":1671341843}]},"active_addr":"10.20.193.187:6801/1671341843","active_change":"2026-02-12T08:48:14.914266+0000","active_mgr_features":4544132024016699391,"available":true,"standbys":[],"modules":["iostat","nfs"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_pg_upmap_activity":{"name":"update_pg_upmap_activity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Updates pg_upmap activity stats to be used in `balancer status detail`","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often the agent on each host will gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"The first TCP port the agent will try to bind to. Up to 1000 subsequent ports will be attempted if this port cannot be bound","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this option can facilitate debugging daemons that encounter runtime problems.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"How frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"Fraction of total system memory to divide among autotuned daemons. Converged systems hosting both Ceph and compute should set a lower value; systems with more than 10 GiB of memory per OSD may benefit from a slightly larger value so that the abundance of RAM can be fully utilized","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command persistent logging","long_desc":"","tags":[],"see_also":[]},"certificate_automated_rotation_enabled":{"name":"certificate_automated_rotation_enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"This flag controls whether cephadm automatically rotates certificates upon expiration.","long_desc":"","tags":[],"see_also":[]},"certificate_check_debug_mode":{"name":"certificate_check_debug_mode","type":"bool","level":"dev","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"FOR TESTING ONLY: This flag forces the certificate check instead of waiting for certificate_check_period.","long_desc":"","tags":[],"see_also":[]},"certificate_check_period":{"name":"certificate_check_period","type":"int","level":"advanced","flags":0,"default_value":"1","min":"0","max":"30","enum_allowed":[],"desc":"Specifies how often (in days) the certificate should be checked for validity.","long_desc":"","tags":[],"see_also":[]},"certificate_duration_days":{"name":"certificate_duration_days","type":"int","level":"advanced","flags":0,"default_value":"1095","min":"90","max":"3650","enum_allowed":[],"desc":"Specifies the duration of self-signed certificates generated and signed by the cephadm root CA","long_desc":"","tags":[],"see_also":[]},"certificate_renewal_threshold_days":{"name":"certificate_renewal_threshold_days","type":"int","level":"advanced","flags":0,"default_value":"30","min":"10","max":"90","enum_allowed":[],"desc":"Specifies the lead time in days to initiate certificate renewal before expiration.","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently Podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.28.1","min":"","max":"","enum_allowed":[],"desc":"Alertmanager container image","long_desc":"","tags":[],"see_also":[]},"container_image_alloy":{"name":"container_image_alloy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/alloy:latest","min":"","max":"","enum_allowed":[],"desc":"Alloy container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"Elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:12.3.1","min":"","max":"","enum_allowed":[],"desc":"Grafana container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"Haproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.9.1","min":"","max":"","enum_allowed":[],"desc":"Node exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.6","min":"","max":"","enum_allowed":[],"desc":"Nvmeof container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"Oauth2 proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v3.6.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba metrics container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"Snmp gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"quay.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"Location of dashboards to include in Grafana","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"How frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether the hardware monitoring daemon be deployed on every host?","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at the debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"Max number of daemons per service per host: guard against mistakes and runaways","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"Max number of OSDs that will be drained simultaneously when OSDs are removed","long_desc":"","tags":[],"see_also":[]},"max_parallel_osd_upgrades":{"name":"max_parallel_osd_upgrades","type":"int","level":"advanced","flags":0,"default_value":"16","min":"","max":"","enum_allowed":[],"desc":"Maximum number of OSD daemons upgraded in parallel.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"Mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default IP address for RedFish API (OOB management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"Location of alerts to include in Prometheus Alertmanager","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry URL for login purposes. This is not the default registry.","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"Cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Customized SSH config file to distribute to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many sequential SSH connection liveness check failures before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often SSH connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"stray_daemon_check_interval":{"name":"stray_daemon_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"How frequently cephadm should check for the presence of stray daemons","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use the cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest to ensure that all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROM_ALERT_CREDENTIAL_CACHE_TTL":{"name":"PROM_ALERT_CREDENTIAL_CACHE_TTL","type":"int","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_HOSTNAME_PER_DAEMON":{"name":"RGW_HOSTNAME_PER_DAEMON","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crypto_caller":{"name":"crypto_caller","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":true,"error_string":"","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"healthcheck_history_max_entries":{"name":"healthcheck_history_max_entries","type":"int","level":"advanced","flags":1,"default_value":"1000","min":"","max":"","enum_allowed":[],"desc":"Maximum number of health check history entries to keep","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"prometheus_tls_secret_name":{"name":"prometheus_tls_secret_name","type":"str","level":"advanced","flags":0,"default_value":"rook-ceph-prometheus-server-tls","min":"","max":"","enum_allowed":[],"desc":"name of tls secret in k8s for prometheus","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_cloning":{"name":"pause_cloning","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_purging":{"name":"pause_purging","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous subvolume purge threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"tentacle":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"force_disabled_modules":{},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"10.20.193.187:0","nonce":3298033082}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"10.20.193.187:0","nonce":1515298418}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"10.20.193.187:0","nonce":3135177255}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"10.20.193.187:0","nonce":1468250079}]}]} 2026-02-12T08:48:32.205 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2026-02-12T08:48:32.205 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2026-02-12T08:48:32.206 DEBUG:teuthology.orchestra.run.trial187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-12T08:48:32.282 INFO:teuthology.orchestra.run.trial187.stderr:2026-02-12T08:48:32.277+0000 7fd36045a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T08:48:32.283 INFO:teuthology.orchestra.run.trial187.stderr:2026-02-12T08:48:32.280+0000 7fd36045a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T08:48:32.393 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:48:32.393 INFO:teuthology.orchestra.run.trial187.stdout:{"epoch":17,"fsid":"7b7dfe4c-4997-4a67-b28c-45558d8d7840","created":"2026-02-12T08:48:12.461759+0000","modified":"2026-02-12T08:48:31.701996+0000","last_up_change":"2026-02-12T08:48:23.678422+0000","last_in_change":"2026-02-12T08:48:14.209663+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-12T08:48:24.928999+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-12T08:48:28.195744+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"47e6ade1-ddda-4d98-b3f2-d5299ec8c7e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.187:6807","nonce":2609858288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.187:6802","nonce":2609858288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.187:6812","nonce":2609858288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.187:6813","nonce":2609858288}]},"public_addr":"10.20.193.187:6807/2609858288","cluster_addr":"10.20.193.187:6802/2609858288","heartbeat_back_addr":"10.20.193.187:6812/2609858288","heartbeat_front_addr":"10.20.193.187:6813/2609858288","state":["exists","up"]},{"osd":1,"uuid":"11ebae76-9d8b-4a71-bc2e-69d4dc7d6e9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.187:6806","nonce":1640918308}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.187:6805","nonce":1640918308}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.187:6810","nonce":1640918308}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.187:6811","nonce":1640918308}]},"public_addr":"10.20.193.187:6806/1640918308","cluster_addr":"10.20.193.187:6805/1640918308","heartbeat_back_addr":"10.20.193.187:6810/1640918308","heartbeat_front_addr":"10.20.193.187:6811/1640918308","state":["exists","up"]},{"osd":2,"uuid":"b8bf0d6d-c4c2-4ca1-a416-5e2fc282efa3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.187:6804","nonce":1581523883}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.187:6803","nonce":1581523883}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.187:6809","nonce":1581523883}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.187:6808","nonce":1581523883}]},"public_addr":"10.20.193.187:6804/1581523883","cluster_addr":"10.20.193.187:6803/1581523883","heartbeat_back_addr":"10.20.193.187:6809/1581523883","heartbeat_front_addr":"10.20.193.187:6808/1581523883","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-12T08:48:32.403 INFO:tasks.ceph.ceph_manager.ceph:all up! 2026-02-12T08:48:32.403 DEBUG:teuthology.orchestra.run.trial187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-12T08:48:32.480 INFO:teuthology.orchestra.run.trial187.stderr:2026-02-12T08:48:32.475+0000 7fadbc6cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T08:48:32.481 INFO:teuthology.orchestra.run.trial187.stderr:2026-02-12T08:48:32.478+0000 7fadbc6cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T08:48:32.591 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:48:32.592 INFO:teuthology.orchestra.run.trial187.stdout:{"epoch":17,"fsid":"7b7dfe4c-4997-4a67-b28c-45558d8d7840","created":"2026-02-12T08:48:12.461759+0000","modified":"2026-02-12T08:48:31.701996+0000","last_up_change":"2026-02-12T08:48:23.678422+0000","last_in_change":"2026-02-12T08:48:14.209663+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-12T08:48:24.928999+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-12T08:48:28.195744+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"47e6ade1-ddda-4d98-b3f2-d5299ec8c7e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.187:6807","nonce":2609858288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.187:6802","nonce":2609858288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.187:6812","nonce":2609858288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.187:6813","nonce":2609858288}]},"public_addr":"10.20.193.187:6807/2609858288","cluster_addr":"10.20.193.187:6802/2609858288","heartbeat_back_addr":"10.20.193.187:6812/2609858288","heartbeat_front_addr":"10.20.193.187:6813/2609858288","state":["exists","up"]},{"osd":1,"uuid":"11ebae76-9d8b-4a71-bc2e-69d4dc7d6e9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.187:6806","nonce":1640918308}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.187:6805","nonce":1640918308}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.187:6810","nonce":1640918308}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.187:6811","nonce":1640918308}]},"public_addr":"10.20.193.187:6806/1640918308","cluster_addr":"10.20.193.187:6805/1640918308","heartbeat_back_addr":"10.20.193.187:6810/1640918308","heartbeat_front_addr":"10.20.193.187:6811/1640918308","state":["exists","up"]},{"osd":2,"uuid":"b8bf0d6d-c4c2-4ca1-a416-5e2fc282efa3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.187:6804","nonce":1581523883}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.187:6803","nonce":1581523883}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.187:6809","nonce":1581523883}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.187:6808","nonce":1581523883}]},"public_addr":"10.20.193.187:6804/1581523883","cluster_addr":"10.20.193.187:6803/1581523883","heartbeat_back_addr":"10.20.193.187:6809/1581523883","heartbeat_front_addr":"10.20.193.187:6808/1581523883","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-12T08:48:32.602 DEBUG:teuthology.orchestra.run.trial187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-12T08:48:32.602 DEBUG:teuthology.orchestra.run.trial187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-12T08:48:32.603 DEBUG:teuthology.orchestra.run.trial187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-12T08:48:32.683 INFO:teuthology.orchestra.run.trial187.stderr:2026-02-12T08:48:32.678+0000 7fa7d81d3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T08:48:32.684 INFO:teuthology.orchestra.run.trial187.stderr:2026-02-12T08:48:32.678+0000 7fb2b5ccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T08:48:32.684 INFO:teuthology.orchestra.run.trial187.stderr:2026-02-12T08:48:32.678+0000 7fdc63c3d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T08:48:32.685 INFO:teuthology.orchestra.run.trial187.stderr:2026-02-12T08:48:32.682+0000 7fdc63c3d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T08:48:32.685 INFO:teuthology.orchestra.run.trial187.stderr:2026-02-12T08:48:32.682+0000 7fa7d81d3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T08:48:32.685 INFO:teuthology.orchestra.run.trial187.stderr:2026-02-12T08:48:32.682+0000 7fb2b5ccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T08:48:32.699 INFO:teuthology.orchestra.run.trial187.stdout:42949672968 2026-02-12T08:48:32.700 DEBUG:teuthology.orchestra.run.trial187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-12T08:48:32.701 INFO:teuthology.orchestra.run.trial187.stdout:42949672968 2026-02-12T08:48:32.702 DEBUG:teuthology.orchestra.run.trial187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-12T08:48:32.703 INFO:teuthology.orchestra.run.trial187.stdout:38654705673 2026-02-12T08:48:32.703 DEBUG:teuthology.orchestra.run.trial187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-12T08:48:32.821 INFO:teuthology.orchestra.run.trial187.stderr:2026-02-12T08:48:32.815+0000 7ff53fdc1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T08:48:32.822 INFO:teuthology.orchestra.run.trial187.stderr:2026-02-12T08:48:32.816+0000 7f95c643d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T08:48:32.822 INFO:teuthology.orchestra.run.trial187.stderr:2026-02-12T08:48:32.819+0000 7ff53fdc1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T08:48:32.823 INFO:teuthology.orchestra.run.trial187.stderr:2026-02-12T08:48:32.820+0000 7f95c643d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T08:48:32.825 INFO:teuthology.orchestra.run.trial187.stderr:2026-02-12T08:48:32.819+0000 7f0d1761a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T08:48:32.827 INFO:teuthology.orchestra.run.trial187.stderr:2026-02-12T08:48:32.825+0000 7f0d1761a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T08:48:32.938 INFO:teuthology.orchestra.run.trial187.stdout:42949672968 2026-02-12T08:48:32.940 INFO:teuthology.orchestra.run.trial187.stdout:42949672968 2026-02-12T08:48:32.946 INFO:teuthology.orchestra.run.trial187.stdout:38654705673 2026-02-12T08:48:32.949 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672968 got 42949672968 for osd.1 2026-02-12T08:48:32.949 DEBUG:teuthology.parallel:result is None 2026-02-12T08:48:32.951 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672968 got 42949672968 for osd.0 2026-02-12T08:48:32.951 DEBUG:teuthology.parallel:result is None 2026-02-12T08:48:32.958 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705673 got 38654705673 for osd.2 2026-02-12T08:48:32.958 DEBUG:teuthology.parallel:result is None 2026-02-12T08:48:32.958 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-12T08:48:32.958 DEBUG:teuthology.orchestra.run.trial187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T08:48:33.074 INFO:teuthology.orchestra.run.trial187.stderr:2026-02-12T08:48:33.069+0000 7fe834bdc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T08:48:33.075 INFO:teuthology.orchestra.run.trial187.stderr:2026-02-12T08:48:33.073+0000 7fe834bdc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T08:48:33.185 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:48:33.186 INFO:teuthology.orchestra.run.trial187.stderr:dumped all 2026-02-12T08:48:33.194 INFO:teuthology.orchestra.run.trial187.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2026-02-12T08:48:32.918103+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156499968,"kb_used":2492,"kb_used_data":2492,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156497476,"statfs":{"total":160255967232,"available":160253415424,"internally_reserved":0,"allocated":2551808,"data_stored":2551808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.227052"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-12T08:48:31.716949+0000","last_change":"2026-02-12T08:48:31.720609+0000","last_active":"2026-02-12T08:48:31.716949+0000","last_peered":"2026-02-12T08:48:31.716949+0000","last_clean":"2026-02-12T08:48:31.716949+0000","last_became_active":"2026-02-12T08:48:29.733536+0000","last_became_peered":"2026-02-12T08:48:29.733536+0000","last_unstale":"2026-02-12T08:48:31.716949+0000","last_undegraded":"2026-02-12T08:48:31.716949+0000","last_fullsized":"2026-02-12T08:48:31.716949+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T08:48:28.692133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T08:48:28.692133+0000","last_clean_scrub_stamp":"2026-02-12T08:48:28.692133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-12T08:48:31.718468+0000","last_change":"2026-02-12T08:48:31.720274+0000","last_active":"2026-02-12T08:48:31.718468+0000","last_peered":"2026-02-12T08:48:31.718468+0000","last_clean":"2026-02-12T08:48:31.718468+0000","last_became_active":"2026-02-12T08:48:29.729455+0000","last_became_peered":"2026-02-12T08:48:29.729455+0000","last_unstale":"2026-02-12T08:48:31.718468+0000","last_undegraded":"2026-02-12T08:48:31.718468+0000","last_fullsized":"2026-02-12T08:48:31.718468+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T08:48:28.692133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T08:48:28.692133+0000","last_clean_scrub_stamp":"2026-02-12T08:48:28.692133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-12T08:48:31.718474+0000","last_change":"2026-02-12T08:48:31.721742+0000","last_active":"2026-02-12T08:48:31.718474+0000","last_peered":"2026-02-12T08:48:31.718474+0000","last_clean":"2026-02-12T08:48:31.718474+0000","last_became_active":"2026-02-12T08:48:29.736066+0000","last_became_peered":"2026-02-12T08:48:29.736066+0000","last_unstale":"2026-02-12T08:48:31.718474+0000","last_undegraded":"2026-02-12T08:48:31.718474+0000","last_fullsized":"2026-02-12T08:48:31.718474+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T08:48:28.692133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T08:48:28.692133+0000","last_clean_scrub_stamp":"2026-02-12T08:48:28.692133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-12T08:48:31.718501+0000","last_change":"2026-02-12T08:48:31.720346+0000","last_active":"2026-02-12T08:48:31.718501+0000","last_peered":"2026-02-12T08:48:31.718501+0000","last_clean":"2026-02-12T08:48:31.718501+0000","last_became_active":"2026-02-12T08:48:29.733040+0000","last_became_peered":"2026-02-12T08:48:29.733040+0000","last_unstale":"2026-02-12T08:48:31.718501+0000","last_undegraded":"2026-02-12T08:48:31.718501+0000","last_fullsized":"2026-02-12T08:48:31.718501+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T08:48:28.692133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T08:48:28.692133+0000","last_clean_scrub_stamp":"2026-02-12T08:48:28.692133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"17'2","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-12T08:48:31.721781+0000","last_change":"2026-02-12T08:48:31.722619+0000","last_active":"2026-02-12T08:48:31.721781+0000","last_peered":"2026-02-12T08:48:31.721781+0000","last_clean":"2026-02-12T08:48:31.721781+0000","last_became_active":"2026-02-12T08:48:29.744861+0000","last_became_peered":"2026-02-12T08:48:29.744861+0000","last_unstale":"2026-02-12T08:48:31.721781+0000","last_undegraded":"2026-02-12T08:48:31.721781+0000","last_fullsized":"2026-02-12T08:48:31.721781+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T08:48:28.692133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T08:48:28.692133+0000","last_clean_scrub_stamp":"2026-02-12T08:48:28.692133+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-12T08:48:31.715985+0000","last_change":"2026-02-12T08:48:31.717908+0000","last_active":"2026-02-12T08:48:31.715985+0000","last_peered":"2026-02-12T08:48:31.715985+0000","last_clean":"2026-02-12T08:48:31.715985+0000","last_became_active":"2026-02-12T08:48:29.747097+0000","last_became_peered":"2026-02-12T08:48:29.747097+0000","last_unstale":"2026-02-12T08:48:31.715985+0000","last_undegraded":"2026-02-12T08:48:31.715985+0000","last_fullsized":"2026-02-12T08:48:31.715985+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T08:48:28.692133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T08:48:28.692133+0000","last_clean_scrub_stamp":"2026-02-12T08:48:28.692133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-12T08:48:31.714335+0000","last_change":"2026-02-12T08:48:31.716543+0000","last_active":"2026-02-12T08:48:31.714335+0000","last_peered":"2026-02-12T08:48:31.714335+0000","last_clean":"2026-02-12T08:48:31.714335+0000","last_became_active":"2026-02-12T08:48:29.740234+0000","last_became_peered":"2026-02-12T08:48:29.740234+0000","last_unstale":"2026-02-12T08:48:31.714335+0000","last_undegraded":"2026-02-12T08:48:31.714335+0000","last_fullsized":"2026-02-12T08:48:31.714335+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T08:48:28.692133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T08:48:28.692133+0000","last_clean_scrub_stamp":"2026-02-12T08:48:28.692133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":16,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-12T08:48:31.718592+0000","last_change":"2026-02-12T08:48:31.721811+0000","last_active":"2026-02-12T08:48:31.718592+0000","last_peered":"2026-02-12T08:48:31.718592+0000","last_clean":"2026-02-12T08:48:31.718592+0000","last_became_active":"2026-02-12T08:48:29.729376+0000","last_became_peered":"2026-02-12T08:48:29.729376+0000","last_unstale":"2026-02-12T08:48:31.718592+0000","last_undegraded":"2026-02-12T08:48:31.718592+0000","last_fullsized":"2026-02-12T08:48:31.718592+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T08:48:28.692133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T08:48:28.692133+0000","last_clean_scrub_stamp":"2026-02-12T08:48:28.692133+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":90,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-12T08:48:31.716611+0000","last_change":"2026-02-12T08:48:26.718087+0000","last_active":"2026-02-12T08:48:31.716611+0000","last_peered":"2026-02-12T08:48:31.716611+0000","last_clean":"2026-02-12T08:48:31.716611+0000","last_became_active":"2026-02-12T08:48:26.711088+0000","last_became_peered":"2026-02-12T08:48:26.711088+0000","last_unstale":"2026-02-12T08:48:31.716611+0000","last_undegraded":"2026-02-12T08:48:31.716611+0000","last_fullsized":"2026-02-12T08:48:31.716611+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T08:48:25.682332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T08:48:25.682332+0000","last_clean_scrub_stamp":"2026-02-12T08:48:25.682332+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160255967232,"available":160253366272,"internally_reserved":0,"allocated":2600960,"data_stored":2600960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53418655744,"available":53417480192,"internally_reserved":0,"allocated":1175552,"data_stored":1175552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705673,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":308,"kb_used_data":308,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52166348,"statfs":{"total":53418655744,"available":53418340352,"internally_reserved":0,"allocated":315392,"data_stored":315392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":10,"seq":42949672968,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1116,"kb_used_data":1116,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165540,"statfs":{"total":53418655744,"available":53417512960,"internally_reserved":0,"allocated":1142784,"data_stored":1142784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949672968,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1068,"kb_used_data":1068,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165588,"statfs":{"total":53418655744,"available":53417562112,"internally_reserved":0,"allocated":1093632,"data_stored":1093632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53418655744,"available":53417480192,"internally_reserved":0,"allocated":1175552,"data_stored":1175552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53418655744,"available":53417553920,"internally_reserved":0,"allocated":1101824,"data_stored":1101824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53418655744,"available":53417480192,"internally_reserved":0,"allocated":1175552,"data_stored":1175552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53418655744,"available":53418332160,"internally_reserved":0,"allocated":323584,"data_stored":323584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T08:48:33.196 DEBUG:teuthology.orchestra.run.trial187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T08:48:33.273 INFO:teuthology.orchestra.run.trial187.stderr:2026-02-12T08:48:33.268+0000 7f2a8985a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T08:48:33.274 INFO:teuthology.orchestra.run.trial187.stderr:2026-02-12T08:48:33.272+0000 7f2a8985a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T08:48:33.388 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:48:33.389 INFO:teuthology.orchestra.run.trial187.stderr:dumped all 2026-02-12T08:48:33.398 INFO:teuthology.orchestra.run.trial187.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2026-02-12T08:48:32.918103+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156499968,"kb_used":2492,"kb_used_data":2492,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156497476,"statfs":{"total":160255967232,"available":160253415424,"internally_reserved":0,"allocated":2551808,"data_stored":2551808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.227052"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-12T08:48:31.716949+0000","last_change":"2026-02-12T08:48:31.720609+0000","last_active":"2026-02-12T08:48:31.716949+0000","last_peered":"2026-02-12T08:48:31.716949+0000","last_clean":"2026-02-12T08:48:31.716949+0000","last_became_active":"2026-02-12T08:48:29.733536+0000","last_became_peered":"2026-02-12T08:48:29.733536+0000","last_unstale":"2026-02-12T08:48:31.716949+0000","last_undegraded":"2026-02-12T08:48:31.716949+0000","last_fullsized":"2026-02-12T08:48:31.716949+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T08:48:28.692133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T08:48:28.692133+0000","last_clean_scrub_stamp":"2026-02-12T08:48:28.692133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-12T08:48:31.718468+0000","last_change":"2026-02-12T08:48:31.720274+0000","last_active":"2026-02-12T08:48:31.718468+0000","last_peered":"2026-02-12T08:48:31.718468+0000","last_clean":"2026-02-12T08:48:31.718468+0000","last_became_active":"2026-02-12T08:48:29.729455+0000","last_became_peered":"2026-02-12T08:48:29.729455+0000","last_unstale":"2026-02-12T08:48:31.718468+0000","last_undegraded":"2026-02-12T08:48:31.718468+0000","last_fullsized":"2026-02-12T08:48:31.718468+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T08:48:28.692133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T08:48:28.692133+0000","last_clean_scrub_stamp":"2026-02-12T08:48:28.692133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-12T08:48:31.718474+0000","last_change":"2026-02-12T08:48:31.721742+0000","last_active":"2026-02-12T08:48:31.718474+0000","last_peered":"2026-02-12T08:48:31.718474+0000","last_clean":"2026-02-12T08:48:31.718474+0000","last_became_active":"2026-02-12T08:48:29.736066+0000","last_became_peered":"2026-02-12T08:48:29.736066+0000","last_unstale":"2026-02-12T08:48:31.718474+0000","last_undegraded":"2026-02-12T08:48:31.718474+0000","last_fullsized":"2026-02-12T08:48:31.718474+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T08:48:28.692133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T08:48:28.692133+0000","last_clean_scrub_stamp":"2026-02-12T08:48:28.692133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-12T08:48:31.718501+0000","last_change":"2026-02-12T08:48:31.720346+0000","last_active":"2026-02-12T08:48:31.718501+0000","last_peered":"2026-02-12T08:48:31.718501+0000","last_clean":"2026-02-12T08:48:31.718501+0000","last_became_active":"2026-02-12T08:48:29.733040+0000","last_became_peered":"2026-02-12T08:48:29.733040+0000","last_unstale":"2026-02-12T08:48:31.718501+0000","last_undegraded":"2026-02-12T08:48:31.718501+0000","last_fullsized":"2026-02-12T08:48:31.718501+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T08:48:28.692133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T08:48:28.692133+0000","last_clean_scrub_stamp":"2026-02-12T08:48:28.692133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"17'2","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-12T08:48:31.721781+0000","last_change":"2026-02-12T08:48:31.722619+0000","last_active":"2026-02-12T08:48:31.721781+0000","last_peered":"2026-02-12T08:48:31.721781+0000","last_clean":"2026-02-12T08:48:31.721781+0000","last_became_active":"2026-02-12T08:48:29.744861+0000","last_became_peered":"2026-02-12T08:48:29.744861+0000","last_unstale":"2026-02-12T08:48:31.721781+0000","last_undegraded":"2026-02-12T08:48:31.721781+0000","last_fullsized":"2026-02-12T08:48:31.721781+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T08:48:28.692133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T08:48:28.692133+0000","last_clean_scrub_stamp":"2026-02-12T08:48:28.692133+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-12T08:48:31.715985+0000","last_change":"2026-02-12T08:48:31.717908+0000","last_active":"2026-02-12T08:48:31.715985+0000","last_peered":"2026-02-12T08:48:31.715985+0000","last_clean":"2026-02-12T08:48:31.715985+0000","last_became_active":"2026-02-12T08:48:29.747097+0000","last_became_peered":"2026-02-12T08:48:29.747097+0000","last_unstale":"2026-02-12T08:48:31.715985+0000","last_undegraded":"2026-02-12T08:48:31.715985+0000","last_fullsized":"2026-02-12T08:48:31.715985+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T08:48:28.692133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T08:48:28.692133+0000","last_clean_scrub_stamp":"2026-02-12T08:48:28.692133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-12T08:48:31.714335+0000","last_change":"2026-02-12T08:48:31.716543+0000","last_active":"2026-02-12T08:48:31.714335+0000","last_peered":"2026-02-12T08:48:31.714335+0000","last_clean":"2026-02-12T08:48:31.714335+0000","last_became_active":"2026-02-12T08:48:29.740234+0000","last_became_peered":"2026-02-12T08:48:29.740234+0000","last_unstale":"2026-02-12T08:48:31.714335+0000","last_undegraded":"2026-02-12T08:48:31.714335+0000","last_fullsized":"2026-02-12T08:48:31.714335+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T08:48:28.692133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T08:48:28.692133+0000","last_clean_scrub_stamp":"2026-02-12T08:48:28.692133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":16,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-12T08:48:31.718592+0000","last_change":"2026-02-12T08:48:31.721811+0000","last_active":"2026-02-12T08:48:31.718592+0000","last_peered":"2026-02-12T08:48:31.718592+0000","last_clean":"2026-02-12T08:48:31.718592+0000","last_became_active":"2026-02-12T08:48:29.729376+0000","last_became_peered":"2026-02-12T08:48:29.729376+0000","last_unstale":"2026-02-12T08:48:31.718592+0000","last_undegraded":"2026-02-12T08:48:31.718592+0000","last_fullsized":"2026-02-12T08:48:31.718592+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T08:48:28.692133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T08:48:28.692133+0000","last_clean_scrub_stamp":"2026-02-12T08:48:28.692133+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":90,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-12T08:48:31.716611+0000","last_change":"2026-02-12T08:48:26.718087+0000","last_active":"2026-02-12T08:48:31.716611+0000","last_peered":"2026-02-12T08:48:31.716611+0000","last_clean":"2026-02-12T08:48:31.716611+0000","last_became_active":"2026-02-12T08:48:26.711088+0000","last_became_peered":"2026-02-12T08:48:26.711088+0000","last_unstale":"2026-02-12T08:48:31.716611+0000","last_undegraded":"2026-02-12T08:48:31.716611+0000","last_fullsized":"2026-02-12T08:48:31.716611+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T08:48:25.682332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T08:48:25.682332+0000","last_clean_scrub_stamp":"2026-02-12T08:48:25.682332+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160255967232,"available":160253366272,"internally_reserved":0,"allocated":2600960,"data_stored":2600960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53418655744,"available":53417480192,"internally_reserved":0,"allocated":1175552,"data_stored":1175552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705673,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":308,"kb_used_data":308,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52166348,"statfs":{"total":53418655744,"available":53418340352,"internally_reserved":0,"allocated":315392,"data_stored":315392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":10,"seq":42949672968,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1116,"kb_used_data":1116,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165540,"statfs":{"total":53418655744,"available":53417512960,"internally_reserved":0,"allocated":1142784,"data_stored":1142784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949672968,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1068,"kb_used_data":1068,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165588,"statfs":{"total":53418655744,"available":53417562112,"internally_reserved":0,"allocated":1093632,"data_stored":1093632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53418655744,"available":53417480192,"internally_reserved":0,"allocated":1175552,"data_stored":1175552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53418655744,"available":53417553920,"internally_reserved":0,"allocated":1101824,"data_stored":1101824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53418655744,"available":53417480192,"internally_reserved":0,"allocated":1175552,"data_stored":1175552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53418655744,"available":53418332160,"internally_reserved":0,"allocated":323584,"data_stored":323584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T08:48:33.399 INFO:tasks.ceph.ceph_manager.ceph:clean! 2026-02-12T08:48:33.399 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2026-02-12T08:48:33.399 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2026-02-12T08:48:33.400 DEBUG:teuthology.orchestra.run.trial187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-12T08:48:33.474 INFO:teuthology.orchestra.run.trial187.stderr:2026-02-12T08:48:33.470+0000 7fc083ccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T08:48:33.475 INFO:teuthology.orchestra.run.trial187.stderr:2026-02-12T08:48:33.472+0000 7fc083ccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T08:48:33.613 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:48:33.613 INFO:teuthology.orchestra.run.trial187.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2026-02-12T08:48:33.623 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2026-02-12T08:48:33.623 INFO:teuthology.run_tasks:Running task ssh_keys... 2026-02-12T08:48:33.693 DEBUG:teuthology.orchestra.run.trial187:> echo 'StrictHostKeyChecking no 2026-02-12T08:48:33.694 DEBUG:teuthology.orchestra.run.trial187:> ' > /home/ubuntu/.ssh/config && echo 'UserKnownHostsFile ' /dev/null >> /home/ubuntu/.ssh/config && chmod 600 /home/ubuntu/.ssh/config 2026-02-12T08:48:33.710 INFO:teuthology.task.ssh_keys:generated public key AAAAB3NzaC1yc2EAAAADAQABAAABAQC0UJi4nikJ9H0day3qmxMKS7Mtk1Q36QLpKWRlddsxjxzoyPPiV+riPS3GpKzp8h89Rn/c+iIFoP9BIs3pwLDJ4OMxPa2cho73ooqPfsXmijIhLF27VeDyGi03qBSix/IOgMRZqMzGI/02cCGMvv3TOZmFDSu5uiz19+JuFW2yMzBtrbzjbF4BN5zO5ilZsWJyGPXW/RBf4esJzLowVnyll5qwezRmvjqiKWUyF2V1NqpjpUIeEemT5DejfdeDXMfCc2kV26SghK1QF/WebqjsOvxX2wLl14Cm8gdnRYH90GlSBeSWn81LUBspMQ1/jawYuWxIKdHsopXSEsqVcT3F 2026-02-12T08:48:33.710 INFO:teuthology.task.ssh_keys:pushing keys to trial187.front.sepia.ceph.com for ubuntu 2026-02-12T08:48:33.710 DEBUG:teuthology.orchestra.run.trial187:> rm -f -- /home/ubuntu/.ssh/id_rsa 2026-02-12T08:48:33.766 DEBUG:teuthology.orchestra.run.trial187:> set -ex 2026-02-12T08:48:33.766 DEBUG:teuthology.orchestra.run.trial187:> dd of=/home/ubuntu/.ssh/id_rsa 2026-02-12T08:48:33.766 DEBUG:teuthology.orchestra.run.trial187:> chmod 0500 /home/ubuntu/.ssh/id_rsa 2026-02-12T08:48:33.823 DEBUG:teuthology.orchestra.run.trial187:> rm -f -- /home/ubuntu/.ssh/id_rsa.pub 2026-02-12T08:48:33.879 DEBUG:teuthology.orchestra.run.trial187:> set -ex 2026-02-12T08:48:33.879 DEBUG:teuthology.orchestra.run.trial187:> dd of=/home/ubuntu/.ssh/id_rsa.pub 2026-02-12T08:48:33.935 DEBUG:teuthology.orchestra.run.trial187:> cp -v -a /home/ubuntu/.ssh/authorized_keys /home/ubuntu/.ssh/authorized_keys_2026-02-12_08:48:33:934923 2026-02-12T08:48:33.991 INFO:teuthology.orchestra.run.trial187.stdout:'/home/ubuntu/.ssh/authorized_keys' -> '/home/ubuntu/.ssh/authorized_keys_2026-02-12_08:48:33:934923' 2026-02-12T08:48:33.992 DEBUG:teuthology.orchestra.run.trial187:> set -ex 2026-02-12T08:48:33.992 DEBUG:teuthology.orchestra.run.trial187:> dd of=/home/ubuntu/.ssh/authorized_keys conv=notrunc oflag=append 2026-02-12T08:48:34.048 INFO:teuthology.run_tasks:Running task cbt... 2026-02-12T08:48:34.074 INFO:tasks.cbt:cbt configuration is {'cluster': {'user': 'ubuntu', 'head': 'ubuntu@trial187.front.sepia.ceph.com', 'osds': ['ubuntu@trial187.front.sepia.ceph.com'], 'mons': ['ubuntu@trial187.front.sepia.ceph.com'], 'clients': ['ubuntu@trial187.front.sepia.ceph.com'], 'rgws': {'ubuntu@trial187.front.sepia.ceph.com': None}, 'osds_per_node': 3, 'rebuild_every_test': False, 'use_existing': True, 'is_teuthology': True, 'iterations': 1, 'tmp_dir': '/tmp/cbt', 'pool_profiles': {'replicated': {'pg_size': 256, 'pgp_size': 256, 'replication': 'replicated'}}, 'pid_dir': '/var/run/ceph'}, 'benchmarks': {'radosbench': {'concurrent_ops': 4, 'concurrent_procs': 2, 'op_size': [4194304], 'pool_monitoring_list': ['collectl'], 'pool_profile': 'replicated', 'run_monitoring_list': ['collectl'], 'time': 60, 'write_only': False}}, 'client_endpoints': None, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}} 2026-02-12T08:48:34.074 DEBUG:teuthology.orchestra.run.trial187:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/cbt 2026-02-12T08:48:34.108 DEBUG:teuthology.orchestra.run.trial187:> set -ex 2026-02-12T08:48:34.109 DEBUG:teuthology.orchestra.run.trial187:> dd of=/home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2026-02-12T08:48:34.160 DEBUG:teuthology.orchestra.run.trial187:> git clone --depth 1 -b master https://github.com/ceph/cbt.git /home/ubuntu/cephtest/cbt 2026-02-12T08:48:34.224 INFO:teuthology.orchestra.run.trial187.stderr:Cloning into '/home/ubuntu/cephtest/cbt'... 2026-02-12T08:48:34.488 INFO:tasks.cbt:Installing collectl 2026-02-12T08:48:34.489 DEBUG:teuthology.orchestra.run.trial187:> sudo mkdir collectl && cd collectl && sudo wget https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download -O collectl.tar.gz && sudo tar -xvf collectl.tar.gz && cd collectl-4.3.1 && sudo ./INSTALL 2026-02-12T08:48:34.564 INFO:teuthology.orchestra.run.trial187.stderr:--2026-02-12 08:48:34-- https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download 2026-02-12T08:48:34.625 INFO:teuthology.orchestra.run.trial187.stderr:Resolving sourceforge.net (sourceforge.net)... 104.18.13.149, 104.18.12.149, 2606:4700::6812:d95, ... 2026-02-12T08:48:34.635 INFO:teuthology.orchestra.run.trial187.stderr:Connecting to sourceforge.net (sourceforge.net)|104.18.13.149|:443... connected. 2026-02-12T08:48:34.834 INFO:teuthology.orchestra.run.trial187.stderr:HTTP request sent, awaiting response... 302 Found 2026-02-12T08:48:34.835 INFO:teuthology.orchestra.run.trial187.stderr:Location: https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABpjZPiNEvSkbMZb2h-6DwPRC-3oLuPwTXCT17Nr4CPtjRZ1vmoi82UM9bD541o6OJQIkozCnUI5wh_g90bWX0OJmytTA%3D%3D&use_mirror=psychz&r= [following] 2026-02-12T08:48:34.835 INFO:teuthology.orchestra.run.trial187.stderr:--2026-02-12 08:48:34-- https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABpjZPiNEvSkbMZb2h-6DwPRC-3oLuPwTXCT17Nr4CPtjRZ1vmoi82UM9bD541o6OJQIkozCnUI5wh_g90bWX0OJmytTA%3D%3D&use_mirror=psychz&r= 2026-02-12T08:48:34.846 INFO:teuthology.orchestra.run.trial187.stderr:Resolving downloads.sourceforge.net (downloads.sourceforge.net)... 104.18.12.149, 104.18.13.149, 2606:4700::6812:c95, ... 2026-02-12T08:48:34.855 INFO:teuthology.orchestra.run.trial187.stderr:Connecting to downloads.sourceforge.net (downloads.sourceforge.net)|104.18.12.149|:443... connected. 2026-02-12T08:48:35.066 INFO:teuthology.orchestra.run.trial187.stderr:HTTP request sent, awaiting response... 302 Found 2026-02-12T08:48:35.067 INFO:teuthology.orchestra.run.trial187.stderr:Location: https://psychz.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?viasf=1 [following] 2026-02-12T08:48:35.067 INFO:teuthology.orchestra.run.trial187.stderr:--2026-02-12 08:48:35-- https://psychz.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?viasf=1 2026-02-12T08:48:35.078 INFO:teuthology.orchestra.run.trial187.stderr:Resolving psychz.dl.sourceforge.net (psychz.dl.sourceforge.net)... 208.87.241.191 2026-02-12T08:48:35.146 INFO:teuthology.orchestra.run.trial187.stderr:Connecting to psychz.dl.sourceforge.net (psychz.dl.sourceforge.net)|208.87.241.191|:443... connected. 2026-02-12T08:48:35.466 INFO:teuthology.orchestra.run.trial187.stderr:HTTP request sent, awaiting response... 200 OK 2026-02-12T08:48:35.466 INFO:teuthology.orchestra.run.trial187.stderr:Length: 620693 (606K) [application/x-gzip] 2026-02-12T08:48:35.466 INFO:teuthology.orchestra.run.trial187.stderr:Saving to: ‘collectl.tar.gz’ 2026-02-12T08:48:35.467 INFO:teuthology.orchestra.run.trial187.stderr: 2026-02-12T08:48:35.534 INFO:teuthology.orchestra.run.trial187.stderr: 0K .......... .......... .......... .......... .......... 8% 736K 1s 2026-02-12T08:48:35.602 INFO:teuthology.orchestra.run.trial187.stderr: 50K .......... .......... .......... .......... .......... 16% 746K 1s 2026-02-12T08:48:35.602 INFO:teuthology.orchestra.run.trial187.stderr: 100K .......... .......... .......... .......... .......... 24% 63.4M 0s 2026-02-12T08:48:35.602 INFO:teuthology.orchestra.run.trial187.stderr: 150K .......... .......... .......... .......... .......... 32% 114M 0s 2026-02-12T08:48:35.672 INFO:teuthology.orchestra.run.trial187.stderr: 200K .......... .......... .......... .......... .......... 41% 748K 0s 2026-02-12T08:48:35.672 INFO:teuthology.orchestra.run.trial187.stderr: 250K .......... .......... .......... .......... .......... 49% 95.0M 0s 2026-02-12T08:48:35.672 INFO:teuthology.orchestra.run.trial187.stderr: 300K .......... .......... .......... .......... .......... 57% 246M 0s 2026-02-12T08:48:35.672 INFO:teuthology.orchestra.run.trial187.stderr: 350K .......... .......... .......... .......... .......... 65% 214M 0s 2026-02-12T08:48:35.739 INFO:teuthology.orchestra.run.trial187.stderr: 400K .......... .......... .......... .......... .......... 74% 754K 0s 2026-02-12T08:48:35.739 INFO:teuthology.orchestra.run.trial187.stderr: 450K .......... .......... .......... .......... .......... 82% 385M 0s 2026-02-12T08:48:35.739 INFO:teuthology.orchestra.run.trial187.stderr: 500K .......... .......... .......... .......... .......... 90% 121M 0s 2026-02-12T08:48:35.739 INFO:teuthology.orchestra.run.trial187.stderr: 550K .......... .......... .......... .......... .......... 98% 250M 0s 2026-02-12T08:48:35.740 INFO:teuthology.orchestra.run.trial187.stderr: 600K ...... 100% 11.4T=0.3s 2026-02-12T08:48:35.740 INFO:teuthology.orchestra.run.trial187.stderr: 2026-02-12T08:48:35.740 INFO:teuthology.orchestra.run.trial187.stderr:2026-02-12 08:48:35 (2.18 MB/s) - ‘collectl.tar.gz’ saved [620693/620693] 2026-02-12T08:48:35.740 INFO:teuthology.orchestra.run.trial187.stderr: 2026-02-12T08:48:35.758 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/ 2026-02-12T08:48:35.759 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/collectl.conf 2026-02-12T08:48:35.759 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/docs/ 2026-02-12T08:48:35.759 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/docs/SlowProc.html 2026-02-12T08:48:35.759 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/docs/BuddyInfo.html 2026-02-12T08:48:35.759 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/docs/WhySummary.html 2026-02-12T08:48:35.759 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/docs/Socket.html 2026-02-12T08:48:35.759 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/docs/colmux.html 2026-02-12T08:48:35.759 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/docs/Tutorial.html 2026-02-12T08:48:35.759 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/docs/Gexpr.html 2026-02-12T08:48:35.759 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/docs/Import.html 2026-02-12T08:48:35.759 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/docs/style.css 2026-02-12T08:48:35.759 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/docs/Ganglia3.jpg 2026-02-12T08:48:35.760 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/docs/Ganglia2.jpg 2026-02-12T08:48:35.760 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/docs/FileNaming.html 2026-02-12T08:48:35.760 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/docs/TheMath.html 2026-02-12T08:48:35.760 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/docs/Environmental.html 2026-02-12T08:48:35.760 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/docs/Process.html 2026-02-12T08:48:35.760 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/docs/OutputFiles.html 2026-02-12T08:48:35.760 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/docs/HiResTime.html 2026-02-12T08:48:35.760 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/docs/OutputFormats.html 2026-02-12T08:48:35.760 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/docs/slub.jpg 2026-02-12T08:48:35.760 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/docs/Lustre.html 2026-02-12T08:48:35.760 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/docs/Documentation.html 2026-02-12T08:48:35.761 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/docs/Inodes.html 2026-02-12T08:48:35.761 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/docs/Graphite.html 2026-02-12T08:48:35.761 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/docs/InputFiles.html 2026-02-12T08:48:35.761 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/docs/Export.html 2026-02-12T08:48:35.761 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/docs/Interrupts.html 2026-02-12T08:48:35.761 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/docs/Startup.html 2026-02-12T08:48:35.761 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/docs/Performance.html 2026-02-12T08:48:35.761 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/docs/Messages.html 2026-02-12T08:48:35.761 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/docs/Logging.html 2026-02-12T08:48:35.761 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/docs/Data-detail.html 2026-02-12T08:48:35.761 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/docs/CPUs.html 2026-02-12T08:48:35.761 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/docs/Matrix.html 2026-02-12T08:48:35.761 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/docs/Data-brief.html 2026-02-12T08:48:35.762 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/docs/SlabInfo.html 2026-02-12T08:48:35.762 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/docs/Misc.html 2026-02-12T08:48:35.762 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/docs/Examples.html 2026-02-12T08:48:35.762 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/docs/OpenStack.html 2026-02-12T08:48:35.762 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/docs/Plotfiles.html 2026-02-12T08:48:35.762 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/docs/ColmuxCPU.jpg 2026-02-12T08:48:35.762 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/docs/Tutorial-Lustre.html 2026-02-12T08:48:35.762 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/docs/Data-verbose.html 2026-02-12T08:48:35.762 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/docs/Data.html 2026-02-12T08:48:35.762 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/docs/Exceptions.html 2026-02-12T08:48:35.762 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/docs/RunningAsAService.html 2026-02-12T08:48:35.762 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/docs/FAQ-collectl.html 2026-02-12T08:48:35.762 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/docs/Ganglia.jpg 2026-02-12T08:48:35.763 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/docs/OperationalModes.html 2026-02-12T08:48:35.763 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/docs/Playback.html 2026-02-12T08:48:35.763 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/docs/Hello.html 2026-02-12T08:48:35.763 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/docs/Sockets.html 2026-02-12T08:48:35.763 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/docs/Network.html 2026-02-12T08:48:35.763 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/docs/Infiniband.html 2026-02-12T08:48:35.763 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/docs/Memory.html 2026-02-12T08:48:35.763 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/docs/ColmuxLustre.jpg 2026-02-12T08:48:35.763 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/docs/Disks.html 2026-02-12T08:48:35.763 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/docs/NfsInfo.html 2026-02-12T08:48:35.763 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/service/ 2026-02-12T08:48:35.763 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/service/collectl.service 2026-02-12T08:48:35.763 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/initd/ 2026-02-12T08:48:35.764 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/initd/collectl-suse 2026-02-12T08:48:35.764 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/initd/collectl-debian 2026-02-12T08:48:35.764 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/initd/collectl-generic 2026-02-12T08:48:35.764 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/initd/collectl 2026-02-12T08:48:35.764 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/misc.ph 2026-02-12T08:48:35.764 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/gexpr.ph 2026-02-12T08:48:35.764 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/statsd.ph 2026-02-12T08:48:35.764 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/envrules.std 2026-02-12T08:48:35.764 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/hello.ph 2026-02-12T08:48:35.764 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/vmsum.ph 2026-02-12T08:48:35.764 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/proctree.ph 2026-02-12T08:48:35.764 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/README-WINDOWS 2026-02-12T08:48:35.764 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/formatit.ph 2026-02-12T08:48:35.765 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/colmux 2026-02-12T08:48:35.765 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/INSTALL 2026-02-12T08:48:35.765 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/graphite.ph 2026-02-12T08:48:35.765 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/RELEASE-collectl 2026-02-12T08:48:35.765 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/vmstat.ph 2026-02-12T08:48:35.765 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/README 2026-02-12T08:48:35.765 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/man1/ 2026-02-12T08:48:35.765 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/man1/collectl.1 2026-02-12T08:48:35.765 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/man1/colmux.1 2026-02-12T08:48:35.765 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/lexpr.ph 2026-02-12T08:48:35.765 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/client.pl 2026-02-12T08:48:35.765 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/collectl 2026-02-12T08:48:35.765 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/COPYING 2026-02-12T08:48:35.766 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/vnet.ph 2026-02-12T08:48:35.766 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/GPL 2026-02-12T08:48:35.766 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/UNINSTALL 2026-02-12T08:48:35.766 INFO:teuthology.orchestra.run.trial187.stdout:collectl-4.3.1/ARTISTIC 2026-02-12T08:48:35.802 INFO:teuthology.orchestra.run.trial187.stderr:./INSTALL: line 90: chkconfig: command not found 2026-02-12T08:48:35.804 DEBUG:teuthology.orchestra.run.trial187:> sudo yum -y install librbd-devel pdsh pdsh-rcmd-ssh perf 2026-02-12T08:48:36.151 INFO:teuthology.orchestra.run.trial187.stdout:Last metadata expiration check: 0:00:58 ago on Thu 12 Feb 2026 08:47:38 AM UTC. 2026-02-12T08:48:36.235 INFO:teuthology.orchestra.run.trial187.stdout:Dependencies resolved. 2026-02-12T08:48:36.236 INFO:teuthology.orchestra.run.trial187.stdout:================================================================================ 2026-02-12T08:48:36.236 INFO:teuthology.orchestra.run.trial187.stdout: Package Arch Version Repository Size 2026-02-12T08:48:36.236 INFO:teuthology.orchestra.run.trial187.stdout:================================================================================ 2026-02-12T08:48:36.236 INFO:teuthology.orchestra.run.trial187.stdout:Installing: 2026-02-12T08:48:36.236 INFO:teuthology.orchestra.run.trial187.stdout: librbd-devel x86_64 2:20.3.0-5161.g2401a109.el9 ceph 24 k 2026-02-12T08:48:36.236 INFO:teuthology.orchestra.run.trial187.stdout: pdsh x86_64 2.34-7.el9 epel 118 k 2026-02-12T08:48:36.236 INFO:teuthology.orchestra.run.trial187.stdout: pdsh-rcmd-ssh x86_64 2.34-7.el9 epel 13 k 2026-02-12T08:48:36.236 INFO:teuthology.orchestra.run.trial187.stdout: perf x86_64 5.14.0-677.el9 appstream 3.2 M 2026-02-12T08:48:36.236 INFO:teuthology.orchestra.run.trial187.stdout:Installing dependencies: 2026-02-12T08:48:36.236 INFO:teuthology.orchestra.run.trial187.stdout: bzip2 x86_64 1.0.8-10.el9 baseos 56 k 2026-02-12T08:48:36.236 INFO:teuthology.orchestra.run.trial187.stdout: libradospp-devel x86_64 2:20.3.0-5161.g2401a109.el9 ceph 31 k 2026-02-12T08:48:36.236 INFO:teuthology.orchestra.run.trial187.stdout: libtraceevent x86_64 1.8.4-2.el9 baseos 290 k 2026-02-12T08:48:36.236 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:48:36.236 INFO:teuthology.orchestra.run.trial187.stdout:Transaction Summary 2026-02-12T08:48:36.236 INFO:teuthology.orchestra.run.trial187.stdout:================================================================================ 2026-02-12T08:48:36.236 INFO:teuthology.orchestra.run.trial187.stdout:Install 7 Packages 2026-02-12T08:48:36.236 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:48:36.236 INFO:teuthology.orchestra.run.trial187.stdout:Total download size: 3.7 M 2026-02-12T08:48:36.236 INFO:teuthology.orchestra.run.trial187.stdout:Installed size: 14 M 2026-02-12T08:48:36.237 INFO:teuthology.orchestra.run.trial187.stdout:Downloading Packages: 2026-02-12T08:48:36.579 INFO:teuthology.orchestra.run.trial187.stdout:(1/7): bzip2-1.0.8-10.el9.x86_64.rpm 504 kB/s | 56 kB 00:00 2026-02-12T08:48:36.665 INFO:teuthology.orchestra.run.trial187.stdout:(2/7): libtraceevent-1.8.4-2.el9.x86_64.rpm 3.3 MB/s | 290 kB 00:00 2026-02-12T08:48:36.780 INFO:teuthology.orchestra.run.trial187.stdout:(3/7): perf-5.14.0-677.el9.x86_64.rpm 27 MB/s | 3.2 MB 00:00 2026-02-12T08:48:36.811 INFO:teuthology.orchestra.run.trial187.stdout:(4/7): librbd-devel-20.3.0-5161.g2401a109.el9.x 69 kB/s | 24 kB 00:00 2026-02-12T08:48:36.815 INFO:teuthology.orchestra.run.trial187.stdout:(5/7): libradospp-devel-20.3.0-5161.g2401a109.e 90 kB/s | 31 kB 00:00 2026-02-12T08:48:37.007 INFO:teuthology.orchestra.run.trial187.stdout:(6/7): pdsh-rcmd-ssh-2.34-7.el9.x86_64.rpm 67 kB/s | 13 kB 00:00 2026-02-12T08:48:37.171 INFO:teuthology.orchestra.run.trial187.stdout:(7/7): pdsh-2.34-7.el9.x86_64.rpm 303 kB/s | 118 kB 00:00 2026-02-12T08:48:37.174 INFO:teuthology.orchestra.run.trial187.stdout:-------------------------------------------------------------------------------- 2026-02-12T08:48:37.174 INFO:teuthology.orchestra.run.trial187.stdout:Total 3.9 MB/s | 3.7 MB 00:00 2026-02-12T08:48:37.208 INFO:teuthology.orchestra.run.trial187.stdout:Running transaction check 2026-02-12T08:48:37.219 INFO:teuthology.orchestra.run.trial187.stdout:Transaction check succeeded. 2026-02-12T08:48:37.219 INFO:teuthology.orchestra.run.trial187.stdout:Running transaction test 2026-02-12T08:48:37.278 INFO:teuthology.orchestra.run.trial187.stdout:Transaction test succeeded. 2026-02-12T08:48:37.279 INFO:teuthology.orchestra.run.trial187.stdout:Running transaction 2026-02-12T08:48:37.357 INFO:teuthology.orchestra.run.trial187.stdout: Preparing : 1/1 2026-02-12T08:48:37.362 INFO:teuthology.orchestra.run.trial187.stdout: Installing : pdsh-rcmd-ssh-2.34-7.el9.x86_64 1/7 2026-02-12T08:48:37.402 INFO:teuthology.orchestra.run.trial187.stdout: Installing : pdsh-2.34-7.el9.x86_64 2/7 2026-02-12T08:48:37.408 INFO:teuthology.orchestra.run.trial187.stdout: Installing : libtraceevent-1.8.4-2.el9.x86_64 3/7 2026-02-12T08:48:37.412 INFO:teuthology.orchestra.run.trial187.stdout: Installing : bzip2-1.0.8-10.el9.x86_64 4/7 2026-02-12T08:48:37.414 INFO:teuthology.orchestra.run.trial187.stdout: Installing : libradospp-devel-2:20.3.0-5161.g2401a109.el9.x86_64 5/7 2026-02-12T08:48:37.579 INFO:teuthology.orchestra.run.trial187.stdout: Installing : librbd-devel-2:20.3.0-5161.g2401a109.el9.x86_64 6/7 2026-02-12T08:48:37.600 INFO:teuthology.orchestra.run.trial187.stdout: Installing : perf-5.14.0-677.el9.x86_64 7/7 2026-02-12T08:48:37.844 INFO:teuthology.orchestra.run.trial187.stdout: Running scriptlet: perf-5.14.0-677.el9.x86_64 7/7 2026-02-12T08:48:37.844 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : libradospp-devel-2:20.3.0-5161.g2401a109.el9.x86_64 1/7 2026-02-12T08:48:37.844 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : librbd-devel-2:20.3.0-5161.g2401a109.el9.x86_64 2/7 2026-02-12T08:48:37.844 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : bzip2-1.0.8-10.el9.x86_64 3/7 2026-02-12T08:48:37.844 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : libtraceevent-1.8.4-2.el9.x86_64 4/7 2026-02-12T08:48:37.844 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : perf-5.14.0-677.el9.x86_64 5/7 2026-02-12T08:48:37.845 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : pdsh-2.34-7.el9.x86_64 6/7 2026-02-12T08:48:37.903 INFO:teuthology.orchestra.run.trial187.stdout: Verifying : pdsh-rcmd-ssh-2.34-7.el9.x86_64 7/7 2026-02-12T08:48:37.903 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:48:37.903 INFO:teuthology.orchestra.run.trial187.stdout:Installed: 2026-02-12T08:48:37.904 INFO:teuthology.orchestra.run.trial187.stdout: bzip2-1.0.8-10.el9.x86_64 2026-02-12T08:48:37.904 INFO:teuthology.orchestra.run.trial187.stdout: libradospp-devel-2:20.3.0-5161.g2401a109.el9.x86_64 2026-02-12T08:48:37.904 INFO:teuthology.orchestra.run.trial187.stdout: librbd-devel-2:20.3.0-5161.g2401a109.el9.x86_64 2026-02-12T08:48:37.904 INFO:teuthology.orchestra.run.trial187.stdout: libtraceevent-1.8.4-2.el9.x86_64 2026-02-12T08:48:37.904 INFO:teuthology.orchestra.run.trial187.stdout: pdsh-2.34-7.el9.x86_64 2026-02-12T08:48:37.904 INFO:teuthology.orchestra.run.trial187.stdout: pdsh-rcmd-ssh-2.34-7.el9.x86_64 2026-02-12T08:48:37.904 INFO:teuthology.orchestra.run.trial187.stdout: perf-5.14.0-677.el9.x86_64 2026-02-12T08:48:37.904 INFO:teuthology.orchestra.run.trial187.stdout: 2026-02-12T08:48:37.904 INFO:teuthology.orchestra.run.trial187.stdout:Complete! 2026-02-12T08:48:37.934 DEBUG:teuthology.orchestra.run.trial187:> python3 -m pip install -r /home/ubuntu/cephtest/cbt/requirements.txt 2026-02-12T08:48:38.075 INFO:teuthology.orchestra.run.trial187.stdout:Defaulting to user installation because normal site-packages is not writeable 2026-02-12T08:48:38.085 INFO:teuthology.orchestra.run.trial187.stdout:Requirement already satisfied: pyyaml in /usr/lib64/python3.9/site-packages (from -r /home/ubuntu/cephtest/cbt/requirements.txt (line 1)) (5.4.1) 2026-02-12T08:48:38.086 INFO:teuthology.orchestra.run.trial187.stdout:Requirement already satisfied: lxml in /usr/lib64/python3.9/site-packages (from -r /home/ubuntu/cephtest/cbt/requirements.txt (line 2)) (4.6.5) 2026-02-12T08:48:38.407 INFO:teuthology.orchestra.run.trial187.stdout:Collecting matplotlib 2026-02-12T08:48:38.437 INFO:teuthology.orchestra.run.trial187.stdout: Downloading matplotlib-3.9.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (8.3 MB) 2026-02-12T08:48:38.868 INFO:teuthology.orchestra.run.trial187.stdout:Collecting pillow>=8 2026-02-12T08:48:38.875 INFO:teuthology.orchestra.run.trial187.stdout: Downloading pillow-11.3.0-cp39-cp39-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (6.6 MB) 2026-02-12T08:48:38.916 INFO:teuthology.orchestra.run.trial187.stdout:Collecting cycler>=0.10 2026-02-12T08:48:38.920 INFO:teuthology.orchestra.run.trial187.stdout: Downloading cycler-0.12.1-py3-none-any.whl (8.3 kB) 2026-02-12T08:48:38.921 INFO:teuthology.orchestra.run.trial187.stdout:Requirement already satisfied: pyparsing>=2.3.1 in /usr/lib/python3.9/site-packages (from matplotlib->-r /home/ubuntu/cephtest/cbt/requirements.txt (line 3)) (2.4.7) 2026-02-12T08:48:38.921 INFO:teuthology.orchestra.run.trial187.stdout:Requirement already satisfied: python-dateutil>=2.7 in /usr/lib/python3.9/site-packages (from matplotlib->-r /home/ubuntu/cephtest/cbt/requirements.txt (line 3)) (2.9.0.post0) 2026-02-12T08:48:39.048 INFO:teuthology.orchestra.run.trial187.stdout:Collecting kiwisolver>=1.3.1 2026-02-12T08:48:39.055 INFO:teuthology.orchestra.run.trial187.stdout: Downloading kiwisolver-1.4.7-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.6 MB) 2026-02-12T08:48:39.097 INFO:teuthology.orchestra.run.trial187.stdout:Collecting importlib-resources>=3.2.0 2026-02-12T08:48:39.101 INFO:teuthology.orchestra.run.trial187.stdout: Downloading importlib_resources-6.5.2-py3-none-any.whl (37 kB) 2026-02-12T08:48:39.102 INFO:teuthology.orchestra.run.trial187.stdout:Requirement already satisfied: packaging>=20.0 in /usr/lib/python3.9/site-packages (from matplotlib->-r /home/ubuntu/cephtest/cbt/requirements.txt (line 3)) (20.9) 2026-02-12T08:48:39.241 INFO:teuthology.orchestra.run.trial187.stdout:Collecting contourpy>=1.0.1 2026-02-12T08:48:39.244 INFO:teuthology.orchestra.run.trial187.stdout: Downloading contourpy-1.3.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (321 kB) 2026-02-12T08:48:39.582 INFO:teuthology.orchestra.run.trial187.stdout:Collecting fonttools>=4.22.0 2026-02-12T08:48:39.587 INFO:teuthology.orchestra.run.trial187.stdout: Downloading fonttools-4.60.2-cp39-cp39-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (4.8 MB) 2026-02-12T08:48:39.611 INFO:teuthology.orchestra.run.trial187.stdout:Requirement already satisfied: numpy>=1.23 in /usr/lib64/python3.9/site-packages (from matplotlib->-r /home/ubuntu/cephtest/cbt/requirements.txt (line 3)) (1.23.5) 2026-02-12T08:48:39.662 INFO:teuthology.orchestra.run.trial187.stdout:Collecting zipp>=3.1.0 2026-02-12T08:48:39.666 INFO:teuthology.orchestra.run.trial187.stdout: Downloading zipp-3.23.0-py3-none-any.whl (10 kB) 2026-02-12T08:48:39.682 INFO:teuthology.orchestra.run.trial187.stdout:Requirement already satisfied: six>=1.5 in /usr/lib/python3.9/site-packages (from python-dateutil>=2.7->matplotlib->-r /home/ubuntu/cephtest/cbt/requirements.txt (line 3)) (1.15.0) 2026-02-12T08:48:39.812 INFO:teuthology.orchestra.run.trial187.stdout:Installing collected packages: zipp, pillow, kiwisolver, importlib-resources, fonttools, cycler, contourpy, matplotlib 2026-02-12T08:48:40.856 INFO:teuthology.orchestra.run.trial187.stdout:Successfully installed contourpy-1.3.0 cycler-0.12.1 fonttools-4.60.2 importlib-resources-6.5.2 kiwisolver-1.4.7 matplotlib-3.9.4 pillow-11.3.0 zipp-3.23.0 2026-02-12T08:48:40.964 INFO:tasks.cbt:benchmark: radosbench 2026-02-12T08:48:40.965 DEBUG:teuthology.orchestra.run.trial187:> sudo /sbin/sysctl -q -w kernel.perf_event_paranoid=0 2026-02-12T08:48:40.992 DEBUG:teuthology.orchestra.run.trial187:> /home/ubuntu/cephtest/cbt/cbt.py -a /home/ubuntu/cephtest/archive/cbt /home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2026-02-12T08:48:41.107 INFO:teuthology.orchestra.run.trial187.stderr:08:48:41 - DEBUG - cbt - Settings.cluster: 2026-02-12T08:48:41.107 INFO:teuthology.orchestra.run.trial187.stderr: {'archive_dir': '/home/ubuntu/cephtest/archive/cbt', 2026-02-12T08:48:41.107 INFO:teuthology.orchestra.run.trial187.stderr: 'clients': ['ubuntu@trial187.front.sepia.ceph.com'], 2026-02-12T08:48:41.107 INFO:teuthology.orchestra.run.trial187.stderr: 'conf_file': '/etc/ceph/ceph.conf', 2026-02-12T08:48:41.107 INFO:teuthology.orchestra.run.trial187.stderr: 'head': 'ubuntu@trial187.front.sepia.ceph.com', 2026-02-12T08:48:41.107 INFO:teuthology.orchestra.run.trial187.stderr: 'is_teuthology': True, 2026-02-12T08:48:41.107 INFO:teuthology.orchestra.run.trial187.stderr: 'iterations': 1, 2026-02-12T08:48:41.107 INFO:teuthology.orchestra.run.trial187.stderr: 'mons': ['ubuntu@trial187.front.sepia.ceph.com'], 2026-02-12T08:48:41.107 INFO:teuthology.orchestra.run.trial187.stderr: 'osds': ['ubuntu@trial187.front.sepia.ceph.com'], 2026-02-12T08:48:41.108 INFO:teuthology.orchestra.run.trial187.stderr: 'osds_per_node': 3, 2026-02-12T08:48:41.108 INFO:teuthology.orchestra.run.trial187.stderr: 'pid_dir': '/var/run/ceph', 2026-02-12T08:48:41.108 INFO:teuthology.orchestra.run.trial187.stderr: 'pool_profiles': {'replicated': {'pg_size': 256, 2026-02-12T08:48:41.108 INFO:teuthology.orchestra.run.trial187.stderr: 'pgp_size': 256, 2026-02-12T08:48:41.108 INFO:teuthology.orchestra.run.trial187.stderr: 'replication': 'replicated'}}, 2026-02-12T08:48:41.108 INFO:teuthology.orchestra.run.trial187.stderr: 'rebuild_every_test': False, 2026-02-12T08:48:41.108 INFO:teuthology.orchestra.run.trial187.stderr: 'rgws': {'ubuntu@trial187.front.sepia.ceph.com': None}, 2026-02-12T08:48:41.108 INFO:teuthology.orchestra.run.trial187.stderr: 'tmp_dir': '/tmp/cbt', 2026-02-12T08:48:41.108 INFO:teuthology.orchestra.run.trial187.stderr: 'use_existing': True, 2026-02-12T08:48:41.108 INFO:teuthology.orchestra.run.trial187.stderr: 'user': 'ubuntu'} 2026-02-12T08:48:41.108 INFO:teuthology.orchestra.run.trial187.stderr:08:48:41 - INFO - cbt - Results dir: /home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d 2026-02-12T08:48:41.108 INFO:teuthology.orchestra.run.trial187.stderr:08:48:41 - INFO - cbt - Pausing for 60s for idle monitoring. 2026-02-12T08:48:41.109 INFO:teuthology.orchestra.run.trial187.stderr:08:48:41 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial187.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/idle_monitoring/collectl 2026-02-12T08:48:41.284 INFO:teuthology.orchestra.run.trial187.stderr:08:48:41 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial187.front.sepia.ceph.com collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/idle_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2026-02-12T08:48:41.286 INFO:teuthology.orchestra.run.trial187.stderr:08:48:41 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial187.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/idle_monitoring/perf 2026-02-12T08:48:41.449 INFO:teuthology.orchestra.run.trial187.stderr:08:48:41 - DEBUG - cbt - PerfMonitoring: remote_node 2026-02-12T08:48:41.449 INFO:teuthology.orchestra.run.trial187.stderr:08:48:41 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial187.front.sepia.ceph.com for pid in `cat /var/run/ceph/crimson-osd.*.pid`; do perf stat -p ${pid} -o /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/idle_monitoring/perf/perf_stat.${pid} & done 2026-02-12T08:49:41.490 INFO:teuthology.orchestra.run.trial187.stderr:08:49:41 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial187.front.sepia.ceph.com pkill -SIGINT -f collectl 2026-02-12T08:49:41.676 INFO:teuthology.orchestra.run.trial187.stderr:08:49:41 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial187.front.sepia.ceph.com sudo pkill -SIGINT -f perf\ 2026-02-12T08:49:41.868 INFO:teuthology.orchestra.run.trial187.stderr:08:49:41 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial187.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/idle_monitoring/perf/perf.data 2026-02-12T08:49:41.869 INFO:teuthology.orchestra.run.trial187.stderr:08:49:41 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial187.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/idle_monitoring/perf/perf_stat.* 2026-02-12T08:49:41.872 INFO:teuthology.orchestra.run.trial187.stderr:08:49:41 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial187.front.sepia.ceph.com sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/* 2026-02-12T08:49:42.048 INFO:teuthology.orchestra.run.trial187.stderr:08:49:42 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=rpdcp -f 10 -R ssh -w ubuntu@trial187.front.sepia.ceph.com -r /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/* /home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d 2026-02-12T08:49:42.211 INFO:teuthology.orchestra.run.trial187.stderr:08:49:42 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial187.front.sepia.ceph.com sudo killall -9 rados 2026-02-12T08:49:42.388 INFO:teuthology.orchestra.run.trial187.stderr:08:49:42 - INFO - cbt - Results dir: /home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d 2026-02-12T08:49:42.388 INFO:teuthology.orchestra.run.trial187.stderr:08:49:42 - INFO - cbt - Skipping existing test in /home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d. 2026-02-12T08:49:42.388 INFO:teuthology.orchestra.run.trial187.stderr:08:49:42 - INFO - cbt - Running benchmark /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004 2026-02-12T08:49:42.389 INFO:teuthology.orchestra.run.trial187.stderr:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d 2026-02-12T08:49:42.389 INFO:teuthology.orchestra.run.trial187.stderr:{'concurrent_ops': 4, 'concurrent_procs': 2, 'pool_profile': 'replicated', 'time': 60, 'write_only': False, 'op_size': 4194304, 'pool_monitoring_list': 'collectl', 'run_monitoring_list': 'collectl', 'benchmark': 'radosbench', 'iteration': 0} == iteration 0 == 2026-02-12T08:49:42.389 INFO:teuthology.orchestra.run.trial187.stderr:08:49:42 - INFO - cbt - Setting OSD Read Ahead to: 0 2026-02-12T08:49:42.389 INFO:teuthology.orchestra.run.trial187.stderr:08:49:42 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial187.front.sepia.ceph.com find /dev/disk/by-partlabel/osd-device-*data -exec readlink {} \; | cut -d"/" -f 3 | sed "s/[0-9]$//" | xargs -I{} sudo sh -c "echo 0 > /sys/block/'{}'/queue/read_ahead_kb" 2026-02-12T08:49:42.389 INFO:teuthology.orchestra.run.trial187.stderr:08:49:42 - DEBUG - cbt - Cleaning existing temporary run directory: /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004 2026-02-12T08:49:42.390 INFO:teuthology.orchestra.run.trial187.stderr:08:49:42 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial187.front.sepia.ceph.com sudo rm -rf /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004 2026-02-12T08:49:42.564 INFO:teuthology.orchestra.run.trial187.stderr:08:49:42 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial187.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/pool_monitoring/collectl 2026-02-12T08:49:42.727 INFO:teuthology.orchestra.run.trial187.stderr:08:49:42 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial187.front.sepia.ceph.com collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/pool_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2026-02-12T08:49:42.729 INFO:teuthology.orchestra.run.trial187.stderr:08:49:42 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial187.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/pool_monitoring/perf 2026-02-12T08:49:42.899 INFO:teuthology.orchestra.run.trial187.stderr:08:49:42 - DEBUG - cbt - PerfMonitoring: remote_node 2026-02-12T08:49:42.899 INFO:teuthology.orchestra.run.trial187.stderr:08:49:42 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial187.front.sepia.ceph.com for pid in `cat /var/run/ceph/crimson-osd.*.pid`; do perf stat -p ${pid} -o /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/pool_monitoring/perf/perf_stat.${pid} & done 2026-02-12T08:49:42.901 INFO:teuthology.orchestra.run.trial187.stderr:08:49:42 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial187.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool delete rados-bench-cbt rados-bench-cbt --yes-i-really-really-mean-it 2026-02-12T08:49:43.242 INFO:teuthology.orchestra.run.trial187.stderr:08:49:43 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial187.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool create rados-bench-cbt 256 256 2026-02-12T08:49:44.030 INFO:teuthology.orchestra.run.trial187.stderr:08:49:44 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial187.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool application enable rados-bench-cbt radosbench 2026-02-12T08:49:45.032 INFO:teuthology.orchestra.run.trial187.stderr:08:49:45 - INFO - cbt - Checking Health after pool creation. 2026-02-12T08:49:50.037 INFO:teuthology.orchestra.run.trial187.stderr:08:49:50 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial187.front.sepia.ceph.com /usr/bin/ceph -c /etc/ceph/ceph.conf health 2026-02-12T08:49:50.399 INFO:teuthology.orchestra.run.trial187.stderr:08:49:50 - INFO - cbt - Final Pool Health Check. 2026-02-12T08:49:55.405 INFO:teuthology.orchestra.run.trial187.stderr:08:49:55 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial187.front.sepia.ceph.com /usr/bin/ceph -c /etc/ceph/ceph.conf health 2026-02-12T08:49:55.756 INFO:teuthology.orchestra.run.trial187.stderr:08:49:55 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial187.front.sepia.ceph.com pkill -SIGINT -f collectl 2026-02-12T08:49:55.936 INFO:teuthology.orchestra.run.trial187.stderr:08:49:55 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial187.front.sepia.ceph.com sudo pkill -SIGINT -f perf\ 2026-02-12T08:49:56.129 INFO:teuthology.orchestra.run.trial187.stderr:08:49:56 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial187.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/pool_monitoring/perf/perf.data 2026-02-12T08:49:56.130 INFO:teuthology.orchestra.run.trial187.stderr:08:49:56 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial187.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/pool_monitoring/perf/perf_stat.* 2026-02-12T08:49:56.132 INFO:teuthology.orchestra.run.trial187.stderr:08:49:56 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial187.front.sepia.ceph.com sync 2026-02-12T08:49:56.330 INFO:teuthology.orchestra.run.trial187.stderr:08:49:56 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial187.front.sepia.ceph.com echo 3 | sudo tee /proc/sys/vm/drop_caches 2026-02-12T08:49:57.346 INFO:teuthology.orchestra.run.trial187.stderr:08:49:57 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial187.front.sepia.ceph.com /usr/bin/rados -c /etc/ceph/ceph.conf -v 2026-02-12T08:49:57.556 INFO:teuthology.orchestra.run.trial187.stderr:08:49:57 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial187.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/write 2026-02-12T08:49:57.720 INFO:teuthology.orchestra.run.trial187.stderr:08:49:57 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial187.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf daemon osd.0 config show > /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/write/ceph_settings.out 2026-02-12T08:49:58.006 INFO:teuthology.orchestra.run.trial187.stderr:08:49:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial187.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/write/collectl 2026-02-12T08:49:58.171 INFO:teuthology.orchestra.run.trial187.stderr:08:49:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial187.front.sepia.ceph.com collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/write/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2026-02-12T08:49:58.172 INFO:teuthology.orchestra.run.trial187.stderr:08:49:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial187.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/write/perf 2026-02-12T08:49:58.337 INFO:teuthology.orchestra.run.trial187.stderr:08:49:58 - DEBUG - cbt - PerfMonitoring: remote_node 2026-02-12T08:49:58.337 INFO:teuthology.orchestra.run.trial187.stderr:08:49:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial187.front.sepia.ceph.com for pid in `cat /var/run/ceph/crimson-osd.*.pid`; do perf stat -p ${pid} -o /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/write/perf/perf_stat.${pid} & done 2026-02-12T08:49:58.338 INFO:teuthology.orchestra.run.trial187.stderr:08:49:58 - INFO - cbt - Running radosbench write test. 2026-02-12T08:49:58.339 INFO:teuthology.orchestra.run.trial187.stderr:08:49:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial187.front.sepia.ceph.com /usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench -b 4194304 60 write --concurrent-ios 4 --run-name `hostname -f`-0 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/write/objecter.0.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/write/output.0 2026-02-12T08:49:58.340 INFO:teuthology.orchestra.run.trial187.stderr:08:49:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial187.front.sepia.ceph.com /usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench -b 4194304 60 write --concurrent-ios 4 --run-name `hostname -f`-1 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/write/objecter.1.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/write/output.1 2026-02-12T08:50:59.568 INFO:teuthology.orchestra.run.trial187.stderr:08:50:59 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial187.front.sepia.ceph.com pkill -SIGINT -f collectl 2026-02-12T08:50:59.754 INFO:teuthology.orchestra.run.trial187.stderr:08:50:59 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial187.front.sepia.ceph.com sudo pkill -SIGINT -f perf\ 2026-02-12T08:50:59.939 INFO:teuthology.orchestra.run.trial187.stderr:08:50:59 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial187.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/write/perf/perf.data 2026-02-12T08:50:59.941 INFO:teuthology.orchestra.run.trial187.stderr:08:50:59 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial187.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/write/perf/perf_stat.* 2026-02-12T08:50:59.942 INFO:teuthology.orchestra.run.trial187.stderr:08:50:59 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial187.front.sepia.ceph.com find /var/run/ceph/ceph-osd*.asok -maxdepth 1 -exec sudo /usr/bin/ceph --admin-daemon {} dump_historic_ops \; > /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/write/historic_ops.out 2026-02-12T08:51:00.298 INFO:teuthology.orchestra.run.trial187.stderr:08:51:00 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial187.front.sepia.ceph.com sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/write/* 2026-02-12T08:51:00.473 INFO:teuthology.orchestra.run.trial187.stderr:08:51:00 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=rpdcp -f 10 -R ssh -w ubuntu@trial187.front.sepia.ceph.com -r /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/write/* /home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d/write 2026-02-12T08:51:00.637 INFO:teuthology.orchestra.run.trial187.stderr:08:51:00 - INFO - cbt - Convert results to json format. 2026-02-12T08:51:00.638 INFO:teuthology.orchestra.run.trial187.stderr:08:51:00 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial187.front.sepia.ceph.com sync 2026-02-12T08:51:00.855 INFO:teuthology.orchestra.run.trial187.stderr:08:51:00 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial187.front.sepia.ceph.com echo 3 | sudo tee /proc/sys/vm/drop_caches 2026-02-12T08:51:01.174 INFO:teuthology.orchestra.run.trial187.stderr:08:51:01 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial187.front.sepia.ceph.com /usr/bin/rados -c /etc/ceph/ceph.conf -v 2026-02-12T08:51:01.381 INFO:teuthology.orchestra.run.trial187.stderr:08:51:01 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial187.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/seq 2026-02-12T08:51:01.549 INFO:teuthology.orchestra.run.trial187.stderr:08:51:01 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial187.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf daemon osd.0 config show > /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/seq/ceph_settings.out 2026-02-12T08:51:01.835 INFO:teuthology.orchestra.run.trial187.stderr:08:51:01 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial187.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/seq/collectl 2026-02-12T08:51:01.995 INFO:teuthology.orchestra.run.trial187.stderr:08:51:01 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial187.front.sepia.ceph.com collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/seq/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2026-02-12T08:51:01.997 INFO:teuthology.orchestra.run.trial187.stderr:08:51:01 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial187.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/seq/perf 2026-02-12T08:51:02.161 INFO:teuthology.orchestra.run.trial187.stderr:08:51:02 - DEBUG - cbt - PerfMonitoring: remote_node 2026-02-12T08:51:02.161 INFO:teuthology.orchestra.run.trial187.stderr:08:51:02 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial187.front.sepia.ceph.com for pid in `cat /var/run/ceph/crimson-osd.*.pid`; do perf stat -p ${pid} -o /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/seq/perf/perf_stat.${pid} & done 2026-02-12T08:51:02.162 INFO:teuthology.orchestra.run.trial187.stderr:08:51:02 - INFO - cbt - Running radosbench seq test. 2026-02-12T08:51:02.163 INFO:teuthology.orchestra.run.trial187.stderr:08:51:02 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial187.front.sepia.ceph.com /usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench 60 seq --concurrent-ios 4 --run-name `hostname -f`-0 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/seq/objecter.0.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/seq/output.0 2026-02-12T08:51:02.164 INFO:teuthology.orchestra.run.trial187.stderr:08:51:02 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial187.front.sepia.ceph.com /usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench 60 seq --concurrent-ios 4 --run-name `hostname -f`-1 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/seq/objecter.1.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/seq/output.1 2026-02-12T08:51:17.390 INFO:teuthology.orchestra.run.trial187.stderr:08:51:17 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial187.front.sepia.ceph.com pkill -SIGINT -f collectl 2026-02-12T08:51:17.569 INFO:teuthology.orchestra.run.trial187.stderr:08:51:17 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial187.front.sepia.ceph.com sudo pkill -SIGINT -f perf\ 2026-02-12T08:51:17.759 INFO:teuthology.orchestra.run.trial187.stderr:08:51:17 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial187.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/seq/perf/perf.data 2026-02-12T08:51:17.761 INFO:teuthology.orchestra.run.trial187.stderr:08:51:17 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial187.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/seq/perf/perf_stat.* 2026-02-12T08:51:17.763 INFO:teuthology.orchestra.run.trial187.stderr:08:51:17 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial187.front.sepia.ceph.com find /var/run/ceph/ceph-osd*.asok -maxdepth 1 -exec sudo /usr/bin/ceph --admin-daemon {} dump_historic_ops \; > /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/seq/historic_ops.out 2026-02-12T08:51:18.122 INFO:teuthology.orchestra.run.trial187.stderr:08:51:18 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial187.front.sepia.ceph.com sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/seq/* 2026-02-12T08:51:18.289 INFO:teuthology.orchestra.run.trial187.stderr:08:51:18 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=rpdcp -f 10 -R ssh -w ubuntu@trial187.front.sepia.ceph.com -r /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/seq/* /home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d/seq 2026-02-12T08:51:18.453 INFO:teuthology.orchestra.run.trial187.stderr:08:51:18 - INFO - cbt - Convert results to json format. 2026-02-12T08:51:18.464 DEBUG:teuthology.run_tasks:Unwinding manager cbt 2026-02-12T08:51:18.479 DEBUG:teuthology.orchestra.run.trial187:> rm --one-file-system -rf -- /home/ubuntu/cephtest/cbt 2026-02-12T08:51:18.506 INFO:tasks.cbt_performance:Collecting CBT performance data config 2026-02-12T08:51:18.507 INFO:tasks.cbt_performance:reading cbt preferences 2026-02-12T08:51:18.507 DEBUG:teuthology.orchestra.run.trial187:> find /home/ubuntu/cephtest -name 'json_output.*' 2026-02-12T08:51:18.565 INFO:teuthology.orchestra.run.trial187.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d/write/json_output.1.trial187.front.sepia.ceph.com 2026-02-12T08:51:18.565 INFO:teuthology.orchestra.run.trial187.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d/write/json_output.0.trial187.front.sepia.ceph.com 2026-02-12T08:51:18.565 INFO:teuthology.orchestra.run.trial187.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d/seq/json_output.1.trial187.front.sepia.ceph.com 2026-02-12T08:51:18.565 INFO:teuthology.orchestra.run.trial187.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d/seq/json_output.0.trial187.front.sepia.ceph.com 2026-02-12T08:51:18.566 DEBUG:teuthology.orchestra.run.trial187:> set -ex 2026-02-12T08:51:18.567 DEBUG:teuthology.orchestra.run.trial187:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d/write/json_output.1.trial187.front.sepia.ceph.com of=/dev/stdout 2026-02-12T08:51:18.622 INFO:tasks.cbt_performance:reading total cpu cycles 2026-02-12T08:51:18.623 DEBUG:teuthology.orchestra.run.trial187:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d/write -name 'perf_stat.*' 2026-02-12T08:51:18.679 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2026-02-12T08:51:18.680 INFO:tasks.cbt_performance:total cpu cycles: 0 2026-02-12T08:51:18.680 DEBUG:teuthology.orchestra.run.trial187:> set -ex 2026-02-12T08:51:18.680 DEBUG:teuthology.orchestra.run.trial187:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d/write/json_output.0.trial187.front.sepia.ceph.com of=/dev/stdout 2026-02-12T08:51:18.736 INFO:tasks.cbt_performance:reading total cpu cycles 2026-02-12T08:51:18.737 DEBUG:teuthology.orchestra.run.trial187:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d/write -name 'perf_stat.*' 2026-02-12T08:51:18.792 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2026-02-12T08:51:18.792 INFO:tasks.cbt_performance:total cpu cycles: 0 2026-02-12T08:51:18.793 DEBUG:teuthology.orchestra.run.trial187:> set -ex 2026-02-12T08:51:18.793 DEBUG:teuthology.orchestra.run.trial187:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d/seq/json_output.1.trial187.front.sepia.ceph.com of=/dev/stdout 2026-02-12T08:51:18.848 INFO:tasks.cbt_performance:reading total cpu cycles 2026-02-12T08:51:18.849 DEBUG:teuthology.orchestra.run.trial187:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d/seq -name 'perf_stat.*' 2026-02-12T08:51:18.904 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2026-02-12T08:51:18.904 INFO:tasks.cbt_performance:total cpu cycles: 0 2026-02-12T08:51:18.905 DEBUG:teuthology.orchestra.run.trial187:> set -ex 2026-02-12T08:51:18.905 DEBUG:teuthology.orchestra.run.trial187:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d/seq/json_output.0.trial187.front.sepia.ceph.com of=/dev/stdout 2026-02-12T08:51:18.960 INFO:tasks.cbt_performance:reading total cpu cycles 2026-02-12T08:51:18.961 DEBUG:teuthology.orchestra.run.trial187:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d/seq -name 'perf_stat.*' 2026-02-12T08:51:19.017 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2026-02-12T08:51:19.017 INFO:tasks.cbt_performance:total cpu cycles: 0 2026-02-12T08:51:19.022 WARNING:tasks.cbt_performance:Could not send performance data to http://mira118.front.sepia.ceph.com:4000/cbt_performance: HTTPConnectionPool(host='mira118.front.sepia.ceph.com', port=4000): Max retries exceeded with url: /cbt_performance (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -2] Name or service not known')) 2026-02-12T08:51:19.022 WARNING:tasks.cbt_performance:Continuing without performance data collection 2026-02-12T08:51:19.022 DEBUG:teuthology.run_tasks:Unwinding manager ssh_keys 2026-02-12T08:51:19.037 INFO:teuthology.task.ssh_keys:Cleaning up SSH keys 2026-02-12T08:51:19.038 INFO:teuthology.task.ssh_keys:cleaning up keys added for testing 2026-02-12T08:51:19.038 INFO:teuthology.task.ssh_keys: cleaning up keys for user ubuntu on trial187.front.sepia.ceph.com 2026-02-12T08:51:19.038 DEBUG:teuthology.orchestra.run.trial187:> rm -- /home/ubuntu/.ssh/id_rsa 2026-02-12T08:51:19.073 DEBUG:teuthology.orchestra.run.trial187:> rm -- /home/ubuntu/.ssh/id_rsa.pub 2026-02-12T08:51:19.131 DEBUG:teuthology.orchestra.run.trial187:> stat -c '"%a"' /home/ubuntu/.ssh/authorized_keys 2026-02-12T08:51:19.188 INFO:teuthology.orchestra.run.trial187.stdout:"600" 2026-02-12T08:51:19.188 DEBUG:teuthology.orchestra.run.trial187:> mv -- /home/ubuntu/.ssh/authorized_keys_2026-02-12_08:48:33:934923 /home/ubuntu/.ssh/authorized_keys 2026-02-12T08:51:19.245 DEBUG:teuthology.orchestra.run.trial187:> chmod 600 /home/ubuntu/.ssh/authorized_keys 2026-02-12T08:51:19.301 DEBUG:teuthology.orchestra.run.trial187:> rm /home/ubuntu/.ssh/config 2026-02-12T08:51:19.358 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2026-02-12T08:51:19.372 DEBUG:teuthology.orchestra.run.trial187:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2026-02-12T08:51:19.475 INFO:teuthology.orchestra.run.trial187.stderr:2026-02-12T08:51:19.467+0000 7f571885a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T08:51:19.476 INFO:teuthology.orchestra.run.trial187.stderr:2026-02-12T08:51:19.472+0000 7f571885a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T08:51:19.604 INFO:teuthology.misc:Shutting down mds daemons... 2026-02-12T08:51:19.604 INFO:teuthology.misc:Shutting down osd daemons... 2026-02-12T08:51:19.605 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2026-02-12T08:51:19.605 INFO:teuthology.orchestra.run:waiting for 300 2026-02-12T08:51:19.681 INFO:tasks.ceph.osd.0:Stopped 2026-02-12T08:51:19.682 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2026-02-12T08:51:19.682 INFO:teuthology.orchestra.run:waiting for 300 2026-02-12T08:51:19.758 INFO:tasks.ceph.osd.1:Stopped 2026-02-12T08:51:19.759 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2026-02-12T08:51:19.759 INFO:teuthology.orchestra.run:waiting for 300 2026-02-12T08:51:19.840 INFO:tasks.ceph.osd.2:Stopped 2026-02-12T08:51:19.840 INFO:teuthology.misc:Shutting down mgr daemons... 2026-02-12T08:51:19.840 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2026-02-12T08:51:19.841 INFO:teuthology.orchestra.run:waiting for 300 2026-02-12T08:51:19.879 INFO:tasks.ceph.mgr.x:Stopped 2026-02-12T08:51:19.879 INFO:teuthology.misc:Shutting down mon daemons... 2026-02-12T08:51:19.879 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2026-02-12T08:51:19.879 INFO:teuthology.orchestra.run:waiting for 300 2026-02-12T08:51:19.929 INFO:tasks.ceph.mon.a:Stopped 2026-02-12T08:51:19.929 INFO:tasks.ceph:Checking cluster log for badness... 2026-02-12T08:51:19.930 DEBUG:teuthology.orchestra.run.trial187:> sudo grep -E '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | grep -E -v '\(MDS_ALL_DOWN\)' | grep -E -v '\(MDS_UP_LESS_THAN_MAX\)' | grep -E -v '\(POOL_APP_NOT_ENABLED\)' | grep -E -v '\(PG_' | grep -E -v '\(OSD_' | grep -E -v '\(OBJECT_' | grep -E -v 'overall HEALTH' | head -n 1 2026-02-12T08:51:19.996 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@trial187.front.sepia.ceph.com 2026-02-12T08:51:19.997 DEBUG:teuthology.orchestra.run.trial187:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2026-02-12T08:51:20.142 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@trial187.front.sepia.ceph.com 2026-02-12T08:51:20.142 DEBUG:teuthology.orchestra.run.trial187:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2026-02-12T08:51:20.172 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@trial187.front.sepia.ceph.com 2026-02-12T08:51:20.172 DEBUG:teuthology.orchestra.run.trial187:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2026-02-12T08:51:20.241 INFO:tasks.ceph:Archiving mon data... 2026-02-12T08:51:20.243 DEBUG:teuthology.misc:Transferring archived files from trial187:/var/lib/ceph/mon/ceph-a to /home/teuthworker/mnt/teuthology/matan-2026-02-09_15:50:17-crimson-rados-main-distro-debug-trial/41545/data/mon.a.tgz 2026-02-12T08:51:20.243 DEBUG:teuthology.orchestra.run.trial187:> mktemp 2026-02-12T08:51:20.297 INFO:teuthology.orchestra.run.trial187.stdout:/tmp/tmp.TM7a51gcZw 2026-02-12T08:51:20.298 DEBUG:teuthology.orchestra.run.trial187:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.TM7a51gcZw 2026-02-12T08:51:20.388 DEBUG:teuthology.orchestra.run.trial187:> sudo chmod 0666 /tmp/tmp.TM7a51gcZw 2026-02-12T08:51:20.470 DEBUG:teuthology.orchestra.remote:trial187:/tmp/tmp.TM7a51gcZw is 200KB 2026-02-12T08:51:20.522 DEBUG:teuthology.orchestra.run.trial187:> rm -fr /tmp/tmp.TM7a51gcZw 2026-02-12T08:51:20.538 INFO:tasks.ceph:Cleaning ceph cluster... 2026-02-12T08:51:20.538 DEBUG:teuthology.orchestra.run.trial187:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2026-02-12T08:51:20.661 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2026-02-12T08:51:20.661 INFO:tasks.ceph:Archiving crash dumps... 2026-02-12T08:51:20.663 DEBUG:teuthology.misc:Transferring archived files from trial187:/var/lib/ceph/crash to /home/teuthworker/mnt/teuthology/matan-2026-02-09_15:50:17-crimson-rados-main-distro-debug-trial/41545/remote/trial187/crash 2026-02-12T08:51:20.664 DEBUG:teuthology.orchestra.run.trial187:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2026-02-12T08:51:20.727 INFO:tasks.ceph:Compressing logs... 2026-02-12T08:51:20.727 DEBUG:teuthology.orchestra.run.trial187:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2026-02-12T08:51:20.790 INFO:teuthology.orchestra.run.trial187.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22220.log 2026-02-12T08:51:20.803 INFO:teuthology.orchestra.run.trial187.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2026-02-12T08:51:20.803 INFO:teuthology.orchestra.run.trial187.stderr:/var/log/ceph/ceph-client.admin.22220.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2026-02-12T08:51:20.803 INFO:teuthology.orchestra.run.trial187.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.22220.log.gz 2026-02-12T08:51:20.803 INFO:teuthology.orchestra.run.trial187.stderr:gzip -5 --verbose/var/log/ceph/ceph-osd.2.log: -- /var/log/ceph/ceph-osd.0.log 2026-02-12T08:51:20.803 INFO:teuthology.orchestra.run.trial187.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23333.log 2026-02-12T08:51:20.803 INFO:teuthology.orchestra.run.trial187.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23725.log 2026-02-12T08:51:20.804 INFO:teuthology.orchestra.run.trial187.stderr:/var/log/ceph/ceph-client.admin.23333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22546.log 2026-02-12T08:51:20.804 INFO:teuthology.orchestra.run.trial187.stderr:/var/log/ceph/ceph-client.admin.23725.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.23333.log.gz 2026-02-12T08:51:20.804 INFO:teuthology.orchestra.run.trial187.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2026-02-12T08:51:20.804 INFO:teuthology.orchestra.run.trial187.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.20931.log 2026-02-12T08:51:20.804 INFO:teuthology.orchestra.run.trial187.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.23725.log.gz 2026-02-12T08:51:20.804 INFO:teuthology.orchestra.run.trial187.stderr:/var/log/ceph/ceph-client.admin.22546.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26861.log 2026-02-12T08:51:20.804 INFO:teuthology.orchestra.run.trial187.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.22546.log.gz 2026-02-12T08:51:20.804 INFO:teuthology.orchestra.run.trial187.stderr:/var/log/ceph/ceph.tmp-client.admin.20931.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23382.log 2026-02-12T08:51:20.804 INFO:teuthology.orchestra.run.trial187.stderr: 67.5% -- replaced with /var/log/ceph/ceph.tmp-client.admin.20931.log.gz 2026-02-12T08:51:20.804 INFO:teuthology.orchestra.run.trial187.stderr:/var/log/ceph/ceph-client.admin.26861.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23075.log 2026-02-12T08:51:20.804 INFO:teuthology.orchestra.run.trial187.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.26861.log.gz 2026-02-12T08:51:20.804 INFO:teuthology.orchestra.run.trial187.stderr:/var/log/ceph/ceph-client.admin.23382.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23630.log 2026-02-12T08:51:20.804 INFO:teuthology.orchestra.run.trial187.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.23382.log.gz 2026-02-12T08:51:20.805 INFO:teuthology.orchestra.run.trial187.stderr:/var/log/ceph/ceph-client.admin.23075.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23145.log 2026-02-12T08:51:20.805 INFO:teuthology.orchestra.run.trial187.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.23075.log.gz 2026-02-12T08:51:20.805 INFO:teuthology.orchestra.run.trial187.stderr:/var/log/ceph/ceph-client.admin.23630.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26975.log 2026-02-12T08:51:20.805 INFO:teuthology.orchestra.run.trial187.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.23630.log.gz 2026-02-12T08:51:20.805 INFO:teuthology.orchestra.run.trial187.stderr:/var/log/ceph/ceph-client.admin.23145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23170.log 2026-02-12T08:51:20.805 INFO:teuthology.orchestra.run.trial187.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.23145.log.gz 2026-02-12T08:51:20.805 INFO:teuthology.orchestra.run.trial187.stderr:gzip/var/log/ceph/ceph-client.admin.26975.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.23235.log 2026-02-12T08:51:20.805 INFO:teuthology.orchestra.run.trial187.stderr:gzip -5/var/log/ceph/ceph-client.admin.23170.log: --verbose -- /var/log/ceph/ceph-client.admin.26917.log 2026-02-12T08:51:20.805 INFO:teuthology.orchestra.run.trial187.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.26975.log.gz 2026-02-12T08:51:20.805 INFO:teuthology.orchestra.run.trial187.stderr:/var/log/ceph/ceph-client.admin.23235.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27063.log 2026-02-12T08:51:20.805 INFO:teuthology.orchestra.run.trial187.stderr: 66.7% -- replaced with /var/log/ceph/ceph-client.admin.23170.log.gz 2026-02-12T08:51:20.805 INFO:teuthology.orchestra.run.trial187.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.23235.log.gz 2026-02-12T08:51:20.805 INFO:teuthology.orchestra.run.trial187.stderr:/var/log/ceph/ceph-client.admin.26917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23284.log 2026-02-12T08:51:20.806 INFO:teuthology.orchestra.run.trial187.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.26917.log.gz 2026-02-12T08:51:20.806 INFO:teuthology.orchestra.run.trial187.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22362.log 2026-02-12T08:51:20.806 INFO:teuthology.orchestra.run.trial187.stderr:/var/log/ceph/ceph-client.admin.27063.log: /var/log/ceph/ceph-client.admin.23284.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.27063.log.gz 2026-02-12T08:51:20.806 INFO:teuthology.orchestra.run.trial187.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23631.log 2026-02-12T08:51:20.806 INFO:teuthology.orchestra.run.trial187.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.23284.log.gz 2026-02-12T08:51:20.806 INFO:teuthology.orchestra.run.trial187.stderr:/var/log/ceph/ceph-client.admin.22362.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22315.log 2026-02-12T08:51:20.806 INFO:teuthology.orchestra.run.trial187.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.22362.log.gz 2026-02-12T08:51:20.806 INFO:teuthology.orchestra.run.trial187.stderr:/var/log/ceph/ceph-client.admin.23631.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28285.log 2026-02-12T08:51:20.806 INFO:teuthology.orchestra.run.trial187.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.23631.log.gz 2026-02-12T08:51:20.806 INFO:teuthology.orchestra.run.trial187.stderr:/var/log/ceph/ceph-client.admin.22315.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2026-02-12T08:51:20.806 INFO:teuthology.orchestra.run.trial187.stderr:/var/log/ceph/ceph-client.admin.28285.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.22315.log.gz 2026-02-12T08:51:20.806 INFO:teuthology.orchestra.run.trial187.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27618.log 2026-02-12T08:51:20.806 INFO:teuthology.orchestra.run.trial187.stderr: 66.4% -- replaced with /var/log/ceph/ceph-client.admin.28285.log.gz 2026-02-12T08:51:20.807 INFO:teuthology.orchestra.run.trial187.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23483.log 2026-02-12T08:51:20.807 INFO:teuthology.orchestra.run.trial187.stderr:/var/log/ceph/ceph-client.admin.27618.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2026-02-12T08:51:20.807 INFO:teuthology.orchestra.run.trial187.stderr: 87.7% -- replaced with /var/log/ceph/ceph.audit.log.gz 2026-02-12T08:51:20.807 INFO:teuthology.orchestra.run.trial187.stderr: 66.1% -- replaced with /var/log/ceph/ceph-client.admin.27618.log.gz 2026-02-12T08:51:20.807 INFO:teuthology.orchestra.run.trial187.stderr:/var/log/ceph/ceph-client.admin.23483.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28969.log 2026-02-12T08:51:20.807 INFO:teuthology.orchestra.run.trial187.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.23483.log.gz 2026-02-12T08:51:20.807 INFO:teuthology.orchestra.run.trial187.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22812.log 2026-02-12T08:51:20.807 INFO:teuthology.orchestra.run.trial187.stderr:/var/log/ceph/ceph-client.admin.28969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23632.log 2026-02-12T08:51:20.807 INFO:teuthology.orchestra.run.trial187.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28969.log.gz 2026-02-12T08:51:20.807 INFO:teuthology.orchestra.run.trial187.stderr:/var/log/ceph/ceph-client.admin.22812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23823.log 2026-02-12T08:51:20.807 INFO:teuthology.orchestra.run.trial187.stderr:/var/log/ceph/ceph-client.admin.23632.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27007.log 2026-02-12T08:51:20.807 INFO:teuthology.orchestra.run.trial187.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.22812.log.gz 2026-02-12T08:51:20.807 INFO:teuthology.orchestra.run.trial187.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.23632.log.gz 2026-02-12T08:51:20.808 INFO:teuthology.orchestra.run.trial187.stderr:/var/log/ceph/ceph-client.admin.23823.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22499.log 2026-02-12T08:51:20.808 INFO:teuthology.orchestra.run.trial187.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.23823.log.gz 2026-02-12T08:51:20.808 INFO:teuthology.orchestra.run.trial187.stderr:/var/log/ceph/ceph-client.admin.27007.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23484.log 2026-02-12T08:51:20.808 INFO:teuthology.orchestra.run.trial187.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.27007.log.gz 2026-02-12T08:51:20.808 INFO:teuthology.orchestra.run.trial187.stderr:/var/log/ceph/ceph-client.admin.22499.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22993.log 2026-02-12T08:51:20.808 INFO:teuthology.orchestra.run.trial187.stderr:/var/log/ceph/ceph-client.admin.23484.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.22499.log.gz 2026-02-12T08:51:20.808 INFO:teuthology.orchestra.run.trial187.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22593.log 2026-02-12T08:51:20.808 INFO:teuthology.orchestra.run.trial187.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.23484.log.gz 2026-02-12T08:51:20.808 INFO:teuthology.orchestra.run.trial187.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27617.log 2026-02-12T08:51:20.808 INFO:teuthology.orchestra.run.trial187.stderr:/var/log/ceph/ceph-client.admin.22993.log: /var/log/ceph/ceph-client.admin.22593.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.22993.log.gz 2026-02-12T08:51:20.808 INFO:teuthology.orchestra.run.trial187.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2026-02-12T08:51:20.808 INFO:teuthology.orchestra.run.trial187.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.22593.log.gz 2026-02-12T08:51:20.808 INFO:teuthology.orchestra.run.trial187.stderr:/var/log/ceph/ceph-client.admin.27617.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23485.log 2026-02-12T08:51:20.809 INFO:teuthology.orchestra.run.trial187.stderr: 66.7% -- replaced with /var/log/ceph/ceph-client.admin.27617.log.gz 2026-02-12T08:51:20.809 INFO:teuthology.orchestra.run.trial187.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2026-02-12T08:51:20.809 INFO:teuthology.orchestra.run.trial187.stderr:/var/log/ceph/ceph-client.admin.23485.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22843.log 2026-02-12T08:51:20.809 INFO:teuthology.orchestra.run.trial187.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.23485.log.gz 2026-02-12T08:51:20.809 INFO:teuthology.orchestra.run.trial187.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28292.log 2026-02-12T08:51:20.809 INFO:teuthology.orchestra.run.trial187.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23774.log 2026-02-12T08:51:20.809 INFO:teuthology.orchestra.run.trial187.stderr:/var/log/ceph/ceph-client.admin.22843.log: 85.0% -- replaced with /var/log/ceph/ceph.log.gz 2026-02-12T08:51:20.809 INFO:teuthology.orchestra.run.trial187.stderr:/var/log/ceph/ceph-client.admin.28292.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.22843.log.gz 2026-02-12T08:51:20.809 INFO:teuthology.orchestra.run.trial187.stderr: 66.1% -- replaced with /var/log/ceph/ceph-client.admin.28292.log.gz 2026-02-12T08:51:20.809 INFO:teuthology.orchestra.run.trial187.stderr:/var/log/ceph/ceph-client.admin.23774.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.23774.log.gz 2026-02-12T08:51:20.810 INFO:teuthology.orchestra.run.trial187.stderr: 91.9% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2026-02-12T08:51:22.852 INFO:teuthology.orchestra.run.trial187.stderr: 92.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2026-02-12T08:51:22.873 INFO:teuthology.orchestra.run.trial187.stderr: 92.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2026-02-12T08:51:22.970 INFO:teuthology.orchestra.run.trial187.stderr: 92.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2026-02-12T08:51:22.972 INFO:teuthology.orchestra.run.trial187.stderr: 2026-02-12T08:51:22.972 INFO:teuthology.orchestra.run.trial187.stderr:real 0m2.189s 2026-02-12T08:51:22.972 INFO:teuthology.orchestra.run.trial187.stderr:user 0m5.886s 2026-02-12T08:51:22.972 INFO:teuthology.orchestra.run.trial187.stderr:sys 0m0.455s 2026-02-12T08:51:22.973 INFO:tasks.ceph:Archiving logs... 2026-02-12T08:51:22.973 DEBUG:teuthology.misc:Transferring archived files from trial187:/var/log/ceph to /home/teuthworker/mnt/teuthology/matan-2026-02-09_15:50:17-crimson-rados-main-distro-debug-trial/41545/remote/trial187/log 2026-02-12T08:51:22.974 DEBUG:teuthology.orchestra.run.trial187:> sudo tar c -f - -C /var/log/ceph -- . 2026-02-12T08:51:23.685 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-12T08:51:23.755 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-12T08:51:23.767 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2026-02-12T08:51:23.767 DEBUG:teuthology.orchestra.run.trial187:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-12T08:51:23.793 DEBUG:teuthology.run_tasks:Unwinding manager clock 2026-02-12T08:51:23.804 INFO:teuthology.task.clock:Checking final clock skew... 2026-02-12T08:51:23.804 DEBUG:teuthology.orchestra.run.trial187:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-12T08:51:23.848 INFO:teuthology.orchestra.run.trial187.stderr:bash: line 1: ntpq: command not found 2026-02-12T08:51:23.873 INFO:teuthology.orchestra.run.trial187.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-12T08:51:23.873 INFO:teuthology.orchestra.run.trial187.stdout:=============================================================================== 2026-02-12T08:51:23.873 INFO:teuthology.orchestra.run.trial187.stdout:^* 23.157.160.168 2 6 377 28 +675us[ +697us] +/- 42ms 2026-02-12T08:51:23.873 INFO:teuthology.orchestra.run.trial187.stdout:^+ nyc1.starstorm.net 2 6 377 26 -830us[ -830us] +/- 42ms 2026-02-12T08:51:23.873 INFO:teuthology.orchestra.run.trial187.stdout:^- time-usw1.crimpac.net 2 6 375 25 -2004us[-2004us] +/- 103ms 2026-02-12T08:51:23.873 INFO:teuthology.orchestra.run.trial187.stdout:^- ip229.ip-51-81-226.us 2 6 377 26 -1567us[-1567us] +/- 107ms 2026-02-12T08:51:23.875 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2026-02-12T08:51:23.886 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2026-02-12T08:51:23.886 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2026-02-12T08:51:23.900 DEBUG:teuthology.orchestra.run.trial187:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2026-02-12T08:51:23.962 DEBUG:teuthology.orchestra.run.trial187:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-12T08:51:23.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-12T08:51:23.987 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2026-02-12T08:51:23.999 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2026-02-12T08:51:24.013 INFO:teuthology.task.internal:Duration was 947.254032 seconds 2026-02-12T08:51:24.013 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2026-02-12T08:51:24.027 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2026-02-12T08:51:24.027 DEBUG:teuthology.orchestra.run.trial187:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-12T08:51:24.066 INFO:teuthology.orchestra.run.trial187.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-12T08:51:24.248 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2026-02-12T08:51:24.248 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial187.front.sepia.ceph.com 2026-02-12T08:51:24.248 DEBUG:teuthology.orchestra.run.trial187:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-12T08:51:24.306 INFO:teuthology.task.internal.syslog:Gathering journactl... 2026-02-12T08:51:24.306 DEBUG:teuthology.orchestra.run.trial187:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-12T08:51:24.393 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2026-02-12T08:51:24.393 DEBUG:teuthology.orchestra.run.trial187:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-12T08:51:24.416 INFO:teuthology.orchestra.run.trial187.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-12T08:51:24.416 INFO:teuthology.orchestra.run.trial187.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-12T08:51:24.416 INFO:teuthology.orchestra.run.trial187.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-02-12T08:51:24.417 INFO:teuthology.orchestra.run.trial187.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-12T08:51:24.417 INFO:teuthology.orchestra.run.trial187.stderr:/home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-12T08:51:24.421 INFO:teuthology.orchestra.run.trial187.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: 88.6% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-12T08:51:24.423 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2026-02-12T08:51:24.434 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2026-02-12T08:51:24.434 DEBUG:teuthology.orchestra.run.trial187:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-12T08:51:24.488 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2026-02-12T08:51:24.499 DEBUG:teuthology.orchestra.run.trial187:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-12T08:51:24.555 INFO:teuthology.orchestra.run.trial187.stdout:kernel.core_pattern = core 2026-02-12T08:51:24.570 DEBUG:teuthology.orchestra.run.trial187:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-12T08:51:24.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-12T08:51:24.626 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2026-02-12T08:51:24.638 INFO:teuthology.task.internal:Transferring archived files... 2026-02-12T08:51:24.638 DEBUG:teuthology.misc:Transferring archived files from trial187:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/matan-2026-02-09_15:50:17-crimson-rados-main-distro-debug-trial/41545/remote/trial187 2026-02-12T08:51:24.638 DEBUG:teuthology.orchestra.run.trial187:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-12T08:51:24.711 INFO:teuthology.task.internal:Removing archive directory... 2026-02-12T08:51:24.711 DEBUG:teuthology.orchestra.run.trial187:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-12T08:51:24.754 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2026-02-12T08:51:24.769 INFO:teuthology.task.internal:Not uploading archives. 2026-02-12T08:51:24.769 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2026-02-12T08:51:24.782 INFO:teuthology.task.internal:Tidying up after the test... 2026-02-12T08:51:24.782 DEBUG:teuthology.orchestra.run.trial187:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-12T08:51:24.811 INFO:teuthology.orchestra.run.trial187.stdout: 786525 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 12 08:51 /home/ubuntu/cephtest 2026-02-12T08:51:24.811 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2026-02-12T08:51:24.823 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2026-02-12T08:51:24.852 INFO:teuthology.run:Summary data: description: crimson-rados/perf/{0-crimson_install clusters/{crimson-fixed-1} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/segmented$/{crimson_seastore_segmented} perf_ignorelist settings/optimized workloads/radosbench_4M_seq_read} duration: 947.2540323734283 flavor: debug owner: scheduled_matan@soko04.front.sepia.ceph.com success: true 2026-02-12T08:51:24.852 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-12T08:51:24.937 INFO:teuthology.run:pass