2026-02-26T03:03:35.251 INFO:root:teuthology version: 1.2.4.dev11+g6e3d0419c 2026-02-26T03:03:35.266 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-26T03:03:35.351 INFO:teuthology.run:Config: archive_path: /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-26_02:22:56-crimson-rados-main-distro-debug-trial/71162 branch: main description: crimson-rados/perf/{0-crimson_install clusters/{crimson-fixed-1} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/segmented$/{crimson_seastore_segmented} perf_ignorelist settings/optimized workloads/radosbench_4K_rand_read} email: null first_in_suite: false flavor: debug job_id: '71162' kernel: branch: distro kdb: true sha1: distro last_in_suite: false machine_type: trial name: shraddhaag-2026-02-26_02:22:56-crimson-rados-main-distro-debug-trial no_nested_subset: false os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: global: auth client required: none auth cluster required: none auth service required: none auth supported: none debug asok: 0/0 debug auth: 0/0 debug bdev: 0/0 debug bluefs: 0/0 debug bluestore: 0/0 debug buffer: 0/0 debug civetweb: 0/0 debug client: 0/0 debug compressor: 0/0 debug context: 0/0 debug crush: 0/0 debug crypto: 0/0 debug dpdk: 0/0 debug eventtrace: 0/0 debug filer: 0/0 debug filestore: 0/0 debug finisher: 0/0 debug fuse: 0/0 debug heartbeatmap: 0/0 debug javaclient: 0/0 debug journal: 0/0 debug journaler: 0/0 debug kstore: 0/0 debug leveldb: 0/0 debug lockdep: 0/0 debug mds: 0/0 debug mds balancer: 0/0 debug mds locker: 0/0 debug mds log: 0/0 debug mds log expire: 0/0 debug mds migrator: 0/0 debug memdb: 0/0 debug mgr: 0/0 debug mgrc: 0/0 debug mon: 0/0 debug monc: 0/0 debug ms: 0/0 debug objclass: 0/0 debug objectcacher: 0/0 debug objecter: 0/0 debug optracker: 0/0 debug osd: 0/0 debug paxos: 0/0 debug perfcounter: 0/0 debug rados: 0/0 debug rbd: 0/0 debug rbd mirror: 0/0 debug rbd replay: 0/0 debug refs: 0/0 debug rgw: 0/0 debug rgw lifecycle: 0/0 debug rgw notification: 0/0 debug rgw sync: 0/0 debug rocksdb: 0/0 debug striper: 0/0 debug throttle: 0/0 debug timer: 0/0 debug tp: 0/0 enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 0/0 debug ms: 0/0 debug paxos: 0/0 osd pool default crimson: true osd: crimson cpu num: 4 crimson osd obc lru size: 10 debug filestore: 0/0 debug journal: 0/0 debug ms: 0/0 debug osd: 0/0 debug seastore: 20 debug seastore backref: 20 debug seastore cache: 20 debug seastore cleaner: 20 debug seastore device: 20 debug seastore epm: 20 debug seastore fixedkv tree: 20 debug seastore journal: 20 debug seastore lba: 20 debug seastore odata: 20 debug seastore omap: 20 debug seastore onode: 20 debug seastore t: 20 debug seastore tm: 20 osd objectstore: seastore osd shutdown pgref assert: true seastore cachepin size pershard: 64M seastore max concurrent transactions: 8 crimson_compat: true flavor: debug log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) - \(PG_ - \(OSD_ - \(OBJECT_ - overall HEALTH sha1: abffb617683951511fccf3c3cab6e219f2ec83a8 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log global: osd crush chooseleaf type: 0 osd pool default pg num: 128 osd pool default pgp num: 128 osd pool default size: 2 mon: {} install: ceph: flavor: debug sha1: abffb617683951511fccf3c3cab6e219f2ec83a8 workunit: branch: main env: CRIMSON_COMPAT: '1' sha1: 1f87b69b1a8041112e26ae0ea3c0a2a0fd14d996 owner: scheduled_shraddhaag@soko04.front.sepia.ceph.com priority: 101 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 5117 sha1: abffb617683951511fccf3c3cab6e219f2ec83a8 sleep_before_teardown: 0 suite: crimson-rados suite_branch: main suite_path: /home/teuthworker/src/github.com_ceph_ceph_1f87b69b1a8041112e26ae0ea3c0a2a0fd14d996/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph.git suite_sha1: 1f87b69b1a8041112e26ae0ea3c0a2a0fd14d996 targets: trial134.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBdXNEyjnAMdr0HYsmPoZe4BD3+dkvnDAY9Vor0jDPkZfr9+oShjMH/EKgesItdbPYxmG5j/0mDr0hPhMct8YzE= tasks: - install: extra_packages: deb: - ceph-osd-crimson - ceph-osd-crimson-dbg rpm: - ceph-osd-crimson - ceph-osd-crimson-debuginfo - pexec: all: - sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson - install: null - ceph: conf: osd: debug monc: 20 wait-for-scrub: false - ssh_keys: null - cbt: benchmarks: radosbench: concurrent_ops: 4 concurrent_procs: 2 op_size: - 4096 pool_monitoring_list: - collectl pool_profile: replicated readmode: rand run_monitoring_list: - collectl time: 60 write_only: false cluster: iterations: 1 osds_per_node: 3 pool_profiles: replicated: pg_size: 256 pgp_size: 256 replication: replicated user: ubuntu monitoring_profiles: perf: args: stat -p {pid} -o {perf_dir}/perf_stat.{pid} nodes: - osds perf_cmd: perf pid_glob: crimson-osd.*.pid teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6e3d0419ccca0e75a03e08feb708946e198236f4 timestamp: 2026-02-26_02:22:56 tube: trial user: shraddhaag verbose: false worker_log: /home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.904699 2026-02-26T03:03:35.351 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph_1f87b69b1a8041112e26ae0ea3c0a2a0fd14d996/qa; will attempt to use it 2026-02-26T03:03:35.352 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph_1f87b69b1a8041112e26ae0ea3c0a2a0fd14d996/qa/tasks 2026-02-26T03:03:35.352 INFO:teuthology.run_tasks:Running task internal.check_packages... 2026-02-26T03:03:35.354 INFO:teuthology.task.internal:Checking packages... 2026-02-26T03:03:35.354 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'debug' and ceph hash 'abffb617683951511fccf3c3cab6e219f2ec83a8' 2026-02-26T03:03:35.354 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2026-02-26T03:03:35.354 INFO:teuthology.packaging:ref: None 2026-02-26T03:03:35.354 INFO:teuthology.packaging:tag: None 2026-02-26T03:03:35.354 INFO:teuthology.packaging:branch: main 2026-02-26T03:03:35.354 INFO:teuthology.packaging:sha1: abffb617683951511fccf3c3cab6e219f2ec83a8 2026-02-26T03:03:35.354 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&ref=main 2026-02-26T03:03:35.481 INFO:teuthology.task.internal:Found packages for ceph version 20.3.0-5593.g63656871 2026-02-26T03:03:35.482 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2026-02-26T03:03:35.495 INFO:teuthology.task.internal:no buildpackages task found 2026-02-26T03:03:35.495 INFO:teuthology.run_tasks:Running task internal.save_config... 2026-02-26T03:03:35.511 INFO:teuthology.task.internal:Saving configuration 2026-02-26T03:03:35.519 INFO:teuthology.run_tasks:Running task internal.check_lock... 2026-02-26T03:03:35.532 INFO:teuthology.task.internal.check_lock:Checking locks... 2026-02-26T03:03:35.613 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial134.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/shraddhaag-2026-02-26_02:22:56-crimson-rados-main-distro-debug-trial/71162', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-26 03:00:03.946650', 'locked_by': 'scheduled_shraddhaag@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBdXNEyjnAMdr0HYsmPoZe4BD3+dkvnDAY9Vor0jDPkZfr9+oShjMH/EKgesItdbPYxmG5j/0mDr0hPhMct8YzE='} 2026-02-26T03:03:35.614 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2026-02-26T03:03:35.625 INFO:teuthology.task.internal:roles: ubuntu@trial134.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2026-02-26T03:03:35.625 INFO:teuthology.run_tasks:Running task console_log... 2026-02-26T03:03:35.710 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x71e4c7036ca0>, signals=[15]) 2026-02-26T03:03:35.711 INFO:teuthology.run_tasks:Running task internal.connect... 2026-02-26T03:03:35.723 INFO:teuthology.task.internal:Opening connections... 2026-02-26T03:03:35.724 DEBUG:teuthology.task.internal:connecting to ubuntu@trial134.front.sepia.ceph.com 2026-02-26T03:03:35.724 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-26T03:03:35.792 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2026-02-26T03:03:35.851 DEBUG:teuthology.orchestra.run.trial134:> uname -m 2026-02-26T03:03:35.870 INFO:teuthology.orchestra.run.trial134.stdout:x86_64 2026-02-26T03:03:35.870 DEBUG:teuthology.orchestra.run.trial134:> cat /etc/os-release 2026-02-26T03:03:35.925 INFO:teuthology.orchestra.run.trial134.stdout:NAME="CentOS Stream" 2026-02-26T03:03:35.925 INFO:teuthology.orchestra.run.trial134.stdout:VERSION="9" 2026-02-26T03:03:35.925 INFO:teuthology.orchestra.run.trial134.stdout:ID="centos" 2026-02-26T03:03:35.926 INFO:teuthology.orchestra.run.trial134.stdout:ID_LIKE="rhel fedora" 2026-02-26T03:03:35.926 INFO:teuthology.orchestra.run.trial134.stdout:VERSION_ID="9" 2026-02-26T03:03:35.926 INFO:teuthology.orchestra.run.trial134.stdout:PLATFORM_ID="platform:el9" 2026-02-26T03:03:35.926 INFO:teuthology.orchestra.run.trial134.stdout:PRETTY_NAME="CentOS Stream 9" 2026-02-26T03:03:35.926 INFO:teuthology.orchestra.run.trial134.stdout:ANSI_COLOR="0;31" 2026-02-26T03:03:35.926 INFO:teuthology.orchestra.run.trial134.stdout:LOGO="fedora-logo-icon" 2026-02-26T03:03:35.926 INFO:teuthology.orchestra.run.trial134.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2026-02-26T03:03:35.926 INFO:teuthology.orchestra.run.trial134.stdout:HOME_URL="https://centos.org/" 2026-02-26T03:03:35.926 INFO:teuthology.orchestra.run.trial134.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2026-02-26T03:03:35.926 INFO:teuthology.orchestra.run.trial134.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2026-02-26T03:03:35.926 INFO:teuthology.orchestra.run.trial134.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2026-02-26T03:03:35.927 INFO:teuthology.lock.ops:Updating trial134.front.sepia.ceph.com on lock server 2026-02-26T03:03:36.009 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2026-02-26T03:03:36.022 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2026-02-26T03:03:36.034 INFO:teuthology.task.internal:Checking for old test directory... 2026-02-26T03:03:36.034 DEBUG:teuthology.orchestra.run.trial134:> test '!' -e /home/ubuntu/cephtest 2026-02-26T03:03:36.049 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2026-02-26T03:03:36.064 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2026-02-26T03:03:36.064 DEBUG:teuthology.orchestra.run.trial134:> test -z $(ls -A /var/lib/ceph) 2026-02-26T03:03:36.105 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2026-02-26T03:03:36.203 INFO:teuthology.run_tasks:Running task kernel... 2026-02-26T03:03:36.229 INFO:teuthology.task.kernel:normalize config orig: {'branch': 'distro', 'kdb': True, 'sha1': 'distro'} 2026-02-26T03:03:36.229 INFO:teuthology.task.kernel:config {'mon.a': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}}, timeout 300 2026-02-26T03:03:36.229 DEBUG:teuthology.orchestra.run.trial134:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-26T03:03:36.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-26T03:03:36.244 DEBUG:teuthology.orchestra.run.trial134:> uname -r 2026-02-26T03:03:36.300 INFO:teuthology.orchestra.run.trial134.stdout:5.14.0-681.el9.x86_64 2026-02-26T03:03:36.300 INFO:teuthology.task.kernel:Running kernel on trial134: 5.14.0-681.el9.x86_64 2026-02-26T03:03:36.300 DEBUG:teuthology.orchestra.run.trial134:> sudo yum install -y kernel 2026-02-26T03:03:37.667 INFO:teuthology.orchestra.run.trial134.stdout:CentOS Stream 9 - BaseOS 9.1 MB/s | 8.9 MB 00:00 2026-02-26T03:04:05.674 INFO:teuthology.orchestra.run.trial134.stdout:CentOS Stream 9 - AppStream 1.0 MB/s | 27 MB 00:27 2026-02-26T03:04:09.255 INFO:teuthology.orchestra.run.trial134.stdout:CentOS Stream 9 - CRB 6.6 MB/s | 8.0 MB 00:01 2026-02-26T03:04:11.038 INFO:teuthology.orchestra.run.trial134.stdout:CentOS Stream 9 - Extras packages 19 kB/s | 20 kB 00:01 2026-02-26T03:04:12.102 INFO:teuthology.orchestra.run.trial134.stdout:Extra Packages for Enterprise Linux 20 MB/s | 20 MB 00:01 2026-02-26T03:04:15.331 INFO:teuthology.orchestra.run.trial134.stdout:lab-extras 2.4 MB/s | 50 kB 00:00 2026-02-26T03:04:16.357 INFO:teuthology.orchestra.run.trial134.stdout:Package kernel-5.14.0-645.el9.x86_64 is already installed. 2026-02-26T03:04:16.357 INFO:teuthology.orchestra.run.trial134.stdout:Package kernel-5.14.0-681.el9.x86_64 is already installed. 2026-02-26T03:04:16.377 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-26T03:04:16.377 INFO:teuthology.orchestra.run.trial134.stdout:================================================================================ 2026-02-26T03:04:16.377 INFO:teuthology.orchestra.run.trial134.stdout: Package Architecture Version Repository Size 2026-02-26T03:04:16.377 INFO:teuthology.orchestra.run.trial134.stdout:================================================================================ 2026-02-26T03:04:16.377 INFO:teuthology.orchestra.run.trial134.stdout:Installing: 2026-02-26T03:04:16.378 INFO:teuthology.orchestra.run.trial134.stdout: kernel x86_64 5.14.0-686.el9 baseos 901 k 2026-02-26T03:04:16.378 INFO:teuthology.orchestra.run.trial134.stdout:Installing dependencies: 2026-02-26T03:04:16.378 INFO:teuthology.orchestra.run.trial134.stdout: kernel-core x86_64 5.14.0-686.el9 baseos 17 M 2026-02-26T03:04:16.378 INFO:teuthology.orchestra.run.trial134.stdout: kernel-modules x86_64 5.14.0-686.el9 baseos 40 M 2026-02-26T03:04:16.378 INFO:teuthology.orchestra.run.trial134.stdout: kernel-modules-core x86_64 5.14.0-686.el9 baseos 31 M 2026-02-26T03:04:16.378 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:04:16.378 INFO:teuthology.orchestra.run.trial134.stdout:Transaction Summary 2026-02-26T03:04:16.378 INFO:teuthology.orchestra.run.trial134.stdout:================================================================================ 2026-02-26T03:04:16.378 INFO:teuthology.orchestra.run.trial134.stdout:Install 4 Packages 2026-02-26T03:04:16.378 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:04:16.378 INFO:teuthology.orchestra.run.trial134.stdout:Total download size: 89 M 2026-02-26T03:04:16.378 INFO:teuthology.orchestra.run.trial134.stdout:Installed size: 135 M 2026-02-26T03:04:16.379 INFO:teuthology.orchestra.run.trial134.stdout:Downloading Packages: 2026-02-26T03:04:16.746 INFO:teuthology.orchestra.run.trial134.stdout:(1/4): kernel-5.14.0-686.el9.x86_64.rpm 14 MB/s | 901 kB 00:00 2026-02-26T03:04:16.857 INFO:teuthology.orchestra.run.trial134.stdout:(2/4): kernel-core-5.14.0-686.el9.x86_64.rpm 98 MB/s | 17 MB 00:00 2026-02-26T03:04:16.946 INFO:teuthology.orchestra.run.trial134.stdout:(3/4): kernel-modules-core-5.14.0-686.el9.x86_6 155 MB/s | 31 MB 00:00 2026-02-26T03:04:17.004 INFO:teuthology.orchestra.run.trial134.stdout:(4/4): kernel-modules-5.14.0-686.el9.x86_64.rpm 124 MB/s | 40 MB 00:00 2026-02-26T03:04:17.005 INFO:teuthology.orchestra.run.trial134.stdout:-------------------------------------------------------------------------------- 2026-02-26T03:04:17.005 INFO:teuthology.orchestra.run.trial134.stdout:Total 142 MB/s | 89 MB 00:00 2026-02-26T03:04:17.198 INFO:teuthology.orchestra.run.trial134.stdout:Running transaction check 2026-02-26T03:04:17.406 INFO:teuthology.orchestra.run.trial134.stdout:Transaction check succeeded. 2026-02-26T03:04:17.407 INFO:teuthology.orchestra.run.trial134.stdout:Running transaction test 2026-02-26T03:04:17.791 INFO:teuthology.orchestra.run.trial134.stdout:Transaction test succeeded. 2026-02-26T03:04:17.791 INFO:teuthology.orchestra.run.trial134.stdout:Running transaction 2026-02-26T03:04:18.904 INFO:teuthology.orchestra.run.trial134.stdout: Preparing : 1/1 2026-02-26T03:04:19.195 INFO:teuthology.orchestra.run.trial134.stdout: Installing : kernel-modules-core-5.14.0-686.el9.x86_64 1/4 2026-02-26T03:04:19.217 INFO:teuthology.orchestra.run.trial134.stdout: Installing : kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-26T03:04:21.782 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-26T03:04:21.814 INFO:teuthology.orchestra.run.trial134.stdout: Installing : kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-26T03:04:24.854 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-26T03:04:24.861 INFO:teuthology.orchestra.run.trial134.stdout: Installing : kernel-5.14.0-686.el9.x86_64 4/4 2026-02-26T03:04:27.912 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: kernel-modules-core-5.14.0-686.el9.x86_64 4/4 2026-02-26T03:04:45.365 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: kernel-core-5.14.0-686.el9.x86_64 4/4 2026-02-26T03:04:45.388 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: kernel-modules-5.14.0-686.el9.x86_64 4/4 2026-02-26T03:04:45.576 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: kernel-5.14.0-686.el9.x86_64 4/4 2026-02-26T03:04:45.576 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : kernel-5.14.0-686.el9.x86_64 1/4 2026-02-26T03:04:45.576 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-26T03:04:45.576 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-26T03:04:45.633 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : kernel-modules-core-5.14.0-686.el9.x86_64 4/4 2026-02-26T03:04:45.633 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:04:45.634 INFO:teuthology.orchestra.run.trial134.stdout:Installed: 2026-02-26T03:04:45.634 INFO:teuthology.orchestra.run.trial134.stdout: kernel-5.14.0-686.el9.x86_64 2026-02-26T03:04:45.634 INFO:teuthology.orchestra.run.trial134.stdout: kernel-core-5.14.0-686.el9.x86_64 2026-02-26T03:04:45.634 INFO:teuthology.orchestra.run.trial134.stdout: kernel-modules-5.14.0-686.el9.x86_64 2026-02-26T03:04:45.634 INFO:teuthology.orchestra.run.trial134.stdout: kernel-modules-core-5.14.0-686.el9.x86_64 2026-02-26T03:04:45.634 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:04:45.634 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-26T03:04:45.695 DEBUG:teuthology.orchestra.run.trial134:> rpm -q kernel | sort -rV | head -n 1 2026-02-26T03:04:45.723 INFO:teuthology.orchestra.run.trial134.stdout:kernel-5.14.0-686.el9.x86_64 2026-02-26T03:04:45.724 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-686.el9.x86_64 2026-02-26T03:04:45.724 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-681.el9.x86_64 Expected: 5.14.0-686.el9.x86_64 2026-02-26T03:04:45.724 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2026-02-26T03:04:45.724 DEBUG:teuthology.task.kernel:src is distro, skipping download 2026-02-26T03:04:45.725 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2026-02-26T03:04:45.725 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@trial134.front.sepia.ceph.com, path=None, version=distro) 2026-02-26T03:04:45.725 DEBUG:teuthology.orchestra.run.trial134:> rpm -q kernel | sort -rV | head -n 1 2026-02-26T03:04:45.791 INFO:teuthology.orchestra.run.trial134.stdout:kernel-5.14.0-686.el9.x86_64 2026-02-26T03:04:45.792 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-686.el9.x86_64 2026-02-26T03:04:45.792 DEBUG:teuthology.orchestra.run.trial134:> sudo rpm -qi grub2-tools 2026-02-26T03:04:45.862 INFO:teuthology.orchestra.run.trial134.stdout:Name : grub2-tools 2026-02-26T03:04:45.862 INFO:teuthology.orchestra.run.trial134.stdout:Epoch : 1 2026-02-26T03:04:45.862 INFO:teuthology.orchestra.run.trial134.stdout:Version : 2.06 2026-02-26T03:04:45.862 INFO:teuthology.orchestra.run.trial134.stdout:Release : 120.el9 2026-02-26T03:04:45.862 INFO:teuthology.orchestra.run.trial134.stdout:Architecture: x86_64 2026-02-26T03:04:45.862 INFO:teuthology.orchestra.run.trial134.stdout:Install Date: Wed 18 Feb 2026 05:18:42 PM UTC 2026-02-26T03:04:45.862 INFO:teuthology.orchestra.run.trial134.stdout:Group : Unspecified 2026-02-26T03:04:45.862 INFO:teuthology.orchestra.run.trial134.stdout:Size : 9108611 2026-02-26T03:04:45.862 INFO:teuthology.orchestra.run.trial134.stdout:License : GPLv3+ 2026-02-26T03:04:45.862 INFO:teuthology.orchestra.run.trial134.stdout:Signature : RSA/SHA256, Thu 04 Dec 2025 09:42:59 AM UTC, Key ID 05b555b38483c65d 2026-02-26T03:04:45.862 INFO:teuthology.orchestra.run.trial134.stdout:Source RPM : grub2-2.06-120.el9.src.rpm 2026-02-26T03:04:45.862 INFO:teuthology.orchestra.run.trial134.stdout:Build Date : Wed 03 Dec 2025 12:10:38 PM UTC 2026-02-26T03:04:45.862 INFO:teuthology.orchestra.run.trial134.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2026-02-26T03:04:45.862 INFO:teuthology.orchestra.run.trial134.stdout:Packager : builder@centos.org 2026-02-26T03:04:45.862 INFO:teuthology.orchestra.run.trial134.stdout:Vendor : CentOS 2026-02-26T03:04:45.862 INFO:teuthology.orchestra.run.trial134.stdout:URL : http://www.gnu.org/software/grub/ 2026-02-26T03:04:45.862 INFO:teuthology.orchestra.run.trial134.stdout:Summary : Support tools for GRUB. 2026-02-26T03:04:45.862 INFO:teuthology.orchestra.run.trial134.stdout:Description : 2026-02-26T03:04:45.862 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:04:45.862 INFO:teuthology.orchestra.run.trial134.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2026-02-26T03:04:45.862 INFO:teuthology.orchestra.run.trial134.stdout:customizable bootloader with modular architecture. It supports a rich 2026-02-26T03:04:45.862 INFO:teuthology.orchestra.run.trial134.stdout:variety of kernel formats, file systems, computer architectures and 2026-02-26T03:04:45.862 INFO:teuthology.orchestra.run.trial134.stdout:hardware devices. 2026-02-26T03:04:45.862 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:04:45.862 INFO:teuthology.orchestra.run.trial134.stdout:This subpackage provides tools for support of all platforms. 2026-02-26T03:04:45.863 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2026-02-26T03:04:45.864 INFO:teuthology.task.kernel:Updating grub on trial134 to boot 5.14.0-686.el9.x86_64 2026-02-26T03:04:45.864 DEBUG:teuthology.orchestra.run.trial134:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f -print -quit 2026-02-26T03:04:45.927 INFO:teuthology.orchestra.run.trial134.stdout:/boot/loader/entries/b0373be789df49cdafedf423703d2566-5.14.0-681.el9.x86_64.conf 2026-02-26T03:04:45.929 DEBUG:teuthology.orchestra.run.trial134:> sudo command -v grubby && echo yes || echo no 2026-02-26T03:04:45.992 INFO:teuthology.orchestra.run.trial134.stdout:/sbin/grubby 2026-02-26T03:04:45.994 INFO:teuthology.orchestra.run.trial134.stdout:yes 2026-02-26T03:04:45.994 DEBUG:teuthology.orchestra.run.trial134:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f | grep -F "5.14.0-686.el9.x86_64" | head -n 1 2026-02-26T03:04:46.059 INFO:teuthology.orchestra.run.trial134.stdout:/boot/loader/entries/b0373be789df49cdafedf423703d2566-5.14.0-686.el9.x86_64.conf 2026-02-26T03:04:46.059 DEBUG:teuthology.orchestra.run.trial134:> sudo grub2-set-default b0373be789df49cdafedf423703d2566-5.14.0-686.el9.x86_64 2026-02-26T03:04:46.133 DEBUG:teuthology.orchestra.run.trial134:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2026-02-26T03:04:46.341 INFO:teuthology.orchestra.run.trial134.stderr:Generating grub configuration file ... 2026-02-26T03:04:46.410 INFO:teuthology.orchestra.run.trial134.stderr:Found linux image: /boot/vmlinuz-5.14.0-686.el9.x86_64 2026-02-26T03:04:46.417 INFO:teuthology.orchestra.run.trial134.stderr:Found initrd image: /boot/initramfs-5.14.0-686.el9.x86_64.img 2026-02-26T03:04:46.665 INFO:teuthology.orchestra.run.trial134.stderr:Found linux image: /boot/vmlinuz-5.14.0-681.el9.x86_64 2026-02-26T03:04:46.672 INFO:teuthology.orchestra.run.trial134.stderr:Found initrd image: /boot/initramfs-5.14.0-681.el9.x86_64.img 2026-02-26T03:04:46.706 INFO:teuthology.orchestra.run.trial134.stderr:Found linux image: /boot/vmlinuz-5.14.0-645.el9.x86_64 2026-02-26T03:04:46.713 INFO:teuthology.orchestra.run.trial134.stderr:Found initrd image: /boot/initramfs-5.14.0-645.el9.x86_64.img 2026-02-26T03:04:46.744 INFO:teuthology.orchestra.run.trial134.stderr:Found linux image: /boot/vmlinuz-0-rescue-b0373be789df49cdafedf423703d2566 2026-02-26T03:04:46.750 INFO:teuthology.orchestra.run.trial134.stderr:Found initrd image: /boot/initramfs-0-rescue-b0373be789df49cdafedf423703d2566.img 2026-02-26T03:04:46.773 INFO:teuthology.orchestra.run.trial134.stderr:Found linux image: /boot/vmlinuz-0-rescue-2734456005d743dca7f3dc77c35a2d0d 2026-02-26T03:04:46.780 INFO:teuthology.orchestra.run.trial134.stderr:Found initrd image: /boot/initramfs-0-rescue-2734456005d743dca7f3dc77c35a2d0d.img 2026-02-26T03:04:46.868 INFO:teuthology.orchestra.run.trial134.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2026-02-26T03:04:46.877 INFO:teuthology.orchestra.run.trial134.stderr:done 2026-02-26T03:04:46.879 DEBUG:teuthology.orchestra.run.trial134:> test -d /sys/firmware/efi 2026-02-26T03:04:46.896 DEBUG:teuthology.orchestra.run.trial134:> sudo find /boot/efi/EFI -mindepth 1 -maxdepth 1 -type d || true 2026-02-26T03:04:46.964 INFO:teuthology.orchestra.run.trial134.stdout:/boot/efi/EFI/BOOT 2026-02-26T03:04:46.964 INFO:teuthology.orchestra.run.trial134.stdout:/boot/efi/EFI/centos 2026-02-26T03:04:46.965 DEBUG:teuthology.orchestra.run.trial134:> sudo test -f /boot/efi/EFI/BOOT/grub.cfg && echo yes || echo no 2026-02-26T03:04:47.032 INFO:teuthology.orchestra.run.trial134.stdout:no 2026-02-26T03:04:47.033 DEBUG:teuthology.orchestra.run.trial134:> sudo test -f /boot/efi/EFI/centos/grub.cfg && echo yes || echo no 2026-02-26T03:04:47.096 INFO:teuthology.orchestra.run.trial134.stdout:yes 2026-02-26T03:04:47.096 DEBUG:teuthology.orchestra.run.trial134:> sudo cp -f /boot/grub2/grub.cfg /boot/efi/EFI/centos/grub.cfg 2026-02-26T03:04:47.161 INFO:teuthology.orchestra.remote:Triggering safe hard reboot via SysRq on trial134 2026-02-26T03:04:47.161 DEBUG:teuthology.orchestra.run.trial134:> sudo bash -c ' 2026-02-26T03:04:47.161 DEBUG:teuthology.orchestra.run.trial134:> echo pci > /sys/kernel/reboot/type && echo 1 > /sys/kernel/reboot/force && sync && echo s > /proc/sysrq-trigger && echo u > /proc/sysrq-trigger && sleep 2 && echo b > /proc/sysrq-trigger 2026-02-26T03:04:47.161 DEBUG:teuthology.orchestra.run.trial134:> ' 2026-02-26T03:05:17.204 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2026-02-26T03:05:17.205 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial134.front.sepia.ceph.com' 2026-02-26T03:05:17.206 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-26T03:06:17.239 DEBUG:teuthology.orchestra.remote:timed out 2026-02-26T03:06:26.245 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial134.front.sepia.ceph.com' 2026-02-26T03:06:26.246 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-26T03:06:26.369 DEBUG:teuthology.orchestra.run.trial134:> true 2026-02-26T03:06:26.523 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial134.front.sepia.ceph.com' 2026-02-26T03:06:26.523 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-686.el9.x86_64"... 2026-02-26T03:06:26.524 DEBUG:teuthology.orchestra.run.trial134:> uname -r 2026-02-26T03:06:26.580 INFO:teuthology.orchestra.run.trial134.stdout:5.14.0-686.el9.x86_64 2026-02-26T03:06:26.581 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-686.el9.x86_64 vs 5.14.0-686.el9.x86_64 2026-02-26T03:06:26.581 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2026-02-26T03:06:26.581 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2026-02-26T03:06:27.582 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2026-02-26T03:06:27.582 DEBUG:teuthology.orchestra.run.trial134:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-26T03:06:27.612 INFO:teuthology.orchestra.run.trial134.stdout:ttyS1 2026-02-26T03:06:27.632 DEBUG:teuthology.parallel:result is None 2026-02-26T03:06:27.632 INFO:teuthology.run_tasks:Running task internal.base... 2026-02-26T03:06:27.642 INFO:teuthology.task.internal:Creating test directory... 2026-02-26T03:06:27.642 DEBUG:teuthology.orchestra.run.trial134:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-26T03:06:27.687 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2026-02-26T03:06:27.699 INFO:teuthology.run_tasks:Running task internal.archive... 2026-02-26T03:06:27.712 INFO:teuthology.task.internal:Creating archive directory... 2026-02-26T03:06:27.713 DEBUG:teuthology.orchestra.run.trial134:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-26T03:06:27.750 INFO:teuthology.run_tasks:Running task internal.coredump... 2026-02-26T03:06:27.762 INFO:teuthology.task.internal:Enabling coredump saving... 2026-02-26T03:06:27.763 DEBUG:teuthology.orchestra.run.trial134:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-26T03:06:27.808 INFO:teuthology.orchestra.run.trial134.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-26T03:06:27.817 INFO:teuthology.orchestra.run.trial134.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-26T03:06:27.819 INFO:teuthology.run_tasks:Running task internal.sudo... 2026-02-26T03:06:27.831 INFO:teuthology.task.internal:Configuring sudo... 2026-02-26T03:06:27.831 DEBUG:teuthology.orchestra.run.trial134:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-26T03:06:27.883 INFO:teuthology.run_tasks:Running task internal.syslog... 2026-02-26T03:06:27.901 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2026-02-26T03:06:27.901 DEBUG:teuthology.orchestra.run.trial134:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-26T03:06:27.939 DEBUG:teuthology.orchestra.run.trial134:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-26T03:06:28.075 DEBUG:teuthology.orchestra.run.trial134:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-26T03:06:28.100 DEBUG:teuthology.orchestra.run.trial134:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-26T03:06:28.233 DEBUG:teuthology.orchestra.run.trial134:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-26T03:06:28.258 DEBUG:teuthology.orchestra.run.trial134:> set -ex 2026-02-26T03:06:28.259 DEBUG:teuthology.orchestra.run.trial134:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-26T03:06:28.321 DEBUG:teuthology.orchestra.run.trial134:> sudo service rsyslog restart 2026-02-26T03:06:28.388 INFO:teuthology.orchestra.run.trial134.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-26T03:06:28.541 INFO:teuthology.run_tasks:Running task internal.timer... 2026-02-26T03:06:28.555 INFO:teuthology.task.internal:Starting timer... 2026-02-26T03:06:28.555 INFO:teuthology.run_tasks:Running task pcp... 2026-02-26T03:06:28.577 INFO:teuthology.run_tasks:Running task selinux... 2026-02-26T03:06:28.596 DEBUG:teuthology.orchestra.run.trial134:> sudo service auditd rotate 2026-02-26T03:06:28.625 INFO:teuthology.orchestra.run.trial134.stdout:Rotating logs: 2026-02-26T03:06:28.626 DEBUG:teuthology.task.selinux:Getting current SELinux state 2026-02-26T03:06:28.627 DEBUG:teuthology.orchestra.run.trial134:> /usr/sbin/getenforce 2026-02-26T03:06:28.684 INFO:teuthology.orchestra.run.trial134.stdout:Permissive 2026-02-26T03:06:28.684 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@trial134.front.sepia.ceph.com': 'permissive'} 2026-02-26T03:06:28.685 DEBUG:teuthology.orchestra.run.trial134:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-26T03:06:28.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-26T03:06:28.749 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2026-02-26T03:06:28.750 DEBUG:teuthology.orchestra.run.trial134:> sudo /usr/sbin/setenforce permissive 2026-02-26T03:06:28.813 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2026-02-26T03:06:28.829 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2026-02-26T03:06:28.842 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2026-02-26T03:06:28.843 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit trial134.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2026-02-26T03:09:13.071 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@trial134.front.sepia.ceph.com')] 2026-02-26T03:09:13.072 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial134.front.sepia.ceph.com' 2026-02-26T03:09:13.074 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-26T03:09:13.185 DEBUG:teuthology.orchestra.run.trial134:> true 2026-02-26T03:09:13.262 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial134.front.sepia.ceph.com' 2026-02-26T03:09:13.262 INFO:teuthology.run_tasks:Running task clock... 2026-02-26T03:09:13.277 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2026-02-26T03:09:13.277 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-26T03:09:13.278 DEBUG:teuthology.orchestra.run.trial134:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-26T03:09:13.328 INFO:teuthology.orchestra.run.trial134.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2026-02-26T03:09:13.340 INFO:teuthology.orchestra.run.trial134.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2026-02-26T03:09:13.359 INFO:teuthology.orchestra.run.trial134.stderr:sudo: ntpd: command not found 2026-02-26T03:09:13.368 INFO:teuthology.orchestra.run.trial134.stdout:506 Cannot talk to daemon 2026-02-26T03:09:13.380 INFO:teuthology.orchestra.run.trial134.stderr:Failed to start ntp.service: Unit ntp.service not found. 2026-02-26T03:09:13.391 INFO:teuthology.orchestra.run.trial134.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2026-02-26T03:09:13.438 INFO:teuthology.orchestra.run.trial134.stderr:bash: line 1: ntpq: command not found 2026-02-26T03:09:13.439 INFO:teuthology.orchestra.run.trial134.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-26T03:09:13.439 INFO:teuthology.orchestra.run.trial134.stdout:=============================================================================== 2026-02-26T03:09:13.440 INFO:teuthology.run_tasks:Running task install... 2026-02-26T03:09:13.451 DEBUG:teuthology.task.install:project ceph 2026-02-26T03:09:13.451 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'debug', 'sha1': 'abffb617683951511fccf3c3cab6e219f2ec83a8'}} 2026-02-26T03:09:13.451 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'flavor': 'debug', 'sha1': 'abffb617683951511fccf3c3cab6e219f2ec83a8'} 2026-02-26T03:09:13.451 INFO:teuthology.task.install:Using flavor: debug 2026-02-26T03:09:13.461 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-osd', 'ceph-osd-classic', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-osd', 'ceph-osd-classic', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-26T03:09:13.461 INFO:teuthology.task.install:extra packages: {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']} 2026-02-26T03:09:13.462 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'debug', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'abffb617683951511fccf3c3cab6e219f2ec83a8', 'tag': None, 'wait_for_package': False} 2026-02-26T03:09:13.462 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=abffb617683951511fccf3c3cab6e219f2ec83a8 2026-02-26T03:09:13.620 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-shraddhaag-peering-bug/abffb617683951511fccf3c3cab6e219f2ec83a8/centos/9/flavors/debug/ 2026-02-26T03:09:13.620 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5312.gabffb617 2026-02-26T03:09:13.983 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-shraddhaag-peering-bug/abffb617683951511fccf3c3cab6e219f2ec83a8/centos/9/flavors/debug/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-shraddhaag-peering-bug/abffb617683951511fccf3c3cab6e219f2ec83a8/centos/9/flavors/debug/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-shraddhaag-peering-bug/abffb617683951511fccf3c3cab6e219f2ec83a8/centos/9/flavors/debug/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-26T03:09:13.983 DEBUG:teuthology.orchestra.run.trial134:> set -ex 2026-02-26T03:09:13.983 DEBUG:teuthology.orchestra.run.trial134:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-26T03:09:14.009 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-osd-crimson, ceph-osd-crimson-debuginfo on remote rpm x86_64 2026-02-26T03:09:14.009 DEBUG:teuthology.orchestra.run.trial134:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-26T03:09:14.075 DEBUG:teuthology.orchestra.run.trial134:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-26T03:09:14.149 DEBUG:teuthology.orchestra.run.trial134:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-26T03:09:14.214 INFO:teuthology.orchestra.run.trial134.stdout:check_obsoletes = 1 2026-02-26T03:09:14.216 DEBUG:teuthology.orchestra.run.trial134:> sudo yum clean all 2026-02-26T03:09:14.364 INFO:teuthology.orchestra.run.trial134.stdout:41 files removed 2026-02-26T03:09:14.378 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install ceph-radosgw 2026-02-26T03:09:15.173 INFO:teuthology.orchestra.run.trial134.stdout:ceph packages for x86_64 145 kB/s | 94 kB 00:00 2026-02-26T03:09:15.768 INFO:teuthology.orchestra.run.trial134.stdout:ceph noarch packages 43 kB/s | 24 kB 00:00 2026-02-26T03:09:16.277 INFO:teuthology.orchestra.run.trial134.stdout:ceph source packages 4.3 kB/s | 2.1 kB 00:00 2026-02-26T03:09:24.637 INFO:teuthology.orchestra.run.trial134.stdout:CentOS Stream 9 - BaseOS 1.1 MB/s | 8.9 MB 00:08 2026-02-26T03:09:27.709 INFO:teuthology.orchestra.run.trial134.stdout:CentOS Stream 9 - AppStream 11 MB/s | 27 MB 00:02 2026-02-26T03:09:31.351 INFO:teuthology.orchestra.run.trial134.stdout:CentOS Stream 9 - CRB 7.4 MB/s | 8.0 MB 00:01 2026-02-26T03:09:33.210 INFO:teuthology.orchestra.run.trial134.stdout:CentOS Stream 9 - Extras packages 18 kB/s | 20 kB 00:01 2026-02-26T03:09:34.244 INFO:teuthology.orchestra.run.trial134.stdout:Extra Packages for Enterprise Linux 21 MB/s | 20 MB 00:00 2026-02-26T03:09:37.523 INFO:teuthology.orchestra.run.trial134.stdout:lab-extras 2.5 MB/s | 50 kB 00:00 2026-02-26T03:09:38.750 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-26T03:09:38.752 INFO:teuthology.orchestra.run.trial134.stdout:================================================================================ 2026-02-26T03:09:38.752 INFO:teuthology.orchestra.run.trial134.stdout: Package Arch Version Repository Size 2026-02-26T03:09:38.752 INFO:teuthology.orchestra.run.trial134.stdout:================================================================================ 2026-02-26T03:09:38.752 INFO:teuthology.orchestra.run.trial134.stdout:Installing: 2026-02-26T03:09:38.752 INFO:teuthology.orchestra.run.trial134.stdout: ceph-radosgw x86_64 2:20.3.0-5312.gabffb617.el9 ceph 11 M 2026-02-26T03:09:38.752 INFO:teuthology.orchestra.run.trial134.stdout:Upgrading: 2026-02-26T03:09:38.752 INFO:teuthology.orchestra.run.trial134.stdout: librados2 x86_64 2:20.3.0-5312.gabffb617.el9 ceph 3.3 M 2026-02-26T03:09:38.752 INFO:teuthology.orchestra.run.trial134.stdout: librbd1 x86_64 2:20.3.0-5312.gabffb617.el9 ceph 2.7 M 2026-02-26T03:09:38.752 INFO:teuthology.orchestra.run.trial134.stdout:Installing dependencies: 2026-02-26T03:09:38.752 INFO:teuthology.orchestra.run.trial134.stdout: boost-program-options x86_64 1.75.0-13.el9 appstream 104 k 2026-02-26T03:09:38.752 INFO:teuthology.orchestra.run.trial134.stdout: ceph-base x86_64 2:20.3.0-5312.gabffb617.el9 ceph 5.1 M 2026-02-26T03:09:38.752 INFO:teuthology.orchestra.run.trial134.stdout: ceph-common x86_64 2:20.3.0-5312.gabffb617.el9 ceph 21 M 2026-02-26T03:09:38.752 INFO:teuthology.orchestra.run.trial134.stdout: ceph-selinux x86_64 2:20.3.0-5312.gabffb617.el9 ceph 25 k 2026-02-26T03:09:38.752 INFO:teuthology.orchestra.run.trial134.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2026-02-26T03:09:38.752 INFO:teuthology.orchestra.run.trial134.stdout: libarrow x86_64 9.0.0-15.el9 epel 4.4 M 2026-02-26T03:09:38.752 INFO:teuthology.orchestra.run.trial134.stdout: libarrow-doc noarch 9.0.0-15.el9 epel 25 k 2026-02-26T03:09:38.752 INFO:teuthology.orchestra.run.trial134.stdout: libcephfs2 x86_64 2:20.3.0-5312.gabffb617.el9 ceph 859 k 2026-02-26T03:09:38.752 INFO:teuthology.orchestra.run.trial134.stdout: libnbd x86_64 1.20.3-4.el9 appstream 164 k 2026-02-26T03:09:38.752 INFO:teuthology.orchestra.run.trial134.stdout: liboath x86_64 2.6.12-1.el9 epel 49 k 2026-02-26T03:09:38.752 INFO:teuthology.orchestra.run.trial134.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2026-02-26T03:09:38.752 INFO:teuthology.orchestra.run.trial134.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2026-02-26T03:09:38.752 INFO:teuthology.orchestra.run.trial134.stdout: libradosstriper1 x86_64 2:20.3.0-5312.gabffb617.el9 ceph 263 k 2026-02-26T03:09:38.752 INFO:teuthology.orchestra.run.trial134.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2026-02-26T03:09:38.753 INFO:teuthology.orchestra.run.trial134.stdout: librgw2 x86_64 2:20.3.0-5312.gabffb617.el9 ceph 5.6 M 2026-02-26T03:09:38.753 INFO:teuthology.orchestra.run.trial134.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2026-02-26T03:09:38.753 INFO:teuthology.orchestra.run.trial134.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2026-02-26T03:09:38.753 INFO:teuthology.orchestra.run.trial134.stdout: mailcap noarch 2.1.49-5.el9 baseos 33 k 2026-02-26T03:09:38.753 INFO:teuthology.orchestra.run.trial134.stdout: parquet-libs x86_64 9.0.0-15.el9 epel 838 k 2026-02-26T03:09:38.753 INFO:teuthology.orchestra.run.trial134.stdout: pciutils x86_64 3.7.0-7.el9 baseos 93 k 2026-02-26T03:09:38.753 INFO:teuthology.orchestra.run.trial134.stdout: python3-ceph-argparse x86_64 2:20.3.0-5312.gabffb617.el9 ceph 45 k 2026-02-26T03:09:38.753 INFO:teuthology.orchestra.run.trial134.stdout: python3-ceph-common x86_64 2:20.3.0-5312.gabffb617.el9 ceph 185 k 2026-02-26T03:09:38.753 INFO:teuthology.orchestra.run.trial134.stdout: python3-cephfs x86_64 2:20.3.0-5312.gabffb617.el9 ceph 203 k 2026-02-26T03:09:38.753 INFO:teuthology.orchestra.run.trial134.stdout: python3-rados x86_64 2:20.3.0-5312.gabffb617.el9 ceph 324 k 2026-02-26T03:09:38.753 INFO:teuthology.orchestra.run.trial134.stdout: python3-rbd x86_64 2:20.3.0-5312.gabffb617.el9 ceph 305 k 2026-02-26T03:09:38.753 INFO:teuthology.orchestra.run.trial134.stdout: python3-rgw x86_64 2:20.3.0-5312.gabffb617.el9 ceph 100 k 2026-02-26T03:09:38.753 INFO:teuthology.orchestra.run.trial134.stdout: qatlib x86_64 25.08.0-2.el9 appstream 240 k 2026-02-26T03:09:38.753 INFO:teuthology.orchestra.run.trial134.stdout: qatzip-libs x86_64 1.3.1-1.el9 appstream 66 k 2026-02-26T03:09:38.753 INFO:teuthology.orchestra.run.trial134.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2026-02-26T03:09:38.753 INFO:teuthology.orchestra.run.trial134.stdout: thrift x86_64 0.15.0-4.el9 epel 1.6 M 2026-02-26T03:09:38.753 INFO:teuthology.orchestra.run.trial134.stdout:Installing weak dependencies: 2026-02-26T03:09:38.753 INFO:teuthology.orchestra.run.trial134.stdout: libcephfs-daemon x86_64 2:20.3.0-5312.gabffb617.el9 ceph 37 k 2026-02-26T03:09:38.753 INFO:teuthology.orchestra.run.trial134.stdout: libcephfs-proxy2 x86_64 2:20.3.0-5312.gabffb617.el9 ceph 26 k 2026-02-26T03:09:38.753 INFO:teuthology.orchestra.run.trial134.stdout: nvme-cli x86_64 2.16-1.el9 baseos 1.2 M 2026-02-26T03:09:38.753 INFO:teuthology.orchestra.run.trial134.stdout: qatlib-service x86_64 25.08.0-2.el9 appstream 37 k 2026-02-26T03:09:38.753 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:09:38.753 INFO:teuthology.orchestra.run.trial134.stdout:Transaction Summary 2026-02-26T03:09:38.753 INFO:teuthology.orchestra.run.trial134.stdout:================================================================================ 2026-02-26T03:09:38.753 INFO:teuthology.orchestra.run.trial134.stdout:Install 35 Packages 2026-02-26T03:09:38.753 INFO:teuthology.orchestra.run.trial134.stdout:Upgrade 2 Packages 2026-02-26T03:09:38.753 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:09:38.753 INFO:teuthology.orchestra.run.trial134.stdout:Total download size: 61 M 2026-02-26T03:09:38.753 INFO:teuthology.orchestra.run.trial134.stdout:Downloading Packages: 2026-02-26T03:09:41.224 INFO:teuthology.orchestra.run.trial134.stdout:(1/37): ceph-base-20.3.0-5312.gabffb617.el9.x86 2.2 MB/s | 5.1 MB 00:02 2026-02-26T03:09:41.288 INFO:teuthology.orchestra.run.trial134.stdout:(2/37): ceph-selinux-20.3.0-5312.gabffb617.el9. 387 kB/s | 25 kB 00:00 2026-02-26T03:09:41.355 INFO:teuthology.orchestra.run.trial134.stdout:(3/37): libcephfs-daemon-20.3.0-5312.gabffb617. 556 kB/s | 37 kB 00:00 2026-02-26T03:09:41.421 INFO:teuthology.orchestra.run.trial134.stdout:(4/37): libcephfs-proxy2-20.3.0-5312.gabffb617. 387 kB/s | 26 kB 00:00 2026-02-26T03:09:41.767 INFO:teuthology.orchestra.run.trial134.stdout:(5/37): libcephfs2-20.3.0-5312.gabffb617.el9.x8 2.4 MB/s | 859 kB 00:00 2026-02-26T03:09:41.907 INFO:teuthology.orchestra.run.trial134.stdout:(6/37): libradosstriper1-20.3.0-5312.gabffb617. 1.8 MB/s | 263 kB 00:00 2026-02-26T03:09:42.037 INFO:teuthology.orchestra.run.trial134.stdout:(7/37): ceph-radosgw-20.3.0-5312.gabffb617.el9. 3.6 MB/s | 11 MB 00:03 2026-02-26T03:09:42.099 INFO:teuthology.orchestra.run.trial134.stdout:(8/37): python3-ceph-argparse-20.3.0-5312.gabff 730 kB/s | 45 kB 00:00 2026-02-26T03:09:42.163 INFO:teuthology.orchestra.run.trial134.stdout:(9/37): python3-ceph-common-20.3.0-5312.gabffb6 2.8 MB/s | 185 kB 00:00 2026-02-26T03:09:42.227 INFO:teuthology.orchestra.run.trial134.stdout:(10/37): python3-cephfs-20.3.0-5312.gabffb617.e 3.1 MB/s | 203 kB 00:00 2026-02-26T03:09:42.292 INFO:teuthology.orchestra.run.trial134.stdout:(11/37): python3-rados-20.3.0-5312.gabffb617.el 4.9 MB/s | 324 kB 00:00 2026-02-26T03:09:42.355 INFO:teuthology.orchestra.run.trial134.stdout:(12/37): python3-rbd-20.3.0-5312.gabffb617.el9. 4.8 MB/s | 305 kB 00:00 2026-02-26T03:09:42.414 INFO:teuthology.orchestra.run.trial134.stdout:(13/37): python3-rgw-20.3.0-5312.gabffb617.el9. 1.6 MB/s | 100 kB 00:00 2026-02-26T03:09:43.689 INFO:teuthology.orchestra.run.trial134.stdout:(14/37): librgw2-20.3.0-5312.gabffb617.el9.x86_ 3.2 MB/s | 5.6 MB 00:01 2026-02-26T03:09:48.710 INFO:teuthology.orchestra.run.trial134.stdout:(15/37): ceph-common-20.3.0-5312.gabffb617.el9. 2.1 MB/s | 21 MB 00:09 2026-02-26T03:10:15.478 INFO:teuthology.orchestra.run.trial134.stdout:(16/37): mailcap-2.1.49-5.el9.noarch.rpm 1.0 kB/s | 33 kB 00:33 2026-02-26T03:10:15.882 INFO:teuthology.orchestra.run.trial134.stdout:(17/37): boost-program-options-1.75.0-13.el9.x8 258 kB/s | 104 kB 00:00 2026-02-26T03:10:16.062 INFO:teuthology.orchestra.run.trial134.stdout:(18/37): libnbd-1.20.3-4.el9.x86_64.rpm 910 kB/s | 164 kB 00:00 2026-02-26T03:10:16.176 INFO:teuthology.orchestra.run.trial134.stdout:(19/37): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.4 MB/s | 160 kB 00:00 2026-02-26T03:10:16.264 INFO:teuthology.orchestra.run.trial134.stdout:(20/37): librabbitmq-0.11.0-7.el9.x86_64.rpm 517 kB/s | 45 kB 00:00 2026-02-26T03:10:16.488 INFO:teuthology.orchestra.run.trial134.stdout:(21/37): librdkafka-1.6.1-102.el9.x86_64.rpm 2.9 MB/s | 662 kB 00:00 2026-02-26T03:10:16.587 INFO:teuthology.orchestra.run.trial134.stdout:(22/37): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.9 MB/s | 292 kB 00:00 2026-02-26T03:10:16.683 INFO:teuthology.orchestra.run.trial134.stdout:(23/37): qatlib-25.08.0-2.el9.x86_64.rpm 2.5 MB/s | 240 kB 00:00 2026-02-26T03:10:16.762 INFO:teuthology.orchestra.run.trial134.stdout:(24/37): qatlib-service-25.08.0-2.el9.x86_64.rp 466 kB/s | 37 kB 00:00 2026-02-26T03:10:16.845 INFO:teuthology.orchestra.run.trial134.stdout:(25/37): qatzip-libs-1.3.1-1.el9.x86_64.rpm 801 kB/s | 66 kB 00:00 2026-02-26T03:10:17.268 INFO:teuthology.orchestra.run.trial134.stdout:(26/37): gperftools-libs-2.9.1-3.el9.x86_64.rpm 727 kB/s | 308 kB 00:00 2026-02-26T03:10:17.593 INFO:teuthology.orchestra.run.trial134.stdout:(27/37): libarrow-9.0.0-15.el9.x86_64.rpm 14 MB/s | 4.4 MB 00:00 2026-02-26T03:10:17.655 INFO:teuthology.orchestra.run.trial134.stdout:(28/37): libarrow-doc-9.0.0-15.el9.noarch.rpm 401 kB/s | 25 kB 00:00 2026-02-26T03:10:17.717 INFO:teuthology.orchestra.run.trial134.stdout:(29/37): liboath-2.6.12-1.el9.x86_64.rpm 790 kB/s | 49 kB 00:00 2026-02-26T03:10:17.779 INFO:teuthology.orchestra.run.trial134.stdout:(30/37): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2026-02-26T03:10:17.848 INFO:teuthology.orchestra.run.trial134.stdout:(31/37): parquet-libs-9.0.0-15.el9.x86_64.rpm 12 MB/s | 838 kB 00:00 2026-02-26T03:10:17.911 INFO:teuthology.orchestra.run.trial134.stdout:(32/37): re2-20211101-20.el9.x86_64.rpm 3.0 MB/s | 191 kB 00:00 2026-02-26T03:10:17.988 INFO:teuthology.orchestra.run.trial134.stdout:(33/37): thrift-0.15.0-4.el9.x86_64.rpm 21 MB/s | 1.6 MB 00:00 2026-02-26T03:10:19.694 INFO:teuthology.orchestra.run.trial134.stdout:(34/37): librados2-20.3.0-5312.gabffb617.el9.x8 1.9 MB/s | 3.3 MB 00:01 2026-02-26T03:10:21.730 INFO:teuthology.orchestra.run.trial134.stdout:(35/37): librbd1-20.3.0-5312.gabffb617.el9.x86_ 1.3 MB/s | 2.7 MB 00:02 2026-02-26T03:10:51.984 INFO:teuthology.orchestra.run.trial134.stdout:(36/37): nvme-cli-2.16-1.el9.x86_64.rpm 17 kB/s | 1.2 MB 01:08 2026-02-26T03:11:58.821 INFO:teuthology.orchestra.run.trial134.stdout:[MIRROR] pciutils-3.7.0-7.el9.x86_64.rpm: Curl error (56): Failure when receiving data from the peer for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/pciutils-3.7.0-7.el9.x86_64.rpm [Recv failure: Connection reset by peer] 2026-02-26T03:12:08.107 INFO:teuthology.orchestra.run.trial134.stdout:(37/37): pciutils-3.7.0-7.el9.x86_64.rpm 684 B/s | 93 kB 02:19 2026-02-26T03:12:08.107 INFO:teuthology.orchestra.run.trial134.stdout:-------------------------------------------------------------------------------- 2026-02-26T03:12:08.108 INFO:teuthology.orchestra.run.trial134.stdout:Total 421 kB/s | 61 MB 02:29 2026-02-26T03:12:08.192 INFO:teuthology.orchestra.run.trial134.stdout:Running transaction check 2026-02-26T03:12:08.211 INFO:teuthology.orchestra.run.trial134.stdout:Transaction check succeeded. 2026-02-26T03:12:08.211 INFO:teuthology.orchestra.run.trial134.stdout:Running transaction test 2026-02-26T03:12:08.456 INFO:teuthology.orchestra.run.trial134.stdout:Transaction test succeeded. 2026-02-26T03:12:08.457 INFO:teuthology.orchestra.run.trial134.stdout:Running transaction 2026-02-26T03:12:08.755 INFO:teuthology.orchestra.run.trial134.stdout: Preparing : 1/1 2026-02-26T03:12:08.892 INFO:teuthology.orchestra.run.trial134.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/39 2026-02-26T03:12:08.897 INFO:teuthology.orchestra.run.trial134.stdout: Upgrading : librados2-2:20.3.0-5312.gabffb617.el9.x86_64 2/39 2026-02-26T03:12:08.943 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: librados2-2:20.3.0-5312.gabffb617.el9.x86_64 2/39 2026-02-26T03:12:08.944 INFO:teuthology.orchestra.run.trial134.stdout: Installing : libcephfs2-2:20.3.0-5312.gabffb617.el9.x86_64 3/39 2026-02-26T03:12:08.956 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: libcephfs2-2:20.3.0-5312.gabffb617.el9.x86_64 3/39 2026-02-26T03:12:08.958 INFO:teuthology.orchestra.run.trial134.stdout: Installing : libcephfs-daemon-2:20.3.0-5312.gabffb617.el9.x86_6 4/39 2026-02-26T03:12:08.959 INFO:teuthology.orchestra.run.trial134.stdout: Installing : libcephfs-proxy2-2:20.3.0-5312.gabffb617.el9.x86_6 5/39 2026-02-26T03:12:08.984 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: libcephfs-proxy2-2:20.3.0-5312.gabffb617.el9.x86_6 5/39 2026-02-26T03:12:08.987 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-rados-2:20.3.0-5312.gabffb617.el9.x86_64 6/39 2026-02-26T03:12:08.994 INFO:teuthology.orchestra.run.trial134.stdout: Installing : liboath-2.6.12-1.el9.x86_64 7/39 2026-02-26T03:12:08.997 INFO:teuthology.orchestra.run.trial134.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/39 2026-02-26T03:12:09.000 INFO:teuthology.orchestra.run.trial134.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/39 2026-02-26T03:12:09.004 INFO:teuthology.orchestra.run.trial134.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/39 2026-02-26T03:12:09.015 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-ceph-argparse-2:20.3.0-5312.gabffb617.el9. 11/39 2026-02-26T03:12:09.028 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-cephfs-2:20.3.0-5312.gabffb617.el9.x86_64 12/39 2026-02-26T03:12:09.029 INFO:teuthology.orchestra.run.trial134.stdout: Installing : libradosstriper1-2:20.3.0-5312.gabffb617.el9.x86_6 13/39 2026-02-26T03:12:09.051 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: libradosstriper1-2:20.3.0-5312.gabffb617.el9.x86_6 13/39 2026-02-26T03:12:09.055 INFO:teuthology.orchestra.run.trial134.stdout: Installing : thrift-0.15.0-4.el9.x86_64 14/39 2026-02-26T03:12:09.086 INFO:teuthology.orchestra.run.trial134.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/39 2026-02-26T03:12:09.089 INFO:teuthology.orchestra.run.trial134.stdout: Installing : libarrow-9.0.0-15.el9.x86_64 16/39 2026-02-26T03:12:09.097 INFO:teuthology.orchestra.run.trial134.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/39 2026-02-26T03:12:09.099 INFO:teuthology.orchestra.run.trial134.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/39 2026-02-26T03:12:09.104 INFO:teuthology.orchestra.run.trial134.stdout: Installing : libarrow-doc-9.0.0-15.el9.noarch 19/39 2026-02-26T03:12:09.217 INFO:teuthology.orchestra.run.trial134.stdout: Installing : libnbd-1.20.3-4.el9.x86_64 20/39 2026-02-26T03:12:09.218 INFO:teuthology.orchestra.run.trial134.stdout: Upgrading : librbd1-2:20.3.0-5312.gabffb617.el9.x86_64 21/39 2026-02-26T03:12:09.243 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: librbd1-2:20.3.0-5312.gabffb617.el9.x86_64 21/39 2026-02-26T03:12:09.245 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-rbd-2:20.3.0-5312.gabffb617.el9.x86_64 22/39 2026-02-26T03:12:09.252 INFO:teuthology.orchestra.run.trial134.stdout: Installing : boost-program-options-1.75.0-13.el9.x86_64 23/39 2026-02-26T03:12:09.499 INFO:teuthology.orchestra.run.trial134.stdout: Installing : parquet-libs-9.0.0-15.el9.x86_64 24/39 2026-02-26T03:12:09.500 INFO:teuthology.orchestra.run.trial134.stdout: Installing : librgw2-2:20.3.0-5312.gabffb617.el9.x86_64 25/39 2026-02-26T03:12:09.514 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: librgw2-2:20.3.0-5312.gabffb617.el9.x86_64 25/39 2026-02-26T03:12:09.518 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-rgw-2:20.3.0-5312.gabffb617.el9.x86_64 26/39 2026-02-26T03:12:09.523 INFO:teuthology.orchestra.run.trial134.stdout: Installing : pciutils-3.7.0-7.el9.x86_64 27/39 2026-02-26T03:12:09.526 INFO:teuthology.orchestra.run.trial134.stdout: Installing : qatlib-25.08.0-2.el9.x86_64 28/39 2026-02-26T03:12:09.526 INFO:teuthology.orchestra.run.trial134.stdout: Installing : qatlib-service-25.08.0-2.el9.x86_64 29/39 2026-02-26T03:12:09.542 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: qatlib-service-25.08.0-2.el9.x86_64 29/39 2026-02-26T03:12:09.654 INFO:teuthology.orchestra.run.trial134.stdout: Installing : qatzip-libs-1.3.1-1.el9.x86_64 30/39 2026-02-26T03:12:09.657 INFO:teuthology.orchestra.run.trial134.stdout: Installing : nvme-cli-2.16-1.el9.x86_64 31/39 2026-02-26T03:12:09.935 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: nvme-cli-2.16-1.el9.x86_64 31/39 2026-02-26T03:12:09.935 INFO:teuthology.orchestra.run.trial134.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service → /usr/lib/systemd/system/nvmefc-boot-connections.service. 2026-02-26T03:12:09.935 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:12:09.978 INFO:teuthology.orchestra.run.trial134.stdout: Installing : mailcap-2.1.49-5.el9.noarch 32/39 2026-02-26T03:12:09.980 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-ceph-common-2:20.3.0-5312.gabffb617.el9.x8 33/39 2026-02-26T03:12:10.920 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: ceph-common-2:20.3.0-5312.gabffb617.el9.x86_64 34/39 2026-02-26T03:12:10.923 INFO:teuthology.orchestra.run.trial134.stdout: Installing : ceph-common-2:20.3.0-5312.gabffb617.el9.x86_64 34/39 2026-02-26T03:12:11.180 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: ceph-common-2:20.3.0-5312.gabffb617.el9.x86_64 34/39 2026-02-26T03:12:11.184 INFO:teuthology.orchestra.run.trial134.stdout: Installing : ceph-base-2:20.3.0-5312.gabffb617.el9.x86_64 35/39 2026-02-26T03:12:11.223 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: ceph-base-2:20.3.0-5312.gabffb617.el9.x86_64 35/39 2026-02-26T03:12:11.224 INFO:teuthology.orchestra.run.trial134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2026-02-26T03:12:11.224 INFO:teuthology.orchestra.run.trial134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2026-02-26T03:12:11.224 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:12:11.227 INFO:teuthology.orchestra.run.trial134.stdout: Installing : ceph-selinux-2:20.3.0-5312.gabffb617.el9.x86_64 36/39 2026-02-26T03:12:15.658 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: ceph-selinux-2:20.3.0-5312.gabffb617.el9.x86_64 36/39 2026-02-26T03:12:15.658 INFO:teuthology.orchestra.run.trial134.stdout:skipping the directory /sys 2026-02-26T03:12:15.658 INFO:teuthology.orchestra.run.trial134.stdout:skipping the directory /proc 2026-02-26T03:12:15.658 INFO:teuthology.orchestra.run.trial134.stdout:skipping the directory /mnt 2026-02-26T03:12:15.659 INFO:teuthology.orchestra.run.trial134.stdout:skipping the directory /var/tmp 2026-02-26T03:12:15.659 INFO:teuthology.orchestra.run.trial134.stdout:skipping the directory /home 2026-02-26T03:12:15.659 INFO:teuthology.orchestra.run.trial134.stdout:skipping the directory /root 2026-02-26T03:12:15.659 INFO:teuthology.orchestra.run.trial134.stdout:skipping the directory /tmp 2026-02-26T03:12:15.659 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:12:16.148 INFO:teuthology.orchestra.run.trial134.stdout: Installing : ceph-radosgw-2:20.3.0-5312.gabffb617.el9.x86_64 37/39 2026-02-26T03:12:16.171 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: ceph-radosgw-2:20.3.0-5312.gabffb617.el9.x86_64 37/39 2026-02-26T03:12:16.171 INFO:teuthology.orchestra.run.trial134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-26T03:12:16.171 INFO:teuthology.orchestra.run.trial134.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2026-02-26T03:12:16.171 INFO:teuthology.orchestra.run.trial134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-26T03:12:16.171 INFO:teuthology.orchestra.run.trial134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-26T03:12:16.171 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:12:16.173 INFO:teuthology.orchestra.run.trial134.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 38/39 2026-02-26T03:12:16.185 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 38/39 2026-02-26T03:12:16.185 INFO:teuthology.orchestra.run.trial134.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 39/39 2026-02-26T03:12:16.237 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 39/39 2026-02-26T03:12:16.237 INFO:teuthology.orchestra.run.trial134.stdout:Creating group 'qat' with GID 992. 2026-02-26T03:12:16.237 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:12:16.524 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : ceph-base-2:20.3.0-5312.gabffb617.el9.x86_64 1/39 2026-02-26T03:12:16.524 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : ceph-common-2:20.3.0-5312.gabffb617.el9.x86_64 2/39 2026-02-26T03:12:16.524 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : ceph-radosgw-2:20.3.0-5312.gabffb617.el9.x86_64 3/39 2026-02-26T03:12:16.524 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : ceph-selinux-2:20.3.0-5312.gabffb617.el9.x86_64 4/39 2026-02-26T03:12:16.524 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : libcephfs-daemon-2:20.3.0-5312.gabffb617.el9.x86_6 5/39 2026-02-26T03:12:16.524 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : libcephfs-proxy2-2:20.3.0-5312.gabffb617.el9.x86_6 6/39 2026-02-26T03:12:16.524 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : libcephfs2-2:20.3.0-5312.gabffb617.el9.x86_64 7/39 2026-02-26T03:12:16.524 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : libradosstriper1-2:20.3.0-5312.gabffb617.el9.x86_6 8/39 2026-02-26T03:12:16.524 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : librgw2-2:20.3.0-5312.gabffb617.el9.x86_64 9/39 2026-02-26T03:12:16.524 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-ceph-argparse-2:20.3.0-5312.gabffb617.el9. 10/39 2026-02-26T03:12:16.524 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-ceph-common-2:20.3.0-5312.gabffb617.el9.x8 11/39 2026-02-26T03:12:16.524 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-cephfs-2:20.3.0-5312.gabffb617.el9.x86_64 12/39 2026-02-26T03:12:16.524 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-rados-2:20.3.0-5312.gabffb617.el9.x86_64 13/39 2026-02-26T03:12:16.524 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-rbd-2:20.3.0-5312.gabffb617.el9.x86_64 14/39 2026-02-26T03:12:16.524 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-rgw-2:20.3.0-5312.gabffb617.el9.x86_64 15/39 2026-02-26T03:12:16.524 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : mailcap-2.1.49-5.el9.noarch 16/39 2026-02-26T03:12:16.524 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : nvme-cli-2.16-1.el9.x86_64 17/39 2026-02-26T03:12:16.524 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : pciutils-3.7.0-7.el9.x86_64 18/39 2026-02-26T03:12:16.524 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : boost-program-options-1.75.0-13.el9.x86_64 19/39 2026-02-26T03:12:16.524 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : libnbd-1.20.3-4.el9.x86_64 20/39 2026-02-26T03:12:16.524 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 21/39 2026-02-26T03:12:16.524 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 22/39 2026-02-26T03:12:16.524 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 23/39 2026-02-26T03:12:16.524 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 24/39 2026-02-26T03:12:16.524 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : qatlib-25.08.0-2.el9.x86_64 25/39 2026-02-26T03:12:16.524 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : qatlib-service-25.08.0-2.el9.x86_64 26/39 2026-02-26T03:12:16.525 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : qatzip-libs-1.3.1-1.el9.x86_64 27/39 2026-02-26T03:12:16.525 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 28/39 2026-02-26T03:12:16.525 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : libarrow-9.0.0-15.el9.x86_64 29/39 2026-02-26T03:12:16.525 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : libarrow-doc-9.0.0-15.el9.noarch 30/39 2026-02-26T03:12:16.525 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : liboath-2.6.12-1.el9.x86_64 31/39 2026-02-26T03:12:16.525 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 32/39 2026-02-26T03:12:16.525 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : parquet-libs-9.0.0-15.el9.x86_64 33/39 2026-02-26T03:12:16.525 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : re2-1:20211101-20.el9.x86_64 34/39 2026-02-26T03:12:16.525 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : thrift-0.15.0-4.el9.x86_64 35/39 2026-02-26T03:12:16.525 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : librados2-2:20.3.0-5312.gabffb617.el9.x86_64 36/39 2026-02-26T03:12:16.525 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 37/39 2026-02-26T03:12:16.525 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : librbd1-2:20.3.0-5312.gabffb617.el9.x86_64 38/39 2026-02-26T03:12:16.592 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 39/39 2026-02-26T03:12:16.593 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:12:16.593 INFO:teuthology.orchestra.run.trial134.stdout:Upgraded: 2026-02-26T03:12:16.593 INFO:teuthology.orchestra.run.trial134.stdout: librados2-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:12:16.593 INFO:teuthology.orchestra.run.trial134.stdout: librbd1-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:12:16.593 INFO:teuthology.orchestra.run.trial134.stdout:Installed: 2026-02-26T03:12:16.593 INFO:teuthology.orchestra.run.trial134.stdout: boost-program-options-1.75.0-13.el9.x86_64 2026-02-26T03:12:16.593 INFO:teuthology.orchestra.run.trial134.stdout: ceph-base-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:12:16.593 INFO:teuthology.orchestra.run.trial134.stdout: ceph-common-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:12:16.593 INFO:teuthology.orchestra.run.trial134.stdout: ceph-radosgw-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:12:16.593 INFO:teuthology.orchestra.run.trial134.stdout: ceph-selinux-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:12:16.593 INFO:teuthology.orchestra.run.trial134.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2026-02-26T03:12:16.593 INFO:teuthology.orchestra.run.trial134.stdout: libarrow-9.0.0-15.el9.x86_64 2026-02-26T03:12:16.593 INFO:teuthology.orchestra.run.trial134.stdout: libarrow-doc-9.0.0-15.el9.noarch 2026-02-26T03:12:16.593 INFO:teuthology.orchestra.run.trial134.stdout: libcephfs-daemon-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:12:16.593 INFO:teuthology.orchestra.run.trial134.stdout: libcephfs-proxy2-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:12:16.593 INFO:teuthology.orchestra.run.trial134.stdout: libcephfs2-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:12:16.593 INFO:teuthology.orchestra.run.trial134.stdout: libnbd-1.20.3-4.el9.x86_64 2026-02-26T03:12:16.593 INFO:teuthology.orchestra.run.trial134.stdout: liboath-2.6.12-1.el9.x86_64 2026-02-26T03:12:16.593 INFO:teuthology.orchestra.run.trial134.stdout: libpmemobj-1.12.1-1.el9.x86_64 2026-02-26T03:12:16.593 INFO:teuthology.orchestra.run.trial134.stdout: librabbitmq-0.11.0-7.el9.x86_64 2026-02-26T03:12:16.593 INFO:teuthology.orchestra.run.trial134.stdout: libradosstriper1-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:12:16.593 INFO:teuthology.orchestra.run.trial134.stdout: librdkafka-1.6.1-102.el9.x86_64 2026-02-26T03:12:16.593 INFO:teuthology.orchestra.run.trial134.stdout: librgw2-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:12:16.593 INFO:teuthology.orchestra.run.trial134.stdout: libunwind-1.6.2-1.el9.x86_64 2026-02-26T03:12:16.593 INFO:teuthology.orchestra.run.trial134.stdout: lttng-ust-2.12.0-6.el9.x86_64 2026-02-26T03:12:16.594 INFO:teuthology.orchestra.run.trial134.stdout: mailcap-2.1.49-5.el9.noarch 2026-02-26T03:12:16.594 INFO:teuthology.orchestra.run.trial134.stdout: nvme-cli-2.16-1.el9.x86_64 2026-02-26T03:12:16.594 INFO:teuthology.orchestra.run.trial134.stdout: parquet-libs-9.0.0-15.el9.x86_64 2026-02-26T03:12:16.594 INFO:teuthology.orchestra.run.trial134.stdout: pciutils-3.7.0-7.el9.x86_64 2026-02-26T03:12:16.594 INFO:teuthology.orchestra.run.trial134.stdout: python3-ceph-argparse-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:12:16.594 INFO:teuthology.orchestra.run.trial134.stdout: python3-ceph-common-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:12:16.594 INFO:teuthology.orchestra.run.trial134.stdout: python3-cephfs-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:12:16.594 INFO:teuthology.orchestra.run.trial134.stdout: python3-rados-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:12:16.594 INFO:teuthology.orchestra.run.trial134.stdout: python3-rbd-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:12:16.594 INFO:teuthology.orchestra.run.trial134.stdout: python3-rgw-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:12:16.594 INFO:teuthology.orchestra.run.trial134.stdout: qatlib-25.08.0-2.el9.x86_64 2026-02-26T03:12:16.594 INFO:teuthology.orchestra.run.trial134.stdout: qatlib-service-25.08.0-2.el9.x86_64 2026-02-26T03:12:16.594 INFO:teuthology.orchestra.run.trial134.stdout: qatzip-libs-1.3.1-1.el9.x86_64 2026-02-26T03:12:16.594 INFO:teuthology.orchestra.run.trial134.stdout: re2-1:20211101-20.el9.x86_64 2026-02-26T03:12:16.594 INFO:teuthology.orchestra.run.trial134.stdout: thrift-0.15.0-4.el9.x86_64 2026-02-26T03:12:16.594 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:12:16.594 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-26T03:12:16.633 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install ceph-test 2026-02-26T03:12:16.945 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:02:39 ago on Thu 26 Feb 2026 03:09:37 AM UTC. 2026-02-26T03:12:17.030 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-26T03:12:17.031 INFO:teuthology.orchestra.run.trial134.stdout:================================================================================ 2026-02-26T03:12:17.031 INFO:teuthology.orchestra.run.trial134.stdout: Package Arch Version Repository Size 2026-02-26T03:12:17.031 INFO:teuthology.orchestra.run.trial134.stdout:================================================================================ 2026-02-26T03:12:17.031 INFO:teuthology.orchestra.run.trial134.stdout:Installing: 2026-02-26T03:12:17.031 INFO:teuthology.orchestra.run.trial134.stdout: ceph-test x86_64 2:20.3.0-5312.gabffb617.el9 ceph 126 M 2026-02-26T03:12:17.031 INFO:teuthology.orchestra.run.trial134.stdout:Installing dependencies: 2026-02-26T03:12:17.031 INFO:teuthology.orchestra.run.trial134.stdout: hwloc-libs x86_64 2.4.1-6.el9 baseos 2.1 M 2026-02-26T03:12:17.031 INFO:teuthology.orchestra.run.trial134.stdout: libasan8 x86_64 15.2.1-7.el9 appstream 525 k 2026-02-26T03:12:17.031 INFO:teuthology.orchestra.run.trial134.stdout: libatomic x86_64 11.5.0-14.el9 baseos 23 k 2026-02-26T03:12:17.032 INFO:teuthology.orchestra.run.trial134.stdout: libcephsqlite x86_64 2:20.3.0-5312.gabffb617.el9 ceph 153 k 2026-02-26T03:12:17.032 INFO:teuthology.orchestra.run.trial134.stdout: libubsan x86_64 11.5.0-14.el9 appstream 176 k 2026-02-26T03:12:17.032 INFO:teuthology.orchestra.run.trial134.stdout: libxslt x86_64 1.1.34-12.el9 appstream 233 k 2026-02-26T03:12:17.032 INFO:teuthology.orchestra.run.trial134.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2026-02-26T03:12:17.032 INFO:teuthology.orchestra.run.trial134.stdout: protobuf x86_64 3.14.0-17.el9 appstream 1.0 M 2026-02-26T03:12:17.032 INFO:teuthology.orchestra.run.trial134.stdout: socat x86_64 1.7.4.1-8.el9 appstream 303 k 2026-02-26T03:12:17.032 INFO:teuthology.orchestra.run.trial134.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2026-02-26T03:12:17.032 INFO:teuthology.orchestra.run.trial134.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2026-02-26T03:12:17.032 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:12:17.032 INFO:teuthology.orchestra.run.trial134.stdout:Transaction Summary 2026-02-26T03:12:17.032 INFO:teuthology.orchestra.run.trial134.stdout:================================================================================ 2026-02-26T03:12:17.033 INFO:teuthology.orchestra.run.trial134.stdout:Install 12 Packages 2026-02-26T03:12:17.033 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:12:17.033 INFO:teuthology.orchestra.run.trial134.stdout:Total download size: 131 M 2026-02-26T03:12:17.033 INFO:teuthology.orchestra.run.trial134.stdout:Installed size: 976 M 2026-02-26T03:12:17.033 INFO:teuthology.orchestra.run.trial134.stdout:Downloading Packages: 2026-02-26T03:12:18.228 INFO:teuthology.orchestra.run.trial134.stdout:(1/12): hwloc-libs-2.4.1-6.el9.x86_64.rpm 8.6 MB/s | 2.1 MB 00:00 2026-02-26T03:12:18.253 INFO:teuthology.orchestra.run.trial134.stdout:(2/12): libatomic-11.5.0-14.el9.x86_64.rpm 899 kB/s | 23 kB 00:00 2026-02-26T03:12:18.280 INFO:teuthology.orchestra.run.trial134.stdout:(3/12): lksctp-tools-1.0.19-2.el9.x86_64.rpm 3.5 MB/s | 94 kB 00:00 2026-02-26T03:12:18.455 INFO:teuthology.orchestra.run.trial134.stdout:(4/12): libcephsqlite-20.3.0-5312.gabffb617.el9 324 kB/s | 153 kB 00:00 2026-02-26T03:12:18.776 INFO:teuthology.orchestra.run.trial134.stdout:(5/12): libubsan-11.5.0-14.el9.x86_64.rpm 549 kB/s | 176 kB 00:00 2026-02-26T03:12:19.098 INFO:teuthology.orchestra.run.trial134.stdout:(6/12): libxslt-1.1.34-12.el9.x86_64.rpm 725 kB/s | 233 kB 00:00 2026-02-26T03:12:19.522 INFO:teuthology.orchestra.run.trial134.stdout:(7/12): protobuf-3.14.0-17.el9.x86_64.rpm 2.4 MB/s | 1.0 MB 00:00 2026-02-26T03:12:20.033 INFO:teuthology.orchestra.run.trial134.stdout:(8/12): socat-1.7.4.1-8.el9.x86_64.rpm 594 kB/s | 303 kB 00:00 2026-02-26T03:12:20.228 INFO:teuthology.orchestra.run.trial134.stdout:(9/12): libasan8-15.2.1-7.el9.x86_64.rpm 269 kB/s | 525 kB 00:01 2026-02-26T03:12:20.295 INFO:teuthology.orchestra.run.trial134.stdout:(10/12): xmlstarlet-1.6.1-20.el9.x86_64.rpm 243 kB/s | 64 kB 00:00 2026-02-26T03:12:20.321 INFO:teuthology.orchestra.run.trial134.stdout:(11/12): yaml-cpp-0.6.3-5.el9.x86_64.rpm 1.3 MB/s | 122 kB 00:00 2026-02-26T03:13:43.514 INFO:teuthology.orchestra.run.trial134.stdout:(12/12): ceph-test-20.3.0-5312.gabffb617.el9.x8 1.5 MB/s | 126 MB 01:25 2026-02-26T03:13:43.515 INFO:teuthology.orchestra.run.trial134.stdout:-------------------------------------------------------------------------------- 2026-02-26T03:13:43.516 INFO:teuthology.orchestra.run.trial134.stdout:Total 1.5 MB/s | 131 MB 01:26 2026-02-26T03:13:43.587 INFO:teuthology.orchestra.run.trial134.stdout:Running transaction check 2026-02-26T03:13:43.598 INFO:teuthology.orchestra.run.trial134.stdout:Transaction check succeeded. 2026-02-26T03:13:43.598 INFO:teuthology.orchestra.run.trial134.stdout:Running transaction test 2026-02-26T03:13:44.028 INFO:teuthology.orchestra.run.trial134.stdout:Transaction test succeeded. 2026-02-26T03:13:44.028 INFO:teuthology.orchestra.run.trial134.stdout:Running transaction 2026-02-26T03:13:44.484 INFO:teuthology.orchestra.run.trial134.stdout: Preparing : 1/1 2026-02-26T03:13:44.492 INFO:teuthology.orchestra.run.trial134.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/12 2026-02-26T03:13:44.501 INFO:teuthology.orchestra.run.trial134.stdout: Installing : socat-1.7.4.1-8.el9.x86_64 2/12 2026-02-26T03:13:44.506 INFO:teuthology.orchestra.run.trial134.stdout: Installing : protobuf-3.14.0-17.el9.x86_64 3/12 2026-02-26T03:13:44.511 INFO:teuthology.orchestra.run.trial134.stdout: Installing : libxslt-1.1.34-12.el9.x86_64 4/12 2026-02-26T03:13:44.513 INFO:teuthology.orchestra.run.trial134.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 5/12 2026-02-26T03:13:44.518 INFO:teuthology.orchestra.run.trial134.stdout: Installing : libubsan-11.5.0-14.el9.x86_64 6/12 2026-02-26T03:13:44.519 INFO:teuthology.orchestra.run.trial134.stdout: Installing : libasan8-15.2.1-7.el9.x86_64 7/12 2026-02-26T03:13:44.536 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: libasan8-15.2.1-7.el9.x86_64 7/12 2026-02-26T03:13:44.538 INFO:teuthology.orchestra.run.trial134.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 8/12 2026-02-26T03:13:44.551 INFO:teuthology.orchestra.run.trial134.stdout: Installing : libatomic-11.5.0-14.el9.x86_64 9/12 2026-02-26T03:13:44.560 INFO:teuthology.orchestra.run.trial134.stdout: Installing : hwloc-libs-2.4.1-6.el9.x86_64 10/12 2026-02-26T03:13:44.561 INFO:teuthology.orchestra.run.trial134.stdout: Installing : libcephsqlite-2:20.3.0-5312.gabffb617.el9.x86_64 11/12 2026-02-26T03:13:51.431 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: libcephsqlite-2:20.3.0-5312.gabffb617.el9.x86_64 11/12 2026-02-26T03:13:51.450 INFO:teuthology.orchestra.run.trial134.stdout: Installing : ceph-test-2:20.3.0-5312.gabffb617.el9.x86_64 12/12 2026-02-26T03:13:52.260 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: ceph-test-2:20.3.0-5312.gabffb617.el9.x86_64 12/12 2026-02-26T03:13:52.260 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : ceph-test-2:20.3.0-5312.gabffb617.el9.x86_64 1/12 2026-02-26T03:13:52.260 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : libcephsqlite-2:20.3.0-5312.gabffb617.el9.x86_64 2/12 2026-02-26T03:13:52.260 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : hwloc-libs-2.4.1-6.el9.x86_64 3/12 2026-02-26T03:13:52.260 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : libatomic-11.5.0-14.el9.x86_64 4/12 2026-02-26T03:13:52.260 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 5/12 2026-02-26T03:13:52.260 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : libasan8-15.2.1-7.el9.x86_64 6/12 2026-02-26T03:13:52.260 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : libubsan-11.5.0-14.el9.x86_64 7/12 2026-02-26T03:13:52.260 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : libxslt-1.1.34-12.el9.x86_64 8/12 2026-02-26T03:13:52.260 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : protobuf-3.14.0-17.el9.x86_64 9/12 2026-02-26T03:13:52.260 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : socat-1.7.4.1-8.el9.x86_64 10/12 2026-02-26T03:13:52.260 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 11/12 2026-02-26T03:13:52.322 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 12/12 2026-02-26T03:13:52.322 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:13:52.322 INFO:teuthology.orchestra.run.trial134.stdout:Installed: 2026-02-26T03:13:52.322 INFO:teuthology.orchestra.run.trial134.stdout: ceph-test-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:13:52.322 INFO:teuthology.orchestra.run.trial134.stdout: hwloc-libs-2.4.1-6.el9.x86_64 2026-02-26T03:13:52.323 INFO:teuthology.orchestra.run.trial134.stdout: libasan8-15.2.1-7.el9.x86_64 2026-02-26T03:13:52.323 INFO:teuthology.orchestra.run.trial134.stdout: libatomic-11.5.0-14.el9.x86_64 2026-02-26T03:13:52.323 INFO:teuthology.orchestra.run.trial134.stdout: libcephsqlite-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:13:52.323 INFO:teuthology.orchestra.run.trial134.stdout: libubsan-11.5.0-14.el9.x86_64 2026-02-26T03:13:52.323 INFO:teuthology.orchestra.run.trial134.stdout: libxslt-1.1.34-12.el9.x86_64 2026-02-26T03:13:52.323 INFO:teuthology.orchestra.run.trial134.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2026-02-26T03:13:52.323 INFO:teuthology.orchestra.run.trial134.stdout: protobuf-3.14.0-17.el9.x86_64 2026-02-26T03:13:52.323 INFO:teuthology.orchestra.run.trial134.stdout: socat-1.7.4.1-8.el9.x86_64 2026-02-26T03:13:52.323 INFO:teuthology.orchestra.run.trial134.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2026-02-26T03:13:52.323 INFO:teuthology.orchestra.run.trial134.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2026-02-26T03:13:52.323 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:13:52.323 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-26T03:13:52.362 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install ceph 2026-02-26T03:13:52.672 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:04:15 ago on Thu 26 Feb 2026 03:09:37 AM UTC. 2026-02-26T03:13:52.760 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-26T03:13:52.763 INFO:teuthology.orchestra.run.trial134.stdout:======================================================================================= 2026-02-26T03:13:52.763 INFO:teuthology.orchestra.run.trial134.stdout: Package Arch Version Repository Size 2026-02-26T03:13:52.763 INFO:teuthology.orchestra.run.trial134.stdout:======================================================================================= 2026-02-26T03:13:52.763 INFO:teuthology.orchestra.run.trial134.stdout:Installing: 2026-02-26T03:13:52.763 INFO:teuthology.orchestra.run.trial134.stdout: ceph x86_64 2:20.3.0-5312.gabffb617.el9 ceph 6.5 k 2026-02-26T03:13:52.763 INFO:teuthology.orchestra.run.trial134.stdout:Installing dependencies: 2026-02-26T03:13:52.763 INFO:teuthology.orchestra.run.trial134.stdout: abseil-cpp x86_64 20211102.0-4.el9 epel 551 k 2026-02-26T03:13:52.763 INFO:teuthology.orchestra.run.trial134.stdout: ceph-grafana-dashboards noarch 2:20.3.0-5312.gabffb617.el9 ceph-noarch 57 k 2026-02-26T03:13:52.763 INFO:teuthology.orchestra.run.trial134.stdout: ceph-mds x86_64 2:20.3.0-5312.gabffb617.el9 ceph 2.0 M 2026-02-26T03:13:52.763 INFO:teuthology.orchestra.run.trial134.stdout: ceph-mgr x86_64 2:20.3.0-5312.gabffb617.el9 ceph 937 k 2026-02-26T03:13:52.763 INFO:teuthology.orchestra.run.trial134.stdout: ceph-mgr-modules-core noarch 2:20.3.0-5312.gabffb617.el9 ceph-noarch 297 k 2026-02-26T03:13:52.763 INFO:teuthology.orchestra.run.trial134.stdout: ceph-mon x86_64 2:20.3.0-5312.gabffb617.el9 ceph 4.9 M 2026-02-26T03:13:52.763 INFO:teuthology.orchestra.run.trial134.stdout: ceph-osd x86_64 2:20.3.0-5312.gabffb617.el9 ceph 163 k 2026-02-26T03:13:52.763 INFO:teuthology.orchestra.run.trial134.stdout: ceph-osd-classic x86_64 2:20.3.0-5312.gabffb617.el9 ceph 15 M 2026-02-26T03:13:52.763 INFO:teuthology.orchestra.run.trial134.stdout: ceph-osd-crimson x86_64 2:20.3.0-5312.gabffb617.el9 ceph 87 M 2026-02-26T03:13:52.763 INFO:teuthology.orchestra.run.trial134.stdout: ceph-prometheus-alerts noarch 2:20.3.0-5312.gabffb617.el9 ceph-noarch 17 k 2026-02-26T03:13:52.763 INFO:teuthology.orchestra.run.trial134.stdout: cephadm noarch 2:20.3.0-5312.gabffb617.el9 ceph-noarch 1.0 M 2026-02-26T03:13:52.763 INFO:teuthology.orchestra.run.trial134.stdout: cryptsetup x86_64 2.8.1-3.el9 baseos 351 k 2026-02-26T03:13:52.763 INFO:teuthology.orchestra.run.trial134.stdout: flexiblas x86_64 3.0.4-9.el9 appstream 30 k 2026-02-26T03:13:52.763 INFO:teuthology.orchestra.run.trial134.stdout: flexiblas-netlib x86_64 3.0.4-9.el9 appstream 3.0 M 2026-02-26T03:13:52.763 INFO:teuthology.orchestra.run.trial134.stdout: flexiblas-openblas-openmp x86_64 3.0.4-9.el9 appstream 15 k 2026-02-26T03:13:52.763 INFO:teuthology.orchestra.run.trial134.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2026-02-26T03:13:52.763 INFO:teuthology.orchestra.run.trial134.stdout: ledmon-libs x86_64 1.1.0-3.el9 baseos 40 k 2026-02-26T03:13:52.763 INFO:teuthology.orchestra.run.trial134.stdout: libconfig x86_64 1.7.2-9.el9 baseos 72 k 2026-02-26T03:13:52.763 INFO:teuthology.orchestra.run.trial134.stdout: libgfortran x86_64 11.5.0-14.el9 baseos 794 k 2026-02-26T03:13:52.763 INFO:teuthology.orchestra.run.trial134.stdout: libquadmath x86_64 11.5.0-14.el9 baseos 184 k 2026-02-26T03:13:52.763 INFO:teuthology.orchestra.run.trial134.stdout: libstoragemgmt x86_64 1.10.1-1.el9 appstream 246 k 2026-02-26T03:13:52.764 INFO:teuthology.orchestra.run.trial134.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2026-02-26T03:13:52.764 INFO:teuthology.orchestra.run.trial134.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2026-02-26T03:13:52.764 INFO:teuthology.orchestra.run.trial134.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2026-02-26T03:13:52.764 INFO:teuthology.orchestra.run.trial134.stdout: openblas x86_64 0.3.29-1.el9 appstream 42 k 2026-02-26T03:13:52.764 INFO:teuthology.orchestra.run.trial134.stdout: openblas-openmp x86_64 0.3.29-1.el9 appstream 5.3 M 2026-02-26T03:13:52.764 INFO:teuthology.orchestra.run.trial134.stdout: protobuf-compiler x86_64 3.14.0-17.el9 crb 862 k 2026-02-26T03:13:52.764 INFO:teuthology.orchestra.run.trial134.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2026-02-26T03:13:52.764 INFO:teuthology.orchestra.run.trial134.stdout: python3-autocommand noarch 2.2.2-8.el9 epel 29 k 2026-02-26T03:13:52.764 INFO:teuthology.orchestra.run.trial134.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2026-02-26T03:13:52.764 INFO:teuthology.orchestra.run.trial134.stdout: python3-backports-tarfile noarch 1.2.0-1.el9 epel 60 k 2026-02-26T03:13:52.764 INFO:teuthology.orchestra.run.trial134.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2026-02-26T03:13:52.764 INFO:teuthology.orchestra.run.trial134.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2026-02-26T03:13:52.764 INFO:teuthology.orchestra.run.trial134.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2026-02-26T03:13:52.764 INFO:teuthology.orchestra.run.trial134.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2026-02-26T03:13:52.764 INFO:teuthology.orchestra.run.trial134.stdout: python3-cheroot noarch 10.0.1-4.el9 epel 173 k 2026-02-26T03:13:52.764 INFO:teuthology.orchestra.run.trial134.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2026-02-26T03:13:52.764 INFO:teuthology.orchestra.run.trial134.stdout: python3-cryptography x86_64 36.0.1-5.el9 baseos 1.2 M 2026-02-26T03:13:52.764 INFO:teuthology.orchestra.run.trial134.stdout: python3-devel x86_64 3.9.25-3.el9 appstream 244 k 2026-02-26T03:13:52.764 INFO:teuthology.orchestra.run.trial134.stdout: python3-google-auth noarch 1:2.45.0-1.el9 epel 254 k 2026-02-26T03:13:52.764 INFO:teuthology.orchestra.run.trial134.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2026-02-26T03:13:52.764 INFO:teuthology.orchestra.run.trial134.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2026-02-26T03:13:52.764 INFO:teuthology.orchestra.run.trial134.stdout: python3-isodate noarch 0.6.1-3.el9 epel 56 k 2026-02-26T03:13:52.764 INFO:teuthology.orchestra.run.trial134.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2026-02-26T03:13:52.764 INFO:teuthology.orchestra.run.trial134.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2026-02-26T03:13:52.764 INFO:teuthology.orchestra.run.trial134.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2026-02-26T03:13:52.764 INFO:teuthology.orchestra.run.trial134.stdout: python3-jaraco-context noarch 6.0.1-3.el9 epel 20 k 2026-02-26T03:13:52.764 INFO:teuthology.orchestra.run.trial134.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2026-02-26T03:13:52.764 INFO:teuthology.orchestra.run.trial134.stdout: python3-jaraco-text noarch 4.0.0-2.el9 epel 26 k 2026-02-26T03:13:52.764 INFO:teuthology.orchestra.run.trial134.stdout: python3-jinja2 noarch 2.11.3-8.el9 appstream 249 k 2026-02-26T03:13:52.764 INFO:teuthology.orchestra.run.trial134.stdout: python3-jmespath noarch 1.0.1-1.el9 appstream 48 k 2026-02-26T03:13:52.764 INFO:teuthology.orchestra.run.trial134.stdout: python3-kubernetes noarch 1:26.1.0-3.el9 epel 1.0 M 2026-02-26T03:13:52.764 INFO:teuthology.orchestra.run.trial134.stdout: python3-libstoragemgmt x86_64 1.10.1-1.el9 appstream 177 k 2026-02-26T03:13:52.764 INFO:teuthology.orchestra.run.trial134.stdout: python3-lxml x86_64 4.6.5-3.el9 appstream 1.2 M 2026-02-26T03:13:52.764 INFO:teuthology.orchestra.run.trial134.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2026-02-26T03:13:52.764 INFO:teuthology.orchestra.run.trial134.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2026-02-26T03:13:52.764 INFO:teuthology.orchestra.run.trial134.stdout: python3-msgpack x86_64 1.0.3-2.el9 epel 86 k 2026-02-26T03:13:52.764 INFO:teuthology.orchestra.run.trial134.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2026-02-26T03:13:52.764 INFO:teuthology.orchestra.run.trial134.stdout: python3-numpy x86_64 1:1.23.5-2.el9 appstream 6.1 M 2026-02-26T03:13:52.764 INFO:teuthology.orchestra.run.trial134.stdout: python3-numpy-f2py x86_64 1:1.23.5-2.el9 appstream 442 k 2026-02-26T03:13:52.764 INFO:teuthology.orchestra.run.trial134.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2026-02-26T03:13:52.764 INFO:teuthology.orchestra.run.trial134.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2026-02-26T03:13:52.765 INFO:teuthology.orchestra.run.trial134.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2026-02-26T03:13:52.765 INFO:teuthology.orchestra.run.trial134.stdout: python3-protobuf noarch 3.14.0-17.el9 appstream 267 k 2026-02-26T03:13:52.765 INFO:teuthology.orchestra.run.trial134.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2026-02-26T03:13:52.765 INFO:teuthology.orchestra.run.trial134.stdout: python3-pyasn1 noarch 0.4.8-7.el9 appstream 157 k 2026-02-26T03:13:52.765 INFO:teuthology.orchestra.run.trial134.stdout: python3-pyasn1-modules noarch 0.4.8-7.el9 appstream 277 k 2026-02-26T03:13:52.765 INFO:teuthology.orchestra.run.trial134.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2026-02-26T03:13:52.765 INFO:teuthology.orchestra.run.trial134.stdout: python3-pyparsing noarch 2.4.7-9.el9 baseos 150 k 2026-02-26T03:13:52.765 INFO:teuthology.orchestra.run.trial134.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2026-02-26T03:13:52.765 INFO:teuthology.orchestra.run.trial134.stdout: python3-requests noarch 2.25.1-10.el9 baseos 126 k 2026-02-26T03:13:52.765 INFO:teuthology.orchestra.run.trial134.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2026-02-26T03:13:52.765 INFO:teuthology.orchestra.run.trial134.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2026-02-26T03:13:52.765 INFO:teuthology.orchestra.run.trial134.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2026-02-26T03:13:52.765 INFO:teuthology.orchestra.run.trial134.stdout: python3-scipy x86_64 1.9.3-2.el9 appstream 19 M 2026-02-26T03:13:52.765 INFO:teuthology.orchestra.run.trial134.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2026-02-26T03:13:52.765 INFO:teuthology.orchestra.run.trial134.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2026-02-26T03:13:52.765 INFO:teuthology.orchestra.run.trial134.stdout: python3-typing-extensions noarch 4.15.0-1.el9 epel 86 k 2026-02-26T03:13:52.765 INFO:teuthology.orchestra.run.trial134.stdout: python3-urllib3 noarch 1.26.5-7.el9 baseos 218 k 2026-02-26T03:13:52.765 INFO:teuthology.orchestra.run.trial134.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2026-02-26T03:13:52.765 INFO:teuthology.orchestra.run.trial134.stdout: python3-xmlsec x86_64 1.3.13-1.el9 epel 48 k 2026-02-26T03:13:52.765 INFO:teuthology.orchestra.run.trial134.stdout: python3-xmltodict noarch 0.12.0-15.el9 epel 22 k 2026-02-26T03:13:52.765 INFO:teuthology.orchestra.run.trial134.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2026-02-26T03:13:52.765 INFO:teuthology.orchestra.run.trial134.stdout: unzip x86_64 6.0-59.el9 baseos 182 k 2026-02-26T03:13:52.765 INFO:teuthology.orchestra.run.trial134.stdout: xmlsec1 x86_64 1.2.29-13.el9 appstream 189 k 2026-02-26T03:13:52.765 INFO:teuthology.orchestra.run.trial134.stdout: xmlsec1-openssl x86_64 1.2.29-13.el9 appstream 90 k 2026-02-26T03:13:52.765 INFO:teuthology.orchestra.run.trial134.stdout: zip x86_64 3.0-35.el9 baseos 266 k 2026-02-26T03:13:52.765 INFO:teuthology.orchestra.run.trial134.stdout:Installing weak dependencies: 2026-02-26T03:13:52.765 INFO:teuthology.orchestra.run.trial134.stdout: ceph-mgr-cephadm noarch 2:20.3.0-5312.gabffb617.el9 ceph-noarch 184 k 2026-02-26T03:13:52.765 INFO:teuthology.orchestra.run.trial134.stdout: ceph-mgr-dashboard noarch 2:20.3.0-5312.gabffb617.el9 ceph-noarch 51 M 2026-02-26T03:13:52.765 INFO:teuthology.orchestra.run.trial134.stdout: ceph-mgr-diskprediction-local noarch 2:20.3.0-5312.gabffb617.el9 ceph-noarch 7.4 M 2026-02-26T03:13:52.765 INFO:teuthology.orchestra.run.trial134.stdout: ceph-mgr-k8sevents noarch 2:20.3.0-5312.gabffb617.el9 ceph-noarch 22 k 2026-02-26T03:13:52.765 INFO:teuthology.orchestra.run.trial134.stdout: ceph-mgr-rook noarch 2:20.3.0-5312.gabffb617.el9 ceph-noarch 50 k 2026-02-26T03:13:52.765 INFO:teuthology.orchestra.run.trial134.stdout: ceph-volume noarch 2:20.3.0-5312.gabffb617.el9 ceph-noarch 301 k 2026-02-26T03:13:52.765 INFO:teuthology.orchestra.run.trial134.stdout: python3-influxdb noarch 5.3.1-1.el9 epel 139 k 2026-02-26T03:13:52.765 INFO:teuthology.orchestra.run.trial134.stdout: python3-saml noarch 1.16.0-1.el9 epel 125 k 2026-02-26T03:13:52.765 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:13:52.765 INFO:teuthology.orchestra.run.trial134.stdout:Transaction Summary 2026-02-26T03:13:52.765 INFO:teuthology.orchestra.run.trial134.stdout:======================================================================================= 2026-02-26T03:13:52.765 INFO:teuthology.orchestra.run.trial134.stdout:Install 96 Packages 2026-02-26T03:13:52.765 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:13:52.766 INFO:teuthology.orchestra.run.trial134.stdout:Total download size: 225 M 2026-02-26T03:13:52.766 INFO:teuthology.orchestra.run.trial134.stdout:Installed size: 1.5 G 2026-02-26T03:13:52.766 INFO:teuthology.orchestra.run.trial134.stdout:Downloading Packages: 2026-02-26T03:13:53.623 INFO:teuthology.orchestra.run.trial134.stdout:(1/96): ceph-20.3.0-5312.gabffb617.el9.x86_64.r 23 kB/s | 6.5 kB 00:00 2026-02-26T03:13:54.355 INFO:teuthology.orchestra.run.trial134.stdout:(2/96): ceph-mgr-20.3.0-5312.gabffb617.el9.x86_ 928 kB/s | 937 kB 00:01 2026-02-26T03:13:54.558 INFO:teuthology.orchestra.run.trial134.stdout:(3/96): ceph-osd-20.3.0-5312.gabffb617.el9.x86_ 801 kB/s | 163 kB 00:00 2026-02-26T03:13:54.847 INFO:teuthology.orchestra.run.trial134.stdout:(4/96): ceph-mds-20.3.0-5312.gabffb617.el9.x86_ 1.3 MB/s | 2.0 MB 00:01 2026-02-26T03:13:58.211 INFO:teuthology.orchestra.run.trial134.stdout:(5/96): ceph-mon-20.3.0-5312.gabffb617.el9.x86_ 1.1 MB/s | 4.9 MB 00:04 2026-02-26T03:13:58.282 INFO:teuthology.orchestra.run.trial134.stdout:(6/96): ceph-grafana-dashboards-20.3.0-5312.gab 811 kB/s | 57 kB 00:00 2026-02-26T03:13:58.491 INFO:teuthology.orchestra.run.trial134.stdout:(7/96): ceph-mgr-cephadm-20.3.0-5312.gabffb617. 879 kB/s | 184 kB 00:00 2026-02-26T03:14:06.240 INFO:teuthology.orchestra.run.trial134.stdout:(8/96): ceph-osd-classic-20.3.0-5312.gabffb617. 1.3 MB/s | 15 MB 00:11 2026-02-26T03:14:13.934 INFO:teuthology.orchestra.run.trial134.stdout:(9/96): ceph-mgr-diskprediction-local-20.3.0-53 983 kB/s | 7.4 MB 00:07 2026-02-26T03:14:14.003 INFO:teuthology.orchestra.run.trial134.stdout:(10/96): ceph-mgr-k8sevents-20.3.0-5312.gabffb6 324 kB/s | 22 kB 00:00 2026-02-26T03:14:14.356 INFO:teuthology.orchestra.run.trial134.stdout:(11/96): ceph-mgr-modules-core-20.3.0-5312.gabf 839 kB/s | 297 kB 00:00 2026-02-26T03:14:14.773 INFO:teuthology.orchestra.run.trial134.stdout:(12/96): ceph-mgr-rook-20.3.0-5312.gabffb617.el 121 kB/s | 50 kB 00:00 2026-02-26T03:14:14.843 INFO:teuthology.orchestra.run.trial134.stdout:(13/96): ceph-prometheus-alerts-20.3.0-5312.gab 248 kB/s | 17 kB 00:00 2026-02-26T03:14:15.529 INFO:teuthology.orchestra.run.trial134.stdout:(14/96): ceph-volume-20.3.0-5312.gabffb617.el9. 439 kB/s | 301 kB 00:00 2026-02-26T03:14:17.104 INFO:teuthology.orchestra.run.trial134.stdout:(15/96): cephadm-20.3.0-5312.gabffb617.el9.noar 661 kB/s | 1.0 MB 00:01 2026-02-26T03:14:17.131 INFO:teuthology.orchestra.run.trial134.stdout:(16/96): cryptsetup-2.8.1-3.el9.x86_64.rpm 13 MB/s | 351 kB 00:00 2026-02-26T03:14:17.135 INFO:teuthology.orchestra.run.trial134.stdout:(17/96): ledmon-libs-1.1.0-3.el9.x86_64.rpm 8.8 MB/s | 40 kB 00:00 2026-02-26T03:14:17.140 INFO:teuthology.orchestra.run.trial134.stdout:(18/96): libconfig-1.7.2-9.el9.x86_64.rpm 15 MB/s | 72 kB 00:00 2026-02-26T03:14:17.150 INFO:teuthology.orchestra.run.trial134.stdout:(19/96): libgfortran-11.5.0-14.el9.x86_64.rpm 83 MB/s | 794 kB 00:00 2026-02-26T03:14:17.157 INFO:teuthology.orchestra.run.trial134.stdout:(20/96): libquadmath-11.5.0-14.el9.x86_64.rpm 24 MB/s | 184 kB 00:00 2026-02-26T03:14:17.165 INFO:teuthology.orchestra.run.trial134.stdout:(21/96): python3-cffi-1.14.5-5.el9.x86_64.rpm 34 MB/s | 253 kB 00:00 2026-02-26T03:14:17.196 INFO:teuthology.orchestra.run.trial134.stdout:(22/96): python3-cryptography-36.0.1-5.el9.x86_ 40 MB/s | 1.2 MB 00:00 2026-02-26T03:14:17.202 INFO:teuthology.orchestra.run.trial134.stdout:(23/96): python3-ply-3.11-14.el9.noarch.rpm 17 MB/s | 106 kB 00:00 2026-02-26T03:14:17.209 INFO:teuthology.orchestra.run.trial134.stdout:(24/96): python3-pycparser-2.20-6.el9.noarch.rp 21 MB/s | 135 kB 00:00 2026-02-26T03:14:17.216 INFO:teuthology.orchestra.run.trial134.stdout:(25/96): python3-pyparsing-2.4.7-9.el9.noarch.r 23 MB/s | 150 kB 00:00 2026-02-26T03:14:17.222 INFO:teuthology.orchestra.run.trial134.stdout:(26/96): python3-requests-2.25.1-10.el9.noarch. 18 MB/s | 126 kB 00:00 2026-02-26T03:14:17.230 INFO:teuthology.orchestra.run.trial134.stdout:(27/96): python3-urllib3-1.26.5-7.el9.noarch.rp 29 MB/s | 218 kB 00:00 2026-02-26T03:14:17.237 INFO:teuthology.orchestra.run.trial134.stdout:(28/96): unzip-6.0-59.el9.x86_64.rpm 24 MB/s | 182 kB 00:00 2026-02-26T03:14:17.244 INFO:teuthology.orchestra.run.trial134.stdout:(29/96): zip-3.0-35.el9.x86_64.rpm 39 MB/s | 266 kB 00:00 2026-02-26T03:14:17.502 INFO:teuthology.orchestra.run.trial134.stdout:(30/96): flexiblas-3.0.4-9.el9.x86_64.rpm 115 kB/s | 30 kB 00:00 2026-02-26T03:14:18.094 INFO:teuthology.orchestra.run.trial134.stdout:(31/96): flexiblas-netlib-3.0.4-9.el9.x86_64.rp 5.1 MB/s | 3.0 MB 00:00 2026-02-26T03:14:18.172 INFO:teuthology.orchestra.run.trial134.stdout:(32/96): flexiblas-openblas-openmp-3.0.4-9.el9. 190 kB/s | 15 kB 00:00 2026-02-26T03:14:18.253 INFO:teuthology.orchestra.run.trial134.stdout:(33/96): libstoragemgmt-1.10.1-1.el9.x86_64.rpm 3.0 MB/s | 246 kB 00:00 2026-02-26T03:14:18.333 INFO:teuthology.orchestra.run.trial134.stdout:(34/96): lua-5.4.4-4.el9.x86_64.rpm 2.3 MB/s | 188 kB 00:00 2026-02-26T03:14:18.412 INFO:teuthology.orchestra.run.trial134.stdout:(35/96): openblas-0.3.29-1.el9.x86_64.rpm 537 kB/s | 42 kB 00:00 2026-02-26T03:14:18.617 INFO:teuthology.orchestra.run.trial134.stdout:(36/96): openblas-openmp-0.3.29-1.el9.x86_64.rp 26 MB/s | 5.3 MB 00:00 2026-02-26T03:14:18.803 INFO:teuthology.orchestra.run.trial134.stdout:(37/96): python3-babel-2.9.1-2.el9.noarch.rpm 32 MB/s | 6.0 MB 00:00 2026-02-26T03:14:18.884 INFO:teuthology.orchestra.run.trial134.stdout:(38/96): python3-devel-3.9.25-3.el9.x86_64.rpm 2.9 MB/s | 244 kB 00:00 2026-02-26T03:14:18.965 INFO:teuthology.orchestra.run.trial134.stdout:(39/96): python3-jinja2-2.11.3-8.el9.noarch.rpm 3.0 MB/s | 249 kB 00:00 2026-02-26T03:14:19.044 INFO:teuthology.orchestra.run.trial134.stdout:(40/96): python3-jmespath-1.0.1-1.el9.noarch.rp 604 kB/s | 48 kB 00:00 2026-02-26T03:14:19.124 INFO:teuthology.orchestra.run.trial134.stdout:(41/96): python3-libstoragemgmt-1.10.1-1.el9.x8 2.2 MB/s | 177 kB 00:00 2026-02-26T03:14:19.211 INFO:teuthology.orchestra.run.trial134.stdout:(42/96): python3-lxml-4.6.5-3.el9.x86_64.rpm 14 MB/s | 1.2 MB 00:00 2026-02-26T03:14:19.290 INFO:teuthology.orchestra.run.trial134.stdout:(43/96): python3-markupsafe-1.1.1-12.el9.x86_64 442 kB/s | 35 kB 00:00 2026-02-26T03:14:19.476 INFO:teuthology.orchestra.run.trial134.stdout:(44/96): python3-numpy-1.23.5-2.el9.x86_64.rpm 33 MB/s | 6.1 MB 00:00 2026-02-26T03:14:19.557 INFO:teuthology.orchestra.run.trial134.stdout:(45/96): python3-numpy-f2py-1.23.5-2.el9.x86_64 5.3 MB/s | 442 kB 00:00 2026-02-26T03:14:19.637 INFO:teuthology.orchestra.run.trial134.stdout:(46/96): python3-packaging-20.9-5.el9.noarch.rp 974 kB/s | 77 kB 00:00 2026-02-26T03:14:19.716 INFO:teuthology.orchestra.run.trial134.stdout:(47/96): python3-protobuf-3.14.0-17.el9.noarch. 3.3 MB/s | 267 kB 00:00 2026-02-26T03:14:19.797 INFO:teuthology.orchestra.run.trial134.stdout:(48/96): python3-pyasn1-0.4.8-7.el9.noarch.rpm 1.9 MB/s | 157 kB 00:00 2026-02-26T03:14:19.878 INFO:teuthology.orchestra.run.trial134.stdout:(49/96): python3-pyasn1-modules-0.4.8-7.el9.noa 3.3 MB/s | 277 kB 00:00 2026-02-26T03:14:19.957 INFO:teuthology.orchestra.run.trial134.stdout:(50/96): python3-requests-oauthlib-1.3.0-12.el9 679 kB/s | 54 kB 00:00 2026-02-26T03:14:20.571 INFO:teuthology.orchestra.run.trial134.stdout:(51/96): python3-scipy-1.9.3-2.el9.x86_64.rpm 31 MB/s | 19 MB 00:00 2026-02-26T03:14:20.650 INFO:teuthology.orchestra.run.trial134.stdout:(52/96): python3-toml-0.10.2-6.el9.noarch.rpm 527 kB/s | 42 kB 00:00 2026-02-26T03:14:20.731 INFO:teuthology.orchestra.run.trial134.stdout:(53/96): xmlsec1-1.2.29-13.el9.x86_64.rpm 2.3 MB/s | 189 kB 00:00 2026-02-26T03:14:20.810 INFO:teuthology.orchestra.run.trial134.stdout:(54/96): xmlsec1-openssl-1.2.29-13.el9.x86_64.r 1.1 MB/s | 90 kB 00:00 2026-02-26T03:14:20.857 INFO:teuthology.orchestra.run.trial134.stdout:(55/96): lua-devel-5.4.4-4.el9.x86_64.rpm 474 kB/s | 22 kB 00:00 2026-02-26T03:14:20.943 INFO:teuthology.orchestra.run.trial134.stdout:(56/96): protobuf-compiler-3.14.0-17.el9.x86_64 9.8 MB/s | 862 kB 00:00 2026-02-26T03:14:21.077 INFO:teuthology.orchestra.run.trial134.stdout:(57/96): abseil-cpp-20211102.0-4.el9.x86_64.rpm 4.0 MB/s | 551 kB 00:00 2026-02-26T03:14:21.107 INFO:teuthology.orchestra.run.trial134.stdout:(58/96): grpc-data-1.46.7-10.el9.noarch.rpm 655 kB/s | 19 kB 00:00 2026-02-26T03:14:21.132 INFO:teuthology.orchestra.run.trial134.stdout:(59/96): luarocks-3.9.2-5.el9.noarch.rpm 5.8 MB/s | 151 kB 00:00 2026-02-26T03:14:21.175 INFO:teuthology.orchestra.run.trial134.stdout:(60/96): python3-asyncssh-2.13.2-5.el9.noarch.r 13 MB/s | 548 kB 00:00 2026-02-26T03:14:21.204 INFO:teuthology.orchestra.run.trial134.stdout:(61/96): python3-autocommand-2.2.2-8.el9.noarch 993 kB/s | 29 kB 00:00 2026-02-26T03:14:21.234 INFO:teuthology.orchestra.run.trial134.stdout:(62/96): python3-backports-tarfile-1.2.0-1.el9. 2.0 MB/s | 60 kB 00:00 2026-02-26T03:14:21.257 INFO:teuthology.orchestra.run.trial134.stdout:(63/96): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.8 MB/s | 43 kB 00:00 2026-02-26T03:14:21.289 INFO:teuthology.orchestra.run.trial134.stdout:(64/96): python3-cachetools-4.2.4-1.el9.noarch. 1.0 MB/s | 32 kB 00:00 2026-02-26T03:14:21.323 INFO:teuthology.orchestra.run.trial134.stdout:(65/96): python3-certifi-2023.05.07-4.el9.noarc 422 kB/s | 14 kB 00:00 2026-02-26T03:14:21.346 INFO:teuthology.orchestra.run.trial134.stdout:(66/96): python3-cheroot-10.0.1-4.el9.noarch.rp 7.4 MB/s | 173 kB 00:00 2026-02-26T03:14:21.380 INFO:teuthology.orchestra.run.trial134.stdout:(67/96): python3-cherrypy-18.6.1-2.el9.noarch.r 10 MB/s | 358 kB 00:00 2026-02-26T03:14:21.413 INFO:teuthology.orchestra.run.trial134.stdout:(68/96): python3-google-auth-2.45.0-1.el9.noarc 7.4 MB/s | 254 kB 00:00 2026-02-26T03:14:21.509 INFO:teuthology.orchestra.run.trial134.stdout:(69/96): python3-grpcio-1.46.7-10.el9.x86_64.rp 21 MB/s | 2.0 MB 00:00 2026-02-26T03:14:21.542 INFO:teuthology.orchestra.run.trial134.stdout:(70/96): python3-grpcio-tools-1.46.7-10.el9.x86 4.3 MB/s | 144 kB 00:00 2026-02-26T03:14:21.574 INFO:teuthology.orchestra.run.trial134.stdout:(71/96): python3-influxdb-5.3.1-1.el9.noarch.rp 4.2 MB/s | 139 kB 00:00 2026-02-26T03:14:21.604 INFO:teuthology.orchestra.run.trial134.stdout:(72/96): python3-isodate-0.6.1-3.el9.noarch.rpm 1.9 MB/s | 56 kB 00:00 2026-02-26T03:14:21.639 INFO:teuthology.orchestra.run.trial134.stdout:(73/96): python3-jaraco-8.2.1-3.el9.noarch.rpm 307 kB/s | 11 kB 00:00 2026-02-26T03:14:21.674 INFO:teuthology.orchestra.run.trial134.stdout:(74/96): python3-jaraco-classes-3.2.1-5.el9.noa 506 kB/s | 18 kB 00:00 2026-02-26T03:14:21.708 INFO:teuthology.orchestra.run.trial134.stdout:(75/96): python3-jaraco-collections-3.0.0-8.el9 692 kB/s | 23 kB 00:00 2026-02-26T03:14:21.735 INFO:teuthology.orchestra.run.trial134.stdout:(76/96): python3-jaraco-context-6.0.1-3.el9.noa 719 kB/s | 20 kB 00:00 2026-02-26T03:14:21.772 INFO:teuthology.orchestra.run.trial134.stdout:(77/96): python3-jaraco-functools-3.5.0-2.el9.n 528 kB/s | 19 kB 00:00 2026-02-26T03:14:21.797 INFO:teuthology.orchestra.run.trial134.stdout:(78/96): python3-jaraco-text-4.0.0-2.el9.noarch 1.0 MB/s | 26 kB 00:00 2026-02-26T03:14:21.835 INFO:teuthology.orchestra.run.trial134.stdout:(79/96): python3-kubernetes-26.1.0-3.el9.noarch 27 MB/s | 1.0 MB 00:00 2026-02-26T03:14:21.872 INFO:teuthology.orchestra.run.trial134.stdout:(80/96): python3-more-itertools-8.12.0-2.el9.no 2.1 MB/s | 79 kB 00:00 2026-02-26T03:14:21.897 INFO:teuthology.orchestra.run.trial134.stdout:(81/96): python3-msgpack-1.0.3-2.el9.x86_64.rpm 3.4 MB/s | 86 kB 00:00 2026-02-26T03:14:21.936 INFO:teuthology.orchestra.run.trial134.stdout:(82/96): python3-natsort-7.1.1-5.el9.noarch.rpm 1.5 MB/s | 58 kB 00:00 2026-02-26T03:14:21.972 INFO:teuthology.orchestra.run.trial134.stdout:(83/96): python3-portend-3.1.0-2.el9.noarch.rpm 457 kB/s | 16 kB 00:00 2026-02-26T03:14:22.003 INFO:teuthology.orchestra.run.trial134.stdout:(84/96): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.9 MB/s | 90 kB 00:00 2026-02-26T03:14:22.030 INFO:teuthology.orchestra.run.trial134.stdout:(85/96): python3-repoze-lru-0.7-16.el9.noarch.r 1.1 MB/s | 31 kB 00:00 2026-02-26T03:14:22.057 INFO:teuthology.orchestra.run.trial134.stdout:(86/96): python3-routes-2.5.1-5.el9.noarch.rpm 6.9 MB/s | 188 kB 00:00 2026-02-26T03:14:22.082 INFO:teuthology.orchestra.run.trial134.stdout:(87/96): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2026-02-26T03:14:22.119 INFO:teuthology.orchestra.run.trial134.stdout:(88/96): python3-saml-1.16.0-1.el9.noarch.rpm 3.3 MB/s | 125 kB 00:00 2026-02-26T03:14:22.146 INFO:teuthology.orchestra.run.trial134.stdout:(89/96): python3-tempora-5.0.0-2.el9.noarch.rpm 1.3 MB/s | 36 kB 00:00 2026-02-26T03:14:22.171 INFO:teuthology.orchestra.run.trial134.stdout:(90/96): python3-typing-extensions-4.15.0-1.el9 3.4 MB/s | 86 kB 00:00 2026-02-26T03:14:22.200 INFO:teuthology.orchestra.run.trial134.stdout:(91/96): python3-websocket-client-1.2.3-2.el9.n 3.0 MB/s | 90 kB 00:00 2026-02-26T03:14:22.235 INFO:teuthology.orchestra.run.trial134.stdout:(92/96): python3-xmlsec-1.3.13-1.el9.x86_64.rpm 1.3 MB/s | 48 kB 00:00 2026-02-26T03:14:22.263 INFO:teuthology.orchestra.run.trial134.stdout:(93/96): python3-xmltodict-0.12.0-15.el9.noarch 785 kB/s | 22 kB 00:00 2026-02-26T03:14:22.303 INFO:teuthology.orchestra.run.trial134.stdout:(94/96): python3-zc-lockfile-2.0-10.el9.noarch. 501 kB/s | 20 kB 00:00 2026-02-26T03:14:53.634 INFO:teuthology.orchestra.run.trial134.stdout:(95/96): ceph-mgr-dashboard-20.3.0-5312.gabffb6 940 kB/s | 51 MB 00:55 2026-02-26T03:15:46.363 INFO:teuthology.orchestra.run.trial134.stdout:(96/96): ceph-osd-crimson-20.3.0-5312.gabffb617 796 kB/s | 87 MB 01:51 2026-02-26T03:15:46.365 INFO:teuthology.orchestra.run.trial134.stdout:-------------------------------------------------------------------------------- 2026-02-26T03:15:46.365 INFO:teuthology.orchestra.run.trial134.stdout:Total 2.0 MB/s | 225 MB 01:53 2026-02-26T03:15:46.734 INFO:teuthology.orchestra.run.trial134.stdout:Running transaction check 2026-02-26T03:15:46.772 INFO:teuthology.orchestra.run.trial134.stdout:Transaction check succeeded. 2026-02-26T03:15:46.772 INFO:teuthology.orchestra.run.trial134.stdout:Running transaction test 2026-02-26T03:15:47.570 INFO:teuthology.orchestra.run.trial134.stdout:Transaction test succeeded. 2026-02-26T03:15:47.570 INFO:teuthology.orchestra.run.trial134.stdout:Running transaction 2026-02-26T03:15:48.395 INFO:teuthology.orchestra.run.trial134.stdout: Preparing : 1/1 2026-02-26T03:15:48.397 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/96 2026-02-26T03:15:48.401 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/96 2026-02-26T03:15:48.422 INFO:teuthology.orchestra.run.trial134.stdout: Installing : xmlsec1-1.2.29-13.el9.x86_64 3/96 2026-02-26T03:15:48.456 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-pyasn1-0.4.8-7.el9.noarch 4/96 2026-02-26T03:15:48.462 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-lxml-4.6.5-3.el9.x86_64 5/96 2026-02-26T03:15:48.464 INFO:teuthology.orchestra.run.trial134.stdout: Installing : lua-5.4.4-4.el9.x86_64 6/96 2026-02-26T03:15:48.469 INFO:teuthology.orchestra.run.trial134.stdout: Installing : flexiblas-3.0.4-9.el9.x86_64 7/96 2026-02-26T03:15:48.492 INFO:teuthology.orchestra.run.trial134.stdout: Installing : unzip-6.0-59.el9.x86_64 8/96 2026-02-26T03:15:48.505 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-urllib3-1.26.5-7.el9.noarch 9/96 2026-02-26T03:15:48.508 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-requests-2.25.1-10.el9.noarch 10/96 2026-02-26T03:15:48.514 INFO:teuthology.orchestra.run.trial134.stdout: Installing : libquadmath-11.5.0-14.el9.x86_64 11/96 2026-02-26T03:15:48.516 INFO:teuthology.orchestra.run.trial134.stdout: Installing : libgfortran-11.5.0-14.el9.x86_64 12/96 2026-02-26T03:15:48.527 INFO:teuthology.orchestra.run.trial134.stdout: Installing : ledmon-libs-1.1.0-3.el9.x86_64 13/96 2026-02-26T03:15:48.533 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 14/96 2026-02-26T03:15:48.557 INFO:teuthology.orchestra.run.trial134.stdout: Installing : zip-3.0-35.el9.x86_64 15/96 2026-02-26T03:15:48.560 INFO:teuthology.orchestra.run.trial134.stdout: Installing : luarocks-3.9.2-5.el9.noarch 16/96 2026-02-26T03:15:48.611 INFO:teuthology.orchestra.run.trial134.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 17/96 2026-02-26T03:15:48.623 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-pyasn1-modules-0.4.8-7.el9.noarch 18/96 2026-02-26T03:15:48.626 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-rsa-4.9-2.el9.noarch 19/96 2026-02-26T03:15:48.630 INFO:teuthology.orchestra.run.trial134.stdout: Installing : xmlsec1-openssl-1.2.29-13.el9.x86_64 20/96 2026-02-26T03:15:48.636 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-xmlsec-1.3.13-1.el9.x86_64 21/96 2026-02-26T03:15:48.641 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 22/96 2026-02-26T03:15:48.644 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 23/96 2026-02-26T03:15:48.658 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-xmltodict-0.12.0-15.el9.noarch 24/96 2026-02-26T03:15:48.663 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 25/96 2026-02-26T03:15:48.668 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-typing-extensions-4.15.0-1.el9.noarch 26/96 2026-02-26T03:15:48.679 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 27/96 2026-02-26T03:15:48.688 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 28/96 2026-02-26T03:15:48.694 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 29/96 2026-02-26T03:15:48.714 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-msgpack-1.0.3-2.el9.x86_64 30/96 2026-02-26T03:15:48.725 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-influxdb-5.3.1-1.el9.noarch 31/96 2026-02-26T03:15:48.740 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-isodate-0.6.1-3.el9.noarch 32/96 2026-02-26T03:15:48.745 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-saml-1.16.0-1.el9.noarch 33/96 2026-02-26T03:15:48.752 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 34/96 2026-02-26T03:15:48.792 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 35/96 2026-02-26T03:15:49.117 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-google-auth-1:2.45.0-1.el9.noarch 36/96 2026-02-26T03:15:49.127 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-kubernetes-1:26.1.0-3.el9.noarch 37/96 2026-02-26T03:15:49.130 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-backports-tarfile-1.2.0-1.el9.noarch 38/96 2026-02-26T03:15:49.136 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-jaraco-context-6.0.1-3.el9.noarch 39/96 2026-02-26T03:15:49.138 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-autocommand-2.2.2-8.el9.noarch 40/96 2026-02-26T03:15:49.165 INFO:teuthology.orchestra.run.trial134.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 41/96 2026-02-26T03:15:49.209 INFO:teuthology.orchestra.run.trial134.stdout: Installing : abseil-cpp-20211102.0-4.el9.x86_64 42/96 2026-02-26T03:15:49.217 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 43/96 2026-02-26T03:15:49.223 INFO:teuthology.orchestra.run.trial134.stdout: Installing : protobuf-compiler-3.14.0-17.el9.x86_64 44/96 2026-02-26T03:15:49.226 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 45/96 2026-02-26T03:15:49.232 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 46/96 2026-02-26T03:15:49.236 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-jaraco-text-4.0.0-2.el9.noarch 47/96 2026-02-26T03:15:49.243 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 48/96 2026-02-26T03:15:49.246 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 49/96 2026-02-26T03:15:49.274 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 50/96 2026-02-26T03:15:49.284 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-protobuf-3.14.0-17.el9.noarch 51/96 2026-02-26T03:15:49.290 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 52/96 2026-02-26T03:15:49.296 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 53/96 2026-02-26T03:15:49.331 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-jmespath-1.0.1-1.el9.noarch 54/96 2026-02-26T03:15:49.555 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-devel-3.9.25-3.el9.x86_64 55/96 2026-02-26T03:15:49.578 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 56/96 2026-02-26T03:15:49.582 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-jinja2-2.11.3-8.el9.noarch 57/96 2026-02-26T03:15:49.641 INFO:teuthology.orchestra.run.trial134.stdout: Installing : openblas-0.3.29-1.el9.x86_64 58/96 2026-02-26T03:15:49.643 INFO:teuthology.orchestra.run.trial134.stdout: Installing : openblas-openmp-0.3.29-1.el9.x86_64 59/96 2026-02-26T03:15:49.664 INFO:teuthology.orchestra.run.trial134.stdout: Installing : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 60/96 2026-02-26T03:15:49.989 INFO:teuthology.orchestra.run.trial134.stdout: Installing : flexiblas-netlib-3.0.4-9.el9.x86_64 61/96 2026-02-26T03:15:50.062 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-numpy-1:1.23.5-2.el9.x86_64 62/96 2026-02-26T03:15:50.730 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 63/96 2026-02-26T03:15:50.747 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-scipy-1.9.3-2.el9.x86_64 64/96 2026-02-26T03:15:50.756 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-pyparsing-2.4.7-9.el9.noarch 65/96 2026-02-26T03:15:50.763 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-packaging-20.9-5.el9.noarch 66/96 2026-02-26T03:15:50.776 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-ply-3.11-14.el9.noarch 67/96 2026-02-26T03:15:50.792 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 68/96 2026-02-26T03:15:50.868 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 69/96 2026-02-26T03:15:50.876 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-cryptography-36.0.1-5.el9.x86_64 70/96 2026-02-26T03:15:50.900 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 71/96 2026-02-26T03:15:50.929 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-cheroot-10.0.1-4.el9.noarch 72/96 2026-02-26T03:15:50.980 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 73/96 2026-02-26T03:15:50.986 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 74/96 2026-02-26T03:15:50.989 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 75/96 2026-02-26T03:15:50.990 INFO:teuthology.orchestra.run.trial134.stdout: Installing : libconfig-1.7.2-9.el9.x86_64 76/96 2026-02-26T03:15:51.008 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-26T03:15:51.008 INFO:teuthology.orchestra.run.trial134.stdout:Creating group 'libstoragemgmt' with GID 991. 2026-02-26T03:15:51.009 INFO:teuthology.orchestra.run.trial134.stdout:Creating user 'libstoragemgmt' (daemon account for libstoragemgmt) with UID 991 and GID 991. 2026-02-26T03:15:51.009 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:15:51.016 INFO:teuthology.orchestra.run.trial134.stdout: Installing : libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-26T03:15:51.040 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-26T03:15:51.040 INFO:teuthology.orchestra.run.trial134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/libstoragemgmt.service → /usr/lib/systemd/system/libstoragemgmt.service. 2026-02-26T03:15:51.040 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:15:51.078 INFO:teuthology.orchestra.run.trial134.stdout: Installing : python3-libstoragemgmt-1.10.1-1.el9.x86_64 78/96 2026-02-26T03:15:51.676 INFO:teuthology.orchestra.run.trial134.stdout: Installing : cryptsetup-2.8.1-3.el9.x86_64 79/96 2026-02-26T03:15:57.085 INFO:teuthology.orchestra.run.trial134.stdout: Installing : ceph-osd-classic-2:20.3.0-5312.gabffb617.el9.x86_6 80/96 2026-02-26T03:15:57.149 INFO:teuthology.orchestra.run.trial134.stdout: Installing : ceph-osd-crimson-2:20.3.0-5312.gabffb617.el9.x86_6 81/96 2026-02-26T03:15:57.151 INFO:teuthology.orchestra.run.trial134.stdout: Installing : ceph-volume-2:20.3.0-5312.gabffb617.el9.noarch 82/96 2026-02-26T03:15:57.164 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: ceph-volume-2:20.3.0-5312.gabffb617.el9.noarch 82/96 2026-02-26T03:15:57.164 INFO:teuthology.orchestra.run.trial134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-26T03:15:57.164 INFO:teuthology.orchestra.run.trial134.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2026-02-26T03:15:57.165 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:15:57.175 INFO:teuthology.orchestra.run.trial134.stdout: Installing : ceph-osd-2:20.3.0-5312.gabffb617.el9.x86_64 83/96 2026-02-26T03:15:57.198 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: ceph-osd-2:20.3.0-5312.gabffb617.el9.x86_64 83/96 2026-02-26T03:15:57.198 INFO:teuthology.orchestra.run.trial134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-26T03:15:57.198 INFO:teuthology.orchestra.run.trial134.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2026-02-26T03:15:57.199 INFO:teuthology.orchestra.run.trial134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-26T03:15:57.199 INFO:teuthology.orchestra.run.trial134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-26T03:15:57.199 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:15:57.259 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: cephadm-2:20.3.0-5312.gabffb617.el9.noarch 84/96 2026-02-26T03:15:57.261 INFO:teuthology.orchestra.run.trial134.stdout: Installing : cephadm-2:20.3.0-5312.gabffb617.el9.noarch 84/96 2026-02-26T03:15:57.270 INFO:teuthology.orchestra.run.trial134.stdout: Installing : ceph-prometheus-alerts-2:20.3.0-5312.gabffb617.el9 85/96 2026-02-26T03:15:57.295 INFO:teuthology.orchestra.run.trial134.stdout: Installing : ceph-grafana-dashboards-2:20.3.0-5312.gabffb617.el 86/96 2026-02-26T03:15:57.296 INFO:teuthology.orchestra.run.trial134.stdout: Installing : ceph-mgr-cephadm-2:20.3.0-5312.gabffb617.el9.noarc 87/96 2026-02-26T03:16:00.222 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: ceph-mgr-cephadm-2:20.3.0-5312.gabffb617.el9.noarc 87/96 2026-02-26T03:16:00.229 INFO:teuthology.orchestra.run.trial134.stdout: Installing : ceph-mgr-dashboard-2:20.3.0-5312.gabffb617.el9.noa 88/96 2026-02-26T03:16:00.691 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: ceph-mgr-dashboard-2:20.3.0-5312.gabffb617.el9.noa 88/96 2026-02-26T03:16:00.692 INFO:teuthology.orchestra.run.trial134.stdout: Installing : ceph-mgr-diskprediction-local-2:20.3.0-5312.gabffb 89/96 2026-02-26T03:16:00.704 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:20.3.0-5312.gabffb 89/96 2026-02-26T03:16:00.705 INFO:teuthology.orchestra.run.trial134.stdout: Installing : ceph-mgr-k8sevents-2:20.3.0-5312.gabffb617.el9.noa 90/96 2026-02-26T03:16:00.766 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: ceph-mgr-k8sevents-2:20.3.0-5312.gabffb617.el9.noa 90/96 2026-02-26T03:16:00.806 INFO:teuthology.orchestra.run.trial134.stdout: Installing : ceph-mgr-modules-core-2:20.3.0-5312.gabffb617.el9. 91/96 2026-02-26T03:16:00.808 INFO:teuthology.orchestra.run.trial134.stdout: Installing : ceph-mgr-2:20.3.0-5312.gabffb617.el9.x86_64 92/96 2026-02-26T03:16:00.828 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: ceph-mgr-2:20.3.0-5312.gabffb617.el9.x86_64 92/96 2026-02-26T03:16:00.828 INFO:teuthology.orchestra.run.trial134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-26T03:16:00.828 INFO:teuthology.orchestra.run.trial134.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2026-02-26T03:16:00.828 INFO:teuthology.orchestra.run.trial134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-26T03:16:00.828 INFO:teuthology.orchestra.run.trial134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-26T03:16:00.828 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:16:00.841 INFO:teuthology.orchestra.run.trial134.stdout: Installing : ceph-mgr-rook-2:20.3.0-5312.gabffb617.el9.noarch 93/96 2026-02-26T03:16:01.058 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: ceph-mgr-rook-2:20.3.0-5312.gabffb617.el9.noarch 93/96 2026-02-26T03:16:01.060 INFO:teuthology.orchestra.run.trial134.stdout: Installing : ceph-mon-2:20.3.0-5312.gabffb617.el9.x86_64 94/96 2026-02-26T03:16:01.081 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: ceph-mon-2:20.3.0-5312.gabffb617.el9.x86_64 94/96 2026-02-26T03:16:01.081 INFO:teuthology.orchestra.run.trial134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-26T03:16:01.081 INFO:teuthology.orchestra.run.trial134.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2026-02-26T03:16:01.081 INFO:teuthology.orchestra.run.trial134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-26T03:16:01.081 INFO:teuthology.orchestra.run.trial134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-26T03:16:01.081 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:16:01.166 INFO:teuthology.orchestra.run.trial134.stdout: Installing : ceph-mds-2:20.3.0-5312.gabffb617.el9.x86_64 95/96 2026-02-26T03:16:01.186 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: ceph-mds-2:20.3.0-5312.gabffb617.el9.x86_64 95/96 2026-02-26T03:16:01.187 INFO:teuthology.orchestra.run.trial134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-26T03:16:01.187 INFO:teuthology.orchestra.run.trial134.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2026-02-26T03:16:01.187 INFO:teuthology.orchestra.run.trial134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-26T03:16:01.187 INFO:teuthology.orchestra.run.trial134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-26T03:16:01.188 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:16:01.189 INFO:teuthology.orchestra.run.trial134.stdout: Installing : ceph-2:20.3.0-5312.gabffb617.el9.x86_64 96/96 2026-02-26T03:16:01.193 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: ceph-osd-classic-2:20.3.0-5312.gabffb617.el9.x86_6 96/96 2026-02-26T03:16:01.217 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: ceph-osd-crimson-2:20.3.0-5312.gabffb617.el9.x86_6 96/96 2026-02-26T03:16:02.642 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: ceph-2:20.3.0-5312.gabffb617.el9.x86_64 96/96 2026-02-26T03:16:02.643 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : ceph-2:20.3.0-5312.gabffb617.el9.x86_64 1/96 2026-02-26T03:16:02.643 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : ceph-mds-2:20.3.0-5312.gabffb617.el9.x86_64 2/96 2026-02-26T03:16:02.643 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : ceph-mgr-2:20.3.0-5312.gabffb617.el9.x86_64 3/96 2026-02-26T03:16:02.643 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : ceph-mon-2:20.3.0-5312.gabffb617.el9.x86_64 4/96 2026-02-26T03:16:02.643 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : ceph-osd-2:20.3.0-5312.gabffb617.el9.x86_64 5/96 2026-02-26T03:16:02.643 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : ceph-osd-classic-2:20.3.0-5312.gabffb617.el9.x86_6 6/96 2026-02-26T03:16:02.643 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : ceph-osd-crimson-2:20.3.0-5312.gabffb617.el9.x86_6 7/96 2026-02-26T03:16:02.643 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : ceph-grafana-dashboards-2:20.3.0-5312.gabffb617.el 8/96 2026-02-26T03:16:02.643 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : ceph-mgr-cephadm-2:20.3.0-5312.gabffb617.el9.noarc 9/96 2026-02-26T03:16:02.643 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : ceph-mgr-dashboard-2:20.3.0-5312.gabffb617.el9.noa 10/96 2026-02-26T03:16:02.644 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : ceph-mgr-diskprediction-local-2:20.3.0-5312.gabffb 11/96 2026-02-26T03:16:02.644 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : ceph-mgr-k8sevents-2:20.3.0-5312.gabffb617.el9.noa 12/96 2026-02-26T03:16:02.644 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : ceph-mgr-modules-core-2:20.3.0-5312.gabffb617.el9. 13/96 2026-02-26T03:16:02.644 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : ceph-mgr-rook-2:20.3.0-5312.gabffb617.el9.noarch 14/96 2026-02-26T03:16:02.644 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : ceph-prometheus-alerts-2:20.3.0-5312.gabffb617.el9 15/96 2026-02-26T03:16:02.644 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : ceph-volume-2:20.3.0-5312.gabffb617.el9.noarch 16/96 2026-02-26T03:16:02.644 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : cephadm-2:20.3.0-5312.gabffb617.el9.noarch 17/96 2026-02-26T03:16:02.644 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : cryptsetup-2.8.1-3.el9.x86_64 18/96 2026-02-26T03:16:02.644 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : ledmon-libs-1.1.0-3.el9.x86_64 19/96 2026-02-26T03:16:02.644 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : libconfig-1.7.2-9.el9.x86_64 20/96 2026-02-26T03:16:02.645 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : libgfortran-11.5.0-14.el9.x86_64 21/96 2026-02-26T03:16:02.645 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : libquadmath-11.5.0-14.el9.x86_64 22/96 2026-02-26T03:16:02.645 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 23/96 2026-02-26T03:16:02.645 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-cryptography-36.0.1-5.el9.x86_64 24/96 2026-02-26T03:16:02.645 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-ply-3.11-14.el9.noarch 25/96 2026-02-26T03:16:02.645 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 26/96 2026-02-26T03:16:02.645 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-pyparsing-2.4.7-9.el9.noarch 27/96 2026-02-26T03:16:02.645 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-requests-2.25.1-10.el9.noarch 28/96 2026-02-26T03:16:02.645 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-urllib3-1.26.5-7.el9.noarch 29/96 2026-02-26T03:16:02.645 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : unzip-6.0-59.el9.x86_64 30/96 2026-02-26T03:16:02.645 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : zip-3.0-35.el9.x86_64 31/96 2026-02-26T03:16:02.646 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : flexiblas-3.0.4-9.el9.x86_64 32/96 2026-02-26T03:16:02.646 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : flexiblas-netlib-3.0.4-9.el9.x86_64 33/96 2026-02-26T03:16:02.646 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 34/96 2026-02-26T03:16:02.646 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : libstoragemgmt-1.10.1-1.el9.x86_64 35/96 2026-02-26T03:16:02.646 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : lua-5.4.4-4.el9.x86_64 36/96 2026-02-26T03:16:02.646 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : openblas-0.3.29-1.el9.x86_64 37/96 2026-02-26T03:16:02.646 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : openblas-openmp-0.3.29-1.el9.x86_64 38/96 2026-02-26T03:16:02.646 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 39/96 2026-02-26T03:16:02.646 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-devel-3.9.25-3.el9.x86_64 40/96 2026-02-26T03:16:02.646 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-jinja2-2.11.3-8.el9.noarch 41/96 2026-02-26T03:16:02.646 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-jmespath-1.0.1-1.el9.noarch 42/96 2026-02-26T03:16:02.646 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-libstoragemgmt-1.10.1-1.el9.x86_64 43/96 2026-02-26T03:16:02.647 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-lxml-4.6.5-3.el9.x86_64 44/96 2026-02-26T03:16:02.647 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 45/96 2026-02-26T03:16:02.647 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-numpy-1:1.23.5-2.el9.x86_64 46/96 2026-02-26T03:16:02.647 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 47/96 2026-02-26T03:16:02.647 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 48/96 2026-02-26T03:16:02.647 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-protobuf-3.14.0-17.el9.noarch 49/96 2026-02-26T03:16:02.647 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-pyasn1-0.4.8-7.el9.noarch 50/96 2026-02-26T03:16:02.647 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-pyasn1-modules-0.4.8-7.el9.noarch 51/96 2026-02-26T03:16:02.647 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 52/96 2026-02-26T03:16:02.647 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-scipy-1.9.3-2.el9.x86_64 53/96 2026-02-26T03:16:02.647 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 54/96 2026-02-26T03:16:02.647 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : xmlsec1-1.2.29-13.el9.x86_64 55/96 2026-02-26T03:16:02.647 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : xmlsec1-openssl-1.2.29-13.el9.x86_64 56/96 2026-02-26T03:16:02.648 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 57/96 2026-02-26T03:16:02.648 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : protobuf-compiler-3.14.0-17.el9.x86_64 58/96 2026-02-26T03:16:02.648 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : abseil-cpp-20211102.0-4.el9.x86_64 59/96 2026-02-26T03:16:02.648 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 60/96 2026-02-26T03:16:02.648 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 61/96 2026-02-26T03:16:02.648 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 62/96 2026-02-26T03:16:02.648 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-autocommand-2.2.2-8.el9.noarch 63/96 2026-02-26T03:16:02.648 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-backports-tarfile-1.2.0-1.el9.noarch 64/96 2026-02-26T03:16:02.648 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 65/96 2026-02-26T03:16:02.648 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 66/96 2026-02-26T03:16:02.648 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 67/96 2026-02-26T03:16:02.648 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-cheroot-10.0.1-4.el9.noarch 68/96 2026-02-26T03:16:02.648 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 69/96 2026-02-26T03:16:02.649 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-google-auth-1:2.45.0-1.el9.noarch 70/96 2026-02-26T03:16:02.649 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 71/96 2026-02-26T03:16:02.649 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 72/96 2026-02-26T03:16:02.649 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-influxdb-5.3.1-1.el9.noarch 73/96 2026-02-26T03:16:02.649 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-isodate-0.6.1-3.el9.noarch 74/96 2026-02-26T03:16:02.649 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 75/96 2026-02-26T03:16:02.649 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 76/96 2026-02-26T03:16:02.649 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 77/96 2026-02-26T03:16:02.649 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-jaraco-context-6.0.1-3.el9.noarch 78/96 2026-02-26T03:16:02.649 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 79/96 2026-02-26T03:16:02.649 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-jaraco-text-4.0.0-2.el9.noarch 80/96 2026-02-26T03:16:02.649 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-kubernetes-1:26.1.0-3.el9.noarch 81/96 2026-02-26T03:16:02.650 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 82/96 2026-02-26T03:16:02.650 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-msgpack-1.0.3-2.el9.x86_64 83/96 2026-02-26T03:16:02.650 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 84/96 2026-02-26T03:16:02.650 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 85/96 2026-02-26T03:16:02.650 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/96 2026-02-26T03:16:02.650 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 87/96 2026-02-26T03:16:02.650 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 88/96 2026-02-26T03:16:02.650 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 89/96 2026-02-26T03:16:02.650 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-saml-1.16.0-1.el9.noarch 90/96 2026-02-26T03:16:02.650 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 91/96 2026-02-26T03:16:02.650 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-typing-extensions-4.15.0-1.el9.noarch 92/96 2026-02-26T03:16:02.650 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 93/96 2026-02-26T03:16:02.650 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-xmlsec-1.3.13-1.el9.x86_64 94/96 2026-02-26T03:16:02.651 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-xmltodict-0.12.0-15.el9.noarch 95/96 2026-02-26T03:16:02.712 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 96/96 2026-02-26T03:16:02.713 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:16:02.713 INFO:teuthology.orchestra.run.trial134.stdout:Installed: 2026-02-26T03:16:02.713 INFO:teuthology.orchestra.run.trial134.stdout: abseil-cpp-20211102.0-4.el9.x86_64 2026-02-26T03:16:02.713 INFO:teuthology.orchestra.run.trial134.stdout: ceph-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:16:02.713 INFO:teuthology.orchestra.run.trial134.stdout: ceph-grafana-dashboards-2:20.3.0-5312.gabffb617.el9.noarch 2026-02-26T03:16:02.713 INFO:teuthology.orchestra.run.trial134.stdout: ceph-mds-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:16:02.713 INFO:teuthology.orchestra.run.trial134.stdout: ceph-mgr-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:16:02.713 INFO:teuthology.orchestra.run.trial134.stdout: ceph-mgr-cephadm-2:20.3.0-5312.gabffb617.el9.noarch 2026-02-26T03:16:02.713 INFO:teuthology.orchestra.run.trial134.stdout: ceph-mgr-dashboard-2:20.3.0-5312.gabffb617.el9.noarch 2026-02-26T03:16:02.714 INFO:teuthology.orchestra.run.trial134.stdout: ceph-mgr-diskprediction-local-2:20.3.0-5312.gabffb617.el9.noarch 2026-02-26T03:16:02.714 INFO:teuthology.orchestra.run.trial134.stdout: ceph-mgr-k8sevents-2:20.3.0-5312.gabffb617.el9.noarch 2026-02-26T03:16:02.714 INFO:teuthology.orchestra.run.trial134.stdout: ceph-mgr-modules-core-2:20.3.0-5312.gabffb617.el9.noarch 2026-02-26T03:16:02.714 INFO:teuthology.orchestra.run.trial134.stdout: ceph-mgr-rook-2:20.3.0-5312.gabffb617.el9.noarch 2026-02-26T03:16:02.714 INFO:teuthology.orchestra.run.trial134.stdout: ceph-mon-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:16:02.714 INFO:teuthology.orchestra.run.trial134.stdout: ceph-osd-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:16:02.715 INFO:teuthology.orchestra.run.trial134.stdout: ceph-osd-classic-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:16:02.715 INFO:teuthology.orchestra.run.trial134.stdout: ceph-osd-crimson-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:16:02.715 INFO:teuthology.orchestra.run.trial134.stdout: ceph-prometheus-alerts-2:20.3.0-5312.gabffb617.el9.noarch 2026-02-26T03:16:02.715 INFO:teuthology.orchestra.run.trial134.stdout: ceph-volume-2:20.3.0-5312.gabffb617.el9.noarch 2026-02-26T03:16:02.715 INFO:teuthology.orchestra.run.trial134.stdout: cephadm-2:20.3.0-5312.gabffb617.el9.noarch 2026-02-26T03:16:02.715 INFO:teuthology.orchestra.run.trial134.stdout: cryptsetup-2.8.1-3.el9.x86_64 2026-02-26T03:16:02.715 INFO:teuthology.orchestra.run.trial134.stdout: flexiblas-3.0.4-9.el9.x86_64 2026-02-26T03:16:02.715 INFO:teuthology.orchestra.run.trial134.stdout: flexiblas-netlib-3.0.4-9.el9.x86_64 2026-02-26T03:16:02.715 INFO:teuthology.orchestra.run.trial134.stdout: flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 2026-02-26T03:16:02.715 INFO:teuthology.orchestra.run.trial134.stdout: grpc-data-1.46.7-10.el9.noarch 2026-02-26T03:16:02.715 INFO:teuthology.orchestra.run.trial134.stdout: ledmon-libs-1.1.0-3.el9.x86_64 2026-02-26T03:16:02.715 INFO:teuthology.orchestra.run.trial134.stdout: libconfig-1.7.2-9.el9.x86_64 2026-02-26T03:16:02.716 INFO:teuthology.orchestra.run.trial134.stdout: libgfortran-11.5.0-14.el9.x86_64 2026-02-26T03:16:02.716 INFO:teuthology.orchestra.run.trial134.stdout: libquadmath-11.5.0-14.el9.x86_64 2026-02-26T03:16:02.716 INFO:teuthology.orchestra.run.trial134.stdout: libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-26T03:16:02.716 INFO:teuthology.orchestra.run.trial134.stdout: lua-5.4.4-4.el9.x86_64 2026-02-26T03:16:02.716 INFO:teuthology.orchestra.run.trial134.stdout: lua-devel-5.4.4-4.el9.x86_64 2026-02-26T03:16:02.716 INFO:teuthology.orchestra.run.trial134.stdout: luarocks-3.9.2-5.el9.noarch 2026-02-26T03:16:02.716 INFO:teuthology.orchestra.run.trial134.stdout: openblas-0.3.29-1.el9.x86_64 2026-02-26T03:16:02.716 INFO:teuthology.orchestra.run.trial134.stdout: openblas-openmp-0.3.29-1.el9.x86_64 2026-02-26T03:16:02.716 INFO:teuthology.orchestra.run.trial134.stdout: protobuf-compiler-3.14.0-17.el9.x86_64 2026-02-26T03:16:02.716 INFO:teuthology.orchestra.run.trial134.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2026-02-26T03:16:02.716 INFO:teuthology.orchestra.run.trial134.stdout: python3-autocommand-2.2.2-8.el9.noarch 2026-02-26T03:16:02.716 INFO:teuthology.orchestra.run.trial134.stdout: python3-babel-2.9.1-2.el9.noarch 2026-02-26T03:16:02.717 INFO:teuthology.orchestra.run.trial134.stdout: python3-backports-tarfile-1.2.0-1.el9.noarch 2026-02-26T03:16:02.717 INFO:teuthology.orchestra.run.trial134.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2026-02-26T03:16:02.717 INFO:teuthology.orchestra.run.trial134.stdout: python3-cachetools-4.2.4-1.el9.noarch 2026-02-26T03:16:02.717 INFO:teuthology.orchestra.run.trial134.stdout: python3-certifi-2023.05.07-4.el9.noarch 2026-02-26T03:16:02.717 INFO:teuthology.orchestra.run.trial134.stdout: python3-cffi-1.14.5-5.el9.x86_64 2026-02-26T03:16:02.717 INFO:teuthology.orchestra.run.trial134.stdout: python3-cheroot-10.0.1-4.el9.noarch 2026-02-26T03:16:02.717 INFO:teuthology.orchestra.run.trial134.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2026-02-26T03:16:02.717 INFO:teuthology.orchestra.run.trial134.stdout: python3-cryptography-36.0.1-5.el9.x86_64 2026-02-26T03:16:02.717 INFO:teuthology.orchestra.run.trial134.stdout: python3-devel-3.9.25-3.el9.x86_64 2026-02-26T03:16:02.717 INFO:teuthology.orchestra.run.trial134.stdout: python3-google-auth-1:2.45.0-1.el9.noarch 2026-02-26T03:16:02.717 INFO:teuthology.orchestra.run.trial134.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2026-02-26T03:16:02.717 INFO:teuthology.orchestra.run.trial134.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2026-02-26T03:16:02.718 INFO:teuthology.orchestra.run.trial134.stdout: python3-influxdb-5.3.1-1.el9.noarch 2026-02-26T03:16:02.718 INFO:teuthology.orchestra.run.trial134.stdout: python3-isodate-0.6.1-3.el9.noarch 2026-02-26T03:16:02.718 INFO:teuthology.orchestra.run.trial134.stdout: python3-jaraco-8.2.1-3.el9.noarch 2026-02-26T03:16:02.718 INFO:teuthology.orchestra.run.trial134.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2026-02-26T03:16:02.718 INFO:teuthology.orchestra.run.trial134.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2026-02-26T03:16:02.718 INFO:teuthology.orchestra.run.trial134.stdout: python3-jaraco-context-6.0.1-3.el9.noarch 2026-02-26T03:16:02.718 INFO:teuthology.orchestra.run.trial134.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2026-02-26T03:16:02.718 INFO:teuthology.orchestra.run.trial134.stdout: python3-jaraco-text-4.0.0-2.el9.noarch 2026-02-26T03:16:02.718 INFO:teuthology.orchestra.run.trial134.stdout: python3-jinja2-2.11.3-8.el9.noarch 2026-02-26T03:16:02.718 INFO:teuthology.orchestra.run.trial134.stdout: python3-jmespath-1.0.1-1.el9.noarch 2026-02-26T03:16:02.718 INFO:teuthology.orchestra.run.trial134.stdout: python3-kubernetes-1:26.1.0-3.el9.noarch 2026-02-26T03:16:02.718 INFO:teuthology.orchestra.run.trial134.stdout: python3-libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-26T03:16:02.718 INFO:teuthology.orchestra.run.trial134.stdout: python3-lxml-4.6.5-3.el9.x86_64 2026-02-26T03:16:02.719 INFO:teuthology.orchestra.run.trial134.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2026-02-26T03:16:02.719 INFO:teuthology.orchestra.run.trial134.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2026-02-26T03:16:02.719 INFO:teuthology.orchestra.run.trial134.stdout: python3-msgpack-1.0.3-2.el9.x86_64 2026-02-26T03:16:02.719 INFO:teuthology.orchestra.run.trial134.stdout: python3-natsort-7.1.1-5.el9.noarch 2026-02-26T03:16:02.719 INFO:teuthology.orchestra.run.trial134.stdout: python3-numpy-1:1.23.5-2.el9.x86_64 2026-02-26T03:16:02.719 INFO:teuthology.orchestra.run.trial134.stdout: python3-numpy-f2py-1:1.23.5-2.el9.x86_64 2026-02-26T03:16:02.719 INFO:teuthology.orchestra.run.trial134.stdout: python3-packaging-20.9-5.el9.noarch 2026-02-26T03:16:02.719 INFO:teuthology.orchestra.run.trial134.stdout: python3-ply-3.11-14.el9.noarch 2026-02-26T03:16:02.719 INFO:teuthology.orchestra.run.trial134.stdout: python3-portend-3.1.0-2.el9.noarch 2026-02-26T03:16:02.720 INFO:teuthology.orchestra.run.trial134.stdout: python3-protobuf-3.14.0-17.el9.noarch 2026-02-26T03:16:02.720 INFO:teuthology.orchestra.run.trial134.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2026-02-26T03:16:02.720 INFO:teuthology.orchestra.run.trial134.stdout: python3-pyasn1-0.4.8-7.el9.noarch 2026-02-26T03:16:02.720 INFO:teuthology.orchestra.run.trial134.stdout: python3-pyasn1-modules-0.4.8-7.el9.noarch 2026-02-26T03:16:02.720 INFO:teuthology.orchestra.run.trial134.stdout: python3-pycparser-2.20-6.el9.noarch 2026-02-26T03:16:02.720 INFO:teuthology.orchestra.run.trial134.stdout: python3-pyparsing-2.4.7-9.el9.noarch 2026-02-26T03:16:02.720 INFO:teuthology.orchestra.run.trial134.stdout: python3-repoze-lru-0.7-16.el9.noarch 2026-02-26T03:16:02.720 INFO:teuthology.orchestra.run.trial134.stdout: python3-requests-2.25.1-10.el9.noarch 2026-02-26T03:16:02.720 INFO:teuthology.orchestra.run.trial134.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2026-02-26T03:16:02.720 INFO:teuthology.orchestra.run.trial134.stdout: python3-routes-2.5.1-5.el9.noarch 2026-02-26T03:16:02.720 INFO:teuthology.orchestra.run.trial134.stdout: python3-rsa-4.9-2.el9.noarch 2026-02-26T03:16:02.720 INFO:teuthology.orchestra.run.trial134.stdout: python3-saml-1.16.0-1.el9.noarch 2026-02-26T03:16:02.720 INFO:teuthology.orchestra.run.trial134.stdout: python3-scipy-1.9.3-2.el9.x86_64 2026-02-26T03:16:02.721 INFO:teuthology.orchestra.run.trial134.stdout: python3-tempora-5.0.0-2.el9.noarch 2026-02-26T03:16:02.721 INFO:teuthology.orchestra.run.trial134.stdout: python3-toml-0.10.2-6.el9.noarch 2026-02-26T03:16:02.721 INFO:teuthology.orchestra.run.trial134.stdout: python3-typing-extensions-4.15.0-1.el9.noarch 2026-02-26T03:16:02.721 INFO:teuthology.orchestra.run.trial134.stdout: python3-urllib3-1.26.5-7.el9.noarch 2026-02-26T03:16:02.721 INFO:teuthology.orchestra.run.trial134.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2026-02-26T03:16:02.721 INFO:teuthology.orchestra.run.trial134.stdout: python3-xmlsec-1.3.13-1.el9.x86_64 2026-02-26T03:16:02.721 INFO:teuthology.orchestra.run.trial134.stdout: python3-xmltodict-0.12.0-15.el9.noarch 2026-02-26T03:16:02.721 INFO:teuthology.orchestra.run.trial134.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2026-02-26T03:16:02.721 INFO:teuthology.orchestra.run.trial134.stdout: unzip-6.0-59.el9.x86_64 2026-02-26T03:16:02.721 INFO:teuthology.orchestra.run.trial134.stdout: xmlsec1-1.2.29-13.el9.x86_64 2026-02-26T03:16:02.721 INFO:teuthology.orchestra.run.trial134.stdout: xmlsec1-openssl-1.2.29-13.el9.x86_64 2026-02-26T03:16:02.721 INFO:teuthology.orchestra.run.trial134.stdout: zip-3.0-35.el9.x86_64 2026-02-26T03:16:02.722 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:16:02.722 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-26T03:16:02.779 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install ceph-base 2026-02-26T03:16:03.097 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:06:26 ago on Thu 26 Feb 2026 03:09:37 AM UTC. 2026-02-26T03:16:03.162 INFO:teuthology.orchestra.run.trial134.stdout:Package ceph-base-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:16:03.182 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-26T03:16:03.183 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-26T03:16:03.183 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-26T03:16:03.204 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install cephadm 2026-02-26T03:16:03.517 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:06:26 ago on Thu 26 Feb 2026 03:09:37 AM UTC. 2026-02-26T03:16:03.584 INFO:teuthology.orchestra.run.trial134.stdout:Package cephadm-2:20.3.0-5312.gabffb617.el9.noarch is already installed. 2026-02-26T03:16:03.603 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-26T03:16:03.603 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-26T03:16:03.603 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-26T03:16:03.624 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install ceph-immutable-object-cache 2026-02-26T03:16:03.936 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:06:26 ago on Thu 26 Feb 2026 03:09:37 AM UTC. 2026-02-26T03:16:04.021 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-26T03:16:04.021 INFO:teuthology.orchestra.run.trial134.stdout:================================================================================ 2026-02-26T03:16:04.021 INFO:teuthology.orchestra.run.trial134.stdout: Package Arch Version Repo Size 2026-02-26T03:16:04.022 INFO:teuthology.orchestra.run.trial134.stdout:================================================================================ 2026-02-26T03:16:04.022 INFO:teuthology.orchestra.run.trial134.stdout:Installing: 2026-02-26T03:16:04.022 INFO:teuthology.orchestra.run.trial134.stdout: ceph-immutable-object-cache x86_64 2:20.3.0-5312.gabffb617.el9 ceph 149 k 2026-02-26T03:16:04.022 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:16:04.022 INFO:teuthology.orchestra.run.trial134.stdout:Transaction Summary 2026-02-26T03:16:04.022 INFO:teuthology.orchestra.run.trial134.stdout:================================================================================ 2026-02-26T03:16:04.022 INFO:teuthology.orchestra.run.trial134.stdout:Install 1 Package 2026-02-26T03:16:04.022 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:16:04.022 INFO:teuthology.orchestra.run.trial134.stdout:Total download size: 149 k 2026-02-26T03:16:04.022 INFO:teuthology.orchestra.run.trial134.stdout:Installed size: 455 k 2026-02-26T03:16:04.022 INFO:teuthology.orchestra.run.trial134.stdout:Downloading Packages: 2026-02-26T03:16:04.589 INFO:teuthology.orchestra.run.trial134.stdout:ceph-immutable-object-cache-20.3.0-5312.gabffb6 263 kB/s | 149 kB 00:00 2026-02-26T03:16:04.589 INFO:teuthology.orchestra.run.trial134.stdout:-------------------------------------------------------------------------------- 2026-02-26T03:16:04.589 INFO:teuthology.orchestra.run.trial134.stdout:Total 263 kB/s | 149 kB 00:00 2026-02-26T03:16:04.590 INFO:teuthology.orchestra.run.trial134.stdout:Running transaction check 2026-02-26T03:16:04.594 INFO:teuthology.orchestra.run.trial134.stdout:Transaction check succeeded. 2026-02-26T03:16:04.594 INFO:teuthology.orchestra.run.trial134.stdout:Running transaction test 2026-02-26T03:16:04.625 INFO:teuthology.orchestra.run.trial134.stdout:Transaction test succeeded. 2026-02-26T03:16:04.625 INFO:teuthology.orchestra.run.trial134.stdout:Running transaction 2026-02-26T03:16:04.681 INFO:teuthology.orchestra.run.trial134.stdout: Preparing : 1/1 2026-02-26T03:16:04.683 INFO:teuthology.orchestra.run.trial134.stdout: Installing : ceph-immutable-object-cache-2:20.3.0-5312.gabffb617. 1/1 2026-02-26T03:16:04.704 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: ceph-immutable-object-cache-2:20.3.0-5312.gabffb617. 1/1 2026-02-26T03:16:04.704 INFO:teuthology.orchestra.run.trial134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-26T03:16:04.704 INFO:teuthology.orchestra.run.trial134.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2026-02-26T03:16:04.704 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:16:05.003 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : ceph-immutable-object-cache-2:20.3.0-5312.gabffb617. 1/1 2026-02-26T03:16:05.003 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:16:05.003 INFO:teuthology.orchestra.run.trial134.stdout:Installed: 2026-02-26T03:16:05.003 INFO:teuthology.orchestra.run.trial134.stdout: ceph-immutable-object-cache-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:16:05.003 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:16:05.003 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-26T03:16:05.030 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install ceph-mgr 2026-02-26T03:16:05.346 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:06:28 ago on Thu 26 Feb 2026 03:09:37 AM UTC. 2026-02-26T03:16:05.412 INFO:teuthology.orchestra.run.trial134.stdout:Package ceph-mgr-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:16:05.431 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-26T03:16:05.432 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-26T03:16:05.432 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-26T03:16:05.453 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install ceph-mgr-dashboard 2026-02-26T03:16:05.773 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:06:28 ago on Thu 26 Feb 2026 03:09:37 AM UTC. 2026-02-26T03:16:05.838 INFO:teuthology.orchestra.run.trial134.stdout:Package ceph-mgr-dashboard-2:20.3.0-5312.gabffb617.el9.noarch is already installed. 2026-02-26T03:16:05.857 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-26T03:16:05.858 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-26T03:16:05.858 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-26T03:16:05.880 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-26T03:16:06.191 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:06:29 ago on Thu 26 Feb 2026 03:09:37 AM UTC. 2026-02-26T03:16:06.258 INFO:teuthology.orchestra.run.trial134.stdout:Package ceph-mgr-diskprediction-local-2:20.3.0-5312.gabffb617.el9.noarch is already installed. 2026-02-26T03:16:06.276 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-26T03:16:06.277 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-26T03:16:06.277 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-26T03:16:06.300 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install ceph-mgr-rook 2026-02-26T03:16:06.614 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:06:29 ago on Thu 26 Feb 2026 03:09:37 AM UTC. 2026-02-26T03:16:06.680 INFO:teuthology.orchestra.run.trial134.stdout:Package ceph-mgr-rook-2:20.3.0-5312.gabffb617.el9.noarch is already installed. 2026-02-26T03:16:06.698 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-26T03:16:06.699 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-26T03:16:06.699 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-26T03:16:06.721 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install ceph-mgr-cephadm 2026-02-26T03:16:07.034 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:06:30 ago on Thu 26 Feb 2026 03:09:37 AM UTC. 2026-02-26T03:16:07.100 INFO:teuthology.orchestra.run.trial134.stdout:Package ceph-mgr-cephadm-2:20.3.0-5312.gabffb617.el9.noarch is already installed. 2026-02-26T03:16:07.119 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-26T03:16:07.119 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-26T03:16:07.119 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-26T03:16:07.141 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install ceph-osd 2026-02-26T03:16:07.458 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:06:30 ago on Thu 26 Feb 2026 03:09:37 AM UTC. 2026-02-26T03:16:07.525 INFO:teuthology.orchestra.run.trial134.stdout:Package ceph-osd-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:16:07.544 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-26T03:16:07.544 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-26T03:16:07.544 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-26T03:16:07.566 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install ceph-osd-classic 2026-02-26T03:16:07.884 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:06:30 ago on Thu 26 Feb 2026 03:09:37 AM UTC. 2026-02-26T03:16:07.949 INFO:teuthology.orchestra.run.trial134.stdout:Package ceph-osd-classic-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:16:07.968 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-26T03:16:07.969 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-26T03:16:07.969 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-26T03:16:07.991 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install ceph-fuse 2026-02-26T03:16:08.304 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:06:31 ago on Thu 26 Feb 2026 03:09:37 AM UTC. 2026-02-26T03:16:08.389 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-26T03:16:08.390 INFO:teuthology.orchestra.run.trial134.stdout:================================================================================ 2026-02-26T03:16:08.390 INFO:teuthology.orchestra.run.trial134.stdout: Package Arch Version Repo Size 2026-02-26T03:16:08.390 INFO:teuthology.orchestra.run.trial134.stdout:================================================================================ 2026-02-26T03:16:08.390 INFO:teuthology.orchestra.run.trial134.stdout:Installing: 2026-02-26T03:16:08.390 INFO:teuthology.orchestra.run.trial134.stdout: ceph-fuse x86_64 2:20.3.0-5312.gabffb617.el9 ceph 914 k 2026-02-26T03:16:08.390 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:16:08.390 INFO:teuthology.orchestra.run.trial134.stdout:Transaction Summary 2026-02-26T03:16:08.390 INFO:teuthology.orchestra.run.trial134.stdout:================================================================================ 2026-02-26T03:16:08.391 INFO:teuthology.orchestra.run.trial134.stdout:Install 1 Package 2026-02-26T03:16:08.391 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:16:08.391 INFO:teuthology.orchestra.run.trial134.stdout:Total download size: 914 k 2026-02-26T03:16:08.391 INFO:teuthology.orchestra.run.trial134.stdout:Installed size: 2.9 M 2026-02-26T03:16:08.391 INFO:teuthology.orchestra.run.trial134.stdout:Downloading Packages: 2026-02-26T03:16:09.517 INFO:teuthology.orchestra.run.trial134.stdout:ceph-fuse-20.3.0-5312.gabffb617.el9.x86_64.rpm 811 kB/s | 914 kB 00:01 2026-02-26T03:16:09.518 INFO:teuthology.orchestra.run.trial134.stdout:-------------------------------------------------------------------------------- 2026-02-26T03:16:09.518 INFO:teuthology.orchestra.run.trial134.stdout:Total 810 kB/s | 914 kB 00:01 2026-02-26T03:16:09.518 INFO:teuthology.orchestra.run.trial134.stdout:Running transaction check 2026-02-26T03:16:09.524 INFO:teuthology.orchestra.run.trial134.stdout:Transaction check succeeded. 2026-02-26T03:16:09.524 INFO:teuthology.orchestra.run.trial134.stdout:Running transaction test 2026-02-26T03:16:09.557 INFO:teuthology.orchestra.run.trial134.stdout:Transaction test succeeded. 2026-02-26T03:16:09.557 INFO:teuthology.orchestra.run.trial134.stdout:Running transaction 2026-02-26T03:16:09.649 INFO:teuthology.orchestra.run.trial134.stdout: Preparing : 1/1 2026-02-26T03:16:09.667 INFO:teuthology.orchestra.run.trial134.stdout: Installing : ceph-fuse-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:16:09.930 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: ceph-fuse-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:16:09.993 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : ceph-fuse-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:16:09.994 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:16:09.994 INFO:teuthology.orchestra.run.trial134.stdout:Installed: 2026-02-26T03:16:09.994 INFO:teuthology.orchestra.run.trial134.stdout: ceph-fuse-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:16:09.994 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:16:09.994 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-26T03:16:10.022 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install ceph-volume 2026-02-26T03:16:10.337 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:06:33 ago on Thu 26 Feb 2026 03:09:37 AM UTC. 2026-02-26T03:16:10.403 INFO:teuthology.orchestra.run.trial134.stdout:Package ceph-volume-2:20.3.0-5312.gabffb617.el9.noarch is already installed. 2026-02-26T03:16:10.421 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-26T03:16:10.422 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-26T03:16:10.422 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-26T03:16:10.444 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install librados-devel 2026-02-26T03:16:10.760 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:06:33 ago on Thu 26 Feb 2026 03:09:37 AM UTC. 2026-02-26T03:16:10.844 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-26T03:16:10.845 INFO:teuthology.orchestra.run.trial134.stdout:================================================================================ 2026-02-26T03:16:10.845 INFO:teuthology.orchestra.run.trial134.stdout: Package Arch Version Repo Size 2026-02-26T03:16:10.845 INFO:teuthology.orchestra.run.trial134.stdout:================================================================================ 2026-02-26T03:16:10.845 INFO:teuthology.orchestra.run.trial134.stdout:Installing: 2026-02-26T03:16:10.845 INFO:teuthology.orchestra.run.trial134.stdout: librados-devel x86_64 2:20.3.0-5312.gabffb617.el9 ceph 124 k 2026-02-26T03:16:10.845 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:16:10.845 INFO:teuthology.orchestra.run.trial134.stdout:Transaction Summary 2026-02-26T03:16:10.845 INFO:teuthology.orchestra.run.trial134.stdout:================================================================================ 2026-02-26T03:16:10.845 INFO:teuthology.orchestra.run.trial134.stdout:Install 1 Package 2026-02-26T03:16:10.845 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:16:10.845 INFO:teuthology.orchestra.run.trial134.stdout:Total download size: 124 k 2026-02-26T03:16:10.845 INFO:teuthology.orchestra.run.trial134.stdout:Installed size: 434 k 2026-02-26T03:16:10.845 INFO:teuthology.orchestra.run.trial134.stdout:Downloading Packages: 2026-02-26T03:16:11.595 INFO:teuthology.orchestra.run.trial134.stdout:librados-devel-20.3.0-5312.gabffb617.el9.x86_64 165 kB/s | 124 kB 00:00 2026-02-26T03:16:11.596 INFO:teuthology.orchestra.run.trial134.stdout:-------------------------------------------------------------------------------- 2026-02-26T03:16:11.596 INFO:teuthology.orchestra.run.trial134.stdout:Total 165 kB/s | 124 kB 00:00 2026-02-26T03:16:11.596 INFO:teuthology.orchestra.run.trial134.stdout:Running transaction check 2026-02-26T03:16:11.599 INFO:teuthology.orchestra.run.trial134.stdout:Transaction check succeeded. 2026-02-26T03:16:11.600 INFO:teuthology.orchestra.run.trial134.stdout:Running transaction test 2026-02-26T03:16:11.631 INFO:teuthology.orchestra.run.trial134.stdout:Transaction test succeeded. 2026-02-26T03:16:11.631 INFO:teuthology.orchestra.run.trial134.stdout:Running transaction 2026-02-26T03:16:11.686 INFO:teuthology.orchestra.run.trial134.stdout: Preparing : 1/1 2026-02-26T03:16:11.704 INFO:teuthology.orchestra.run.trial134.stdout: Installing : librados-devel-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:16:11.792 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: librados-devel-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:16:11.855 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : librados-devel-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:16:11.856 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:16:11.856 INFO:teuthology.orchestra.run.trial134.stdout:Installed: 2026-02-26T03:16:11.856 INFO:teuthology.orchestra.run.trial134.stdout: librados-devel-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:16:11.856 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:16:11.856 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-26T03:16:11.884 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install libcephfs2 2026-02-26T03:16:12.198 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:06:35 ago on Thu 26 Feb 2026 03:09:37 AM UTC. 2026-02-26T03:16:12.264 INFO:teuthology.orchestra.run.trial134.stdout:Package libcephfs2-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:16:12.283 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-26T03:16:12.283 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-26T03:16:12.283 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-26T03:16:12.306 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install libcephfs-devel 2026-02-26T03:16:12.623 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:06:35 ago on Thu 26 Feb 2026 03:09:37 AM UTC. 2026-02-26T03:16:12.708 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-26T03:16:12.708 INFO:teuthology.orchestra.run.trial134.stdout:================================================================================ 2026-02-26T03:16:12.708 INFO:teuthology.orchestra.run.trial134.stdout: Package Arch Version Repo Size 2026-02-26T03:16:12.708 INFO:teuthology.orchestra.run.trial134.stdout:================================================================================ 2026-02-26T03:16:12.708 INFO:teuthology.orchestra.run.trial134.stdout:Installing: 2026-02-26T03:16:12.708 INFO:teuthology.orchestra.run.trial134.stdout: libcephfs-devel x86_64 2:20.3.0-5312.gabffb617.el9 ceph 37 k 2026-02-26T03:16:12.708 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:16:12.708 INFO:teuthology.orchestra.run.trial134.stdout:Transaction Summary 2026-02-26T03:16:12.708 INFO:teuthology.orchestra.run.trial134.stdout:================================================================================ 2026-02-26T03:16:12.708 INFO:teuthology.orchestra.run.trial134.stdout:Install 1 Package 2026-02-26T03:16:12.708 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:16:12.709 INFO:teuthology.orchestra.run.trial134.stdout:Total download size: 37 k 2026-02-26T03:16:12.709 INFO:teuthology.orchestra.run.trial134.stdout:Installed size: 171 k 2026-02-26T03:16:12.709 INFO:teuthology.orchestra.run.trial134.stdout:Downloading Packages: 2026-02-26T03:16:13.061 INFO:teuthology.orchestra.run.trial134.stdout:libcephfs-devel-20.3.0-5312.gabffb617.el9.x86_6 106 kB/s | 37 kB 00:00 2026-02-26T03:16:13.061 INFO:teuthology.orchestra.run.trial134.stdout:-------------------------------------------------------------------------------- 2026-02-26T03:16:13.062 INFO:teuthology.orchestra.run.trial134.stdout:Total 105 kB/s | 37 kB 00:00 2026-02-26T03:16:13.062 INFO:teuthology.orchestra.run.trial134.stdout:Running transaction check 2026-02-26T03:16:13.065 INFO:teuthology.orchestra.run.trial134.stdout:Transaction check succeeded. 2026-02-26T03:16:13.065 INFO:teuthology.orchestra.run.trial134.stdout:Running transaction test 2026-02-26T03:16:13.067 INFO:teuthology.orchestra.run.trial134.stdout:Transaction test succeeded. 2026-02-26T03:16:13.067 INFO:teuthology.orchestra.run.trial134.stdout:Running transaction 2026-02-26T03:16:13.090 INFO:teuthology.orchestra.run.trial134.stdout: Preparing : 1/1 2026-02-26T03:16:13.107 INFO:teuthology.orchestra.run.trial134.stdout: Installing : libcephfs-devel-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:16:13.173 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: libcephfs-devel-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:16:13.236 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : libcephfs-devel-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:16:13.237 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:16:13.237 INFO:teuthology.orchestra.run.trial134.stdout:Installed: 2026-02-26T03:16:13.237 INFO:teuthology.orchestra.run.trial134.stdout: libcephfs-devel-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:16:13.237 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:16:13.237 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-26T03:16:13.263 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install librados2 2026-02-26T03:16:13.578 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:06:36 ago on Thu 26 Feb 2026 03:09:37 AM UTC. 2026-02-26T03:16:13.644 INFO:teuthology.orchestra.run.trial134.stdout:Package librados2-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:16:13.662 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-26T03:16:13.662 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-26T03:16:13.663 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-26T03:16:13.684 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install librbd1 2026-02-26T03:16:14.001 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:06:36 ago on Thu 26 Feb 2026 03:09:37 AM UTC. 2026-02-26T03:16:14.067 INFO:teuthology.orchestra.run.trial134.stdout:Package librbd1-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:16:14.084 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-26T03:16:14.085 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-26T03:16:14.085 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-26T03:16:14.107 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install python3-rados 2026-02-26T03:16:14.422 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:06:37 ago on Thu 26 Feb 2026 03:09:37 AM UTC. 2026-02-26T03:16:14.487 INFO:teuthology.orchestra.run.trial134.stdout:Package python3-rados-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:16:14.505 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-26T03:16:14.506 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-26T03:16:14.506 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-26T03:16:14.527 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install python3-rgw 2026-02-26T03:16:14.842 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:06:37 ago on Thu 26 Feb 2026 03:09:37 AM UTC. 2026-02-26T03:16:14.907 INFO:teuthology.orchestra.run.trial134.stdout:Package python3-rgw-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:16:14.925 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-26T03:16:14.925 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-26T03:16:14.926 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-26T03:16:14.948 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install python3-cephfs 2026-02-26T03:16:15.270 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:06:38 ago on Thu 26 Feb 2026 03:09:37 AM UTC. 2026-02-26T03:16:15.335 INFO:teuthology.orchestra.run.trial134.stdout:Package python3-cephfs-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:16:15.353 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-26T03:16:15.354 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-26T03:16:15.354 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-26T03:16:15.376 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install python3-rbd 2026-02-26T03:16:15.694 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:06:38 ago on Thu 26 Feb 2026 03:09:37 AM UTC. 2026-02-26T03:16:15.760 INFO:teuthology.orchestra.run.trial134.stdout:Package python3-rbd-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:16:15.778 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-26T03:16:15.779 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-26T03:16:15.779 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-26T03:16:15.800 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install rbd-fuse 2026-02-26T03:16:16.117 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:06:39 ago on Thu 26 Feb 2026 03:09:37 AM UTC. 2026-02-26T03:16:16.202 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-26T03:16:16.202 INFO:teuthology.orchestra.run.trial134.stdout:================================================================================ 2026-02-26T03:16:16.202 INFO:teuthology.orchestra.run.trial134.stdout: Package Architecture Version Repository Size 2026-02-26T03:16:16.202 INFO:teuthology.orchestra.run.trial134.stdout:================================================================================ 2026-02-26T03:16:16.202 INFO:teuthology.orchestra.run.trial134.stdout:Installing: 2026-02-26T03:16:16.202 INFO:teuthology.orchestra.run.trial134.stdout: rbd-fuse x86_64 2:20.3.0-5312.gabffb617.el9 ceph 87 k 2026-02-26T03:16:16.202 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:16:16.202 INFO:teuthology.orchestra.run.trial134.stdout:Transaction Summary 2026-02-26T03:16:16.202 INFO:teuthology.orchestra.run.trial134.stdout:================================================================================ 2026-02-26T03:16:16.202 INFO:teuthology.orchestra.run.trial134.stdout:Install 1 Package 2026-02-26T03:16:16.202 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:16:16.202 INFO:teuthology.orchestra.run.trial134.stdout:Total download size: 87 k 2026-02-26T03:16:16.202 INFO:teuthology.orchestra.run.trial134.stdout:Installed size: 236 k 2026-02-26T03:16:16.202 INFO:teuthology.orchestra.run.trial134.stdout:Downloading Packages: 2026-02-26T03:16:16.630 INFO:teuthology.orchestra.run.trial134.stdout:rbd-fuse-20.3.0-5312.gabffb617.el9.x86_64.rpm 202 kB/s | 87 kB 00:00 2026-02-26T03:16:16.630 INFO:teuthology.orchestra.run.trial134.stdout:-------------------------------------------------------------------------------- 2026-02-26T03:16:16.630 INFO:teuthology.orchestra.run.trial134.stdout:Total 202 kB/s | 87 kB 00:00 2026-02-26T03:16:16.631 INFO:teuthology.orchestra.run.trial134.stdout:Running transaction check 2026-02-26T03:16:16.635 INFO:teuthology.orchestra.run.trial134.stdout:Transaction check succeeded. 2026-02-26T03:16:16.635 INFO:teuthology.orchestra.run.trial134.stdout:Running transaction test 2026-02-26T03:16:16.666 INFO:teuthology.orchestra.run.trial134.stdout:Transaction test succeeded. 2026-02-26T03:16:16.666 INFO:teuthology.orchestra.run.trial134.stdout:Running transaction 2026-02-26T03:16:16.719 INFO:teuthology.orchestra.run.trial134.stdout: Preparing : 1/1 2026-02-26T03:16:16.736 INFO:teuthology.orchestra.run.trial134.stdout: Installing : rbd-fuse-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:16:16.824 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: rbd-fuse-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:16:16.888 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : rbd-fuse-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:16:16.888 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:16:16.888 INFO:teuthology.orchestra.run.trial134.stdout:Installed: 2026-02-26T03:16:16.888 INFO:teuthology.orchestra.run.trial134.stdout: rbd-fuse-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:16:16.889 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:16:16.889 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-26T03:16:16.918 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install rbd-mirror 2026-02-26T03:16:17.235 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:06:40 ago on Thu 26 Feb 2026 03:09:37 AM UTC. 2026-02-26T03:16:17.319 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-26T03:16:17.319 INFO:teuthology.orchestra.run.trial134.stdout:================================================================================ 2026-02-26T03:16:17.319 INFO:teuthology.orchestra.run.trial134.stdout: Package Arch Version Repo Size 2026-02-26T03:16:17.319 INFO:teuthology.orchestra.run.trial134.stdout:================================================================================ 2026-02-26T03:16:17.319 INFO:teuthology.orchestra.run.trial134.stdout:Installing: 2026-02-26T03:16:17.319 INFO:teuthology.orchestra.run.trial134.stdout: rbd-mirror x86_64 2:20.3.0-5312.gabffb617.el9 ceph 2.6 M 2026-02-26T03:16:17.319 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:16:17.320 INFO:teuthology.orchestra.run.trial134.stdout:Transaction Summary 2026-02-26T03:16:17.320 INFO:teuthology.orchestra.run.trial134.stdout:================================================================================ 2026-02-26T03:16:17.320 INFO:teuthology.orchestra.run.trial134.stdout:Install 1 Package 2026-02-26T03:16:17.320 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:16:17.320 INFO:teuthology.orchestra.run.trial134.stdout:Total download size: 2.6 M 2026-02-26T03:16:17.320 INFO:teuthology.orchestra.run.trial134.stdout:Installed size: 11 M 2026-02-26T03:16:17.320 INFO:teuthology.orchestra.run.trial134.stdout:Downloading Packages: 2026-02-26T03:16:18.250 INFO:teuthology.orchestra.run.trial134.stdout:rbd-mirror-20.3.0-5312.gabffb617.el9.x86_64.rpm 2.8 MB/s | 2.6 MB 00:00 2026-02-26T03:16:18.251 INFO:teuthology.orchestra.run.trial134.stdout:-------------------------------------------------------------------------------- 2026-02-26T03:16:18.251 INFO:teuthology.orchestra.run.trial134.stdout:Total 2.8 MB/s | 2.6 MB 00:00 2026-02-26T03:16:18.251 INFO:teuthology.orchestra.run.trial134.stdout:Running transaction check 2026-02-26T03:16:18.256 INFO:teuthology.orchestra.run.trial134.stdout:Transaction check succeeded. 2026-02-26T03:16:18.256 INFO:teuthology.orchestra.run.trial134.stdout:Running transaction test 2026-02-26T03:16:18.294 INFO:teuthology.orchestra.run.trial134.stdout:Transaction test succeeded. 2026-02-26T03:16:18.294 INFO:teuthology.orchestra.run.trial134.stdout:Running transaction 2026-02-26T03:16:18.459 INFO:teuthology.orchestra.run.trial134.stdout: Preparing : 1/1 2026-02-26T03:16:18.461 INFO:teuthology.orchestra.run.trial134.stdout: Installing : rbd-mirror-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:16:18.484 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: rbd-mirror-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:16:18.484 INFO:teuthology.orchestra.run.trial134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-26T03:16:18.484 INFO:teuthology.orchestra.run.trial134.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2026-02-26T03:16:18.484 INFO:teuthology.orchestra.run.trial134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-26T03:16:18.484 INFO:teuthology.orchestra.run.trial134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-26T03:16:18.484 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:16:18.806 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : rbd-mirror-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:16:18.806 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:16:18.806 INFO:teuthology.orchestra.run.trial134.stdout:Installed: 2026-02-26T03:16:18.806 INFO:teuthology.orchestra.run.trial134.stdout: rbd-mirror-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:16:18.807 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:16:18.807 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-26T03:16:18.836 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install rbd-nbd 2026-02-26T03:16:19.158 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:06:42 ago on Thu 26 Feb 2026 03:09:37 AM UTC. 2026-02-26T03:16:19.243 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-26T03:16:19.243 INFO:teuthology.orchestra.run.trial134.stdout:================================================================================ 2026-02-26T03:16:19.243 INFO:teuthology.orchestra.run.trial134.stdout: Package Architecture Version Repository Size 2026-02-26T03:16:19.243 INFO:teuthology.orchestra.run.trial134.stdout:================================================================================ 2026-02-26T03:16:19.243 INFO:teuthology.orchestra.run.trial134.stdout:Installing: 2026-02-26T03:16:19.244 INFO:teuthology.orchestra.run.trial134.stdout: rbd-nbd x86_64 2:20.3.0-5312.gabffb617.el9 ceph 166 k 2026-02-26T03:16:19.244 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:16:19.244 INFO:teuthology.orchestra.run.trial134.stdout:Transaction Summary 2026-02-26T03:16:19.244 INFO:teuthology.orchestra.run.trial134.stdout:================================================================================ 2026-02-26T03:16:19.244 INFO:teuthology.orchestra.run.trial134.stdout:Install 1 Package 2026-02-26T03:16:19.244 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:16:19.244 INFO:teuthology.orchestra.run.trial134.stdout:Total download size: 166 k 2026-02-26T03:16:19.244 INFO:teuthology.orchestra.run.trial134.stdout:Installed size: 496 k 2026-02-26T03:16:19.244 INFO:teuthology.orchestra.run.trial134.stdout:Downloading Packages: 2026-02-26T03:16:19.850 INFO:teuthology.orchestra.run.trial134.stdout:rbd-nbd-20.3.0-5312.gabffb617.el9.x86_64.rpm 274 kB/s | 166 kB 00:00 2026-02-26T03:16:19.851 INFO:teuthology.orchestra.run.trial134.stdout:-------------------------------------------------------------------------------- 2026-02-26T03:16:19.851 INFO:teuthology.orchestra.run.trial134.stdout:Total 273 kB/s | 166 kB 00:00 2026-02-26T03:16:19.851 INFO:teuthology.orchestra.run.trial134.stdout:Running transaction check 2026-02-26T03:16:19.856 INFO:teuthology.orchestra.run.trial134.stdout:Transaction check succeeded. 2026-02-26T03:16:19.856 INFO:teuthology.orchestra.run.trial134.stdout:Running transaction test 2026-02-26T03:16:19.887 INFO:teuthology.orchestra.run.trial134.stdout:Transaction test succeeded. 2026-02-26T03:16:19.887 INFO:teuthology.orchestra.run.trial134.stdout:Running transaction 2026-02-26T03:16:19.944 INFO:teuthology.orchestra.run.trial134.stdout: Preparing : 1/1 2026-02-26T03:16:19.961 INFO:teuthology.orchestra.run.trial134.stdout: Installing : rbd-nbd-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:16:20.049 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: rbd-nbd-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:16:20.113 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : rbd-nbd-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:16:20.113 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:16:20.113 INFO:teuthology.orchestra.run.trial134.stdout:Installed: 2026-02-26T03:16:20.113 INFO:teuthology.orchestra.run.trial134.stdout: rbd-nbd-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:16:20.113 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:16:20.113 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-26T03:16:20.141 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install ceph-osd-crimson 2026-02-26T03:16:20.460 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:06:43 ago on Thu 26 Feb 2026 03:09:37 AM UTC. 2026-02-26T03:16:20.525 INFO:teuthology.orchestra.run.trial134.stdout:Package ceph-osd-crimson-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:16:20.543 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-26T03:16:20.544 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-26T03:16:20.544 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-26T03:16:20.567 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install ceph-osd-crimson-debuginfo 2026-02-26T03:16:20.885 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:06:43 ago on Thu 26 Feb 2026 03:09:37 AM UTC. 2026-02-26T03:16:20.968 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-26T03:16:20.968 INFO:teuthology.orchestra.run.trial134.stdout:================================================================================ 2026-02-26T03:16:20.968 INFO:teuthology.orchestra.run.trial134.stdout: Package Arch Version Repo Size 2026-02-26T03:16:20.968 INFO:teuthology.orchestra.run.trial134.stdout:================================================================================ 2026-02-26T03:16:20.968 INFO:teuthology.orchestra.run.trial134.stdout:Installing: 2026-02-26T03:16:20.968 INFO:teuthology.orchestra.run.trial134.stdout: ceph-osd-crimson-debuginfo x86_64 2:20.3.0-5312.gabffb617.el9 ceph 429 M 2026-02-26T03:16:20.968 INFO:teuthology.orchestra.run.trial134.stdout:Installing weak dependencies: 2026-02-26T03:16:20.968 INFO:teuthology.orchestra.run.trial134.stdout: ceph-debugsource x86_64 2:20.3.0-5312.gabffb617.el9 ceph 12 M 2026-02-26T03:16:20.968 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:16:20.968 INFO:teuthology.orchestra.run.trial134.stdout:Transaction Summary 2026-02-26T03:16:20.968 INFO:teuthology.orchestra.run.trial134.stdout:================================================================================ 2026-02-26T03:16:20.968 INFO:teuthology.orchestra.run.trial134.stdout:Install 2 Packages 2026-02-26T03:16:20.968 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:16:20.969 INFO:teuthology.orchestra.run.trial134.stdout:Total download size: 441 M 2026-02-26T03:16:20.969 INFO:teuthology.orchestra.run.trial134.stdout:Installed size: 2.7 G 2026-02-26T03:16:20.969 INFO:teuthology.orchestra.run.trial134.stdout:Downloading Packages: 2026-02-26T03:16:30.417 INFO:teuthology.orchestra.run.trial134.stdout:(1/2): ceph-debugsource-20.3.0-5312.gabffb617.e 1.3 MB/s | 12 MB 00:09 2026-02-26T03:31:44.504 INFO:teuthology.orchestra.run.trial134.stdout:(2/2): ceph-osd-crimson-debuginfo-20.3.0-5312.g 475 kB/s | 429 MB 15:23 2026-02-26T03:31:44.504 INFO:teuthology.orchestra.run.trial134.stdout:-------------------------------------------------------------------------------- 2026-02-26T03:31:44.505 INFO:teuthology.orchestra.run.trial134.stdout:Total 489 kB/s | 441 MB 15:23 2026-02-26T03:31:44.505 INFO:teuthology.orchestra.run.trial134.stdout:Running transaction check 2026-02-26T03:31:44.511 INFO:teuthology.orchestra.run.trial134.stdout:Transaction check succeeded. 2026-02-26T03:31:44.511 INFO:teuthology.orchestra.run.trial134.stdout:Running transaction test 2026-02-26T03:31:45.875 INFO:teuthology.orchestra.run.trial134.stdout:Transaction test succeeded. 2026-02-26T03:31:45.876 INFO:teuthology.orchestra.run.trial134.stdout:Running transaction 2026-02-26T03:31:49.472 INFO:teuthology.orchestra.run.trial134.stdout: Preparing : 1/1 2026-02-26T03:32:11.532 INFO:teuthology.orchestra.run.trial134.stdout: Installing : ceph-debugsource-2:20.3.0-5312.gabffb617.el9.x86_64 1/2 2026-02-26T03:32:11.552 INFO:teuthology.orchestra.run.trial134.stdout: Installing : ceph-osd-crimson-debuginfo-2:20.3.0-5312.gabffb617.e 2/2 2026-02-26T03:32:13.450 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: ceph-osd-crimson-debuginfo-2:20.3.0-5312.gabffb617.e 2/2 2026-02-26T03:32:13.450 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : ceph-debugsource-2:20.3.0-5312.gabffb617.el9.x86_64 1/2 2026-02-26T03:32:13.514 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : ceph-osd-crimson-debuginfo-2:20.3.0-5312.gabffb617.e 2/2 2026-02-26T03:32:13.515 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:32:13.515 INFO:teuthology.orchestra.run.trial134.stdout:Installed: 2026-02-26T03:32:13.515 INFO:teuthology.orchestra.run.trial134.stdout: ceph-debugsource-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:32:13.515 INFO:teuthology.orchestra.run.trial134.stdout: ceph-osd-crimson-debuginfo-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:32:13.515 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:32:13.515 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-26T03:32:13.601 DEBUG:teuthology.parallel:result is None 2026-02-26T03:32:13.603 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=abffb617683951511fccf3c3cab6e219f2ec83a8 2026-02-26T03:32:13.776 DEBUG:teuthology.orchestra.run.trial134:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-26T03:32:13.799 INFO:teuthology.orchestra.run.trial134.stdout:20.3.0-5312.gabffb617.el9 2026-02-26T03:32:13.800 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5312.gabffb617.el9 2026-02-26T03:32:13.800 INFO:teuthology.task.install:The correct ceph version 20.3.0-5312.gabffb617 is installed. 2026-02-26T03:32:13.802 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2026-02-26T03:32:13.802 DEBUG:teuthology.orchestra.run.trial134:> set -ex 2026-02-26T03:32:13.802 DEBUG:teuthology.orchestra.run.trial134:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-26T03:32:13.872 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2026-02-26T03:32:13.873 DEBUG:teuthology.orchestra.run.trial134:> set -ex 2026-02-26T03:32:13.873 DEBUG:teuthology.orchestra.run.trial134:> sudo dd of=/usr/bin/daemon-helper 2026-02-26T03:32:13.938 DEBUG:teuthology.orchestra.run.trial134:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-26T03:32:14.008 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2026-02-26T03:32:14.008 DEBUG:teuthology.orchestra.run.trial134:> set -ex 2026-02-26T03:32:14.008 DEBUG:teuthology.orchestra.run.trial134:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-26T03:32:14.072 DEBUG:teuthology.orchestra.run.trial134:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-26T03:32:14.136 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2026-02-26T03:32:14.136 DEBUG:teuthology.orchestra.run.trial134:> set -ex 2026-02-26T03:32:14.136 DEBUG:teuthology.orchestra.run.trial134:> sudo dd of=/usr/bin/stdin-killer 2026-02-26T03:32:14.200 DEBUG:teuthology.orchestra.run.trial134:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-26T03:32:14.263 INFO:teuthology.run_tasks:Running task pexec... 2026-02-26T03:32:14.290 INFO:teuthology.task.pexec:Executing custom commands... 2026-02-26T03:32:14.291 DEBUG:teuthology.orchestra.run.trial134:> TESTDIR=/home/ubuntu/cephtest bash -s 2026-02-26T03:32:14.305 INFO:teuthology.task.pexec:Running commands on host ubuntu@trial134.front.sepia.ceph.com 2026-02-26T03:32:14.305 INFO:teuthology.task.pexec:sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson 2026-02-26T03:32:14.357 DEBUG:teuthology.parallel:result is None 2026-02-26T03:32:14.357 INFO:teuthology.run_tasks:Running task install... 2026-02-26T03:32:14.370 DEBUG:teuthology.task.install:project ceph 2026-02-26T03:32:14.371 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'debug', 'sha1': 'abffb617683951511fccf3c3cab6e219f2ec83a8'}} 2026-02-26T03:32:14.371 DEBUG:teuthology.task.install:config {'flavor': 'debug', 'sha1': 'abffb617683951511fccf3c3cab6e219f2ec83a8'} 2026-02-26T03:32:14.371 INFO:teuthology.task.install:Using flavor: debug 2026-02-26T03:32:14.380 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-osd', 'ceph-osd-classic', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-osd', 'ceph-osd-classic', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-26T03:32:14.380 INFO:teuthology.task.install:extra packages: [] 2026-02-26T03:32:14.381 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'debug', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'abffb617683951511fccf3c3cab6e219f2ec83a8', 'tag': None, 'wait_for_package': False} 2026-02-26T03:32:14.381 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=abffb617683951511fccf3c3cab6e219f2ec83a8 2026-02-26T03:32:14.546 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-shraddhaag-peering-bug/abffb617683951511fccf3c3cab6e219f2ec83a8/centos/9/flavors/debug/ 2026-02-26T03:32:14.546 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5312.gabffb617 2026-02-26T03:32:14.925 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-shraddhaag-peering-bug/abffb617683951511fccf3c3cab6e219f2ec83a8/centos/9/flavors/debug/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-shraddhaag-peering-bug/abffb617683951511fccf3c3cab6e219f2ec83a8/centos/9/flavors/debug/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-shraddhaag-peering-bug/abffb617683951511fccf3c3cab6e219f2ec83a8/centos/9/flavors/debug/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-26T03:32:14.925 DEBUG:teuthology.orchestra.run.trial134:> set -ex 2026-02-26T03:32:14.925 DEBUG:teuthology.orchestra.run.trial134:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-26T03:32:14.951 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2026-02-26T03:32:14.951 DEBUG:teuthology.orchestra.run.trial134:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-26T03:32:15.015 DEBUG:teuthology.orchestra.run.trial134:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-26T03:32:15.079 DEBUG:teuthology.orchestra.run.trial134:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-26T03:32:15.134 INFO:teuthology.orchestra.run.trial134.stdout:check_obsoletes = 1 2026-02-26T03:32:15.145 DEBUG:teuthology.orchestra.run.trial134:> sudo yum clean all 2026-02-26T03:32:15.304 INFO:teuthology.orchestra.run.trial134.stdout:56 files removed 2026-02-26T03:32:15.319 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install ceph-radosgw 2026-02-26T03:32:16.196 INFO:teuthology.orchestra.run.trial134.stdout:ceph packages for x86_64 131 kB/s | 94 kB 00:00 2026-02-26T03:32:16.863 INFO:teuthology.orchestra.run.trial134.stdout:ceph noarch packages 38 kB/s | 24 kB 00:00 2026-02-26T03:32:17.458 INFO:teuthology.orchestra.run.trial134.stdout:ceph source packages 3.6 kB/s | 2.1 kB 00:00 2026-02-26T03:32:19.265 INFO:teuthology.orchestra.run.trial134.stdout:CentOS Stream 9 - BaseOS 5.0 MB/s | 8.9 MB 00:01 2026-02-26T03:32:22.520 INFO:teuthology.orchestra.run.trial134.stdout:CentOS Stream 9 - AppStream 10 MB/s | 27 MB 00:02 2026-02-26T03:32:26.573 INFO:teuthology.orchestra.run.trial134.stdout:CentOS Stream 9 - CRB 4.9 MB/s | 8.0 MB 00:01 2026-02-26T03:32:27.544 INFO:teuthology.orchestra.run.trial134.stdout:CentOS Stream 9 - Extras packages 92 kB/s | 20 kB 00:00 2026-02-26T03:32:28.022 INFO:teuthology.orchestra.run.trial134.stdout:Extra Packages for Enterprise Linux 49 MB/s | 20 MB 00:00 2026-02-26T03:32:31.332 INFO:teuthology.orchestra.run.trial134.stdout:lab-extras 1.4 MB/s | 50 kB 00:00 2026-02-26T03:32:32.525 INFO:teuthology.orchestra.run.trial134.stdout:Package ceph-radosgw-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:32:32.543 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-26T03:32:32.544 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-26T03:32:32.544 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-26T03:32:32.568 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install ceph-test 2026-02-26T03:32:32.889 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:00:01 ago on Thu 26 Feb 2026 03:32:31 AM UTC. 2026-02-26T03:32:32.954 INFO:teuthology.orchestra.run.trial134.stdout:Package ceph-test-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:32:32.971 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-26T03:32:32.972 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-26T03:32:32.972 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-26T03:32:32.994 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install ceph 2026-02-26T03:32:33.317 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:00:02 ago on Thu 26 Feb 2026 03:32:31 AM UTC. 2026-02-26T03:32:33.383 INFO:teuthology.orchestra.run.trial134.stdout:Package ceph-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:32:33.401 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-26T03:32:33.402 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-26T03:32:33.402 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-26T03:32:33.424 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install ceph-base 2026-02-26T03:32:33.744 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:00:02 ago on Thu 26 Feb 2026 03:32:31 AM UTC. 2026-02-26T03:32:33.809 INFO:teuthology.orchestra.run.trial134.stdout:Package ceph-base-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:32:33.827 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-26T03:32:33.827 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-26T03:32:33.827 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-26T03:32:33.848 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install cephadm 2026-02-26T03:32:34.164 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:00:03 ago on Thu 26 Feb 2026 03:32:31 AM UTC. 2026-02-26T03:32:34.229 INFO:teuthology.orchestra.run.trial134.stdout:Package cephadm-2:20.3.0-5312.gabffb617.el9.noarch is already installed. 2026-02-26T03:32:34.247 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-26T03:32:34.248 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-26T03:32:34.248 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-26T03:32:34.269 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install ceph-immutable-object-cache 2026-02-26T03:32:34.591 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:00:03 ago on Thu 26 Feb 2026 03:32:31 AM UTC. 2026-02-26T03:32:34.657 INFO:teuthology.orchestra.run.trial134.stdout:Package ceph-immutable-object-cache-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:32:34.674 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-26T03:32:34.675 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-26T03:32:34.675 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-26T03:32:34.696 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install ceph-mgr 2026-02-26T03:32:35.015 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:00:04 ago on Thu 26 Feb 2026 03:32:31 AM UTC. 2026-02-26T03:32:35.079 INFO:teuthology.orchestra.run.trial134.stdout:Package ceph-mgr-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:32:35.097 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-26T03:32:35.098 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-26T03:32:35.098 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-26T03:32:35.120 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install ceph-mgr-dashboard 2026-02-26T03:32:35.444 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:00:04 ago on Thu 26 Feb 2026 03:32:31 AM UTC. 2026-02-26T03:32:35.510 INFO:teuthology.orchestra.run.trial134.stdout:Package ceph-mgr-dashboard-2:20.3.0-5312.gabffb617.el9.noarch is already installed. 2026-02-26T03:32:35.528 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-26T03:32:35.528 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-26T03:32:35.529 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-26T03:32:35.551 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-26T03:32:35.869 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:00:04 ago on Thu 26 Feb 2026 03:32:31 AM UTC. 2026-02-26T03:32:35.934 INFO:teuthology.orchestra.run.trial134.stdout:Package ceph-mgr-diskprediction-local-2:20.3.0-5312.gabffb617.el9.noarch is already installed. 2026-02-26T03:32:35.952 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-26T03:32:35.952 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-26T03:32:35.952 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-26T03:32:35.975 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install ceph-mgr-rook 2026-02-26T03:32:36.293 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:00:05 ago on Thu 26 Feb 2026 03:32:31 AM UTC. 2026-02-26T03:32:36.359 INFO:teuthology.orchestra.run.trial134.stdout:Package ceph-mgr-rook-2:20.3.0-5312.gabffb617.el9.noarch is already installed. 2026-02-26T03:32:36.377 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-26T03:32:36.378 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-26T03:32:36.378 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-26T03:32:36.400 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install ceph-mgr-cephadm 2026-02-26T03:32:36.718 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:00:05 ago on Thu 26 Feb 2026 03:32:31 AM UTC. 2026-02-26T03:32:36.785 INFO:teuthology.orchestra.run.trial134.stdout:Package ceph-mgr-cephadm-2:20.3.0-5312.gabffb617.el9.noarch is already installed. 2026-02-26T03:32:36.803 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-26T03:32:36.804 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-26T03:32:36.804 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-26T03:32:36.826 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install ceph-osd 2026-02-26T03:32:37.145 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:00:06 ago on Thu 26 Feb 2026 03:32:31 AM UTC. 2026-02-26T03:32:37.209 INFO:teuthology.orchestra.run.trial134.stdout:Package ceph-osd-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:32:37.227 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-26T03:32:37.228 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-26T03:32:37.229 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-26T03:32:37.251 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install ceph-osd-classic 2026-02-26T03:32:37.571 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:00:06 ago on Thu 26 Feb 2026 03:32:31 AM UTC. 2026-02-26T03:32:37.636 INFO:teuthology.orchestra.run.trial134.stdout:Package ceph-osd-classic-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:32:37.654 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-26T03:32:37.654 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-26T03:32:37.654 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-26T03:32:37.677 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install ceph-fuse 2026-02-26T03:32:37.993 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:00:06 ago on Thu 26 Feb 2026 03:32:31 AM UTC. 2026-02-26T03:32:38.057 INFO:teuthology.orchestra.run.trial134.stdout:Package ceph-fuse-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:32:38.075 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-26T03:32:38.076 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-26T03:32:38.076 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-26T03:32:38.099 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install ceph-volume 2026-02-26T03:32:38.419 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:00:07 ago on Thu 26 Feb 2026 03:32:31 AM UTC. 2026-02-26T03:32:38.484 INFO:teuthology.orchestra.run.trial134.stdout:Package ceph-volume-2:20.3.0-5312.gabffb617.el9.noarch is already installed. 2026-02-26T03:32:38.502 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-26T03:32:38.503 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-26T03:32:38.503 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-26T03:32:38.526 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install librados-devel 2026-02-26T03:32:38.847 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:00:07 ago on Thu 26 Feb 2026 03:32:31 AM UTC. 2026-02-26T03:32:38.912 INFO:teuthology.orchestra.run.trial134.stdout:Package librados-devel-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:32:38.930 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-26T03:32:38.930 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-26T03:32:38.931 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-26T03:32:38.953 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install libcephfs2 2026-02-26T03:32:39.274 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:00:08 ago on Thu 26 Feb 2026 03:32:31 AM UTC. 2026-02-26T03:32:39.339 INFO:teuthology.orchestra.run.trial134.stdout:Package libcephfs2-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:32:39.357 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-26T03:32:39.358 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-26T03:32:39.358 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-26T03:32:39.380 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install libcephfs-devel 2026-02-26T03:32:39.698 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:00:08 ago on Thu 26 Feb 2026 03:32:31 AM UTC. 2026-02-26T03:32:39.763 INFO:teuthology.orchestra.run.trial134.stdout:Package libcephfs-devel-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:32:39.781 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-26T03:32:39.781 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-26T03:32:39.781 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-26T03:32:39.803 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install librados2 2026-02-26T03:32:40.123 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:00:09 ago on Thu 26 Feb 2026 03:32:31 AM UTC. 2026-02-26T03:32:40.188 INFO:teuthology.orchestra.run.trial134.stdout:Package librados2-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:32:40.206 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-26T03:32:40.206 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-26T03:32:40.206 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-26T03:32:40.229 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install librbd1 2026-02-26T03:32:40.544 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:00:09 ago on Thu 26 Feb 2026 03:32:31 AM UTC. 2026-02-26T03:32:40.609 INFO:teuthology.orchestra.run.trial134.stdout:Package librbd1-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:32:40.626 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-26T03:32:40.627 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-26T03:32:40.627 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-26T03:32:40.649 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install python3-rados 2026-02-26T03:32:40.970 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:00:09 ago on Thu 26 Feb 2026 03:32:31 AM UTC. 2026-02-26T03:32:41.036 INFO:teuthology.orchestra.run.trial134.stdout:Package python3-rados-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:32:41.054 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-26T03:32:41.055 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-26T03:32:41.055 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-26T03:32:41.078 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install python3-rgw 2026-02-26T03:32:41.398 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:00:10 ago on Thu 26 Feb 2026 03:32:31 AM UTC. 2026-02-26T03:32:41.464 INFO:teuthology.orchestra.run.trial134.stdout:Package python3-rgw-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:32:41.481 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-26T03:32:41.482 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-26T03:32:41.482 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-26T03:32:41.505 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install python3-cephfs 2026-02-26T03:32:41.826 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:00:10 ago on Thu 26 Feb 2026 03:32:31 AM UTC. 2026-02-26T03:32:41.891 INFO:teuthology.orchestra.run.trial134.stdout:Package python3-cephfs-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:32:41.909 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-26T03:32:41.910 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-26T03:32:41.910 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-26T03:32:41.932 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install python3-rbd 2026-02-26T03:32:42.253 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:00:11 ago on Thu 26 Feb 2026 03:32:31 AM UTC. 2026-02-26T03:32:42.320 INFO:teuthology.orchestra.run.trial134.stdout:Package python3-rbd-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:32:42.337 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-26T03:32:42.338 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-26T03:32:42.338 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-26T03:32:42.361 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install rbd-fuse 2026-02-26T03:32:42.684 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:00:11 ago on Thu 26 Feb 2026 03:32:31 AM UTC. 2026-02-26T03:32:42.749 INFO:teuthology.orchestra.run.trial134.stdout:Package rbd-fuse-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:32:42.767 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-26T03:32:42.767 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-26T03:32:42.767 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-26T03:32:42.789 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install rbd-mirror 2026-02-26T03:32:43.108 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:00:12 ago on Thu 26 Feb 2026 03:32:31 AM UTC. 2026-02-26T03:32:43.174 INFO:teuthology.orchestra.run.trial134.stdout:Package rbd-mirror-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:32:43.191 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-26T03:32:43.192 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-26T03:32:43.192 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-26T03:32:43.215 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install rbd-nbd 2026-02-26T03:32:43.534 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:00:12 ago on Thu 26 Feb 2026 03:32:31 AM UTC. 2026-02-26T03:32:43.598 INFO:teuthology.orchestra.run.trial134.stdout:Package rbd-nbd-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:32:43.616 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-26T03:32:43.617 INFO:teuthology.orchestra.run.trial134.stdout:Nothing to do. 2026-02-26T03:32:43.617 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-26T03:32:43.642 DEBUG:teuthology.parallel:result is None 2026-02-26T03:32:43.643 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=abffb617683951511fccf3c3cab6e219f2ec83a8 2026-02-26T03:32:43.814 DEBUG:teuthology.orchestra.run.trial134:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-26T03:32:43.835 INFO:teuthology.orchestra.run.trial134.stdout:20.3.0-5312.gabffb617.el9 2026-02-26T03:32:43.836 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5312.gabffb617.el9 2026-02-26T03:32:43.836 INFO:teuthology.task.install:The correct ceph version 20.3.0-5312.gabffb617 is installed. 2026-02-26T03:32:43.839 INFO:teuthology.task.install.util:Utilities already shipped, skip it... 2026-02-26T03:32:43.839 INFO:teuthology.run_tasks:Running task ceph... 2026-02-26T03:32:43.947 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2026-02-26T03:32:43.948 DEBUG:teuthology.orchestra.run.trial134:> sudo chmod 777 /var/log/ceph 2026-02-26T03:32:43.973 INFO:tasks.ceph:Disabling ceph logrotate... 2026-02-26T03:32:43.973 DEBUG:teuthology.orchestra.run.trial134:> sudo rm -f -- /etc/logrotate.d/ceph 2026-02-26T03:32:44.037 INFO:tasks.ceph:Creating extra log directories... 2026-02-26T03:32:44.037 DEBUG:teuthology.orchestra.run.trial134:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2026-02-26T03:32:44.113 INFO:tasks.ceph:Creating ceph cluster ceph... 2026-02-26T03:32:44.114 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore max concurrent transactions': 8}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2026-02-26T03:32:44.114 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/mnt/teuthology/shraddhaag-2026-02-26_02:22:56-crimson-rados-main-distro-debug-trial/71162', 'branch': 'main', 'description': 'crimson-rados/perf/{0-crimson_install clusters/{crimson-fixed-1} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/segmented$/{crimson_seastore_segmented} perf_ignorelist settings/optimized workloads/radosbench_4K_rand_read}', 'email': None, 'first_in_suite': False, 'flavor': 'debug', 'job_id': '71162', 'kernel': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'trial', 'name': 'shraddhaag-2026-02-26_02:22:56-crimson-rados-main-distro-debug-trial', 'no_nested_subset': False, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd': {'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore max concurrent transactions': 8}}, 'crimson_compat': True, 'flavor': 'debug', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH'], 'sha1': 'abffb617683951511fccf3c3cab6e219f2ec83a8'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'global': {'osd crush chooseleaf type': 0, 'osd pool default pg num': 128, 'osd pool default pgp num': 128, 'osd pool default size': 2}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'debug', 'sha1': 'abffb617683951511fccf3c3cab6e219f2ec83a8'}}, 'workunit': {'branch': 'main', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '1f87b69b1a8041112e26ae0ea3c0a2a0fd14d996'}}, 'owner': 'scheduled_shraddhaag@soko04.front.sepia.ceph.com', 'priority': 101, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 5117, 'sha1': 'abffb617683951511fccf3c3cab6e219f2ec83a8', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph_1f87b69b1a8041112e26ae0ea3c0a2a0fd14d996/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph.git', 'suite_sha1': '1f87b69b1a8041112e26ae0ea3c0a2a0fd14d996', 'targets': {'trial134.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBdXNEyjnAMdr0HYsmPoZe4BD3+dkvnDAY9Vor0jDPkZfr9+oShjMH/EKgesItdbPYxmG5j/0mDr0hPhMct8YzE='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'flavor': 'debug', 'sha1': 'abffb617683951511fccf3c3cab6e219f2ec83a8'}}, {'pexec': {'all': ['sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson']}}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore max concurrent transactions': 8}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}}, 'wait-for-scrub': False, 'crimson_compat': True, 'flavor': 'debug', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH'], 'sha1': 'abffb617683951511fccf3c3cab6e219f2ec83a8', 'cluster': 'ceph'}}, {'ssh_keys': None}, {'cbt': {'benchmarks': {'radosbench': {'concurrent_ops': 4, 'concurrent_procs': 2, 'op_size': [4096], 'pool_monitoring_list': ['collectl'], 'pool_profile': 'replicated', 'readmode': 'rand', 'run_monitoring_list': ['collectl'], 'time': 60, 'write_only': False}}, 'cluster': {'iterations': 1, 'osds_per_node': 3, 'pool_profiles': {'replicated': {'pg_size': 256, 'pgp_size': 256, 'replication': 'replicated'}}, 'user': 'ubuntu'}, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '6e3d0419ccca0e75a03e08feb708946e198236f4', 'timestamp': '2026-02-26_02:22:56', 'tube': 'trial', 'user': 'shraddhaag', 'verbose': False, 'worker_log': '/home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.904699'} 2026-02-26T03:32:44.115 DEBUG:teuthology.orchestra.run.trial134:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2026-02-26T03:32:44.159 DEBUG:teuthology.orchestra.run.trial134:> sudo install -d -m0777 -- /var/run/ceph 2026-02-26T03:32:44.225 DEBUG:teuthology.orchestra.run.trial134:> set -ex 2026-02-26T03:32:44.225 DEBUG:teuthology.orchestra.run.trial134:> dd if=/scratch_devs of=/dev/stdout 2026-02-26T03:32:44.282 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2026-02-26T03:32:44.282 DEBUG:teuthology.orchestra.run.trial134:> stat /dev/vg_nvme/lv_1 2026-02-26T03:32:44.337 INFO:teuthology.orchestra.run.trial134.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2026-02-26T03:32:44.337 INFO:teuthology.orchestra.run.trial134.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-26T03:32:44.337 INFO:teuthology.orchestra.run.trial134.stdout:Device: 6h/6d Inode: 1312 Links: 1 2026-02-26T03:32:44.337 INFO:teuthology.orchestra.run.trial134.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-26T03:32:44.337 INFO:teuthology.orchestra.run.trial134.stdout:Context: system_u:object_r:device_t:s0 2026-02-26T03:32:44.337 INFO:teuthology.orchestra.run.trial134.stdout:Access: 2026-02-26 03:16:18.623247101 +0000 2026-02-26T03:32:44.337 INFO:teuthology.orchestra.run.trial134.stdout:Modify: 2026-02-26 03:12:09.962994854 +0000 2026-02-26T03:32:44.337 INFO:teuthology.orchestra.run.trial134.stdout:Change: 2026-02-26 03:12:09.962994854 +0000 2026-02-26T03:32:44.337 INFO:teuthology.orchestra.run.trial134.stdout: Birth: 2026-02-26 03:12:09.962994854 +0000 2026-02-26T03:32:44.337 DEBUG:teuthology.orchestra.run.trial134:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2026-02-26T03:32:44.399 INFO:teuthology.orchestra.run.trial134.stderr:1+0 records in 2026-02-26T03:32:44.399 INFO:teuthology.orchestra.run.trial134.stderr:1+0 records out 2026-02-26T03:32:44.399 INFO:teuthology.orchestra.run.trial134.stderr:512 bytes copied, 0.000150091 s, 3.4 MB/s 2026-02-26T03:32:44.401 DEBUG:teuthology.orchestra.run.trial134:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2026-02-26T03:32:44.458 DEBUG:teuthology.orchestra.run.trial134:> stat /dev/vg_nvme/lv_2 2026-02-26T03:32:44.512 INFO:teuthology.orchestra.run.trial134.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2026-02-26T03:32:44.513 INFO:teuthology.orchestra.run.trial134.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-26T03:32:44.513 INFO:teuthology.orchestra.run.trial134.stdout:Device: 6h/6d Inode: 1293 Links: 1 2026-02-26T03:32:44.513 INFO:teuthology.orchestra.run.trial134.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-26T03:32:44.513 INFO:teuthology.orchestra.run.trial134.stdout:Context: system_u:object_r:device_t:s0 2026-02-26T03:32:44.513 INFO:teuthology.orchestra.run.trial134.stdout:Access: 2026-02-26 03:16:18.623247101 +0000 2026-02-26T03:32:44.513 INFO:teuthology.orchestra.run.trial134.stdout:Modify: 2026-02-26 03:12:09.962994854 +0000 2026-02-26T03:32:44.513 INFO:teuthology.orchestra.run.trial134.stdout:Change: 2026-02-26 03:12:09.962994854 +0000 2026-02-26T03:32:44.513 INFO:teuthology.orchestra.run.trial134.stdout: Birth: 2026-02-26 03:12:09.962994854 +0000 2026-02-26T03:32:44.513 DEBUG:teuthology.orchestra.run.trial134:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2026-02-26T03:32:44.575 INFO:teuthology.orchestra.run.trial134.stderr:1+0 records in 2026-02-26T03:32:44.575 INFO:teuthology.orchestra.run.trial134.stderr:1+0 records out 2026-02-26T03:32:44.575 INFO:teuthology.orchestra.run.trial134.stderr:512 bytes copied, 0.000163565 s, 3.1 MB/s 2026-02-26T03:32:44.576 DEBUG:teuthology.orchestra.run.trial134:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2026-02-26T03:32:44.633 DEBUG:teuthology.orchestra.run.trial134:> stat /dev/vg_nvme/lv_3 2026-02-26T03:32:44.688 INFO:teuthology.orchestra.run.trial134.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2026-02-26T03:32:44.688 INFO:teuthology.orchestra.run.trial134.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-26T03:32:44.688 INFO:teuthology.orchestra.run.trial134.stdout:Device: 6h/6d Inode: 1292 Links: 1 2026-02-26T03:32:44.688 INFO:teuthology.orchestra.run.trial134.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-26T03:32:44.688 INFO:teuthology.orchestra.run.trial134.stdout:Context: system_u:object_r:device_t:s0 2026-02-26T03:32:44.688 INFO:teuthology.orchestra.run.trial134.stdout:Access: 2026-02-26 03:16:18.623247101 +0000 2026-02-26T03:32:44.688 INFO:teuthology.orchestra.run.trial134.stdout:Modify: 2026-02-26 03:12:09.962994854 +0000 2026-02-26T03:32:44.688 INFO:teuthology.orchestra.run.trial134.stdout:Change: 2026-02-26 03:12:09.962994854 +0000 2026-02-26T03:32:44.688 INFO:teuthology.orchestra.run.trial134.stdout: Birth: 2026-02-26 03:12:09.962994854 +0000 2026-02-26T03:32:44.688 DEBUG:teuthology.orchestra.run.trial134:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2026-02-26T03:32:44.750 INFO:teuthology.orchestra.run.trial134.stderr:1+0 records in 2026-02-26T03:32:44.751 INFO:teuthology.orchestra.run.trial134.stderr:1+0 records out 2026-02-26T03:32:44.751 INFO:teuthology.orchestra.run.trial134.stderr:512 bytes copied, 0.0001501 s, 3.4 MB/s 2026-02-26T03:32:44.752 DEBUG:teuthology.orchestra.run.trial134:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2026-02-26T03:32:44.809 DEBUG:teuthology.orchestra.run.trial134:> stat /dev/vg_nvme/lv_4 2026-02-26T03:32:44.864 INFO:teuthology.orchestra.run.trial134.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2026-02-26T03:32:44.864 INFO:teuthology.orchestra.run.trial134.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-26T03:32:44.864 INFO:teuthology.orchestra.run.trial134.stdout:Device: 6h/6d Inode: 1297 Links: 1 2026-02-26T03:32:44.864 INFO:teuthology.orchestra.run.trial134.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-26T03:32:44.864 INFO:teuthology.orchestra.run.trial134.stdout:Context: system_u:object_r:device_t:s0 2026-02-26T03:32:44.864 INFO:teuthology.orchestra.run.trial134.stdout:Access: 2026-02-26 03:16:18.624247086 +0000 2026-02-26T03:32:44.864 INFO:teuthology.orchestra.run.trial134.stdout:Modify: 2026-02-26 03:12:09.962994854 +0000 2026-02-26T03:32:44.864 INFO:teuthology.orchestra.run.trial134.stdout:Change: 2026-02-26 03:12:09.962994854 +0000 2026-02-26T03:32:44.864 INFO:teuthology.orchestra.run.trial134.stdout: Birth: 2026-02-26 03:12:09.962994854 +0000 2026-02-26T03:32:44.865 DEBUG:teuthology.orchestra.run.trial134:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2026-02-26T03:32:44.927 INFO:teuthology.orchestra.run.trial134.stderr:1+0 records in 2026-02-26T03:32:44.927 INFO:teuthology.orchestra.run.trial134.stderr:1+0 records out 2026-02-26T03:32:44.927 INFO:teuthology.orchestra.run.trial134.stderr:512 bytes copied, 0.000147887 s, 3.5 MB/s 2026-02-26T03:32:44.928 DEBUG:teuthology.orchestra.run.trial134:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2026-02-26T03:32:44.984 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-26T03:32:44.984 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@trial134.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2026-02-26T03:32:44.984 INFO:tasks.ceph:Generating config... 2026-02-26T03:32:44.987 INFO:tasks.ceph:[osd] debug monc = 20 2026-02-26T03:32:44.987 INFO:tasks.ceph:[osd] crimson cpu num = 4 2026-02-26T03:32:44.987 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2026-02-26T03:32:44.987 INFO:tasks.ceph:[osd] debug filestore = 0/0 2026-02-26T03:32:44.987 INFO:tasks.ceph:[osd] debug journal = 0/0 2026-02-26T03:32:44.987 INFO:tasks.ceph:[osd] debug ms = 0/0 2026-02-26T03:32:44.987 INFO:tasks.ceph:[osd] debug osd = 0/0 2026-02-26T03:32:44.988 INFO:tasks.ceph:[osd] debug seastore = 20 2026-02-26T03:32:44.988 INFO:tasks.ceph:[osd] debug seastore backref = 20 2026-02-26T03:32:44.988 INFO:tasks.ceph:[osd] debug seastore cache = 20 2026-02-26T03:32:44.988 INFO:tasks.ceph:[osd] debug seastore cleaner = 20 2026-02-26T03:32:44.988 INFO:tasks.ceph:[osd] debug seastore device = 20 2026-02-26T03:32:44.988 INFO:tasks.ceph:[osd] debug seastore epm = 20 2026-02-26T03:32:44.988 INFO:tasks.ceph:[osd] debug seastore fixedkv tree = 20 2026-02-26T03:32:44.988 INFO:tasks.ceph:[osd] debug seastore journal = 20 2026-02-26T03:32:44.988 INFO:tasks.ceph:[osd] debug seastore lba = 20 2026-02-26T03:32:44.988 INFO:tasks.ceph:[osd] debug seastore odata = 20 2026-02-26T03:32:44.988 INFO:tasks.ceph:[osd] debug seastore omap = 20 2026-02-26T03:32:44.988 INFO:tasks.ceph:[osd] debug seastore onode = 20 2026-02-26T03:32:44.989 INFO:tasks.ceph:[osd] debug seastore t = 20 2026-02-26T03:32:44.989 INFO:tasks.ceph:[osd] debug seastore tm = 20 2026-02-26T03:32:44.989 INFO:tasks.ceph:[osd] osd objectstore = seastore 2026-02-26T03:32:44.989 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2026-02-26T03:32:44.989 INFO:tasks.ceph:[osd] seastore cachepin size pershard = 64M 2026-02-26T03:32:44.989 INFO:tasks.ceph:[osd] seastore max concurrent transactions = 8 2026-02-26T03:32:44.989 INFO:tasks.ceph:[global] auth client required = none 2026-02-26T03:32:44.989 INFO:tasks.ceph:[global] auth cluster required = none 2026-02-26T03:32:44.989 INFO:tasks.ceph:[global] auth service required = none 2026-02-26T03:32:44.989 INFO:tasks.ceph:[global] auth supported = none 2026-02-26T03:32:44.989 INFO:tasks.ceph:[global] debug asok = 0/0 2026-02-26T03:32:44.989 INFO:tasks.ceph:[global] debug auth = 0/0 2026-02-26T03:32:44.990 INFO:tasks.ceph:[global] debug bdev = 0/0 2026-02-26T03:32:44.990 INFO:tasks.ceph:[global] debug bluefs = 0/0 2026-02-26T03:32:44.990 INFO:tasks.ceph:[global] debug bluestore = 0/0 2026-02-26T03:32:44.990 INFO:tasks.ceph:[global] debug buffer = 0/0 2026-02-26T03:32:44.990 INFO:tasks.ceph:[global] debug civetweb = 0/0 2026-02-26T03:32:44.990 INFO:tasks.ceph:[global] debug client = 0/0 2026-02-26T03:32:44.990 INFO:tasks.ceph:[global] debug compressor = 0/0 2026-02-26T03:32:44.990 INFO:tasks.ceph:[global] debug context = 0/0 2026-02-26T03:32:44.990 INFO:tasks.ceph:[global] debug crush = 0/0 2026-02-26T03:32:44.990 INFO:tasks.ceph:[global] debug crypto = 0/0 2026-02-26T03:32:44.990 INFO:tasks.ceph:[global] debug dpdk = 0/0 2026-02-26T03:32:44.990 INFO:tasks.ceph:[global] debug eventtrace = 0/0 2026-02-26T03:32:44.990 INFO:tasks.ceph:[global] debug filer = 0/0 2026-02-26T03:32:44.991 INFO:tasks.ceph:[global] debug filestore = 0/0 2026-02-26T03:32:44.991 INFO:tasks.ceph:[global] debug finisher = 0/0 2026-02-26T03:32:44.991 INFO:tasks.ceph:[global] debug fuse = 0/0 2026-02-26T03:32:44.991 INFO:tasks.ceph:[global] debug heartbeatmap = 0/0 2026-02-26T03:32:44.991 INFO:tasks.ceph:[global] debug javaclient = 0/0 2026-02-26T03:32:44.991 INFO:tasks.ceph:[global] debug journal = 0/0 2026-02-26T03:32:44.991 INFO:tasks.ceph:[global] debug journaler = 0/0 2026-02-26T03:32:44.991 INFO:tasks.ceph:[global] debug kstore = 0/0 2026-02-26T03:32:44.991 INFO:tasks.ceph:[global] debug leveldb = 0/0 2026-02-26T03:32:44.991 INFO:tasks.ceph:[global] debug lockdep = 0/0 2026-02-26T03:32:44.991 INFO:tasks.ceph:[global] debug mds = 0/0 2026-02-26T03:32:44.991 INFO:tasks.ceph:[global] debug mds balancer = 0/0 2026-02-26T03:32:44.991 INFO:tasks.ceph:[global] debug mds locker = 0/0 2026-02-26T03:32:44.992 INFO:tasks.ceph:[global] debug mds log = 0/0 2026-02-26T03:32:44.992 INFO:tasks.ceph:[global] debug mds log expire = 0/0 2026-02-26T03:32:44.992 INFO:tasks.ceph:[global] debug mds migrator = 0/0 2026-02-26T03:32:44.992 INFO:tasks.ceph:[global] debug memdb = 0/0 2026-02-26T03:32:44.992 INFO:tasks.ceph:[global] debug mgr = 0/0 2026-02-26T03:32:44.992 INFO:tasks.ceph:[global] debug mgrc = 0/0 2026-02-26T03:32:44.992 INFO:tasks.ceph:[global] debug mon = 0/0 2026-02-26T03:32:44.992 INFO:tasks.ceph:[global] debug monc = 0/0 2026-02-26T03:32:44.992 INFO:tasks.ceph:[global] debug ms = 0/0 2026-02-26T03:32:44.992 INFO:tasks.ceph:[global] debug objclass = 0/0 2026-02-26T03:32:44.992 INFO:tasks.ceph:[global] debug objectcacher = 0/0 2026-02-26T03:32:44.992 INFO:tasks.ceph:[global] debug objecter = 0/0 2026-02-26T03:32:44.992 INFO:tasks.ceph:[global] debug optracker = 0/0 2026-02-26T03:32:44.992 INFO:tasks.ceph:[global] debug osd = 0/0 2026-02-26T03:32:44.993 INFO:tasks.ceph:[global] debug paxos = 0/0 2026-02-26T03:32:44.993 INFO:tasks.ceph:[global] debug perfcounter = 0/0 2026-02-26T03:32:44.993 INFO:tasks.ceph:[global] debug rados = 0/0 2026-02-26T03:32:44.993 INFO:tasks.ceph:[global] debug rbd = 0/0 2026-02-26T03:32:44.993 INFO:tasks.ceph:[global] debug rbd mirror = 0/0 2026-02-26T03:32:44.993 INFO:tasks.ceph:[global] debug rbd replay = 0/0 2026-02-26T03:32:44.993 INFO:tasks.ceph:[global] debug refs = 0/0 2026-02-26T03:32:44.993 INFO:tasks.ceph:[global] debug rgw = 0/0 2026-02-26T03:32:44.993 INFO:tasks.ceph:[global] debug rgw lifecycle = 0/0 2026-02-26T03:32:44.993 INFO:tasks.ceph:[global] debug rgw notification = 0/0 2026-02-26T03:32:44.993 INFO:tasks.ceph:[global] debug rgw sync = 0/0 2026-02-26T03:32:44.993 INFO:tasks.ceph:[global] debug rocksdb = 0/0 2026-02-26T03:32:44.993 INFO:tasks.ceph:[global] debug striper = 0/0 2026-02-26T03:32:44.994 INFO:tasks.ceph:[global] debug throttle = 0/0 2026-02-26T03:32:44.994 INFO:tasks.ceph:[global] debug timer = 0/0 2026-02-26T03:32:44.994 INFO:tasks.ceph:[global] debug tp = 0/0 2026-02-26T03:32:44.994 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2026-02-26T03:32:44.994 INFO:tasks.ceph:[global] ms client mode = crc 2026-02-26T03:32:44.994 INFO:tasks.ceph:[global] ms cluster mode = crc 2026-02-26T03:32:44.994 INFO:tasks.ceph:[global] ms mon client mode = crc 2026-02-26T03:32:44.994 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2026-02-26T03:32:44.994 INFO:tasks.ceph:[global] ms mon service mode = crc 2026-02-26T03:32:44.994 INFO:tasks.ceph:[global] ms service mode = crc 2026-02-26T03:32:44.994 INFO:tasks.ceph:[mgr] debug mgr = 20 2026-02-26T03:32:44.994 INFO:tasks.ceph:[mgr] debug ms = 1 2026-02-26T03:32:44.994 INFO:tasks.ceph:[mon] debug mon = 0/0 2026-02-26T03:32:44.995 INFO:tasks.ceph:[mon] debug ms = 0/0 2026-02-26T03:32:44.995 INFO:tasks.ceph:[mon] debug paxos = 0/0 2026-02-26T03:32:44.995 INFO:tasks.ceph:[mon] osd pool default crimson = True 2026-02-26T03:32:44.995 INFO:tasks.ceph:Setting up mon.a... 2026-02-26T03:32:44.995 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2026-02-26T03:32:45.065 INFO:teuthology.orchestra.run.trial134.stdout:creating /etc/ceph/ceph.keyring 2026-02-26T03:32:45.067 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2026-02-26T03:32:45.151 DEBUG:teuthology.orchestra.run.trial134:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-26T03:32:45.216 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '10.20.193.134')] 2026-02-26T03:32:45.217 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'none', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '10.20.193.134', 'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': 'true', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore max concurrent transactions': 8}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': '0/0', 'debug mon': '0/0', 'debug paxos': '0/0', 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}} 2026-02-26T03:32:45.218 DEBUG:teuthology.orchestra.run.trial134:> set -ex 2026-02-26T03:32:45.219 DEBUG:teuthology.orchestra.run.trial134:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2026-02-26T03:32:45.274 DEBUG:teuthology.orchestra.run.trial134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 10.20.193.134 --print /home/ubuntu/cephtest/ceph.monmap 2026-02-26T03:32:45.355 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-26T03:32:45.353+0000 7f3405687f40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:32:45.355 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-26T03:32:45.353+0000 7f3405687f40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:32:45.355 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-26T03:32:45.353+0000 7f3405687f40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:32:45.356 INFO:teuthology.orchestra.run.trial134.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2026-02-26T03:32:45.357 INFO:teuthology.orchestra.run.trial134.stdout:monmaptool: generated fsid 6ee9ec32-e74b-407c-ab79-d45b46c3ba91 2026-02-26T03:32:45.357 INFO:teuthology.orchestra.run.trial134.stdout:setting min_mon_release = tentacle 2026-02-26T03:32:45.357 INFO:teuthology.orchestra.run.trial134.stdout:epoch 0 2026-02-26T03:32:45.357 INFO:teuthology.orchestra.run.trial134.stdout:fsid 6ee9ec32-e74b-407c-ab79-d45b46c3ba91 2026-02-26T03:32:45.357 INFO:teuthology.orchestra.run.trial134.stdout:last_changed 2026-02-26T03:32:45.354733+0000 2026-02-26T03:32:45.357 INFO:teuthology.orchestra.run.trial134.stdout:created 2026-02-26T03:32:45.354733+0000 2026-02-26T03:32:45.357 INFO:teuthology.orchestra.run.trial134.stdout:min_mon_release 20 (tentacle) 2026-02-26T03:32:45.357 INFO:teuthology.orchestra.run.trial134.stdout:election_strategy: 1 2026-02-26T03:32:45.357 INFO:teuthology.orchestra.run.trial134.stdout:0: [v2:10.20.193.134:3300/0,v1:10.20.193.134:6789/0] mon.a 2026-02-26T03:32:45.358 INFO:teuthology.orchestra.run.trial134.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2026-02-26T03:32:45.358 DEBUG:teuthology.orchestra.run.trial134:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2026-02-26T03:32:45.412 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 6ee9ec32-e74b-407c-ab79-d45b46c3ba91... 2026-02-26T03:32:45.414 DEBUG:teuthology.orchestra.run.trial134:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2026-02-26T03:32:45.493 INFO:teuthology.orchestra.run.trial134.stdout:[global] 2026-02-26T03:32:45.493 INFO:teuthology.orchestra.run.trial134.stdout: chdir = "" 2026-02-26T03:32:45.493 INFO:teuthology.orchestra.run.trial134.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2026-02-26T03:32:45.493 INFO:teuthology.orchestra.run.trial134.stdout: auth supported = none 2026-02-26T03:32:45.493 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:32:45.493 INFO:teuthology.orchestra.run.trial134.stdout: filestore xattr use omap = true 2026-02-26T03:32:45.493 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:32:45.493 INFO:teuthology.orchestra.run.trial134.stdout: mon clock drift allowed = 1.000 2026-02-26T03:32:45.494 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:32:45.494 INFO:teuthology.orchestra.run.trial134.stdout: osd crush chooseleaf type = 0 2026-02-26T03:32:45.494 INFO:teuthology.orchestra.run.trial134.stdout: auth debug = true 2026-02-26T03:32:45.494 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:32:45.494 INFO:teuthology.orchestra.run.trial134.stdout: ms die on old message = true 2026-02-26T03:32:45.494 INFO:teuthology.orchestra.run.trial134.stdout: ms die on bug = true 2026-02-26T03:32:45.494 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:32:45.494 INFO:teuthology.orchestra.run.trial134.stdout: mon max pg per osd = 10000 # >= luminous 2026-02-26T03:32:45.494 INFO:teuthology.orchestra.run.trial134.stdout: mon pg warn max object skew = 0 2026-02-26T03:32:45.494 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:32:45.494 INFO:teuthology.orchestra.run.trial134.stdout: # disable pg_autoscaler by default for new pools 2026-02-26T03:32:45.495 INFO:teuthology.orchestra.run.trial134.stdout: osd_pool_default_pg_autoscale_mode = off 2026-02-26T03:32:45.495 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:32:45.495 INFO:teuthology.orchestra.run.trial134.stdout: osd pool default size = 2 2026-02-26T03:32:45.495 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:32:45.495 INFO:teuthology.orchestra.run.trial134.stdout: mon osd allow primary affinity = true 2026-02-26T03:32:45.495 INFO:teuthology.orchestra.run.trial134.stdout: mon osd allow pg remap = true 2026-02-26T03:32:45.495 INFO:teuthology.orchestra.run.trial134.stdout: mon warn on legacy crush tunables = false 2026-02-26T03:32:45.495 INFO:teuthology.orchestra.run.trial134.stdout: mon warn on crush straw calc version zero = false 2026-02-26T03:32:45.495 INFO:teuthology.orchestra.run.trial134.stdout: mon warn on no sortbitwise = false 2026-02-26T03:32:45.495 INFO:teuthology.orchestra.run.trial134.stdout: mon warn on osd down out interval zero = false 2026-02-26T03:32:45.495 INFO:teuthology.orchestra.run.trial134.stdout: mon warn on too few osds = false 2026-02-26T03:32:45.495 INFO:teuthology.orchestra.run.trial134.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2026-02-26T03:32:45.496 INFO:teuthology.orchestra.run.trial134.stdout: mon_warn_on_pool_no_redundancy = false 2026-02-26T03:32:45.496 INFO:teuthology.orchestra.run.trial134.stdout: mon_allow_pool_size_one = true 2026-02-26T03:32:45.496 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:32:45.496 INFO:teuthology.orchestra.run.trial134.stdout: osd pool default erasure code profile = plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2026-02-26T03:32:45.496 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:32:45.496 INFO:teuthology.orchestra.run.trial134.stdout: osd default data pool replay window = 5 2026-02-26T03:32:45.496 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:32:45.496 INFO:teuthology.orchestra.run.trial134.stdout: mon allow pool delete = true 2026-02-26T03:32:45.496 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:32:45.496 INFO:teuthology.orchestra.run.trial134.stdout: mon cluster log file level = debug 2026-02-26T03:32:45.496 INFO:teuthology.orchestra.run.trial134.stdout: debug asserts on shutdown = true 2026-02-26T03:32:45.496 INFO:teuthology.orchestra.run.trial134.stdout: mon health detail to clog = false 2026-02-26T03:32:45.497 INFO:teuthology.orchestra.run.trial134.stdout: mon host = 10.20.193.134 2026-02-26T03:32:45.497 INFO:teuthology.orchestra.run.trial134.stdout: auth client required = none 2026-02-26T03:32:45.497 INFO:teuthology.orchestra.run.trial134.stdout: auth cluster required = none 2026-02-26T03:32:45.497 INFO:teuthology.orchestra.run.trial134.stdout: auth service required = none 2026-02-26T03:32:45.497 INFO:teuthology.orchestra.run.trial134.stdout: debug asok = 0/0 2026-02-26T03:32:45.497 INFO:teuthology.orchestra.run.trial134.stdout: debug auth = 0/0 2026-02-26T03:32:45.497 INFO:teuthology.orchestra.run.trial134.stdout: debug bdev = 0/0 2026-02-26T03:32:45.497 INFO:teuthology.orchestra.run.trial134.stdout: debug bluefs = 0/0 2026-02-26T03:32:45.497 INFO:teuthology.orchestra.run.trial134.stdout: debug bluestore = 0/0 2026-02-26T03:32:45.497 INFO:teuthology.orchestra.run.trial134.stdout: debug buffer = 0/0 2026-02-26T03:32:45.497 INFO:teuthology.orchestra.run.trial134.stdout: debug civetweb = 0/0 2026-02-26T03:32:45.497 INFO:teuthology.orchestra.run.trial134.stdout: debug client = 0/0 2026-02-26T03:32:45.497 INFO:teuthology.orchestra.run.trial134.stdout: debug compressor = 0/0 2026-02-26T03:32:45.497 INFO:teuthology.orchestra.run.trial134.stdout: debug context = 0/0 2026-02-26T03:32:45.498 INFO:teuthology.orchestra.run.trial134.stdout: debug crush = 0/0 2026-02-26T03:32:45.498 INFO:teuthology.orchestra.run.trial134.stdout: debug crypto = 0/0 2026-02-26T03:32:45.498 INFO:teuthology.orchestra.run.trial134.stdout: debug dpdk = 0/0 2026-02-26T03:32:45.498 INFO:teuthology.orchestra.run.trial134.stdout: debug eventtrace = 0/0 2026-02-26T03:32:45.498 INFO:teuthology.orchestra.run.trial134.stdout: debug filer = 0/0 2026-02-26T03:32:45.498 INFO:teuthology.orchestra.run.trial134.stdout: debug filestore = 0/0 2026-02-26T03:32:45.498 INFO:teuthology.orchestra.run.trial134.stdout: debug finisher = 0/0 2026-02-26T03:32:45.498 INFO:teuthology.orchestra.run.trial134.stdout: debug fuse = 0/0 2026-02-26T03:32:45.498 INFO:teuthology.orchestra.run.trial134.stdout: debug heartbeatmap = 0/0 2026-02-26T03:32:45.498 INFO:teuthology.orchestra.run.trial134.stdout: debug javaclient = 0/0 2026-02-26T03:32:45.498 INFO:teuthology.orchestra.run.trial134.stdout: debug journal = 0/0 2026-02-26T03:32:45.498 INFO:teuthology.orchestra.run.trial134.stdout: debug journaler = 0/0 2026-02-26T03:32:45.498 INFO:teuthology.orchestra.run.trial134.stdout: debug kstore = 0/0 2026-02-26T03:32:45.499 INFO:teuthology.orchestra.run.trial134.stdout: debug leveldb = 0/0 2026-02-26T03:32:45.499 INFO:teuthology.orchestra.run.trial134.stdout: debug lockdep = 0/0 2026-02-26T03:32:45.499 INFO:teuthology.orchestra.run.trial134.stdout: debug mds = 0/0 2026-02-26T03:32:45.499 INFO:teuthology.orchestra.run.trial134.stdout: debug mds balancer = 0/0 2026-02-26T03:32:45.499 INFO:teuthology.orchestra.run.trial134.stdout: debug mds locker = 0/0 2026-02-26T03:32:45.499 INFO:teuthology.orchestra.run.trial134.stdout: debug mds log = 0/0 2026-02-26T03:32:45.499 INFO:teuthology.orchestra.run.trial134.stdout: debug mds log expire = 0/0 2026-02-26T03:32:45.499 INFO:teuthology.orchestra.run.trial134.stdout: debug mds migrator = 0/0 2026-02-26T03:32:45.499 INFO:teuthology.orchestra.run.trial134.stdout: debug memdb = 0/0 2026-02-26T03:32:45.499 INFO:teuthology.orchestra.run.trial134.stdout: debug mgr = 0/0 2026-02-26T03:32:45.499 INFO:teuthology.orchestra.run.trial134.stdout: debug mgrc = 0/0 2026-02-26T03:32:45.500 INFO:teuthology.orchestra.run.trial134.stdout: debug mon = 0/0 2026-02-26T03:32:45.500 INFO:teuthology.orchestra.run.trial134.stdout: debug monc = 0/0 2026-02-26T03:32:45.500 INFO:teuthology.orchestra.run.trial134.stdout: debug ms = 0/0 2026-02-26T03:32:45.500 INFO:teuthology.orchestra.run.trial134.stdout: debug objclass = 0/0 2026-02-26T03:32:45.500 INFO:teuthology.orchestra.run.trial134.stdout: debug objectcacher = 0/0 2026-02-26T03:32:45.500 INFO:teuthology.orchestra.run.trial134.stdout: debug objecter = 0/0 2026-02-26T03:32:45.500 INFO:teuthology.orchestra.run.trial134.stdout: debug optracker = 0/0 2026-02-26T03:32:45.500 INFO:teuthology.orchestra.run.trial134.stdout: debug osd = 0/0 2026-02-26T03:32:45.500 INFO:teuthology.orchestra.run.trial134.stdout: debug paxos = 0/0 2026-02-26T03:32:45.500 INFO:teuthology.orchestra.run.trial134.stdout: debug perfcounter = 0/0 2026-02-26T03:32:45.500 INFO:teuthology.orchestra.run.trial134.stdout: debug rados = 0/0 2026-02-26T03:32:45.500 INFO:teuthology.orchestra.run.trial134.stdout: debug rbd = 0/0 2026-02-26T03:32:45.500 INFO:teuthology.orchestra.run.trial134.stdout: debug rbd mirror = 0/0 2026-02-26T03:32:45.501 INFO:teuthology.orchestra.run.trial134.stdout: debug rbd replay = 0/0 2026-02-26T03:32:45.501 INFO:teuthology.orchestra.run.trial134.stdout: debug refs = 0/0 2026-02-26T03:32:45.501 INFO:teuthology.orchestra.run.trial134.stdout: debug rgw = 0/0 2026-02-26T03:32:45.501 INFO:teuthology.orchestra.run.trial134.stdout: debug rgw lifecycle = 0/0 2026-02-26T03:32:45.501 INFO:teuthology.orchestra.run.trial134.stdout: debug rgw notification = 0/0 2026-02-26T03:32:45.501 INFO:teuthology.orchestra.run.trial134.stdout: debug rgw sync = 0/0 2026-02-26T03:32:45.501 INFO:teuthology.orchestra.run.trial134.stdout: debug rocksdb = 0/0 2026-02-26T03:32:45.501 INFO:teuthology.orchestra.run.trial134.stdout: debug striper = 0/0 2026-02-26T03:32:45.501 INFO:teuthology.orchestra.run.trial134.stdout: debug throttle = 0/0 2026-02-26T03:32:45.501 INFO:teuthology.orchestra.run.trial134.stdout: debug timer = 0/0 2026-02-26T03:32:45.501 INFO:teuthology.orchestra.run.trial134.stdout: debug tp = 0/0 2026-02-26T03:32:45.501 INFO:teuthology.orchestra.run.trial134.stdout: enable experimental unrecoverable data corrupting features = crimson 2026-02-26T03:32:45.501 INFO:teuthology.orchestra.run.trial134.stdout: ms client mode = crc 2026-02-26T03:32:45.502 INFO:teuthology.orchestra.run.trial134.stdout: ms cluster mode = crc 2026-02-26T03:32:45.502 INFO:teuthology.orchestra.run.trial134.stdout: ms mon client mode = crc 2026-02-26T03:32:45.502 INFO:teuthology.orchestra.run.trial134.stdout: ms mon cluster mode = crc 2026-02-26T03:32:45.502 INFO:teuthology.orchestra.run.trial134.stdout: ms mon service mode = crc 2026-02-26T03:32:45.502 INFO:teuthology.orchestra.run.trial134.stdout: ms service mode = crc 2026-02-26T03:32:45.502 INFO:teuthology.orchestra.run.trial134.stdout: fsid = 6ee9ec32-e74b-407c-ab79-d45b46c3ba91 2026-02-26T03:32:45.502 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:32:45.502 INFO:teuthology.orchestra.run.trial134.stdout:[osd] 2026-02-26T03:32:45.502 INFO:teuthology.orchestra.run.trial134.stdout: osd journal size = 100 2026-02-26T03:32:45.502 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:32:45.502 INFO:teuthology.orchestra.run.trial134.stdout: osd scrub load threshold = 5.0 2026-02-26T03:32:45.502 INFO:teuthology.orchestra.run.trial134.stdout: osd scrub max interval = 600 2026-02-26T03:32:45.503 INFO:teuthology.orchestra.run.trial134.stdout: osd mclock profile = high_recovery_ops 2026-02-26T03:32:45.503 INFO:teuthology.orchestra.run.trial134.stdout: osd mclock skip benchmark = true 2026-02-26T03:32:45.503 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:32:45.503 INFO:teuthology.orchestra.run.trial134.stdout: osd recover clone overlap = true 2026-02-26T03:32:45.503 INFO:teuthology.orchestra.run.trial134.stdout: osd recovery max chunk = 1048576 2026-02-26T03:32:45.503 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:32:45.503 INFO:teuthology.orchestra.run.trial134.stdout: osd debug shutdown = true 2026-02-26T03:32:45.503 INFO:teuthology.orchestra.run.trial134.stdout: osd debug op order = true 2026-02-26T03:32:45.503 INFO:teuthology.orchestra.run.trial134.stdout: osd debug verify stray on activate = true 2026-02-26T03:32:45.503 INFO:teuthology.orchestra.run.trial134.stdout: osd debug trim objects = true 2026-02-26T03:32:45.503 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:32:45.503 INFO:teuthology.orchestra.run.trial134.stdout: osd open classes on start = true 2026-02-26T03:32:45.503 INFO:teuthology.orchestra.run.trial134.stdout: osd debug pg log writeout = true 2026-02-26T03:32:45.504 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:32:45.504 INFO:teuthology.orchestra.run.trial134.stdout: osd deep scrub update digest min age = 30 2026-02-26T03:32:45.504 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:32:45.504 INFO:teuthology.orchestra.run.trial134.stdout: osd map max advance = 10 2026-02-26T03:32:45.504 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:32:45.504 INFO:teuthology.orchestra.run.trial134.stdout: journal zero on create = true 2026-02-26T03:32:45.504 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:32:45.504 INFO:teuthology.orchestra.run.trial134.stdout: filestore ondisk finisher threads = 3 2026-02-26T03:32:45.504 INFO:teuthology.orchestra.run.trial134.stdout: filestore apply finisher threads = 3 2026-02-26T03:32:45.504 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:32:45.504 INFO:teuthology.orchestra.run.trial134.stdout: bdev debug aio = true 2026-02-26T03:32:45.504 INFO:teuthology.orchestra.run.trial134.stdout: osd debug misdirected ops = true 2026-02-26T03:32:45.504 INFO:teuthology.orchestra.run.trial134.stdout: debug monc = 20 2026-02-26T03:32:45.505 INFO:teuthology.orchestra.run.trial134.stdout: crimson cpu num = 4 2026-02-26T03:32:45.505 INFO:teuthology.orchestra.run.trial134.stdout: crimson osd obc lru size = 10 2026-02-26T03:32:45.505 INFO:teuthology.orchestra.run.trial134.stdout: debug filestore = 0/0 2026-02-26T03:32:45.505 INFO:teuthology.orchestra.run.trial134.stdout: debug journal = 0/0 2026-02-26T03:32:45.505 INFO:teuthology.orchestra.run.trial134.stdout: debug ms = 0/0 2026-02-26T03:32:45.505 INFO:teuthology.orchestra.run.trial134.stdout: debug osd = 0/0 2026-02-26T03:32:45.505 INFO:teuthology.orchestra.run.trial134.stdout: debug seastore = 20 2026-02-26T03:32:45.505 INFO:teuthology.orchestra.run.trial134.stdout: debug seastore backref = 20 2026-02-26T03:32:45.505 INFO:teuthology.orchestra.run.trial134.stdout: debug seastore cache = 20 2026-02-26T03:32:45.505 INFO:teuthology.orchestra.run.trial134.stdout: debug seastore cleaner = 20 2026-02-26T03:32:45.505 INFO:teuthology.orchestra.run.trial134.stdout: debug seastore device = 20 2026-02-26T03:32:45.505 INFO:teuthology.orchestra.run.trial134.stdout: debug seastore epm = 20 2026-02-26T03:32:45.505 INFO:teuthology.orchestra.run.trial134.stdout: debug seastore fixedkv tree = 20 2026-02-26T03:32:45.506 INFO:teuthology.orchestra.run.trial134.stdout: debug seastore journal = 20 2026-02-26T03:32:45.506 INFO:teuthology.orchestra.run.trial134.stdout: debug seastore lba = 20 2026-02-26T03:32:45.506 INFO:teuthology.orchestra.run.trial134.stdout: debug seastore odata = 20 2026-02-26T03:32:45.506 INFO:teuthology.orchestra.run.trial134.stdout: debug seastore omap = 20 2026-02-26T03:32:45.506 INFO:teuthology.orchestra.run.trial134.stdout: debug seastore onode = 20 2026-02-26T03:32:45.506 INFO:teuthology.orchestra.run.trial134.stdout: debug seastore t = 20 2026-02-26T03:32:45.506 INFO:teuthology.orchestra.run.trial134.stdout: debug seastore tm = 20 2026-02-26T03:32:45.506 INFO:teuthology.orchestra.run.trial134.stdout: osd objectstore = seastore 2026-02-26T03:32:45.506 INFO:teuthology.orchestra.run.trial134.stdout: osd shutdown pgref assert = True 2026-02-26T03:32:45.506 INFO:teuthology.orchestra.run.trial134.stdout: seastore cachepin size pershard = 64M 2026-02-26T03:32:45.506 INFO:teuthology.orchestra.run.trial134.stdout: seastore max concurrent transactions = 8 2026-02-26T03:32:45.506 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:32:45.506 INFO:teuthology.orchestra.run.trial134.stdout:[mgr] 2026-02-26T03:32:45.507 INFO:teuthology.orchestra.run.trial134.stdout: debug ms = 1 2026-02-26T03:32:45.507 INFO:teuthology.orchestra.run.trial134.stdout: debug mgr = 20 2026-02-26T03:32:45.507 INFO:teuthology.orchestra.run.trial134.stdout: debug mon = 20 2026-02-26T03:32:45.507 INFO:teuthology.orchestra.run.trial134.stdout: debug auth = 20 2026-02-26T03:32:45.507 INFO:teuthology.orchestra.run.trial134.stdout: mon reweight min pgs per osd = 4 2026-02-26T03:32:45.507 INFO:teuthology.orchestra.run.trial134.stdout: mon reweight min bytes per osd = 10 2026-02-26T03:32:45.507 INFO:teuthology.orchestra.run.trial134.stdout: mgr/telemetry/nag = false 2026-02-26T03:32:45.507 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:32:45.507 INFO:teuthology.orchestra.run.trial134.stdout:[mon] 2026-02-26T03:32:45.507 INFO:teuthology.orchestra.run.trial134.stdout: debug ms = 0/0 2026-02-26T03:32:45.507 INFO:teuthology.orchestra.run.trial134.stdout: debug mon = 0/0 2026-02-26T03:32:45.507 INFO:teuthology.orchestra.run.trial134.stdout: debug paxos = 0/0 2026-02-26T03:32:45.508 INFO:teuthology.orchestra.run.trial134.stdout: debug auth = 20 2026-02-26T03:32:45.508 INFO:teuthology.orchestra.run.trial134.stdout: mon data avail warn = 5 2026-02-26T03:32:45.508 INFO:teuthology.orchestra.run.trial134.stdout: mon mgr mkfs grace = 240 2026-02-26T03:32:45.508 INFO:teuthology.orchestra.run.trial134.stdout: mon reweight min pgs per osd = 4 2026-02-26T03:32:45.508 INFO:teuthology.orchestra.run.trial134.stdout: mon osd reporter subtree level = osd 2026-02-26T03:32:45.508 INFO:teuthology.orchestra.run.trial134.stdout: mon osd prime pg temp = true 2026-02-26T03:32:45.508 INFO:teuthology.orchestra.run.trial134.stdout: mon reweight min bytes per osd = 10 2026-02-26T03:32:45.508 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:32:45.508 INFO:teuthology.orchestra.run.trial134.stdout: # rotate auth tickets quickly to exercise renewal paths 2026-02-26T03:32:45.508 INFO:teuthology.orchestra.run.trial134.stdout: auth mon ticket ttl = 660 # 11m 2026-02-26T03:32:45.508 INFO:teuthology.orchestra.run.trial134.stdout: auth service ticket ttl = 240 # 4m 2026-02-26T03:32:45.508 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:32:45.508 INFO:teuthology.orchestra.run.trial134.stdout: # don't complain about insecure global_id in the test suite 2026-02-26T03:32:45.508 INFO:teuthology.orchestra.run.trial134.stdout: mon_warn_on_insecure_global_id_reclaim = false 2026-02-26T03:32:45.509 INFO:teuthology.orchestra.run.trial134.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2026-02-26T03:32:45.509 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:32:45.509 INFO:teuthology.orchestra.run.trial134.stdout: # 1m isn't quite enough 2026-02-26T03:32:45.509 INFO:teuthology.orchestra.run.trial134.stdout: mon_down_mkfs_grace = 2m 2026-02-26T03:32:45.509 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:32:45.509 INFO:teuthology.orchestra.run.trial134.stdout: mon_warn_on_filestore_osds = false 2026-02-26T03:32:45.509 INFO:teuthology.orchestra.run.trial134.stdout: osd pool default crimson = True 2026-02-26T03:32:45.509 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:32:45.509 INFO:teuthology.orchestra.run.trial134.stdout:[client] 2026-02-26T03:32:45.509 INFO:teuthology.orchestra.run.trial134.stdout: rgw cache enabled = true 2026-02-26T03:32:45.509 INFO:teuthology.orchestra.run.trial134.stdout: rgw enable ops log = true 2026-02-26T03:32:45.509 INFO:teuthology.orchestra.run.trial134.stdout: rgw enable usage log = true 2026-02-26T03:32:45.509 INFO:teuthology.orchestra.run.trial134.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2026-02-26T03:32:45.510 INFO:teuthology.orchestra.run.trial134.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2026-02-26T03:32:45.510 INFO:teuthology.orchestra.run.trial134.stdout:[mon.a] 2026-02-26T03:32:45.511 INFO:tasks.ceph:Creating admin key on mon.a... 2026-02-26T03:32:45.511 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2026-02-26T03:32:45.602 INFO:tasks.ceph:Copying monmap to all nodes... 2026-02-26T03:32:45.603 DEBUG:teuthology.orchestra.run.trial134:> set -ex 2026-02-26T03:32:45.603 DEBUG:teuthology.orchestra.run.trial134:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2026-02-26T03:32:45.619 DEBUG:teuthology.orchestra.run.trial134:> set -ex 2026-02-26T03:32:45.619 DEBUG:teuthology.orchestra.run.trial134:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2026-02-26T03:32:45.676 INFO:tasks.ceph:Sending monmap to node ubuntu@trial134.front.sepia.ceph.com 2026-02-26T03:32:45.676 DEBUG:teuthology.orchestra.run.trial134:> set -ex 2026-02-26T03:32:45.676 DEBUG:teuthology.orchestra.run.trial134:> sudo dd of=/etc/ceph/ceph.keyring 2026-02-26T03:32:45.676 DEBUG:teuthology.orchestra.run.trial134:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-26T03:32:45.750 DEBUG:teuthology.orchestra.run.trial134:> set -ex 2026-02-26T03:32:45.750 DEBUG:teuthology.orchestra.run.trial134:> dd of=/home/ubuntu/cephtest/ceph.monmap 2026-02-26T03:32:45.806 INFO:tasks.ceph:Setting up mon nodes... 2026-02-26T03:32:45.806 INFO:tasks.ceph:Setting up mgr nodes... 2026-02-26T03:32:45.807 DEBUG:teuthology.orchestra.run.trial134:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2026-02-26T03:32:45.895 INFO:teuthology.orchestra.run.trial134.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2026-02-26T03:32:45.898 INFO:tasks.ceph:Setting up mds nodes... 2026-02-26T03:32:45.898 INFO:tasks.ceph_client:Setting up client nodes... 2026-02-26T03:32:45.898 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2026-02-26T03:32:45.940 INFO:teuthology.orchestra.run.trial134.stdout:creating /etc/ceph/ceph.client.0.keyring 2026-02-26T03:32:45.951 INFO:tasks.ceph:Running mkfs on osd nodes... 2026-02-26T03:32:45.952 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@trial134.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2026-02-26T03:32:45.952 DEBUG:teuthology.orchestra.run.trial134:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2026-02-26T03:32:46.015 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-26T03:32:46.015 INFO:tasks.ceph:role: osd.0 2026-02-26T03:32:46.016 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@trial134.front.sepia.ceph.com 2026-02-26T03:32:46.016 DEBUG:teuthology.orchestra.run.trial134:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2026-02-26T03:32:46.080 INFO:teuthology.orchestra.run.trial134.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=45875200 blks 2026-02-26T03:32:46.080 INFO:teuthology.orchestra.run.trial134.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-26T03:32:46.080 INFO:teuthology.orchestra.run.trial134.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-26T03:32:46.081 INFO:teuthology.orchestra.run.trial134.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-26T03:32:46.081 INFO:teuthology.orchestra.run.trial134.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-26T03:32:46.081 INFO:teuthology.orchestra.run.trial134.stdout: = sunit=0 swidth=0 blks 2026-02-26T03:32:46.081 INFO:teuthology.orchestra.run.trial134.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-26T03:32:46.081 INFO:teuthology.orchestra.run.trial134.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-26T03:32:46.081 INFO:teuthology.orchestra.run.trial134.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-26T03:32:46.081 INFO:teuthology.orchestra.run.trial134.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-26T03:32:46.650 INFO:teuthology.orchestra.run.trial134.stdout:Discarding blocks...Done. 2026-02-26T03:32:46.652 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@trial134.front.sepia.ceph.com -o noatime 2026-02-26T03:32:46.653 DEBUG:teuthology.orchestra.run.trial134:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2026-02-26T03:32:46.722 DEBUG:teuthology.orchestra.run.trial134:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2026-02-26T03:32:46.788 DEBUG:teuthology.orchestra.run.trial134:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2026-02-26T03:32:46.853 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-26T03:32:46.854 INFO:tasks.ceph:role: osd.1 2026-02-26T03:32:46.854 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@trial134.front.sepia.ceph.com 2026-02-26T03:32:46.854 DEBUG:teuthology.orchestra.run.trial134:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2026-02-26T03:32:46.920 INFO:teuthology.orchestra.run.trial134.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=45875200 blks 2026-02-26T03:32:46.920 INFO:teuthology.orchestra.run.trial134.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-26T03:32:46.920 INFO:teuthology.orchestra.run.trial134.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-26T03:32:46.920 INFO:teuthology.orchestra.run.trial134.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-26T03:32:46.920 INFO:teuthology.orchestra.run.trial134.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-26T03:32:46.920 INFO:teuthology.orchestra.run.trial134.stdout: = sunit=0 swidth=0 blks 2026-02-26T03:32:46.920 INFO:teuthology.orchestra.run.trial134.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-26T03:32:46.920 INFO:teuthology.orchestra.run.trial134.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-26T03:32:46.920 INFO:teuthology.orchestra.run.trial134.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-26T03:32:46.921 INFO:teuthology.orchestra.run.trial134.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-26T03:32:47.479 INFO:teuthology.orchestra.run.trial134.stdout:Discarding blocks...Done. 2026-02-26T03:32:47.481 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@trial134.front.sepia.ceph.com -o noatime 2026-02-26T03:32:47.482 DEBUG:teuthology.orchestra.run.trial134:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2026-02-26T03:32:47.554 DEBUG:teuthology.orchestra.run.trial134:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2026-02-26T03:32:47.624 DEBUG:teuthology.orchestra.run.trial134:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2026-02-26T03:32:47.687 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-26T03:32:47.687 INFO:tasks.ceph:role: osd.2 2026-02-26T03:32:47.687 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@trial134.front.sepia.ceph.com 2026-02-26T03:32:47.687 DEBUG:teuthology.orchestra.run.trial134:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2026-02-26T03:32:47.752 INFO:teuthology.orchestra.run.trial134.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=45875200 blks 2026-02-26T03:32:47.753 INFO:teuthology.orchestra.run.trial134.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-26T03:32:47.753 INFO:teuthology.orchestra.run.trial134.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-26T03:32:47.753 INFO:teuthology.orchestra.run.trial134.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-26T03:32:47.753 INFO:teuthology.orchestra.run.trial134.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-26T03:32:47.753 INFO:teuthology.orchestra.run.trial134.stdout: = sunit=0 swidth=0 blks 2026-02-26T03:32:47.753 INFO:teuthology.orchestra.run.trial134.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-26T03:32:47.753 INFO:teuthology.orchestra.run.trial134.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-26T03:32:47.753 INFO:teuthology.orchestra.run.trial134.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-26T03:32:47.753 INFO:teuthology.orchestra.run.trial134.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-26T03:32:48.309 INFO:teuthology.orchestra.run.trial134.stdout:Discarding blocks...Done. 2026-02-26T03:32:48.311 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@trial134.front.sepia.ceph.com -o noatime 2026-02-26T03:32:48.311 DEBUG:teuthology.orchestra.run.trial134:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2026-02-26T03:32:48.384 DEBUG:teuthology.orchestra.run.trial134:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2026-02-26T03:32:48.454 DEBUG:teuthology.orchestra.run.trial134:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-26T03:32:48.603 INFO:teuthology.orchestra.run.trial134.stderr:INFO  osd - OSD::main: parsing early config 2026-02-26T03:32:48.657 INFO:teuthology.orchestra.run.trial134.stderr:INFO 2026-02-26 03:32:48,657 seastar - Reactor backend: linux-aio 2026-02-26T03:32:48.669 INFO:teuthology.orchestra.run.trial134.stderr:INFO 2026-02-26 03:32:48,668 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:32:48.669 INFO:teuthology.orchestra.run.trial134.stderr:INFO 2026-02-26 03:32:48,668 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:32:48.671 INFO:teuthology.orchestra.run.trial134.stderr:==39933==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-26T03:32:48.730 INFO:teuthology.orchestra.run.trial134.stderr:DEBUG 2026-02-26 03:32:48,730 [shard 0:main] seastar - reactor::drain 2026-02-26T03:32:50.769 INFO:teuthology.orchestra.run.trial134.stderr:----------------------------------------------------- 2026-02-26T03:32:50.769 INFO:teuthology.orchestra.run.trial134.stderr:Suppressions used: 2026-02-26T03:32:50.769 INFO:teuthology.orchestra.run.trial134.stderr: count bytes template 2026-02-26T03:32:50.770 INFO:teuthology.orchestra.run.trial134.stderr: 1 8 MallocExtension::Register 2026-02-26T03:32:50.770 INFO:teuthology.orchestra.run.trial134.stderr:----------------------------------------------------- 2026-02-26T03:32:50.770 INFO:teuthology.orchestra.run.trial134.stderr: 2026-02-26T03:32:50.788 INFO:teuthology.orchestra.run.trial134.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-26T03:32:50.788 INFO:teuthology.orchestra.run.trial134.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-26T03:32:50.830 INFO:teuthology.orchestra.run.trial134.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-26T03:32:50.830 INFO:teuthology.orchestra.run.trial134.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-26T03:32:50.837 INFO:teuthology.orchestra.run.trial134.stderr:INFO 2026-02-26 03:32:50,836 seastar - Reactor backend: linux-aio 2026-02-26T03:32:50.851 INFO:teuthology.orchestra.run.trial134.stderr:INFO 2026-02-26 03:32:50,851 [shard 2:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:32:50.851 INFO:teuthology.orchestra.run.trial134.stderr:INFO 2026-02-26 03:32:50,851 [shard 2:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:32:50.855 INFO:teuthology.orchestra.run.trial134.stderr:==39931==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-26T03:32:50.855 INFO:teuthology.orchestra.run.trial134.stderr:INFO 2026-02-26 03:32:50,855 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-26T03:32:50.999 INFO:teuthology.orchestra.run.trial134.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1583b 0xc95eb2 0xca741d 0x9c98955 0x9d47a94 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6fa06ea 0x6fa44eb 0x6faa095 0x7419054 0x743ba5c 0x728f369 0x728f461 0x736153a 0x73615f3 0x73617aa 0x743badd 0x728f4de 0x728f563 0x72bb301 0x7423ee4 0x7423f69 0x7424047 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:32:50.999 INFO:teuthology.orchestra.run.trial134.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x9c0739a 0x9d1b543 0x9d47a30 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6fa06ea 0x6fa44eb 0x6faa095 0x7419054 0x743ba5c 0x728f369 0x728f461 0x736153a 0x73615f3 0x73617aa 0x743badd 0x728f4de 0x728f563 0x72bb301 0x7423ee4 0x7423f69 0x7424047 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:32:51.000 INFO:teuthology.orchestra.run.trial134.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffa832a157 0xffffffffa7ff9006 0xffffffffa7ffc9c0 0xffffffffa7ffdd21 0xffffffffa8025a53 0xffffffffa8025c15 0xffffffffa7fcc8ce 0xffffffffa7fd4bab 0xffffffffa7fd500b 0xffffffffa7c910d2 0xffffffffa8950451 0xffffffffa8a00ba2 2026-02-26T03:32:51.000 INFO:teuthology.orchestra.run.trial134.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xf4e8f /lib64/libasan.so.8+0x10069c /lib64/libasan.so.8+0xc65b9 /lib64/libasan.so.8+0x15197 0x1125428 0xf2e0d1 0x9c50c5c 0x9c50f59 0x9c511cc 0x9c51482 0x9c51f31 0x9c5242f 0x9c52a5f 0x9c52bc9 0x9d32f39 0x9d45d3c 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6fa06ea 0x6fa44eb 0x6faa095 0x7419054 0x743ba5c 0x728f369 0x728f461 0x736153a 0x73615f3 0x73617aa 0x743badd 0x728f4de 0x728f563 0x72bb301 0x7423ee4 0x7423f69 0x7424047 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:32:51.031 INFO:teuthology.orchestra.run.trial134.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0xb7580d 0x10da81f 0x7e645ef 0x8f5d61d 0x8f5ff8b 0x787cd7f 0x78b3782 0x6fa18b5 0x755a4ec 0x755a640 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f7a738 0x9bf83c1 0x9bf8bfc 0xecf14d /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb46f44 2026-02-26T03:32:51.072 INFO:teuthology.orchestra.run.trial134.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc6583 /lib64/libasan.so.8+0x157c7 0x9c126a3 0x9ca9bf1 0x9ca9dc8 0x9d383b7 0x9d38b76 0x9d390bc 0x9d44f45 0x9d477c3 0x9d07d91 0x9d06d91 0x8f5bc05 0x8f5ce1a 0x8f5ff8b 0x787cd7f 0x78b3782 0x6fa18b5 0x755a4ec 0x755a640 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:32:51.097 INFO:teuthology.orchestra.run.trial134.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x2a552f8 0x7d837f4 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f7a738 0x9bf83c1 0x9bf8bfc 0xecf14d /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb46f44 2026-02-26T03:32:51.134 INFO:teuthology.orchestra.run.trial134.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x2a00bec 0x2a559e0 0x7d837b9 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:32:51.136 INFO:teuthology.orchestra.run.trial134.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x2a55425 0x7d837f4 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:32:51.138 INFO:teuthology.orchestra.run.trial134.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x151f5 0x2a55a3f 0x7d837b9 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:32:51.266 INFO:teuthology.orchestra.run.trial134.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x15e1f 0x9d331b6 0x9d45d3c 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6f43702 0x6f45a6a 0x7483f52 0x72a690f 0x72a6a11 0x72a6b09 0x7360706 0x73607bf 0x7360972 0x7360aaa 0x7360b25 0x7360baa 0x7360d05 0x7423961 0x7483fdb 0x728dda4 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:32:51.266 INFO:teuthology.orchestra.run.trial134.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1294d2 /lib64/libasan.so.8+0x1297ad /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x15197 0x1125428 0xf2e0d1 0x9c50a8f 0x9c50f59 0x9c511cc 0x9c521cc 0x9c5a24f 0x9c5a5cb 0x9d43118 0x9d477c3 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6f43702 0x6f45a6a 0x7483f52 0x72a690f 0x72a6a11 0x72a6b09 0x7360706 0x73607bf 0x7360972 0x7360aaa 0x7360b25 0x7360baa 0x7360d05 0x7423961 0x7483fdb 0x728dda4 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:32:51.266 INFO:teuthology.orchestra.run.trial134.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0xe06a1e 0x11253d1 0xf2e0d1 0x9c50a8f 0x9c50f59 0x9c511cc 0x9c51482 0x9c51f31 0x9c5242f 0x9c52a5f 0x9c52bc9 0x9d32f39 0x9d45d3c 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6f43702 0x6f45a6a 0x7483f52 0x72a690f 0x72a6a11 0x72a6b09 0x7360706 0x73607bf 0x7360972 0x7360aaa 0x7360b25 0x7360baa 0x7360d05 0x7423961 0x7483fdb 0x728dda4 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:32:51.348 INFO:teuthology.orchestra.run.trial134.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x8cc08 /lib64/libasan.so.8+0x10069c /lib64/libasan.so.8+0xc65b9 /lib64/libasan.so.8+0x15197 0x2a55a3f 0x7d837b9 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f7a738 0x9bf83c1 0x9bf8bfc 0xecf14d /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb46f44 2026-02-26T03:32:51.363 INFO:teuthology.orchestra.run.trial134.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc6592 /lib64/libasan.so.8+0x15197 0x2a55a3f 0x7d837b9 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:32:51.363 INFO:teuthology.orchestra.run.trial134.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x2a00bea 0x2a559e0 0x7d837b9 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:32:51.363 INFO:teuthology.orchestra.run.trial134.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1297e4 /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x15197 0x2a55a3f 0x7d837b9 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:32:53.511 INFO:teuthology.orchestra.run.trial134.stderr:----------------------------------------------------- 2026-02-26T03:32:53.511 INFO:teuthology.orchestra.run.trial134.stderr:Suppressions used: 2026-02-26T03:32:53.512 INFO:teuthology.orchestra.run.trial134.stderr: count bytes template 2026-02-26T03:32:53.512 INFO:teuthology.orchestra.run.trial134.stderr: 1 8 MallocExtension::Register 2026-02-26T03:32:53.512 INFO:teuthology.orchestra.run.trial134.stderr:----------------------------------------------------- 2026-02-26T03:32:53.512 INFO:teuthology.orchestra.run.trial134.stderr: 2026-02-26T03:32:53.546 DEBUG:teuthology.orchestra.run.trial134:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2026-02-26T03:32:53.573 DEBUG:teuthology.orchestra.run.trial134:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-26T03:32:53.724 INFO:teuthology.orchestra.run.trial134.stderr:INFO  osd - OSD::main: parsing early config 2026-02-26T03:32:53.775 INFO:teuthology.orchestra.run.trial134.stderr:INFO 2026-02-26 03:32:53,774 seastar - Reactor backend: linux-aio 2026-02-26T03:32:53.789 INFO:teuthology.orchestra.run.trial134.stderr:INFO 2026-02-26 03:32:53,788 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:32:53.789 INFO:teuthology.orchestra.run.trial134.stderr:INFO 2026-02-26 03:32:53,788 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:32:53.791 INFO:teuthology.orchestra.run.trial134.stderr:==39999==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-26T03:32:53.851 INFO:teuthology.orchestra.run.trial134.stderr:DEBUG 2026-02-26 03:32:53,851 [shard 0:main] seastar - reactor::drain 2026-02-26T03:32:55.903 INFO:teuthology.orchestra.run.trial134.stderr:----------------------------------------------------- 2026-02-26T03:32:55.903 INFO:teuthology.orchestra.run.trial134.stderr:Suppressions used: 2026-02-26T03:32:55.903 INFO:teuthology.orchestra.run.trial134.stderr: count bytes template 2026-02-26T03:32:55.903 INFO:teuthology.orchestra.run.trial134.stderr: 1 8 MallocExtension::Register 2026-02-26T03:32:55.903 INFO:teuthology.orchestra.run.trial134.stderr:----------------------------------------------------- 2026-02-26T03:32:55.903 INFO:teuthology.orchestra.run.trial134.stderr: 2026-02-26T03:32:55.922 INFO:teuthology.orchestra.run.trial134.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-26T03:32:55.922 INFO:teuthology.orchestra.run.trial134.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-26T03:32:55.967 INFO:teuthology.orchestra.run.trial134.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-26T03:32:55.967 INFO:teuthology.orchestra.run.trial134.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-26T03:32:55.974 INFO:teuthology.orchestra.run.trial134.stderr:INFO 2026-02-26 03:32:55,973 seastar - Reactor backend: linux-aio 2026-02-26T03:32:55.985 INFO:teuthology.orchestra.run.trial134.stderr:INFO 2026-02-26 03:32:55,984 [shard 2:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:32:55.985 INFO:teuthology.orchestra.run.trial134.stderr:INFO 2026-02-26 03:32:55,984 [shard 2:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:32:55.989 INFO:teuthology.orchestra.run.trial134.stderr:==39997==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-26T03:32:55.989 INFO:teuthology.orchestra.run.trial134.stderr:INFO 2026-02-26 03:32:55,989 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-26T03:32:56.131 INFO:teuthology.orchestra.run.trial134.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/ld-linux-x86-64.so.2+0x16ed3 /lib64/libasan.so.8+0xc65b1 /lib64/libasan.so.8+0x15197 0x1125428 0xf2e0d1 0x9c50a8f 0x9c50f59 0x9c511cc 0x9c52ef3 0x9d320be 0x9d32837 0x9d32d22 0x9d330b5 0x9d45d3c 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6fa06ea 0x6fa44eb 0x6faa095 0x7419054 0x743ba5c 0x728f369 0x728f461 0x736153a 0x73615f3 0x73617aa 0x743badd 0x728f4de 0x728f563 0x72bb301 0x7423ee4 0x7423f69 0x7424047 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:32:56.132 INFO:teuthology.orchestra.run.trial134.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc6592 /lib64/libasan.so.8+0x15197 0x1125428 0xf2e0d1 0x9c50a8f 0x9c50f59 0x9c511cc 0x9c51482 0x9c51a44 0x9c59eeb 0x9c5a5cb 0x9d43118 0x9d477c3 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6fa06ea 0x6fa44eb 0x6faa095 0x7419054 0x743ba5c 0x728f369 0x728f461 0x736153a 0x73615f3 0x73617aa 0x743badd 0x728f4de 0x728f563 0x72bb301 0x7423ee4 0x7423f69 0x7424047 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:32:56.132 INFO:teuthology.orchestra.run.trial134.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x9d33984 0x9d33e86 0x9d34209 0x9d44360 0x9d477c3 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6fa06ea 0x6fa44eb 0x6faa095 0x7419054 0x743ba5c 0x728f369 0x728f461 0x736153a 0x73615f3 0x73617aa 0x743badd 0x728f4de 0x728f563 0x72bb301 0x7423ee4 0x7423f69 0x7424047 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:32:56.164 INFO:teuthology.orchestra.run.trial134.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xb04e6 /lib64/libasan.so.8+0xb1437 0xb69757 0x1090878 0xf2e0dc 0x9c554b6 0x9c55df5 0x9d41dc5 0x9d445a1 0x9d477c3 0x9d07d91 0x9d06d91 0x7e61695 0x7e64490 0x8f5d61d 0x8f5ff8b 0x787cd7f 0x78b3782 0x6fa18b5 0x755a4ec 0x755a640 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f7a738 0x9bf83c1 0x9bf8bfc 0xecf14d /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb46f44 2026-02-26T03:32:56.229 INFO:teuthology.orchestra.run.trial134.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1521e 0x2a55a3f 0x7d837b9 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f7a738 0x9bf83c1 0x9bf8bfc 0xecf14d /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb46f44 2026-02-26T03:32:56.269 INFO:teuthology.orchestra.run.trial134.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x2a55954 0x7d837b9 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:32:56.270 INFO:teuthology.orchestra.run.trial134.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x151f5 0x2a55a3f 0x7d837b9 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:32:56.270 INFO:teuthology.orchestra.run.trial134.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1006a8 /lib64/libasan.so.8+0xc65b9 /lib64/libasan.so.8+0x15197 0x2a55a3f 0x7d837b9 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:32:56.395 INFO:teuthology.orchestra.run.trial134.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x119067 /lib64/libasan.so.8+0xebbf /lib64/libasan.so.8+0xec776 0xcb04f1 0x789489e 0x7bea228 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6f43702 0x6f45a6a 0x7483f52 0x72a690f 0x72a6a11 0x72a6b09 0x7360706 0x73607bf 0x7360972 0x7360aaa 0x7360b25 0x7360baa 0x7360d05 0x7423961 0x7483fdb 0x728dda4 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:32:56.395 INFO:teuthology.orchestra.run.trial134.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x129896 /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x15e37 0x9c50e1a 0x9c511cc 0x9c51482 0x9c51a44 0x9c59eeb 0x9c5a5cb 0x9d43118 0x9d477c3 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6f43702 0x6f45a6a 0x7483f52 0x72a690f 0x72a6a11 0x72a6b09 0x7360706 0x73607bf 0x7360972 0x7360aaa 0x7360b25 0x7360baa 0x7360d05 0x7423961 0x7483fdb 0x728dda4 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:32:56.396 INFO:teuthology.orchestra.run.trial134.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x8cc0c /lib64/libasan.so.8+0x10069c /lib64/libasan.so.8+0xc65b9 /lib64/libasan.so.8+0x15197 0x9c50af8 0x9c50f59 0x9c511cc 0x9c51482 0x9c51f31 0x9c5242f 0x9c52a5f 0x9c52bc9 0x9d32f39 0x9d45d3c 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6f43702 0x6f45a6a 0x7483f52 0x72a690f 0x72a6a11 0x72a6b09 0x7360706 0x73607bf 0x7360972 0x7360aaa 0x7360b25 0x7360baa 0x7360d05 0x7423961 0x7483fdb 0x728dda4 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:32:56.478 INFO:teuthology.orchestra.run.trial134.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/ld-linux-x86-64.so.2+0x16ec6 /lib64/libasan.so.8+0xc65b1 /lib64/libasan.so.8+0x15197 0x2a55a3f 0x7d837b9 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f7a738 0x9bf83c1 0x9bf8bfc 0xecf14d /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb46f44 2026-02-26T03:32:56.493 INFO:teuthology.orchestra.run.trial134.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x2a558cb 0x7d837b9 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:32:56.493 INFO:teuthology.orchestra.run.trial134.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x12976b /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x15197 0x2a55a3f 0x7d837b9 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:32:56.493 INFO:teuthology.orchestra.run.trial134.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x2a55990 0x7d837b9 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:32:58.671 INFO:teuthology.orchestra.run.trial134.stderr:----------------------------------------------------- 2026-02-26T03:32:58.671 INFO:teuthology.orchestra.run.trial134.stderr:Suppressions used: 2026-02-26T03:32:58.671 INFO:teuthology.orchestra.run.trial134.stderr: count bytes template 2026-02-26T03:32:58.671 INFO:teuthology.orchestra.run.trial134.stderr: 1 8 MallocExtension::Register 2026-02-26T03:32:58.671 INFO:teuthology.orchestra.run.trial134.stderr:----------------------------------------------------- 2026-02-26T03:32:58.672 INFO:teuthology.orchestra.run.trial134.stderr: 2026-02-26T03:32:58.708 DEBUG:teuthology.orchestra.run.trial134:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2026-02-26T03:32:58.733 DEBUG:teuthology.orchestra.run.trial134:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-26T03:32:58.884 INFO:teuthology.orchestra.run.trial134.stderr:INFO  osd - OSD::main: parsing early config 2026-02-26T03:32:58.943 INFO:teuthology.orchestra.run.trial134.stderr:INFO 2026-02-26 03:32:58,943 seastar - Reactor backend: linux-aio 2026-02-26T03:32:58.961 INFO:teuthology.orchestra.run.trial134.stderr:INFO 2026-02-26 03:32:58,960 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:32:58.961 INFO:teuthology.orchestra.run.trial134.stderr:INFO 2026-02-26 03:32:58,960 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:32:58.963 INFO:teuthology.orchestra.run.trial134.stderr:==40070==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-26T03:32:59.023 INFO:teuthology.orchestra.run.trial134.stderr:DEBUG 2026-02-26 03:32:59,022 [shard 0:main] seastar - reactor::drain 2026-02-26T03:33:01.073 INFO:teuthology.orchestra.run.trial134.stderr:----------------------------------------------------- 2026-02-26T03:33:01.073 INFO:teuthology.orchestra.run.trial134.stderr:Suppressions used: 2026-02-26T03:33:01.074 INFO:teuthology.orchestra.run.trial134.stderr: count bytes template 2026-02-26T03:33:01.074 INFO:teuthology.orchestra.run.trial134.stderr: 1 8 MallocExtension::Register 2026-02-26T03:33:01.074 INFO:teuthology.orchestra.run.trial134.stderr:----------------------------------------------------- 2026-02-26T03:33:01.074 INFO:teuthology.orchestra.run.trial134.stderr: 2026-02-26T03:33:01.093 INFO:teuthology.orchestra.run.trial134.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-26T03:33:01.093 INFO:teuthology.orchestra.run.trial134.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-26T03:33:01.141 INFO:teuthology.orchestra.run.trial134.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-26T03:33:01.141 INFO:teuthology.orchestra.run.trial134.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-26T03:33:01.148 INFO:teuthology.orchestra.run.trial134.stderr:INFO 2026-02-26 03:33:01,147 seastar - Reactor backend: linux-aio 2026-02-26T03:33:01.159 INFO:teuthology.orchestra.run.trial134.stderr:INFO 2026-02-26 03:33:01,158 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:33:01.159 INFO:teuthology.orchestra.run.trial134.stderr:INFO 2026-02-26 03:33:01,158 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:33:01.163 INFO:teuthology.orchestra.run.trial134.stderr:==40068==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-26T03:33:01.164 INFO:teuthology.orchestra.run.trial134.stderr:INFO 2026-02-26 03:33:01,163 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-26T03:33:01.305 INFO:teuthology.orchestra.run.trial134.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0xb9b054 0x9d3ec28 0x9d3f21a 0x9d496e7 0x78947dd 0x7bea228 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6fa06ea 0x6fa44eb 0x6faa095 0x7419054 0x743ba5c 0x728f369 0x728f461 0x736153a 0x73615f3 0x73617aa 0x743badd 0x728f4de 0x728f563 0x72bb301 0x7423ee4 0x7423f69 0x7424047 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:33:01.305 INFO:teuthology.orchestra.run.trial134.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1297e4 /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x15197 0x1125428 0xf2e0d1 0x9c50a8f 0x9c50f59 0x9c511cc 0x9c51482 0x9c51f31 0x9c5242f 0x9c52a5f 0x9c52bc9 0x9d32f39 0x9d45d3c 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6fa06ea 0x6fa44eb 0x6faa095 0x7419054 0x743ba5c 0x728f369 0x728f461 0x736153a 0x73615f3 0x73617aa 0x743badd 0x728f4de 0x728f563 0x72bb301 0x7423ee4 0x7423f69 0x7424047 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:33:01.305 INFO:teuthology.orchestra.run.trial134.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x191c1b /lib64/libasan.so.8+0xc748 /lib64/libasan.so.8+0x91c1 /lib64/libasan.so.8+0xee7dc 0x9b2559b 0x9ba2d36 0x9ba2e4c 0x9ba2f28 0x9ba31a4 0x9ba325d 0x9ba374b 0x9ba3b4f 0x9ba3e0c 0x9ba4c54 0x9d1b824 0x9d1b91d 0x7be9c60 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6fa06ea 0x6fa44eb 0x6faa095 0x7419054 0x743ba5c 0x728f369 0x728f461 0x736153a 0x73615f3 0x73617aa 0x743badd 0x728f4de 0x728f563 0x72bb301 0x7423ee4 0x7423f69 0x7424047 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:33:01.380 INFO:teuthology.orchestra.run.trial134.stderr:Reactor stalled for 67 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1190d8 /lib64/libasan.so.8+0xc5f4 /lib64/libasan.so.8+0x91c1 /lib64/libasan.so.8+0xee7dc 0x9b2559b 0x9ba2d36 0x9ba2e4c 0x9ba2f28 0x9ba31a4 0x9ba374b 0x9ba3b4f 0x9ba3e0c 0x9d45991 0x9d07d91 0x9d06d91 0x7beb0f2 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6fa14d6 0x755a4ec 0x755a640 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:33:01.402 INFO:teuthology.orchestra.run.trial134.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x2a54d8c 0x7d837e7 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f7a738 0x9bf83c1 0x9bf8bfc 0xecf14d /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb46f44 2026-02-26T03:33:01.442 INFO:teuthology.orchestra.run.trial134.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1297e4 /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x15197 0x2a55a3f 0x7d837b9 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:33:01.445 INFO:teuthology.orchestra.run.trial134.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1521a 0x2a55a3f 0x7d837b9 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:33:01.450 INFO:teuthology.orchestra.run.trial134.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x2a54da0 0x7d837e7 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:33:01.569 INFO:teuthology.orchestra.run.trial134.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0xcb05fd 0x11253a2 0xf2e0d1 0x9c554b6 0x9c55763 0x9d42c55 0x9d477c3 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6f43702 0x6f45a6a 0x7483f52 0x72a690f 0x72a6a11 0x72a6b09 0x7360706 0x73607bf 0x7360972 0x7360aaa 0x7360b25 0x7360baa 0x7360d05 0x7423961 0x7483fdb 0x728dda4 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:33:01.570 INFO:teuthology.orchestra.run.trial134.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xfcaef /lib64/libasan.so.8+0xc6592 /lib64/libasan.so.8+0x15197 0x9c0cf71 0x9d35546 0x9d39cb8 0x9d3ebdc 0x9d3f21a 0x9d496e7 0x78947dd 0x7be9c23 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6f43702 0x6f45a6a 0x7483f52 0x72a690f 0x72a6a11 0x72a6b09 0x7360706 0x73607bf 0x7360972 0x7360aaa 0x7360b25 0x7360baa 0x7360d05 0x7423961 0x7483fdb 0x728dda4 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:33:01.570 INFO:teuthology.orchestra.run.trial134.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xb13a7 0xa5d7026 0x9ccfb92 0x9ccfe53 0x9d489df 0x9d48b2c 0x9d48bbf 0x9d49a04 0x78947dd 0x7bea228 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6f43702 0x6f45a6a 0x7483f52 0x72a690f 0x72a6a11 0x72a6b09 0x7360706 0x73607bf 0x7360972 0x7360aaa 0x7360b25 0x7360baa 0x7360d05 0x7423961 0x7483fdb 0x728dda4 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:33:01.634 INFO:teuthology.orchestra.run.trial134.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x12976b /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x15197 0x2a55a3f 0x7d837b9 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:33:01.652 INFO:teuthology.orchestra.run.trial134.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1517f 0x2a55a3f 0x7d837b9 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f7a738 0x9bf83c1 0x9bf8bfc 0xecf14d /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb46f44 2026-02-26T03:33:01.666 INFO:teuthology.orchestra.run.trial134.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc6592 /lib64/libasan.so.8+0x15197 0x2a55a3f 0x7d837b9 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:33:01.666 INFO:teuthology.orchestra.run.trial134.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1521a 0x2a55a3f 0x7d837b9 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:33:03.792 INFO:teuthology.orchestra.run.trial134.stderr:----------------------------------------------------- 2026-02-26T03:33:03.793 INFO:teuthology.orchestra.run.trial134.stderr:Suppressions used: 2026-02-26T03:33:03.793 INFO:teuthology.orchestra.run.trial134.stderr: count bytes template 2026-02-26T03:33:03.793 INFO:teuthology.orchestra.run.trial134.stderr: 1 8 MallocExtension::Register 2026-02-26T03:33:03.793 INFO:teuthology.orchestra.run.trial134.stderr:----------------------------------------------------- 2026-02-26T03:33:03.793 INFO:teuthology.orchestra.run.trial134.stderr: 2026-02-26T03:33:03.830 DEBUG:teuthology.orchestra.run.trial134:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2026-02-26T03:33:03.855 INFO:tasks.ceph:Reading keys from all nodes... 2026-02-26T03:33:03.855 DEBUG:teuthology.orchestra.run.trial134:> set -ex 2026-02-26T03:33:03.855 DEBUG:teuthology.orchestra.run.trial134:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2026-02-26T03:33:03.920 DEBUG:teuthology.orchestra.run.trial134:> set -ex 2026-02-26T03:33:03.920 DEBUG:teuthology.orchestra.run.trial134:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2026-02-26T03:33:03.984 DEBUG:teuthology.orchestra.run.trial134:> set -ex 2026-02-26T03:33:03.985 DEBUG:teuthology.orchestra.run.trial134:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2026-02-26T03:33:04.049 DEBUG:teuthology.orchestra.run.trial134:> set -ex 2026-02-26T03:33:04.049 DEBUG:teuthology.orchestra.run.trial134:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2026-02-26T03:33:04.115 DEBUG:teuthology.orchestra.run.trial134:> set -ex 2026-02-26T03:33:04.115 DEBUG:teuthology.orchestra.run.trial134:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2026-02-26T03:33:04.172 INFO:tasks.ceph:Adding keys to all mons... 2026-02-26T03:33:04.172 DEBUG:teuthology.orchestra.run.trial134:> sudo tee -a /etc/ceph/ceph.keyring 2026-02-26T03:33:04.235 INFO:teuthology.orchestra.run.trial134.stdout:[mgr.x] 2026-02-26T03:33:04.235 INFO:teuthology.orchestra.run.trial134.stdout: key = AQDdvp9p4fNbNRAA0vByDuy8w7S9XJCDSYz6jw== 2026-02-26T03:33:04.235 INFO:teuthology.orchestra.run.trial134.stdout:[osd.0] 2026-02-26T03:33:04.235 INFO:teuthology.orchestra.run.trial134.stdout: key = AQDivp9pFIT6NhAADHSRqRqK6ET1whSMfZ6aiQ== 2026-02-26T03:33:04.235 INFO:teuthology.orchestra.run.trial134.stdout:[osd.1] 2026-02-26T03:33:04.235 INFO:teuthology.orchestra.run.trial134.stdout: key = AQDovp9pFmlEAxAAgioRi0AsBqeC8oSqJyY3tQ== 2026-02-26T03:33:04.235 INFO:teuthology.orchestra.run.trial134.stdout:[osd.2] 2026-02-26T03:33:04.235 INFO:teuthology.orchestra.run.trial134.stdout: key = AQDtvp9pY2euDRAAYgZRsLzomvUmveyggl0MkQ== 2026-02-26T03:33:04.235 INFO:teuthology.orchestra.run.trial134.stdout:[client.0] 2026-02-26T03:33:04.236 INFO:teuthology.orchestra.run.trial134.stdout: key = AQDdvp9pHlIBOBAAoDckbz7VJSZhyPm9+0Dyjw== 2026-02-26T03:33:04.237 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2026-02-26T03:33:04.324 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-26T03:33:04.372 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-26T03:33:04.420 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-26T03:33:04.469 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-26T03:33:04.520 INFO:tasks.ceph:Running mkfs on mon nodes... 2026-02-26T03:33:04.520 DEBUG:teuthology.orchestra.run.trial134:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2026-02-26T03:33:04.545 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2026-02-26T03:33:04.631 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-26T03:33:04.630+0000 7fdaf5838c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:33:04.632 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-26T03:33:04.630+0000 7fdaf5838c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:33:04.639 DEBUG:teuthology.orchestra.run.trial134:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2026-02-26T03:33:04.663 DEBUG:teuthology.orchestra.run.trial134:> rm -- /home/ubuntu/cephtest/ceph.monmap 2026-02-26T03:33:04.719 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2026-02-26T03:33:04.720 INFO:tasks.ceph.mon.a:Restarting daemon 2026-02-26T03:33:04.720 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2026-02-26T03:33:04.762 INFO:tasks.ceph.mon.a:Started 2026-02-26T03:33:04.762 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2026-02-26T03:33:04.762 INFO:tasks.ceph.mgr.x:Restarting daemon 2026-02-26T03:33:04.762 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2026-02-26T03:33:04.765 INFO:tasks.ceph.mgr.x:Started 2026-02-26T03:33:04.765 DEBUG:tasks.ceph:set 0 configs 2026-02-26T03:33:04.765 DEBUG:teuthology.orchestra.run.trial134:> sudo ceph --cluster ceph config dump 2026-02-26T03:33:04.825 INFO:tasks.ceph.mon.a.trial134.stderr:2026-02-26T03:33:04.823+0000 7f5b39e2fc00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:33:04.825 INFO:tasks.ceph.mon.a.trial134.stderr:2026-02-26T03:33:04.823+0000 7f5b39e2fc00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:33:04.825 INFO:tasks.ceph.mon.a.trial134.stderr:2026-02-26T03:33:04.823+0000 7f5b39e2fc00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:33:04.829 INFO:tasks.ceph.mgr.x.trial134.stderr:2026-02-26T03:33:04.827+0000 7f0420e87fc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:33:04.842 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-26T03:33:04.838+0000 7efe12ccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:33:04.843 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-26T03:33:04.842+0000 7efe12ccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:33:04.874 INFO:teuthology.orchestra.run.trial134.stdout:WHO MASK LEVEL OPTION VALUE RO 2026-02-26T03:33:04.880 INFO:tasks.ceph:Setting crush tunables to default 2026-02-26T03:33:04.881 DEBUG:teuthology.orchestra.run.trial134:> sudo ceph --cluster ceph osd crush tunables default 2026-02-26T03:33:04.955 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-26T03:33:04.950+0000 7f95623dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:33:04.956 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-26T03:33:04.954+0000 7f95623dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:33:04.989 INFO:teuthology.orchestra.run.trial134.stderr:adjusted tunables profile to default 2026-02-26T03:33:04.997 INFO:tasks.ceph:check_enable_crimson: True 2026-02-26T03:33:04.997 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2026-02-26T03:33:04.997 DEBUG:teuthology.orchestra.run.trial134:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2026-02-26T03:33:05.032 INFO:tasks.ceph.mgr.x.trial134.stderr:2026-02-26T03:33:05.031+0000 7f0420e87fc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:33:05.033 INFO:tasks.ceph.mgr.x.trial134.stderr:2026-02-26T03:33:05.031+0000 7f0420e87fc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:33:05.072 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-26T03:33:05.068+0000 7f2d29dc1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:33:05.073 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-26T03:33:05.072+0000 7f2d29dc1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:33:05.110 INFO:tasks.ceph.mon.a.trial134.stderr:2026-02-26T03:33:05.109+0000 7f5b311f1640 -1 WARNING: experimental feature 'crimson' is enabled 2026-02-26T03:33:05.110 INFO:tasks.ceph.mon.a.trial134.stderr:Please be aware that this feature is experimental, untested, 2026-02-26T03:33:05.110 INFO:tasks.ceph.mon.a.trial134.stderr:unsupported, and may result in data corruption, data loss, 2026-02-26T03:33:05.111 INFO:tasks.ceph.mon.a.trial134.stderr:and/or irreparable damage to your cluster. Do not use 2026-02-26T03:33:05.111 INFO:tasks.ceph.mon.a.trial134.stderr:feature with important data. 2026-02-26T03:33:05.111 INFO:tasks.ceph.mon.a.trial134.stderr: 2026-02-26T03:33:05.875 INFO:tasks.ceph.mgr.x.trial134.stderr:/usr/lib64/python3.9/site-packages/scipy/__init__.py:73: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2026-02-26T03:33:05.875 INFO:tasks.ceph.mgr.x.trial134.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2026-02-26T03:33:05.875 INFO:tasks.ceph.mgr.x.trial134.stderr: from numpy import show_config as show_numpy_config 2026-02-26T03:33:05.999 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2026-02-26T03:33:06.000 DEBUG:teuthology.orchestra.run.trial134:> set -ex 2026-02-26T03:33:06.000 DEBUG:teuthology.orchestra.run.trial134:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2026-02-26T03:33:06.064 DEBUG:teuthology.orchestra.run.trial134:> set -ex 2026-02-26T03:33:06.064 DEBUG:teuthology.orchestra.run.trial134:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2026-02-26T03:33:06.129 DEBUG:teuthology.orchestra.run.trial134:> set -ex 2026-02-26T03:33:06.129 DEBUG:teuthology.orchestra.run.trial134:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2026-02-26T03:33:06.192 DEBUG:teuthology.orchestra.run.trial134:> sudo ceph --cluster ceph osd new e46311d8-c45b-48fa-9e05-d0fdb7538c00 0 2026-02-26T03:33:06.307 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-26T03:33:06.302+0000 7f3b73e5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:33:06.308 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-26T03:33:06.306+0000 7f3b73e5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:33:06.345 INFO:teuthology.orchestra.run.trial134.stdout:0 2026-02-26T03:33:06.354 DEBUG:teuthology.orchestra.run.trial134:> sudo ceph --cluster ceph osd new e23cd147-21ce-4038-9c6f-b897e06ca54b 1 2026-02-26T03:33:06.428 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-26T03:33:06.423+0000 7f2f4665a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:33:06.429 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-26T03:33:06.427+0000 7f2f4665a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:33:06.468 INFO:teuthology.orchestra.run.trial134.stdout:1 2026-02-26T03:33:06.477 DEBUG:teuthology.orchestra.run.trial134:> sudo ceph --cluster ceph osd new 73f15d42-4d54-4304-bdc0-dc208c74576e 2 2026-02-26T03:33:06.554 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-26T03:33:06.549+0000 7f57bfb74640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:33:06.555 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-26T03:33:06.553+0000 7f57bfb74640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:33:06.592 INFO:teuthology.orchestra.run.trial134.stdout:2 2026-02-26T03:33:06.601 DEBUG:teuthology.orchestra.run.trial134:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2026-02-26T03:33:06.630 INFO:tasks.ceph.osd.0:Restarting daemon 2026-02-26T03:33:06.630 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2026-02-26T03:33:06.672 INFO:tasks.ceph.osd.0:Started 2026-02-26T03:33:06.672 DEBUG:teuthology.orchestra.run.trial134:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2026-02-26T03:33:06.700 INFO:tasks.ceph.osd.1:Restarting daemon 2026-02-26T03:33:06.701 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2026-02-26T03:33:06.744 INFO:tasks.ceph.osd.1:Started 2026-02-26T03:33:06.744 DEBUG:teuthology.orchestra.run.trial134:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2026-02-26T03:33:06.770 INFO:tasks.ceph.osd.2:Restarting daemon 2026-02-26T03:33:06.770 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2026-02-26T03:33:06.813 INFO:tasks.ceph.osd.2:Started 2026-02-26T03:33:06.813 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-26T03:33:06.892 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-26T03:33:06.888+0000 7f081bd5c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:33:06.893 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-26T03:33:06.892+0000 7f081bd5c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:33:06.926 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:33:06.926 INFO:teuthology.orchestra.run.trial134.stdout:{"epoch":6,"fsid":"6ee9ec32-e74b-407c-ab79-d45b46c3ba91","created":"2026-02-26T03:33:04.835545+0000","modified":"2026-02-26T03:33:06.590744+0000","last_up_change":"0.000000","last_in_change":"2026-02-26T03:33:06.590744+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"e46311d8-c45b-48fa-9e05-d0fdb7538c00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e23cd147-21ce-4038-9c6f-b897e06ca54b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"73f15d42-4d54-4304-bdc0-dc208c74576e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-26T03:33:06.935 INFO:tasks.ceph.ceph_manager.ceph:[] 2026-02-26T03:33:06.936 INFO:tasks.ceph:Waiting for OSDs to come up 2026-02-26T03:33:07.238 DEBUG:teuthology.orchestra.run.trial134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-26T03:33:07.308 INFO:teuthology.misc.health.trial134.stderr:2026-02-26T03:33:07.305+0000 7fcf00611640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:33:07.310 INFO:teuthology.misc.health.trial134.stderr:2026-02-26T03:33:07.308+0000 7fcf00611640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:33:07.341 INFO:teuthology.misc.health.trial134.stdout: 2026-02-26T03:33:07.341 INFO:teuthology.misc.health.trial134.stdout:{"epoch":6,"fsid":"6ee9ec32-e74b-407c-ab79-d45b46c3ba91","created":"2026-02-26T03:33:04.835545+0000","modified":"2026-02-26T03:33:06.590744+0000","last_up_change":"0.000000","last_in_change":"2026-02-26T03:33:06.590744+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"e46311d8-c45b-48fa-9e05-d0fdb7538c00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e23cd147-21ce-4038-9c6f-b897e06ca54b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"73f15d42-4d54-4304-bdc0-dc208c74576e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-26T03:33:07.348 DEBUG:teuthology.misc:0 of 3 OSDs are up 2026-02-26T03:33:08.261 INFO:tasks.ceph.mgr.x.trial134.stderr:2026-02-26T03:33:08.259+0000 7f033c36a640 -1 mgr.server handle_report got status from non-daemon mon.a 2026-02-26T03:33:13.651 DEBUG:teuthology.orchestra.run.trial134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-26T03:33:13.737 INFO:teuthology.misc.health.trial134.stderr:2026-02-26T03:33:13.733+0000 7f334b85a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:33:13.739 INFO:teuthology.misc.health.trial134.stderr:2026-02-26T03:33:13.737+0000 7f334b85a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:33:13.879 INFO:teuthology.misc.health.trial134.stdout: 2026-02-26T03:33:13.879 INFO:teuthology.misc.health.trial134.stdout:{"epoch":8,"fsid":"6ee9ec32-e74b-407c-ab79-d45b46c3ba91","created":"2026-02-26T03:33:04.835545+0000","modified":"2026-02-26T03:33:13.718610+0000","last_up_change":"0.000000","last_in_change":"2026-02-26T03:33:06.590744+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"e46311d8-c45b-48fa-9e05-d0fdb7538c00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e23cd147-21ce-4038-9c6f-b897e06ca54b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"73f15d42-4d54-4304-bdc0-dc208c74576e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-26T03:33:13.895 DEBUG:teuthology.misc:0 of 3 OSDs are up 2026-02-26T03:33:20.197 DEBUG:teuthology.orchestra.run.trial134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-26T03:33:20.268 INFO:teuthology.misc.health.trial134.stderr:2026-02-26T03:33:20.264+0000 7efc2cf93640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:33:20.269 INFO:teuthology.misc.health.trial134.stderr:2026-02-26T03:33:20.267+0000 7efc2cf93640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:33:20.378 INFO:teuthology.misc.health.trial134.stdout: 2026-02-26T03:33:20.379 INFO:teuthology.misc.health.trial134.stdout:{"epoch":13,"fsid":"6ee9ec32-e74b-407c-ab79-d45b46c3ba91","created":"2026-02-26T03:33:04.835545+0000","modified":"2026-02-26T03:33:19.735906+0000","last_up_change":"2026-02-26T03:33:15.726975+0000","last_in_change":"2026-02-26T03:33:06.590744+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-26T03:33:17.270492+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e46311d8-c45b-48fa-9e05-d0fdb7538c00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6810","nonce":758418296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6813","nonce":758418296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6809","nonce":758418296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6808","nonce":758418296}]},"public_addr":"10.20.193.134:6810/758418296","cluster_addr":"10.20.193.134:6813/758418296","heartbeat_back_addr":"10.20.193.134:6809/758418296","heartbeat_front_addr":"10.20.193.134:6808/758418296","state":["exists","up"]},{"osd":1,"uuid":"e23cd147-21ce-4038-9c6f-b897e06ca54b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6811","nonce":4031171987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6812","nonce":4031171987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6806","nonce":4031171987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6805","nonce":4031171987}]},"public_addr":"10.20.193.134:6811/4031171987","cluster_addr":"10.20.193.134:6812/4031171987","heartbeat_back_addr":"10.20.193.134:6806/4031171987","heartbeat_front_addr":"10.20.193.134:6805/4031171987","state":["exists","up"]},{"osd":2,"uuid":"73f15d42-4d54-4304-bdc0-dc208c74576e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6803","nonce":2766065478}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6804","nonce":2766065478}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6802","nonce":2766065478}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6807","nonce":2766065478}]},"public_addr":"10.20.193.134:6803/2766065478","cluster_addr":"10.20.193.134:6804/2766065478","heartbeat_back_addr":"10.20.193.134:6802/2766065478","heartbeat_front_addr":"10.20.193.134:6807/2766065478","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-26T03:33:20.387 DEBUG:teuthology.misc:3 of 3 OSDs are up 2026-02-26T03:33:20.387 INFO:tasks.ceph:Creating RBD pool 2026-02-26T03:33:20.387 DEBUG:teuthology.orchestra.run.trial134:> sudo ceph --cluster ceph osd pool create rbd 8 2026-02-26T03:33:20.463 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-26T03:33:20.458+0000 7fe55f051640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:33:20.464 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-26T03:33:20.463+0000 7fe55f051640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:33:21.071 INFO:teuthology.orchestra.run.trial134.stderr:pool 'rbd' created 2026-02-26T03:33:21.085 DEBUG:teuthology.orchestra.run.trial134:> rbd --cluster ceph pool init rbd 2026-02-26T03:33:21.130 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-26T03:33:21.129+0000 7fd7aae8d240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:33:21.131 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-26T03:33:21.129+0000 7fd7aae8d240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:33:21.132 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-26T03:33:21.130+0000 7fd7aae8d240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:33:24.106 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2026-02-26T03:33:24.108 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2026-02-26T03:33:24.108 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2026-02-26T03:33:24.185 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-26T03:33:24.181+0000 7f3946f0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:33:24.187 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-26T03:33:24.185+0000 7f3946f0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:33:24.297 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:33:24.306 INFO:teuthology.orchestra.run.trial134.stdout:[{"version":1,"timestamp":"0.000000","name":"","changes":[]}] 2026-02-26T03:33:24.306 INFO:tasks.ceph_manager:config epoch is 1 2026-02-26T03:33:24.307 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2026-02-26T03:33:24.307 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2026-02-26T03:33:24.307 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2026-02-26T03:33:24.383 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-26T03:33:24.379+0000 7f428425a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:33:24.384 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-26T03:33:24.382+0000 7f428425a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:33:24.524 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:33:24.533 INFO:teuthology.orchestra.run.trial134.stdout:{"epoch":5,"flags":0,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6800","nonce":957341771},{"type":"v1","addr":"10.20.193.134:6801","nonce":957341771}]},"active_addr":"10.20.193.134:6801/957341771","active_change":"2026-02-26T03:33:07.254901+0000","active_mgr_features":4544132024016699391,"available":true,"standbys":[],"modules":["iostat","nfs"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_pg_upmap_activity":{"name":"update_pg_upmap_activity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Updates pg_upmap activity stats to be used in `balancer status detail`","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often the agent on each host will gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"The first TCP port the agent will try to bind to. Up to 1000 subsequent ports will be attempted if this port cannot be bound","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this option can facilitate debugging daemons that encounter runtime problems.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"How frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"Fraction of total system memory to divide among autotuned daemons. Converged systems hosting both Ceph and compute should set a lower value; systems with more than 10 GiB of memory per OSD may benefit from a slightly larger value so that the abundance of RAM can be fully utilized","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command persistent logging","long_desc":"","tags":[],"see_also":[]},"certificate_automated_rotation_enabled":{"name":"certificate_automated_rotation_enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"This flag controls whether cephadm automatically rotates certificates upon expiration.","long_desc":"","tags":[],"see_also":[]},"certificate_check_debug_mode":{"name":"certificate_check_debug_mode","type":"bool","level":"dev","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"FOR TESTING ONLY: This flag forces the certificate check instead of waiting for certificate_check_period.","long_desc":"","tags":[],"see_also":[]},"certificate_check_period":{"name":"certificate_check_period","type":"int","level":"advanced","flags":0,"default_value":"1","min":"0","max":"30","enum_allowed":[],"desc":"Specifies how often (in days) the certificate should be checked for validity.","long_desc":"","tags":[],"see_also":[]},"certificate_duration_days":{"name":"certificate_duration_days","type":"int","level":"advanced","flags":0,"default_value":"1095","min":"90","max":"3650","enum_allowed":[],"desc":"Specifies the duration of self-signed certificates generated and signed by the cephadm root CA","long_desc":"","tags":[],"see_also":[]},"certificate_renewal_threshold_days":{"name":"certificate_renewal_threshold_days","type":"int","level":"advanced","flags":0,"default_value":"30","min":"10","max":"90","enum_allowed":[],"desc":"Specifies the lead time in days to initiate certificate renewal before expiration.","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently Podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.28.1","min":"","max":"","enum_allowed":[],"desc":"Alertmanager container image","long_desc":"","tags":[],"see_also":[]},"container_image_alloy":{"name":"container_image_alloy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/alloy:latest","min":"","max":"","enum_allowed":[],"desc":"Alloy container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"Elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:12.3.1","min":"","max":"","enum_allowed":[],"desc":"Grafana container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"Haproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.9.1","min":"","max":"","enum_allowed":[],"desc":"Node exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.6","min":"","max":"","enum_allowed":[],"desc":"Nvmeof container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"Oauth2 proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v3.6.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba metrics container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"Snmp gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"quay.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"Location of dashboards to include in Grafana","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"How frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether the hardware monitoring daemon be deployed on every host?","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at the debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"Max number of daemons per service per host: guard against mistakes and runaways","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"Max number of OSDs that will be drained simultaneously when OSDs are removed","long_desc":"","tags":[],"see_also":[]},"max_parallel_osd_upgrades":{"name":"max_parallel_osd_upgrades","type":"int","level":"advanced","flags":0,"default_value":"16","min":"","max":"","enum_allowed":[],"desc":"Maximum number of OSD daemons upgraded in parallel.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"Mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default IP address for RedFish API (OOB management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"Location of alerts to include in Prometheus Alertmanager","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry URL for login purposes. This is not the default registry.","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"Cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Customized SSH config file to distribute to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many sequential SSH connection liveness check failures before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often SSH connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"stray_daemon_check_interval":{"name":"stray_daemon_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"How frequently cephadm should check for the presence of stray daemons","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use the cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest to ensure that all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROM_ALERT_CREDENTIAL_CACHE_TTL":{"name":"PROM_ALERT_CREDENTIAL_CACHE_TTL","type":"int","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_HOSTNAME_PER_DAEMON":{"name":"RGW_HOSTNAME_PER_DAEMON","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crypto_caller":{"name":"crypto_caller","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":true,"error_string":"","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"healthcheck_history_max_entries":{"name":"healthcheck_history_max_entries","type":"int","level":"advanced","flags":1,"default_value":"1000","min":"","max":"","enum_allowed":[],"desc":"Maximum number of health check history entries to keep","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"prometheus_tls_secret_name":{"name":"prometheus_tls_secret_name","type":"str","level":"advanced","flags":0,"default_value":"rook-ceph-prometheus-server-tls","min":"","max":"","enum_allowed":[],"desc":"name of tls secret in k8s for prometheus","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_cloning":{"name":"pause_cloning","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_purging":{"name":"pause_purging","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous subvolume purge threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"tentacle":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"force_disabled_modules":{},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"10.20.193.134:0","nonce":3330661496}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"10.20.193.134:0","nonce":1734097247}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"10.20.193.134:0","nonce":299089646}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"10.20.193.134:0","nonce":662192478}]}]} 2026-02-26T03:33:24.537 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2026-02-26T03:33:24.537 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2026-02-26T03:33:24.538 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-26T03:33:24.615 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-26T03:33:24.611+0000 7f2be637e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:33:24.616 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-26T03:33:24.614+0000 7f2be637e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:33:24.725 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:33:24.726 INFO:teuthology.orchestra.run.trial134.stdout:{"epoch":17,"fsid":"6ee9ec32-e74b-407c-ab79-d45b46c3ba91","created":"2026-02-26T03:33:04.835545+0000","modified":"2026-02-26T03:33:24.079953+0000","last_up_change":"2026-02-26T03:33:15.726975+0000","last_in_change":"2026-02-26T03:33:06.590744+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-26T03:33:17.270492+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-26T03:33:20.576604+0000","flags":270337,"flags_names":"hashpspool,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e46311d8-c45b-48fa-9e05-d0fdb7538c00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6810","nonce":758418296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6813","nonce":758418296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6809","nonce":758418296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6808","nonce":758418296}]},"public_addr":"10.20.193.134:6810/758418296","cluster_addr":"10.20.193.134:6813/758418296","heartbeat_back_addr":"10.20.193.134:6809/758418296","heartbeat_front_addr":"10.20.193.134:6808/758418296","state":["exists","up"]},{"osd":1,"uuid":"e23cd147-21ce-4038-9c6f-b897e06ca54b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6811","nonce":4031171987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6812","nonce":4031171987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6806","nonce":4031171987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6805","nonce":4031171987}]},"public_addr":"10.20.193.134:6811/4031171987","cluster_addr":"10.20.193.134:6812/4031171987","heartbeat_back_addr":"10.20.193.134:6806/4031171987","heartbeat_front_addr":"10.20.193.134:6805/4031171987","state":["exists","up"]},{"osd":2,"uuid":"73f15d42-4d54-4304-bdc0-dc208c74576e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6803","nonce":2766065478}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6804","nonce":2766065478}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6802","nonce":2766065478}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6807","nonce":2766065478}]},"public_addr":"10.20.193.134:6803/2766065478","cluster_addr":"10.20.193.134:6804/2766065478","heartbeat_back_addr":"10.20.193.134:6802/2766065478","heartbeat_front_addr":"10.20.193.134:6807/2766065478","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-26T03:33:24.735 INFO:tasks.ceph.ceph_manager.ceph:all up! 2026-02-26T03:33:24.735 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-26T03:33:24.812 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-26T03:33:24.809+0000 7fc7b730b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:33:24.813 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-26T03:33:24.812+0000 7fc7b730b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:33:24.927 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:33:24.927 INFO:teuthology.orchestra.run.trial134.stdout:{"epoch":17,"fsid":"6ee9ec32-e74b-407c-ab79-d45b46c3ba91","created":"2026-02-26T03:33:04.835545+0000","modified":"2026-02-26T03:33:24.079953+0000","last_up_change":"2026-02-26T03:33:15.726975+0000","last_in_change":"2026-02-26T03:33:06.590744+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-26T03:33:17.270492+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-26T03:33:20.576604+0000","flags":270337,"flags_names":"hashpspool,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e46311d8-c45b-48fa-9e05-d0fdb7538c00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6810","nonce":758418296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6813","nonce":758418296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6809","nonce":758418296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6808","nonce":758418296}]},"public_addr":"10.20.193.134:6810/758418296","cluster_addr":"10.20.193.134:6813/758418296","heartbeat_back_addr":"10.20.193.134:6809/758418296","heartbeat_front_addr":"10.20.193.134:6808/758418296","state":["exists","up"]},{"osd":1,"uuid":"e23cd147-21ce-4038-9c6f-b897e06ca54b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6811","nonce":4031171987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6812","nonce":4031171987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6806","nonce":4031171987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6805","nonce":4031171987}]},"public_addr":"10.20.193.134:6811/4031171987","cluster_addr":"10.20.193.134:6812/4031171987","heartbeat_back_addr":"10.20.193.134:6806/4031171987","heartbeat_front_addr":"10.20.193.134:6805/4031171987","state":["exists","up"]},{"osd":2,"uuid":"73f15d42-4d54-4304-bdc0-dc208c74576e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6803","nonce":2766065478}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6804","nonce":2766065478}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6802","nonce":2766065478}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.134:6807","nonce":2766065478}]},"public_addr":"10.20.193.134:6803/2766065478","cluster_addr":"10.20.193.134:6804/2766065478","heartbeat_back_addr":"10.20.193.134:6802/2766065478","heartbeat_front_addr":"10.20.193.134:6807/2766065478","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-26T03:33:24.938 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-26T03:33:24.939 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-26T03:33:24.939 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-26T03:33:25.021 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-26T03:33:25.017+0000 7f91b323d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:33:25.022 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-26T03:33:25.016+0000 7fe52410b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:33:25.022 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-26T03:33:25.016+0000 7fd398dec640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:33:25.023 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-26T03:33:25.021+0000 7f91b323d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:33:25.023 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-26T03:33:25.022+0000 7fd398dec640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:33:25.024 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-26T03:33:25.022+0000 7fe52410b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:33:25.038 INFO:teuthology.orchestra.run.trial134.stdout:38654705674 2026-02-26T03:33:25.039 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-26T03:33:25.039 INFO:teuthology.orchestra.run.trial134.stdout:42949672969 2026-02-26T03:33:25.040 INFO:teuthology.orchestra.run.trial134.stdout:42949672969 2026-02-26T03:33:25.040 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-26T03:33:25.041 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-26T03:33:25.160 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-26T03:33:25.156+0000 7f66dfbfd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:33:25.161 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-26T03:33:25.156+0000 7f5162d0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:33:25.161 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-26T03:33:25.156+0000 7f306ff62640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:33:25.162 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-26T03:33:25.160+0000 7f66dfbfd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:33:25.162 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-26T03:33:25.160+0000 7f5162d0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:33:25.162 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-26T03:33:25.160+0000 7f306ff62640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:33:25.270 INFO:teuthology.orchestra.run.trial134.stdout:38654705673 2026-02-26T03:33:25.273 INFO:teuthology.orchestra.run.trial134.stdout:42949672969 2026-02-26T03:33:25.278 INFO:teuthology.orchestra.run.trial134.stdout:42949672969 2026-02-26T03:33:25.281 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705674 got 38654705673 for osd.1 2026-02-26T03:33:25.284 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672969 got 42949672969 for osd.0 2026-02-26T03:33:25.285 DEBUG:teuthology.parallel:result is None 2026-02-26T03:33:25.289 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672969 got 42949672969 for osd.2 2026-02-26T03:33:25.289 DEBUG:teuthology.parallel:result is None 2026-02-26T03:33:26.282 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-26T03:33:26.362 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-26T03:33:26.358+0000 7fc76f1dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:33:26.363 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-26T03:33:26.362+0000 7fc76f1dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:33:26.473 INFO:teuthology.orchestra.run.trial134.stdout:38654705674 2026-02-26T03:33:26.482 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705674 got 38654705674 for osd.1 2026-02-26T03:33:26.482 DEBUG:teuthology.parallel:result is None 2026-02-26T03:33:26.483 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-26T03:33:26.483 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-26T03:33:26.558 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-26T03:33:26.554+0000 7f2d2150b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:33:26.559 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-26T03:33:26.557+0000 7f2d2150b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:33:26.665 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:33:26.666 INFO:teuthology.orchestra.run.trial134.stderr:dumped all 2026-02-26T03:33:26.674 INFO:teuthology.orchestra.run.trial134.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2026-02-26T03:33:25.258700+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156499968,"kb_used":2540,"kb_used_data":2540,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156497428,"statfs":{"total":160255967232,"available":160253366272,"internally_reserved":0,"allocated":2600960,"data_stored":2600960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.187380"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:33:24.093170+0000","last_change":"2026-02-26T03:33:24.095792+0000","last_active":"2026-02-26T03:33:24.093170+0000","last_peered":"2026-02-26T03:33:24.093170+0000","last_clean":"2026-02-26T03:33:24.093170+0000","last_became_active":"2026-02-26T03:33:22.110938+0000","last_became_peered":"2026-02-26T03:33:22.110938+0000","last_unstale":"2026-02-26T03:33:24.093170+0000","last_undegraded":"2026-02-26T03:33:24.093170+0000","last_fullsized":"2026-02-26T03:33:24.093170+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:33:21.070421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:33:21.070421+0000","last_clean_scrub_stamp":"2026-02-26T03:33:21.070421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:33:24.094602+0000","last_change":"2026-02-26T03:33:24.096392+0000","last_active":"2026-02-26T03:33:24.094602+0000","last_peered":"2026-02-26T03:33:24.094602+0000","last_clean":"2026-02-26T03:33:24.094602+0000","last_became_active":"2026-02-26T03:33:22.107588+0000","last_became_peered":"2026-02-26T03:33:22.107588+0000","last_unstale":"2026-02-26T03:33:24.094602+0000","last_undegraded":"2026-02-26T03:33:24.094602+0000","last_fullsized":"2026-02-26T03:33:24.094602+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:33:21.070421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:33:21.070421+0000","last_clean_scrub_stamp":"2026-02-26T03:33:21.070421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:33:24.094673+0000","last_change":"2026-02-26T03:33:24.097778+0000","last_active":"2026-02-26T03:33:24.094673+0000","last_peered":"2026-02-26T03:33:24.094673+0000","last_clean":"2026-02-26T03:33:24.094673+0000","last_became_active":"2026-02-26T03:33:22.108984+0000","last_became_peered":"2026-02-26T03:33:22.108984+0000","last_unstale":"2026-02-26T03:33:24.094673+0000","last_undegraded":"2026-02-26T03:33:24.094673+0000","last_fullsized":"2026-02-26T03:33:24.094673+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:33:21.070421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:33:21.070421+0000","last_clean_scrub_stamp":"2026-02-26T03:33:21.070421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:33:24.094593+0000","last_change":"2026-02-26T03:33:24.096485+0000","last_active":"2026-02-26T03:33:24.094593+0000","last_peered":"2026-02-26T03:33:24.094593+0000","last_clean":"2026-02-26T03:33:24.094593+0000","last_became_active":"2026-02-26T03:33:22.110411+0000","last_became_peered":"2026-02-26T03:33:22.110411+0000","last_unstale":"2026-02-26T03:33:24.094593+0000","last_undegraded":"2026-02-26T03:33:24.094593+0000","last_fullsized":"2026-02-26T03:33:24.094593+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:33:21.070421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:33:21.070421+0000","last_clean_scrub_stamp":"2026-02-26T03:33:21.070421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'1","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:33:24.094742+0000","last_change":"2026-02-26T03:33:24.094742+0000","last_active":"2026-02-26T03:33:24.094742+0000","last_peered":"2026-02-26T03:33:24.094742+0000","last_clean":"2026-02-26T03:33:24.094742+0000","last_became_active":"2026-02-26T03:33:22.121618+0000","last_became_peered":"2026-02-26T03:33:22.121618+0000","last_unstale":"2026-02-26T03:33:24.094742+0000","last_undegraded":"2026-02-26T03:33:24.094742+0000","last_fullsized":"2026-02-26T03:33:24.094742+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:33:21.070421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:33:21.070421+0000","last_clean_scrub_stamp":"2026-02-26T03:33:21.070421+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:33:24.092886+0000","last_change":"2026-02-26T03:33:24.094840+0000","last_active":"2026-02-26T03:33:24.092886+0000","last_peered":"2026-02-26T03:33:24.092886+0000","last_clean":"2026-02-26T03:33:24.092886+0000","last_became_active":"2026-02-26T03:33:22.118089+0000","last_became_peered":"2026-02-26T03:33:22.118089+0000","last_unstale":"2026-02-26T03:33:24.092886+0000","last_undegraded":"2026-02-26T03:33:24.092886+0000","last_fullsized":"2026-02-26T03:33:24.092886+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:33:21.070421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:33:21.070421+0000","last_clean_scrub_stamp":"2026-02-26T03:33:21.070421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:33:24.091706+0000","last_change":"2026-02-26T03:33:24.094034+0000","last_active":"2026-02-26T03:33:24.091706+0000","last_peered":"2026-02-26T03:33:24.091706+0000","last_clean":"2026-02-26T03:33:24.091706+0000","last_became_active":"2026-02-26T03:33:22.113927+0000","last_became_peered":"2026-02-26T03:33:22.113927+0000","last_unstale":"2026-02-26T03:33:24.091706+0000","last_undegraded":"2026-02-26T03:33:24.091706+0000","last_fullsized":"2026-02-26T03:33:24.091706+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:33:21.070421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:33:21.070421+0000","last_clean_scrub_stamp":"2026-02-26T03:33:21.070421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":16,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:33:24.094576+0000","last_change":"2026-02-26T03:33:24.097770+0000","last_active":"2026-02-26T03:33:24.094576+0000","last_peered":"2026-02-26T03:33:24.094576+0000","last_clean":"2026-02-26T03:33:24.094576+0000","last_became_active":"2026-02-26T03:33:22.110305+0000","last_became_peered":"2026-02-26T03:33:22.110305+0000","last_unstale":"2026-02-26T03:33:24.094576+0000","last_undegraded":"2026-02-26T03:33:24.094576+0000","last_fullsized":"2026-02-26T03:33:24.094576+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:33:21.070421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:33:21.070421+0000","last_clean_scrub_stamp":"2026-02-26T03:33:21.070421+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":90,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:33:24.093885+0000","last_change":"2026-02-26T03:33:18.767167+0000","last_active":"2026-02-26T03:33:24.093885+0000","last_peered":"2026-02-26T03:33:24.093885+0000","last_clean":"2026-02-26T03:33:24.093885+0000","last_became_active":"2026-02-26T03:33:18.760057+0000","last_became_peered":"2026-02-26T03:33:18.760057+0000","last_unstale":"2026-02-26T03:33:24.093885+0000","last_undegraded":"2026-02-26T03:33:24.093885+0000","last_fullsized":"2026-02-26T03:33:24.093885+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:33:17.728653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:33:17.728653+0000","last_clean_scrub_stamp":"2026-02-26T03:33:17.728653+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160255967232,"available":160253366272,"internally_reserved":0,"allocated":2600960,"data_stored":2600960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53418655744,"available":53417480192,"internally_reserved":0,"allocated":1175552,"data_stored":1175552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":316,"kb_used_data":316,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52166340,"statfs":{"total":53418655744,"available":53418332160,"internally_reserved":0,"allocated":323584,"data_stored":323584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1076,"kb_used_data":1076,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165580,"statfs":{"total":53418655744,"available":53417553920,"internally_reserved":0,"allocated":1101824,"data_stored":1101824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1148,"kb_used_data":1148,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165508,"statfs":{"total":53418655744,"available":53417480192,"internally_reserved":0,"allocated":1175552,"data_stored":1175552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53418655744,"available":53417480192,"internally_reserved":0,"allocated":1175552,"data_stored":1175552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53418655744,"available":53417553920,"internally_reserved":0,"allocated":1101824,"data_stored":1101824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53418655744,"available":53417480192,"internally_reserved":0,"allocated":1175552,"data_stored":1175552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53418655744,"available":53418332160,"internally_reserved":0,"allocated":323584,"data_stored":323584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-26T03:33:26.675 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-26T03:33:26.752 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-26T03:33:26.749+0000 7ff9a1bd3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:33:26.753 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-26T03:33:26.752+0000 7ff9a1bd3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:33:26.862 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:33:26.863 INFO:teuthology.orchestra.run.trial134.stderr:dumped all 2026-02-26T03:33:26.871 INFO:teuthology.orchestra.run.trial134.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2026-02-26T03:33:25.258700+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156499968,"kb_used":2540,"kb_used_data":2540,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156497428,"statfs":{"total":160255967232,"available":160253366272,"internally_reserved":0,"allocated":2600960,"data_stored":2600960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.187380"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:33:24.093170+0000","last_change":"2026-02-26T03:33:24.095792+0000","last_active":"2026-02-26T03:33:24.093170+0000","last_peered":"2026-02-26T03:33:24.093170+0000","last_clean":"2026-02-26T03:33:24.093170+0000","last_became_active":"2026-02-26T03:33:22.110938+0000","last_became_peered":"2026-02-26T03:33:22.110938+0000","last_unstale":"2026-02-26T03:33:24.093170+0000","last_undegraded":"2026-02-26T03:33:24.093170+0000","last_fullsized":"2026-02-26T03:33:24.093170+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:33:21.070421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:33:21.070421+0000","last_clean_scrub_stamp":"2026-02-26T03:33:21.070421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:33:24.094602+0000","last_change":"2026-02-26T03:33:24.096392+0000","last_active":"2026-02-26T03:33:24.094602+0000","last_peered":"2026-02-26T03:33:24.094602+0000","last_clean":"2026-02-26T03:33:24.094602+0000","last_became_active":"2026-02-26T03:33:22.107588+0000","last_became_peered":"2026-02-26T03:33:22.107588+0000","last_unstale":"2026-02-26T03:33:24.094602+0000","last_undegraded":"2026-02-26T03:33:24.094602+0000","last_fullsized":"2026-02-26T03:33:24.094602+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:33:21.070421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:33:21.070421+0000","last_clean_scrub_stamp":"2026-02-26T03:33:21.070421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:33:24.094673+0000","last_change":"2026-02-26T03:33:24.097778+0000","last_active":"2026-02-26T03:33:24.094673+0000","last_peered":"2026-02-26T03:33:24.094673+0000","last_clean":"2026-02-26T03:33:24.094673+0000","last_became_active":"2026-02-26T03:33:22.108984+0000","last_became_peered":"2026-02-26T03:33:22.108984+0000","last_unstale":"2026-02-26T03:33:24.094673+0000","last_undegraded":"2026-02-26T03:33:24.094673+0000","last_fullsized":"2026-02-26T03:33:24.094673+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:33:21.070421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:33:21.070421+0000","last_clean_scrub_stamp":"2026-02-26T03:33:21.070421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:33:24.094593+0000","last_change":"2026-02-26T03:33:24.096485+0000","last_active":"2026-02-26T03:33:24.094593+0000","last_peered":"2026-02-26T03:33:24.094593+0000","last_clean":"2026-02-26T03:33:24.094593+0000","last_became_active":"2026-02-26T03:33:22.110411+0000","last_became_peered":"2026-02-26T03:33:22.110411+0000","last_unstale":"2026-02-26T03:33:24.094593+0000","last_undegraded":"2026-02-26T03:33:24.094593+0000","last_fullsized":"2026-02-26T03:33:24.094593+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:33:21.070421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:33:21.070421+0000","last_clean_scrub_stamp":"2026-02-26T03:33:21.070421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'1","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:33:24.094742+0000","last_change":"2026-02-26T03:33:24.094742+0000","last_active":"2026-02-26T03:33:24.094742+0000","last_peered":"2026-02-26T03:33:24.094742+0000","last_clean":"2026-02-26T03:33:24.094742+0000","last_became_active":"2026-02-26T03:33:22.121618+0000","last_became_peered":"2026-02-26T03:33:22.121618+0000","last_unstale":"2026-02-26T03:33:24.094742+0000","last_undegraded":"2026-02-26T03:33:24.094742+0000","last_fullsized":"2026-02-26T03:33:24.094742+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:33:21.070421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:33:21.070421+0000","last_clean_scrub_stamp":"2026-02-26T03:33:21.070421+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:33:24.092886+0000","last_change":"2026-02-26T03:33:24.094840+0000","last_active":"2026-02-26T03:33:24.092886+0000","last_peered":"2026-02-26T03:33:24.092886+0000","last_clean":"2026-02-26T03:33:24.092886+0000","last_became_active":"2026-02-26T03:33:22.118089+0000","last_became_peered":"2026-02-26T03:33:22.118089+0000","last_unstale":"2026-02-26T03:33:24.092886+0000","last_undegraded":"2026-02-26T03:33:24.092886+0000","last_fullsized":"2026-02-26T03:33:24.092886+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:33:21.070421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:33:21.070421+0000","last_clean_scrub_stamp":"2026-02-26T03:33:21.070421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:33:24.091706+0000","last_change":"2026-02-26T03:33:24.094034+0000","last_active":"2026-02-26T03:33:24.091706+0000","last_peered":"2026-02-26T03:33:24.091706+0000","last_clean":"2026-02-26T03:33:24.091706+0000","last_became_active":"2026-02-26T03:33:22.113927+0000","last_became_peered":"2026-02-26T03:33:22.113927+0000","last_unstale":"2026-02-26T03:33:24.091706+0000","last_undegraded":"2026-02-26T03:33:24.091706+0000","last_fullsized":"2026-02-26T03:33:24.091706+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:33:21.070421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:33:21.070421+0000","last_clean_scrub_stamp":"2026-02-26T03:33:21.070421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":16,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:33:24.094576+0000","last_change":"2026-02-26T03:33:24.097770+0000","last_active":"2026-02-26T03:33:24.094576+0000","last_peered":"2026-02-26T03:33:24.094576+0000","last_clean":"2026-02-26T03:33:24.094576+0000","last_became_active":"2026-02-26T03:33:22.110305+0000","last_became_peered":"2026-02-26T03:33:22.110305+0000","last_unstale":"2026-02-26T03:33:24.094576+0000","last_undegraded":"2026-02-26T03:33:24.094576+0000","last_fullsized":"2026-02-26T03:33:24.094576+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:33:21.070421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:33:21.070421+0000","last_clean_scrub_stamp":"2026-02-26T03:33:21.070421+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":90,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:33:24.093885+0000","last_change":"2026-02-26T03:33:18.767167+0000","last_active":"2026-02-26T03:33:24.093885+0000","last_peered":"2026-02-26T03:33:24.093885+0000","last_clean":"2026-02-26T03:33:24.093885+0000","last_became_active":"2026-02-26T03:33:18.760057+0000","last_became_peered":"2026-02-26T03:33:18.760057+0000","last_unstale":"2026-02-26T03:33:24.093885+0000","last_undegraded":"2026-02-26T03:33:24.093885+0000","last_fullsized":"2026-02-26T03:33:24.093885+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:33:17.728653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:33:17.728653+0000","last_clean_scrub_stamp":"2026-02-26T03:33:17.728653+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160255967232,"available":160253366272,"internally_reserved":0,"allocated":2600960,"data_stored":2600960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53418655744,"available":53417480192,"internally_reserved":0,"allocated":1175552,"data_stored":1175552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":316,"kb_used_data":316,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52166340,"statfs":{"total":53418655744,"available":53418332160,"internally_reserved":0,"allocated":323584,"data_stored":323584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1076,"kb_used_data":1076,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165580,"statfs":{"total":53418655744,"available":53417553920,"internally_reserved":0,"allocated":1101824,"data_stored":1101824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1148,"kb_used_data":1148,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165508,"statfs":{"total":53418655744,"available":53417480192,"internally_reserved":0,"allocated":1175552,"data_stored":1175552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53418655744,"available":53417480192,"internally_reserved":0,"allocated":1175552,"data_stored":1175552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53418655744,"available":53417553920,"internally_reserved":0,"allocated":1101824,"data_stored":1101824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53418655744,"available":53417480192,"internally_reserved":0,"allocated":1175552,"data_stored":1175552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53418655744,"available":53418332160,"internally_reserved":0,"allocated":323584,"data_stored":323584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-26T03:33:26.873 INFO:tasks.ceph.ceph_manager.ceph:clean! 2026-02-26T03:33:26.873 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2026-02-26T03:33:26.873 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2026-02-26T03:33:26.873 DEBUG:teuthology.orchestra.run.trial134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-26T03:33:26.949 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-26T03:33:26.947+0000 7fc37a6cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:33:26.950 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-26T03:33:26.949+0000 7fc37a6cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:33:27.086 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:33:27.086 INFO:teuthology.orchestra.run.trial134.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2026-02-26T03:33:27.096 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2026-02-26T03:33:27.096 INFO:teuthology.run_tasks:Running task ssh_keys... 2026-02-26T03:33:27.216 DEBUG:teuthology.orchestra.run.trial134:> echo 'StrictHostKeyChecking no 2026-02-26T03:33:27.216 DEBUG:teuthology.orchestra.run.trial134:> ' > /home/ubuntu/.ssh/config && echo 'UserKnownHostsFile ' /dev/null >> /home/ubuntu/.ssh/config && chmod 600 /home/ubuntu/.ssh/config 2026-02-26T03:33:27.234 INFO:teuthology.task.ssh_keys:generated public key AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ8xphHPFRAl/RhZOhN9ud5RUBRwL/g2gGBbv/xCr0HRVqju85ZVYQHQM5H0+rmMAFWh9ylW3XQbz/bkAulpoBIjayCDYNL7lddFmL9Oa8OhNPzQwWBvU981BRS8ViY++vv8k4t+mwyt8pmxG4TwCrkHR5BDX9+igu9e/w2WcK+D3covD1Df/DrZNUpsch56X9i+WkamtQEDzbTZEvcBtu/QwRGgFcUSUhKVPuEn3iWAkGsnYLX+WGpKSph1oVC8/610IC6YabPsZvof1Kx0g1tPmVsbZlWbKGIDFFZjPZq+E9UM8xRM0JApDWFF285WwO8liYgqJDu05plIImqgvr 2026-02-26T03:33:27.234 INFO:teuthology.task.ssh_keys:pushing keys to trial134.front.sepia.ceph.com for ubuntu 2026-02-26T03:33:27.234 DEBUG:teuthology.orchestra.run.trial134:> rm -f -- /home/ubuntu/.ssh/id_rsa 2026-02-26T03:33:27.290 DEBUG:teuthology.orchestra.run.trial134:> set -ex 2026-02-26T03:33:27.290 DEBUG:teuthology.orchestra.run.trial134:> dd of=/home/ubuntu/.ssh/id_rsa 2026-02-26T03:33:27.290 DEBUG:teuthology.orchestra.run.trial134:> chmod 0500 /home/ubuntu/.ssh/id_rsa 2026-02-26T03:33:27.346 DEBUG:teuthology.orchestra.run.trial134:> rm -f -- /home/ubuntu/.ssh/id_rsa.pub 2026-02-26T03:33:27.403 DEBUG:teuthology.orchestra.run.trial134:> set -ex 2026-02-26T03:33:27.403 DEBUG:teuthology.orchestra.run.trial134:> dd of=/home/ubuntu/.ssh/id_rsa.pub 2026-02-26T03:33:27.459 DEBUG:teuthology.orchestra.run.trial134:> cp -v -a /home/ubuntu/.ssh/authorized_keys /home/ubuntu/.ssh/authorized_keys_2026-02-26_03:33:27:459389 2026-02-26T03:33:27.517 INFO:teuthology.orchestra.run.trial134.stdout:'/home/ubuntu/.ssh/authorized_keys' -> '/home/ubuntu/.ssh/authorized_keys_2026-02-26_03:33:27:459389' 2026-02-26T03:33:27.518 DEBUG:teuthology.orchestra.run.trial134:> set -ex 2026-02-26T03:33:27.518 DEBUG:teuthology.orchestra.run.trial134:> dd of=/home/ubuntu/.ssh/authorized_keys conv=notrunc oflag=append 2026-02-26T03:33:27.574 INFO:teuthology.run_tasks:Running task cbt... 2026-02-26T03:33:27.598 INFO:tasks.cbt:cbt configuration is {'cluster': {'user': 'ubuntu', 'head': 'ubuntu@trial134.front.sepia.ceph.com', 'osds': ['ubuntu@trial134.front.sepia.ceph.com'], 'mons': ['ubuntu@trial134.front.sepia.ceph.com'], 'clients': ['ubuntu@trial134.front.sepia.ceph.com'], 'rgws': {'ubuntu@trial134.front.sepia.ceph.com': None}, 'osds_per_node': 3, 'rebuild_every_test': False, 'use_existing': True, 'is_teuthology': True, 'iterations': 1, 'tmp_dir': '/tmp/cbt', 'pool_profiles': {'replicated': {'pg_size': 256, 'pgp_size': 256, 'replication': 'replicated'}}, 'pid_dir': '/var/run/ceph'}, 'benchmarks': {'radosbench': {'concurrent_ops': 4, 'concurrent_procs': 2, 'op_size': [4096], 'pool_monitoring_list': ['collectl'], 'pool_profile': 'replicated', 'readmode': 'rand', 'run_monitoring_list': ['collectl'], 'time': 60, 'write_only': False}}, 'client_endpoints': None, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}} 2026-02-26T03:33:27.599 DEBUG:teuthology.orchestra.run.trial134:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/cbt 2026-02-26T03:33:27.634 DEBUG:teuthology.orchestra.run.trial134:> set -ex 2026-02-26T03:33:27.634 DEBUG:teuthology.orchestra.run.trial134:> dd of=/home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2026-02-26T03:33:27.686 DEBUG:teuthology.orchestra.run.trial134:> git clone --depth 1 -b master https://github.com/ceph/cbt.git /home/ubuntu/cephtest/cbt 2026-02-26T03:33:27.748 INFO:teuthology.orchestra.run.trial134.stderr:Cloning into '/home/ubuntu/cephtest/cbt'... 2026-02-26T03:33:27.983 INFO:tasks.cbt:Installing collectl 2026-02-26T03:33:27.984 DEBUG:teuthology.orchestra.run.trial134:> sudo mkdir collectl && cd collectl && sudo wget https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download -O collectl.tar.gz && sudo tar -xvf collectl.tar.gz && cd collectl-4.3.1 && sudo ./INSTALL 2026-02-26T03:33:28.058 INFO:teuthology.orchestra.run.trial134.stderr:--2026-02-26 03:33:28-- https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download 2026-02-26T03:33:28.111 INFO:teuthology.orchestra.run.trial134.stderr:Resolving sourceforge.net (sourceforge.net)... 104.18.12.149, 104.18.13.149, 2606:4700::6812:d95, ... 2026-02-26T03:33:28.115 INFO:teuthology.orchestra.run.trial134.stderr:Connecting to sourceforge.net (sourceforge.net)|104.18.12.149|:443... connected. 2026-02-26T03:33:28.319 INFO:teuthology.orchestra.run.trial134.stderr:HTTP request sent, awaiting response... 302 Found 2026-02-26T03:33:28.320 INFO:teuthology.orchestra.run.trial134.stderr:Location: https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABpn78IkYYS_450WfdlcImXu5AguRetiuWyIbzyeQf6zadRZumkXZPpGLn-iEMXljEaH72U2TY5e5c8s5ay8GV4Bj-m6A%3D%3D&use_mirror=gigenet&r= [following] 2026-02-26T03:33:28.320 INFO:teuthology.orchestra.run.trial134.stderr:--2026-02-26 03:33:28-- https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABpn78IkYYS_450WfdlcImXu5AguRetiuWyIbzyeQf6zadRZumkXZPpGLn-iEMXljEaH72U2TY5e5c8s5ay8GV4Bj-m6A%3D%3D&use_mirror=gigenet&r= 2026-02-26T03:33:28.359 INFO:teuthology.orchestra.run.trial134.stderr:Resolving downloads.sourceforge.net (downloads.sourceforge.net)... 104.18.13.149, 104.18.12.149, 2606:4700::6812:d95, ... 2026-02-26T03:33:28.362 INFO:teuthology.orchestra.run.trial134.stderr:Connecting to downloads.sourceforge.net (downloads.sourceforge.net)|104.18.13.149|:443... connected. 2026-02-26T03:33:28.565 INFO:teuthology.orchestra.run.trial134.stderr:HTTP request sent, awaiting response... 302 Found 2026-02-26T03:33:28.566 INFO:teuthology.orchestra.run.trial134.stderr:Location: https://gigenet.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?viasf=1 [following] 2026-02-26T03:33:28.566 INFO:teuthology.orchestra.run.trial134.stderr:--2026-02-26 03:33:28-- https://gigenet.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?viasf=1 2026-02-26T03:33:28.572 INFO:teuthology.orchestra.run.trial134.stderr:Resolving gigenet.dl.sourceforge.net (gigenet.dl.sourceforge.net)... 69.65.16.141 2026-02-26T03:33:28.592 INFO:teuthology.orchestra.run.trial134.stderr:Connecting to gigenet.dl.sourceforge.net (gigenet.dl.sourceforge.net)|69.65.16.141|:443... connected. 2026-02-26T03:33:28.875 INFO:teuthology.orchestra.run.trial134.stderr:HTTP request sent, awaiting response... 200 OK 2026-02-26T03:33:28.875 INFO:teuthology.orchestra.run.trial134.stderr:Length: 620693 (606K) [application/x-gzip] 2026-02-26T03:33:28.875 INFO:teuthology.orchestra.run.trial134.stderr:Saving to: ‘collectl.tar.gz’ 2026-02-26T03:33:28.875 INFO:teuthology.orchestra.run.trial134.stderr: 2026-02-26T03:33:28.894 INFO:teuthology.orchestra.run.trial134.stderr: 0K .......... .......... .......... .......... .......... 8% 2.47M 0s 2026-02-26T03:33:28.915 INFO:teuthology.orchestra.run.trial134.stderr: 50K .......... .......... .......... .......... .......... 16% 2.56M 0s 2026-02-26T03:33:28.915 INFO:teuthology.orchestra.run.trial134.stderr: 100K .......... .......... .......... .......... .......... 24% 76.2M 0s 2026-02-26T03:33:28.915 INFO:teuthology.orchestra.run.trial134.stderr: 150K .......... .......... .......... .......... .......... 32% 76.6M 0s 2026-02-26T03:33:28.935 INFO:teuthology.orchestra.run.trial134.stderr: 200K .......... .......... .......... .......... .......... 41% 2.65M 0s 2026-02-26T03:33:28.935 INFO:teuthology.orchestra.run.trial134.stderr: 250K .......... .......... .......... .......... .......... 49% 97.5M 0s 2026-02-26T03:33:28.935 INFO:teuthology.orchestra.run.trial134.stderr: 300K .......... .......... .......... .......... .......... 57% 108M 0s 2026-02-26T03:33:28.936 INFO:teuthology.orchestra.run.trial134.stderr: 350K .......... .......... .......... .......... .......... 65% 87.7M 0s 2026-02-26T03:33:28.956 INFO:teuthology.orchestra.run.trial134.stderr: 400K .......... .......... .......... .......... .......... 74% 2.72M 0s 2026-02-26T03:33:28.956 INFO:teuthology.orchestra.run.trial134.stderr: 450K .......... .......... .......... .......... .......... 82% 96.8M 0s 2026-02-26T03:33:28.956 INFO:teuthology.orchestra.run.trial134.stderr: 500K .......... .......... .......... .......... .......... 90% 133M 0s 2026-02-26T03:33:28.956 INFO:teuthology.orchestra.run.trial134.stderr: 550K .......... .......... .......... .......... .......... 98% 109M 0s 2026-02-26T03:33:28.956 INFO:teuthology.orchestra.run.trial134.stderr: 600K ...... 100% 11.4T=0.08s 2026-02-26T03:33:28.956 INFO:teuthology.orchestra.run.trial134.stderr: 2026-02-26T03:33:28.956 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-26 03:33:28 (7.46 MB/s) - ‘collectl.tar.gz’ saved [620693/620693] 2026-02-26T03:33:28.956 INFO:teuthology.orchestra.run.trial134.stderr: 2026-02-26T03:33:28.976 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/ 2026-02-26T03:33:28.976 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/collectl.conf 2026-02-26T03:33:28.976 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/docs/ 2026-02-26T03:33:28.976 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/docs/SlowProc.html 2026-02-26T03:33:28.976 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/docs/BuddyInfo.html 2026-02-26T03:33:28.976 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/docs/WhySummary.html 2026-02-26T03:33:28.976 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/docs/Socket.html 2026-02-26T03:33:28.977 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/docs/colmux.html 2026-02-26T03:33:28.977 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/docs/Tutorial.html 2026-02-26T03:33:28.977 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/docs/Gexpr.html 2026-02-26T03:33:28.977 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/docs/Import.html 2026-02-26T03:33:28.977 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/docs/style.css 2026-02-26T03:33:28.977 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/docs/Ganglia3.jpg 2026-02-26T03:33:28.977 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/docs/Ganglia2.jpg 2026-02-26T03:33:28.977 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/docs/FileNaming.html 2026-02-26T03:33:28.977 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/docs/TheMath.html 2026-02-26T03:33:28.977 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/docs/Environmental.html 2026-02-26T03:33:28.977 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/docs/Process.html 2026-02-26T03:33:28.977 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/docs/OutputFiles.html 2026-02-26T03:33:28.977 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/docs/HiResTime.html 2026-02-26T03:33:28.978 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/docs/OutputFormats.html 2026-02-26T03:33:28.978 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/docs/slub.jpg 2026-02-26T03:33:28.978 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/docs/Lustre.html 2026-02-26T03:33:28.978 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/docs/Documentation.html 2026-02-26T03:33:28.978 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/docs/Inodes.html 2026-02-26T03:33:28.978 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/docs/Graphite.html 2026-02-26T03:33:28.978 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/docs/InputFiles.html 2026-02-26T03:33:28.978 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/docs/Export.html 2026-02-26T03:33:28.978 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/docs/Interrupts.html 2026-02-26T03:33:28.978 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/docs/Startup.html 2026-02-26T03:33:28.978 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/docs/Performance.html 2026-02-26T03:33:28.978 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/docs/Messages.html 2026-02-26T03:33:28.978 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/docs/Logging.html 2026-02-26T03:33:28.979 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/docs/Data-detail.html 2026-02-26T03:33:28.979 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/docs/CPUs.html 2026-02-26T03:33:28.979 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/docs/Matrix.html 2026-02-26T03:33:28.979 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/docs/Data-brief.html 2026-02-26T03:33:28.979 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/docs/SlabInfo.html 2026-02-26T03:33:28.979 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/docs/Misc.html 2026-02-26T03:33:28.979 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/docs/Examples.html 2026-02-26T03:33:28.979 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/docs/OpenStack.html 2026-02-26T03:33:28.979 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/docs/Plotfiles.html 2026-02-26T03:33:28.979 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/docs/ColmuxCPU.jpg 2026-02-26T03:33:28.979 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/docs/Tutorial-Lustre.html 2026-02-26T03:33:28.979 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/docs/Data-verbose.html 2026-02-26T03:33:28.979 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/docs/Data.html 2026-02-26T03:33:28.979 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/docs/Exceptions.html 2026-02-26T03:33:28.980 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/docs/RunningAsAService.html 2026-02-26T03:33:28.980 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/docs/FAQ-collectl.html 2026-02-26T03:33:28.980 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/docs/Ganglia.jpg 2026-02-26T03:33:28.980 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/docs/OperationalModes.html 2026-02-26T03:33:28.980 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/docs/Playback.html 2026-02-26T03:33:28.980 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/docs/Hello.html 2026-02-26T03:33:28.980 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/docs/Sockets.html 2026-02-26T03:33:28.980 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/docs/Network.html 2026-02-26T03:33:28.980 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/docs/Infiniband.html 2026-02-26T03:33:28.980 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/docs/Memory.html 2026-02-26T03:33:28.980 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/docs/ColmuxLustre.jpg 2026-02-26T03:33:28.980 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/docs/Disks.html 2026-02-26T03:33:28.980 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/docs/NfsInfo.html 2026-02-26T03:33:28.981 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/service/ 2026-02-26T03:33:28.981 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/service/collectl.service 2026-02-26T03:33:28.981 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/initd/ 2026-02-26T03:33:28.981 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/initd/collectl-suse 2026-02-26T03:33:28.981 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/initd/collectl-debian 2026-02-26T03:33:28.981 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/initd/collectl-generic 2026-02-26T03:33:28.981 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/initd/collectl 2026-02-26T03:33:28.981 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/misc.ph 2026-02-26T03:33:28.981 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/gexpr.ph 2026-02-26T03:33:28.981 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/statsd.ph 2026-02-26T03:33:28.981 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/envrules.std 2026-02-26T03:33:28.981 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/hello.ph 2026-02-26T03:33:28.981 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/vmsum.ph 2026-02-26T03:33:28.981 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/proctree.ph 2026-02-26T03:33:28.982 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/README-WINDOWS 2026-02-26T03:33:28.982 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/formatit.ph 2026-02-26T03:33:28.982 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/colmux 2026-02-26T03:33:28.982 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/INSTALL 2026-02-26T03:33:28.982 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/graphite.ph 2026-02-26T03:33:28.982 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/RELEASE-collectl 2026-02-26T03:33:28.982 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/vmstat.ph 2026-02-26T03:33:28.982 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/README 2026-02-26T03:33:28.982 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/man1/ 2026-02-26T03:33:28.982 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/man1/collectl.1 2026-02-26T03:33:28.982 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/man1/colmux.1 2026-02-26T03:33:28.982 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/lexpr.ph 2026-02-26T03:33:28.982 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/client.pl 2026-02-26T03:33:28.983 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/collectl 2026-02-26T03:33:28.983 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/COPYING 2026-02-26T03:33:28.983 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/vnet.ph 2026-02-26T03:33:28.983 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/GPL 2026-02-26T03:33:28.983 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/UNINSTALL 2026-02-26T03:33:28.983 INFO:teuthology.orchestra.run.trial134.stdout:collectl-4.3.1/ARTISTIC 2026-02-26T03:33:29.018 INFO:teuthology.orchestra.run.trial134.stderr:./INSTALL: line 90: chkconfig: command not found 2026-02-26T03:33:29.020 DEBUG:teuthology.orchestra.run.trial134:> sudo yum -y install librbd-devel pdsh pdsh-rcmd-ssh perf 2026-02-26T03:33:29.379 INFO:teuthology.orchestra.run.trial134.stdout:Last metadata expiration check: 0:00:58 ago on Thu 26 Feb 2026 03:32:31 AM UTC. 2026-02-26T03:33:29.466 INFO:teuthology.orchestra.run.trial134.stdout:Dependencies resolved. 2026-02-26T03:33:29.467 INFO:teuthology.orchestra.run.trial134.stdout:================================================================================ 2026-02-26T03:33:29.467 INFO:teuthology.orchestra.run.trial134.stdout: Package Arch Version Repository Size 2026-02-26T03:33:29.467 INFO:teuthology.orchestra.run.trial134.stdout:================================================================================ 2026-02-26T03:33:29.467 INFO:teuthology.orchestra.run.trial134.stdout:Installing: 2026-02-26T03:33:29.467 INFO:teuthology.orchestra.run.trial134.stdout: librbd-devel x86_64 2:20.3.0-5312.gabffb617.el9 ceph 24 k 2026-02-26T03:33:29.467 INFO:teuthology.orchestra.run.trial134.stdout: pdsh x86_64 2.34-7.el9 epel 118 k 2026-02-26T03:33:29.467 INFO:teuthology.orchestra.run.trial134.stdout: pdsh-rcmd-ssh x86_64 2.34-7.el9 epel 13 k 2026-02-26T03:33:29.467 INFO:teuthology.orchestra.run.trial134.stdout: perf x86_64 5.14.0-686.el9 appstream 3.3 M 2026-02-26T03:33:29.467 INFO:teuthology.orchestra.run.trial134.stdout:Installing dependencies: 2026-02-26T03:33:29.468 INFO:teuthology.orchestra.run.trial134.stdout: libradospp-devel x86_64 2:20.3.0-5312.gabffb617.el9 ceph 31 k 2026-02-26T03:33:29.468 INFO:teuthology.orchestra.run.trial134.stdout: libtraceevent x86_64 1.8.4-2.el9 baseos 290 k 2026-02-26T03:33:29.468 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:33:29.468 INFO:teuthology.orchestra.run.trial134.stdout:Transaction Summary 2026-02-26T03:33:29.468 INFO:teuthology.orchestra.run.trial134.stdout:================================================================================ 2026-02-26T03:33:29.468 INFO:teuthology.orchestra.run.trial134.stdout:Install 6 Packages 2026-02-26T03:33:29.468 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:33:29.468 INFO:teuthology.orchestra.run.trial134.stdout:Total download size: 3.8 M 2026-02-26T03:33:29.468 INFO:teuthology.orchestra.run.trial134.stdout:Installed size: 14 M 2026-02-26T03:33:29.468 INFO:teuthology.orchestra.run.trial134.stdout:Downloading Packages: 2026-02-26T03:33:30.038 INFO:teuthology.orchestra.run.trial134.stdout:(1/6): libradospp-devel-20.3.0-5312.gabffb617.e 88 kB/s | 31 kB 00:00 2026-02-26T03:33:30.040 INFO:teuthology.orchestra.run.trial134.stdout:(2/6): librbd-devel-20.3.0-5312.gabffb617.el9.x 67 kB/s | 24 kB 00:00 2026-02-26T03:33:30.207 INFO:teuthology.orchestra.run.trial134.stdout:(3/6): libtraceevent-1.8.4-2.el9.x86_64.rpm 554 kB/s | 290 kB 00:00 2026-02-26T03:33:30.215 INFO:teuthology.orchestra.run.trial134.stdout:(4/6): pdsh-2.34-7.el9.x86_64.rpm 675 kB/s | 118 kB 00:00 2026-02-26T03:33:30.272 INFO:teuthology.orchestra.run.trial134.stdout:(5/6): pdsh-rcmd-ssh-2.34-7.el9.x86_64.rpm 201 kB/s | 13 kB 00:00 2026-02-26T03:33:31.137 INFO:teuthology.orchestra.run.trial134.stdout:(6/6): perf-5.14.0-686.el9.x86_64.rpm 3.0 MB/s | 3.3 MB 00:01 2026-02-26T03:33:31.138 INFO:teuthology.orchestra.run.trial134.stdout:-------------------------------------------------------------------------------- 2026-02-26T03:33:31.139 INFO:teuthology.orchestra.run.trial134.stdout:Total 2.3 MB/s | 3.8 MB 00:01 2026-02-26T03:33:31.163 INFO:teuthology.orchestra.run.trial134.stdout:Running transaction check 2026-02-26T03:33:31.173 INFO:teuthology.orchestra.run.trial134.stdout:Transaction check succeeded. 2026-02-26T03:33:31.173 INFO:teuthology.orchestra.run.trial134.stdout:Running transaction test 2026-02-26T03:33:31.231 INFO:teuthology.orchestra.run.trial134.stdout:Transaction test succeeded. 2026-02-26T03:33:31.232 INFO:teuthology.orchestra.run.trial134.stdout:Running transaction 2026-02-26T03:33:31.307 INFO:teuthology.orchestra.run.trial134.stdout: Preparing : 1/1 2026-02-26T03:33:31.312 INFO:teuthology.orchestra.run.trial134.stdout: Installing : pdsh-rcmd-ssh-2.34-7.el9.x86_64 1/6 2026-02-26T03:33:31.352 INFO:teuthology.orchestra.run.trial134.stdout: Installing : pdsh-2.34-7.el9.x86_64 2/6 2026-02-26T03:33:31.357 INFO:teuthology.orchestra.run.trial134.stdout: Installing : libtraceevent-1.8.4-2.el9.x86_64 3/6 2026-02-26T03:33:31.359 INFO:teuthology.orchestra.run.trial134.stdout: Installing : libradospp-devel-2:20.3.0-5312.gabffb617.el9.x86_64 4/6 2026-02-26T03:33:31.522 INFO:teuthology.orchestra.run.trial134.stdout: Installing : librbd-devel-2:20.3.0-5312.gabffb617.el9.x86_64 5/6 2026-02-26T03:33:31.544 INFO:teuthology.orchestra.run.trial134.stdout: Installing : perf-5.14.0-686.el9.x86_64 6/6 2026-02-26T03:33:31.780 INFO:teuthology.orchestra.run.trial134.stdout: Running scriptlet: perf-5.14.0-686.el9.x86_64 6/6 2026-02-26T03:33:31.780 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : libradospp-devel-2:20.3.0-5312.gabffb617.el9.x86_64 1/6 2026-02-26T03:33:31.781 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : librbd-devel-2:20.3.0-5312.gabffb617.el9.x86_64 2/6 2026-02-26T03:33:31.781 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : libtraceevent-1.8.4-2.el9.x86_64 3/6 2026-02-26T03:33:31.781 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : perf-5.14.0-686.el9.x86_64 4/6 2026-02-26T03:33:31.781 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : pdsh-2.34-7.el9.x86_64 5/6 2026-02-26T03:33:31.847 INFO:teuthology.orchestra.run.trial134.stdout: Verifying : pdsh-rcmd-ssh-2.34-7.el9.x86_64 6/6 2026-02-26T03:33:31.847 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:33:31.847 INFO:teuthology.orchestra.run.trial134.stdout:Installed: 2026-02-26T03:33:31.848 INFO:teuthology.orchestra.run.trial134.stdout: libradospp-devel-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:33:31.848 INFO:teuthology.orchestra.run.trial134.stdout: librbd-devel-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:33:31.848 INFO:teuthology.orchestra.run.trial134.stdout: libtraceevent-1.8.4-2.el9.x86_64 2026-02-26T03:33:31.848 INFO:teuthology.orchestra.run.trial134.stdout: pdsh-2.34-7.el9.x86_64 2026-02-26T03:33:31.848 INFO:teuthology.orchestra.run.trial134.stdout: pdsh-rcmd-ssh-2.34-7.el9.x86_64 2026-02-26T03:33:31.848 INFO:teuthology.orchestra.run.trial134.stdout: perf-5.14.0-686.el9.x86_64 2026-02-26T03:33:31.848 INFO:teuthology.orchestra.run.trial134.stdout: 2026-02-26T03:33:31.848 INFO:teuthology.orchestra.run.trial134.stdout:Complete! 2026-02-26T03:33:31.879 DEBUG:teuthology.orchestra.run.trial134:> python3 -m pip install -r /home/ubuntu/cephtest/cbt/requirements.txt 2026-02-26T03:33:32.017 INFO:teuthology.orchestra.run.trial134.stdout:Defaulting to user installation because normal site-packages is not writeable 2026-02-26T03:33:32.028 INFO:teuthology.orchestra.run.trial134.stdout:Requirement already satisfied: pyyaml in /usr/lib64/python3.9/site-packages (from -r /home/ubuntu/cephtest/cbt/requirements.txt (line 1)) (5.4.1) 2026-02-26T03:33:32.028 INFO:teuthology.orchestra.run.trial134.stdout:Requirement already satisfied: lxml in /usr/lib64/python3.9/site-packages (from -r /home/ubuntu/cephtest/cbt/requirements.txt (line 2)) (4.6.5) 2026-02-26T03:33:32.359 INFO:teuthology.orchestra.run.trial134.stdout:Collecting matplotlib 2026-02-26T03:33:32.380 INFO:teuthology.orchestra.run.trial134.stdout: Downloading matplotlib-3.9.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (8.3 MB) 2026-02-26T03:33:32.792 INFO:teuthology.orchestra.run.trial134.stdout:Collecting fonttools>=4.22.0 2026-02-26T03:33:32.801 INFO:teuthology.orchestra.run.trial134.stdout: Downloading fonttools-4.60.2-cp39-cp39-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (4.8 MB) 2026-02-26T03:33:33.258 INFO:teuthology.orchestra.run.trial134.stdout:Collecting pillow>=8 2026-02-26T03:33:33.270 INFO:teuthology.orchestra.run.trial134.stdout: Downloading pillow-11.3.0-cp39-cp39-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (6.6 MB) 2026-02-26T03:33:33.311 INFO:teuthology.orchestra.run.trial134.stdout:Requirement already satisfied: python-dateutil>=2.7 in /usr/lib/python3.9/site-packages (from matplotlib->-r /home/ubuntu/cephtest/cbt/requirements.txt (line 3)) (2.9.0.post0) 2026-02-26T03:33:33.361 INFO:teuthology.orchestra.run.trial134.stdout:Collecting importlib-resources>=3.2.0 2026-02-26T03:33:33.365 INFO:teuthology.orchestra.run.trial134.stdout: Downloading importlib_resources-6.5.2-py3-none-any.whl (37 kB) 2026-02-26T03:33:33.507 INFO:teuthology.orchestra.run.trial134.stdout:Collecting kiwisolver>=1.3.1 2026-02-26T03:33:33.514 INFO:teuthology.orchestra.run.trial134.stdout: Downloading kiwisolver-1.4.7-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.6 MB) 2026-02-26T03:33:33.523 INFO:teuthology.orchestra.run.trial134.stdout:Requirement already satisfied: packaging>=20.0 in /usr/lib/python3.9/site-packages (from matplotlib->-r /home/ubuntu/cephtest/cbt/requirements.txt (line 3)) (20.9) 2026-02-26T03:33:33.523 INFO:teuthology.orchestra.run.trial134.stdout:Requirement already satisfied: pyparsing>=2.3.1 in /usr/lib/python3.9/site-packages (from matplotlib->-r /home/ubuntu/cephtest/cbt/requirements.txt (line 3)) (2.4.7) 2026-02-26T03:33:33.523 INFO:teuthology.orchestra.run.trial134.stdout:Requirement already satisfied: numpy>=1.23 in /usr/lib64/python3.9/site-packages (from matplotlib->-r /home/ubuntu/cephtest/cbt/requirements.txt (line 3)) (1.23.5) 2026-02-26T03:33:33.696 INFO:teuthology.orchestra.run.trial134.stdout:Collecting contourpy>=1.0.1 2026-02-26T03:33:33.700 INFO:teuthology.orchestra.run.trial134.stdout: Downloading contourpy-1.3.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (321 kB) 2026-02-26T03:33:33.715 INFO:teuthology.orchestra.run.trial134.stdout:Collecting cycler>=0.10 2026-02-26T03:33:33.719 INFO:teuthology.orchestra.run.trial134.stdout: Downloading cycler-0.12.1-py3-none-any.whl (8.3 kB) 2026-02-26T03:33:33.769 INFO:teuthology.orchestra.run.trial134.stdout:Collecting zipp>=3.1.0 2026-02-26T03:33:33.773 INFO:teuthology.orchestra.run.trial134.stdout: Downloading zipp-3.23.0-py3-none-any.whl (10 kB) 2026-02-26T03:33:33.784 INFO:teuthology.orchestra.run.trial134.stdout:Requirement already satisfied: six>=1.5 in /usr/lib/python3.9/site-packages (from python-dateutil>=2.7->matplotlib->-r /home/ubuntu/cephtest/cbt/requirements.txt (line 3)) (1.15.0) 2026-02-26T03:33:33.898 INFO:teuthology.orchestra.run.trial134.stdout:Installing collected packages: zipp, pillow, kiwisolver, importlib-resources, fonttools, cycler, contourpy, matplotlib 2026-02-26T03:33:34.962 INFO:teuthology.orchestra.run.trial134.stdout:Successfully installed contourpy-1.3.0 cycler-0.12.1 fonttools-4.60.2 importlib-resources-6.5.2 kiwisolver-1.4.7 matplotlib-3.9.4 pillow-11.3.0 zipp-3.23.0 2026-02-26T03:33:35.071 INFO:tasks.cbt:benchmark: radosbench 2026-02-26T03:33:35.071 DEBUG:teuthology.orchestra.run.trial134:> sudo /sbin/sysctl -q -w kernel.perf_event_paranoid=0 2026-02-26T03:33:35.098 DEBUG:teuthology.orchestra.run.trial134:> /home/ubuntu/cephtest/cbt/cbt.py -a /home/ubuntu/cephtest/archive/cbt /home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2026-02-26T03:33:35.210 INFO:teuthology.orchestra.run.trial134.stderr:03:33:35 - DEBUG - cbt - Settings.cluster: 2026-02-26T03:33:35.210 INFO:teuthology.orchestra.run.trial134.stderr: {'archive_dir': '/home/ubuntu/cephtest/archive/cbt', 2026-02-26T03:33:35.210 INFO:teuthology.orchestra.run.trial134.stderr: 'clients': ['ubuntu@trial134.front.sepia.ceph.com'], 2026-02-26T03:33:35.210 INFO:teuthology.orchestra.run.trial134.stderr: 'conf_file': '/etc/ceph/ceph.conf', 2026-02-26T03:33:35.210 INFO:teuthology.orchestra.run.trial134.stderr: 'head': 'ubuntu@trial134.front.sepia.ceph.com', 2026-02-26T03:33:35.210 INFO:teuthology.orchestra.run.trial134.stderr: 'is_teuthology': True, 2026-02-26T03:33:35.210 INFO:teuthology.orchestra.run.trial134.stderr: 'iterations': 1, 2026-02-26T03:33:35.210 INFO:teuthology.orchestra.run.trial134.stderr: 'mons': ['ubuntu@trial134.front.sepia.ceph.com'], 2026-02-26T03:33:35.211 INFO:teuthology.orchestra.run.trial134.stderr: 'osds': ['ubuntu@trial134.front.sepia.ceph.com'], 2026-02-26T03:33:35.211 INFO:teuthology.orchestra.run.trial134.stderr: 'osds_per_node': 3, 2026-02-26T03:33:35.211 INFO:teuthology.orchestra.run.trial134.stderr: 'pid_dir': '/var/run/ceph', 2026-02-26T03:33:35.211 INFO:teuthology.orchestra.run.trial134.stderr: 'pool_profiles': {'replicated': {'pg_size': 256, 2026-02-26T03:33:35.211 INFO:teuthology.orchestra.run.trial134.stderr: 'pgp_size': 256, 2026-02-26T03:33:35.211 INFO:teuthology.orchestra.run.trial134.stderr: 'replication': 'replicated'}}, 2026-02-26T03:33:35.211 INFO:teuthology.orchestra.run.trial134.stderr: 'rebuild_every_test': False, 2026-02-26T03:33:35.211 INFO:teuthology.orchestra.run.trial134.stderr: 'rgws': {'ubuntu@trial134.front.sepia.ceph.com': None}, 2026-02-26T03:33:35.211 INFO:teuthology.orchestra.run.trial134.stderr: 'tmp_dir': '/tmp/cbt', 2026-02-26T03:33:35.211 INFO:teuthology.orchestra.run.trial134.stderr: 'use_existing': True, 2026-02-26T03:33:35.211 INFO:teuthology.orchestra.run.trial134.stderr: 'user': 'ubuntu'} 2026-02-26T03:33:35.211 INFO:teuthology.orchestra.run.trial134.stderr:03:33:35 - INFO - cbt - Results dir: /home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3 2026-02-26T03:33:35.211 INFO:teuthology.orchestra.run.trial134.stderr:03:33:35 - INFO - cbt - Pausing for 60s for idle monitoring. 2026-02-26T03:33:35.212 INFO:teuthology.orchestra.run.trial134.stderr:03:33:35 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial134.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/idle_monitoring/collectl 2026-02-26T03:33:35.375 INFO:teuthology.orchestra.run.trial134.stderr:03:33:35 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial134.front.sepia.ceph.com collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/idle_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2026-02-26T03:33:35.377 INFO:teuthology.orchestra.run.trial134.stderr:03:33:35 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial134.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/idle_monitoring/perf 2026-02-26T03:33:35.539 INFO:teuthology.orchestra.run.trial134.stderr:03:33:35 - DEBUG - cbt - PerfMonitoring: remote_node 2026-02-26T03:33:35.539 INFO:teuthology.orchestra.run.trial134.stderr:03:33:35 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial134.front.sepia.ceph.com for pid in `cat /var/run/ceph/crimson-osd.*.pid`; do perf stat -p ${pid} -o /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/idle_monitoring/perf/perf_stat.${pid} & done 2026-02-26T03:34:35.601 INFO:teuthology.orchestra.run.trial134.stderr:03:34:35 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial134.front.sepia.ceph.com pkill -SIGINT -f collectl 2026-02-26T03:34:35.780 INFO:teuthology.orchestra.run.trial134.stderr:03:34:35 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial134.front.sepia.ceph.com sudo pkill -SIGINT -f perf\ 2026-02-26T03:34:35.961 INFO:teuthology.orchestra.run.trial134.stderr:03:34:35 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial134.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/idle_monitoring/perf/perf.data 2026-02-26T03:34:35.962 INFO:teuthology.orchestra.run.trial134.stderr:03:34:35 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial134.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/idle_monitoring/perf/perf_stat.* 2026-02-26T03:34:35.965 INFO:teuthology.orchestra.run.trial134.stderr:03:34:35 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial134.front.sepia.ceph.com sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/* 2026-02-26T03:34:36.138 INFO:teuthology.orchestra.run.trial134.stderr:03:34:36 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=rpdcp -f 10 -R ssh -w ubuntu@trial134.front.sepia.ceph.com -r /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/* /home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3 2026-02-26T03:34:36.300 INFO:teuthology.orchestra.run.trial134.stderr:03:34:36 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial134.front.sepia.ceph.com sudo killall -9 rados 2026-02-26T03:34:36.473 INFO:teuthology.orchestra.run.trial134.stderr:03:34:36 - INFO - cbt - Results dir: /home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3 2026-02-26T03:34:36.474 INFO:teuthology.orchestra.run.trial134.stderr:03:34:36 - INFO - cbt - Skipping existing test in /home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3. 2026-02-26T03:34:36.474 INFO:teuthology.orchestra.run.trial134.stderr:03:34:36 - INFO - cbt - Running benchmark /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004 2026-02-26T03:34:36.474 INFO:teuthology.orchestra.run.trial134.stderr:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3 2026-02-26T03:34:36.474 INFO:teuthology.orchestra.run.trial134.stderr:{'concurrent_ops': 4, 'concurrent_procs': 2, 'pool_profile': 'replicated', 'readmode': 'rand', 'time': 60, 'write_only': False, 'op_size': 4096, 'pool_monitoring_list': 'collectl', 'run_monitoring_list': 'collectl', 'benchmark': 'radosbench', 'iteration': 0} == iteration 0 == 2026-02-26T03:34:36.474 INFO:teuthology.orchestra.run.trial134.stderr:03:34:36 - INFO - cbt - Setting OSD Read Ahead to: 0 2026-02-26T03:34:36.474 INFO:teuthology.orchestra.run.trial134.stderr:03:34:36 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial134.front.sepia.ceph.com find /dev/disk/by-partlabel/osd-device-*data -exec readlink {} \; | cut -d"/" -f 3 | sed "s/[0-9]$//" | xargs -I{} sudo sh -c "echo 0 > /sys/block/'{}'/queue/read_ahead_kb" 2026-02-26T03:34:36.474 INFO:teuthology.orchestra.run.trial134.stderr:03:34:36 - DEBUG - cbt - Cleaning existing temporary run directory: /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004 2026-02-26T03:34:36.475 INFO:teuthology.orchestra.run.trial134.stderr:03:34:36 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial134.front.sepia.ceph.com sudo rm -rf /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004 2026-02-26T03:34:36.643 INFO:teuthology.orchestra.run.trial134.stderr:03:34:36 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial134.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/pool_monitoring/collectl 2026-02-26T03:34:36.806 INFO:teuthology.orchestra.run.trial134.stderr:03:34:36 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial134.front.sepia.ceph.com collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/pool_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2026-02-26T03:34:36.808 INFO:teuthology.orchestra.run.trial134.stderr:03:34:36 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial134.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/pool_monitoring/perf 2026-02-26T03:34:36.972 INFO:teuthology.orchestra.run.trial134.stderr:03:34:36 - DEBUG - cbt - PerfMonitoring: remote_node 2026-02-26T03:34:36.973 INFO:teuthology.orchestra.run.trial134.stderr:03:34:36 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial134.front.sepia.ceph.com for pid in `cat /var/run/ceph/crimson-osd.*.pid`; do perf stat -p ${pid} -o /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/pool_monitoring/perf/perf_stat.${pid} & done 2026-02-26T03:34:36.974 INFO:teuthology.orchestra.run.trial134.stderr:03:34:36 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial134.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool delete rados-bench-cbt rados-bench-cbt --yes-i-really-really-mean-it 2026-02-26T03:34:37.312 INFO:teuthology.orchestra.run.trial134.stderr:03:34:37 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial134.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool create rados-bench-cbt 256 256 2026-02-26T03:34:38.367 INFO:teuthology.orchestra.run.trial134.stderr:03:34:38 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial134.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool application enable rados-bench-cbt radosbench 2026-02-26T03:34:39.370 INFO:teuthology.orchestra.run.trial134.stderr:03:34:39 - INFO - cbt - Checking Health after pool creation. 2026-02-26T03:34:44.375 INFO:teuthology.orchestra.run.trial134.stderr:03:34:44 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial134.front.sepia.ceph.com /usr/bin/ceph -c /etc/ceph/ceph.conf health 2026-02-26T03:34:44.736 INFO:teuthology.orchestra.run.trial134.stderr:03:34:44 - INFO - cbt - Final Pool Health Check. 2026-02-26T03:34:49.741 INFO:teuthology.orchestra.run.trial134.stderr:03:34:49 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial134.front.sepia.ceph.com /usr/bin/ceph -c /etc/ceph/ceph.conf health 2026-02-26T03:34:50.100 INFO:teuthology.orchestra.run.trial134.stderr:03:34:50 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial134.front.sepia.ceph.com pkill -SIGINT -f collectl 2026-02-26T03:34:50.275 INFO:teuthology.orchestra.run.trial134.stderr:03:34:50 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial134.front.sepia.ceph.com sudo pkill -SIGINT -f perf\ 2026-02-26T03:34:50.462 INFO:teuthology.orchestra.run.trial134.stderr:03:34:50 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial134.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/pool_monitoring/perf/perf.data 2026-02-26T03:34:50.464 INFO:teuthology.orchestra.run.trial134.stderr:03:34:50 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial134.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/pool_monitoring/perf/perf_stat.* 2026-02-26T03:34:50.466 INFO:teuthology.orchestra.run.trial134.stderr:03:34:50 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial134.front.sepia.ceph.com sync 2026-02-26T03:34:50.632 INFO:teuthology.orchestra.run.trial134.stderr:03:34:50 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial134.front.sepia.ceph.com echo 3 | sudo tee /proc/sys/vm/drop_caches 2026-02-26T03:34:51.671 INFO:teuthology.orchestra.run.trial134.stderr:03:34:51 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial134.front.sepia.ceph.com /usr/bin/rados -c /etc/ceph/ceph.conf -v 2026-02-26T03:34:51.893 INFO:teuthology.orchestra.run.trial134.stderr:03:34:51 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial134.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/write 2026-02-26T03:34:52.052 INFO:teuthology.orchestra.run.trial134.stderr:03:34:52 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial134.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf daemon osd.0 config show > /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/write/ceph_settings.out 2026-02-26T03:34:52.339 INFO:teuthology.orchestra.run.trial134.stderr:03:34:52 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial134.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/write/collectl 2026-02-26T03:34:52.499 INFO:teuthology.orchestra.run.trial134.stderr:03:34:52 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial134.front.sepia.ceph.com collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/write/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2026-02-26T03:34:52.501 INFO:teuthology.orchestra.run.trial134.stderr:03:34:52 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial134.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/write/perf 2026-02-26T03:34:52.671 INFO:teuthology.orchestra.run.trial134.stderr:03:34:52 - DEBUG - cbt - PerfMonitoring: remote_node 2026-02-26T03:34:52.671 INFO:teuthology.orchestra.run.trial134.stderr:03:34:52 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial134.front.sepia.ceph.com for pid in `cat /var/run/ceph/crimson-osd.*.pid`; do perf stat -p ${pid} -o /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/write/perf/perf_stat.${pid} & done 2026-02-26T03:34:52.673 INFO:teuthology.orchestra.run.trial134.stderr:03:34:52 - INFO - cbt - Running radosbench write test. 2026-02-26T03:34:52.673 INFO:teuthology.orchestra.run.trial134.stderr:03:34:52 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial134.front.sepia.ceph.com /usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench -b 4096 60 write --concurrent-ios 4 --run-name `hostname -f`-0 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/write/objecter.0.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/write/output.0 2026-02-26T03:34:52.674 INFO:teuthology.orchestra.run.trial134.stderr:03:34:52 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial134.front.sepia.ceph.com /usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench -b 4096 60 write --concurrent-ios 4 --run-name `hostname -f`-1 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/write/objecter.1.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/write/output.1 2026-02-26T03:35:53.897 INFO:teuthology.orchestra.run.trial134.stderr:03:35:53 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial134.front.sepia.ceph.com pkill -SIGINT -f collectl 2026-02-26T03:35:54.103 INFO:teuthology.orchestra.run.trial134.stderr:03:35:54 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial134.front.sepia.ceph.com sudo pkill -SIGINT -f perf\ 2026-02-26T03:35:54.285 INFO:teuthology.orchestra.run.trial134.stderr:03:35:54 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial134.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/write/perf/perf.data 2026-02-26T03:35:54.286 INFO:teuthology.orchestra.run.trial134.stderr:03:35:54 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial134.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/write/perf/perf_stat.* 2026-02-26T03:35:54.288 INFO:teuthology.orchestra.run.trial134.stderr:03:35:54 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial134.front.sepia.ceph.com find /var/run/ceph/ceph-osd*.asok -maxdepth 1 -exec sudo /usr/bin/ceph --admin-daemon {} dump_historic_ops \; > /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/write/historic_ops.out 2026-02-26T03:35:54.649 INFO:teuthology.orchestra.run.trial134.stderr:03:35:54 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial134.front.sepia.ceph.com sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/write/* 2026-02-26T03:35:54.823 INFO:teuthology.orchestra.run.trial134.stderr:03:35:54 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=rpdcp -f 10 -R ssh -w ubuntu@trial134.front.sepia.ceph.com -r /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/write/* /home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/write 2026-02-26T03:35:54.982 INFO:teuthology.orchestra.run.trial134.stderr:03:35:54 - INFO - cbt - Convert results to json format. 2026-02-26T03:35:54.983 INFO:teuthology.orchestra.run.trial134.stderr:03:35:54 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial134.front.sepia.ceph.com sync 2026-02-26T03:35:55.140 INFO:teuthology.orchestra.run.trial134.stderr:03:35:55 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial134.front.sepia.ceph.com echo 3 | sudo tee /proc/sys/vm/drop_caches 2026-02-26T03:35:55.647 INFO:teuthology.orchestra.run.trial134.stderr:03:35:55 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial134.front.sepia.ceph.com /usr/bin/rados -c /etc/ceph/ceph.conf -v 2026-02-26T03:35:55.860 INFO:teuthology.orchestra.run.trial134.stderr:03:35:55 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial134.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/rand 2026-02-26T03:35:56.021 INFO:teuthology.orchestra.run.trial134.stderr:03:35:56 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial134.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf daemon osd.0 config show > /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/rand/ceph_settings.out 2026-02-26T03:35:56.301 INFO:teuthology.orchestra.run.trial134.stderr:03:35:56 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial134.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/rand/collectl 2026-02-26T03:35:56.467 INFO:teuthology.orchestra.run.trial134.stderr:03:35:56 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial134.front.sepia.ceph.com collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/rand/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2026-02-26T03:35:56.469 INFO:teuthology.orchestra.run.trial134.stderr:03:35:56 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial134.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/rand/perf 2026-02-26T03:35:56.630 INFO:teuthology.orchestra.run.trial134.stderr:03:35:56 - DEBUG - cbt - PerfMonitoring: remote_node 2026-02-26T03:35:56.630 INFO:teuthology.orchestra.run.trial134.stderr:03:35:56 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial134.front.sepia.ceph.com for pid in `cat /var/run/ceph/crimson-osd.*.pid`; do perf stat -p ${pid} -o /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/rand/perf/perf_stat.${pid} & done 2026-02-26T03:35:56.631 INFO:teuthology.orchestra.run.trial134.stderr:03:35:56 - INFO - cbt - Running radosbench rand test. 2026-02-26T03:35:56.632 INFO:teuthology.orchestra.run.trial134.stderr:03:35:56 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial134.front.sepia.ceph.com /usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench 60 rand --concurrent-ios 4 --run-name `hostname -f`-0 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/rand/objecter.0.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/rand/output.0 2026-02-26T03:35:56.633 INFO:teuthology.orchestra.run.trial134.stderr:03:35:56 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial134.front.sepia.ceph.com /usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench 60 rand --concurrent-ios 4 --run-name `hostname -f`-1 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/rand/objecter.1.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/rand/output.1 2026-02-26T03:36:57.844 INFO:teuthology.orchestra.run.trial134.stderr:03:36:57 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial134.front.sepia.ceph.com pkill -SIGINT -f collectl 2026-02-26T03:36:58.023 INFO:teuthology.orchestra.run.trial134.stderr:03:36:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial134.front.sepia.ceph.com sudo pkill -SIGINT -f perf\ 2026-02-26T03:36:58.210 INFO:teuthology.orchestra.run.trial134.stderr:03:36:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial134.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/rand/perf/perf.data 2026-02-26T03:36:58.212 INFO:teuthology.orchestra.run.trial134.stderr:03:36:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial134.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/rand/perf/perf_stat.* 2026-02-26T03:36:58.213 INFO:teuthology.orchestra.run.trial134.stderr:03:36:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial134.front.sepia.ceph.com find /var/run/ceph/ceph-osd*.asok -maxdepth 1 -exec sudo /usr/bin/ceph --admin-daemon {} dump_historic_ops \; > /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/rand/historic_ops.out 2026-02-26T03:36:58.606 INFO:teuthology.orchestra.run.trial134.stderr:03:36:58 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial134.front.sepia.ceph.com sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/rand/* 2026-02-26T03:36:58.779 INFO:teuthology.orchestra.run.trial134.stderr:03:36:58 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=rpdcp -f 10 -R ssh -w ubuntu@trial134.front.sepia.ceph.com -r /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/rand/* /home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/rand 2026-02-26T03:36:58.939 INFO:teuthology.orchestra.run.trial134.stderr:03:36:58 - INFO - cbt - Convert results to json format. 2026-02-26T03:36:58.950 DEBUG:teuthology.run_tasks:Unwinding manager cbt 2026-02-26T03:36:58.965 DEBUG:teuthology.orchestra.run.trial134:> rm --one-file-system -rf -- /home/ubuntu/cephtest/cbt 2026-02-26T03:36:58.992 INFO:tasks.cbt_performance:Collecting CBT performance data config 2026-02-26T03:36:58.992 INFO:tasks.cbt_performance:reading cbt preferences 2026-02-26T03:36:58.993 DEBUG:teuthology.orchestra.run.trial134:> find /home/ubuntu/cephtest -name 'json_output.*' 2026-02-26T03:36:59.050 INFO:teuthology.orchestra.run.trial134.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/write/json_output.0.trial134.front.sepia.ceph.com 2026-02-26T03:36:59.051 INFO:teuthology.orchestra.run.trial134.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/write/json_output.1.trial134.front.sepia.ceph.com 2026-02-26T03:36:59.051 INFO:teuthology.orchestra.run.trial134.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/rand/json_output.0.trial134.front.sepia.ceph.com 2026-02-26T03:36:59.051 INFO:teuthology.orchestra.run.trial134.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/rand/json_output.1.trial134.front.sepia.ceph.com 2026-02-26T03:36:59.052 DEBUG:teuthology.orchestra.run.trial134:> set -ex 2026-02-26T03:36:59.052 DEBUG:teuthology.orchestra.run.trial134:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/write/json_output.0.trial134.front.sepia.ceph.com of=/dev/stdout 2026-02-26T03:36:59.109 INFO:tasks.cbt_performance:reading total cpu cycles 2026-02-26T03:36:59.109 DEBUG:teuthology.orchestra.run.trial134:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/write -name 'perf_stat.*' 2026-02-26T03:36:59.165 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2026-02-26T03:36:59.165 INFO:tasks.cbt_performance:total cpu cycles: 0 2026-02-26T03:36:59.165 DEBUG:teuthology.orchestra.run.trial134:> set -ex 2026-02-26T03:36:59.165 DEBUG:teuthology.orchestra.run.trial134:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/write/json_output.1.trial134.front.sepia.ceph.com of=/dev/stdout 2026-02-26T03:36:59.221 INFO:tasks.cbt_performance:reading total cpu cycles 2026-02-26T03:36:59.222 DEBUG:teuthology.orchestra.run.trial134:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/write -name 'perf_stat.*' 2026-02-26T03:36:59.279 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2026-02-26T03:36:59.279 INFO:tasks.cbt_performance:total cpu cycles: 0 2026-02-26T03:36:59.279 DEBUG:teuthology.orchestra.run.trial134:> set -ex 2026-02-26T03:36:59.279 DEBUG:teuthology.orchestra.run.trial134:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/rand/json_output.0.trial134.front.sepia.ceph.com of=/dev/stdout 2026-02-26T03:36:59.335 INFO:tasks.cbt_performance:reading total cpu cycles 2026-02-26T03:36:59.336 DEBUG:teuthology.orchestra.run.trial134:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/rand -name 'perf_stat.*' 2026-02-26T03:36:59.392 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2026-02-26T03:36:59.392 INFO:tasks.cbt_performance:total cpu cycles: 0 2026-02-26T03:36:59.393 DEBUG:teuthology.orchestra.run.trial134:> set -ex 2026-02-26T03:36:59.393 DEBUG:teuthology.orchestra.run.trial134:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/rand/json_output.1.trial134.front.sepia.ceph.com of=/dev/stdout 2026-02-26T03:36:59.448 INFO:tasks.cbt_performance:reading total cpu cycles 2026-02-26T03:36:59.449 DEBUG:teuthology.orchestra.run.trial134:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/rand -name 'perf_stat.*' 2026-02-26T03:36:59.505 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2026-02-26T03:36:59.505 INFO:tasks.cbt_performance:total cpu cycles: 0 2026-02-26T03:36:59.509 WARNING:tasks.cbt_performance:Could not send performance data to http://mira118.front.sepia.ceph.com:4000/cbt_performance: HTTPConnectionPool(host='mira118.front.sepia.ceph.com', port=4000): Max retries exceeded with url: /cbt_performance (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -2] Name or service not known')) 2026-02-26T03:36:59.509 WARNING:tasks.cbt_performance:Continuing without performance data collection 2026-02-26T03:36:59.509 DEBUG:teuthology.run_tasks:Unwinding manager ssh_keys 2026-02-26T03:36:59.521 INFO:teuthology.task.ssh_keys:Cleaning up SSH keys 2026-02-26T03:36:59.521 INFO:teuthology.task.ssh_keys:cleaning up keys added for testing 2026-02-26T03:36:59.522 INFO:teuthology.task.ssh_keys: cleaning up keys for user ubuntu on trial134.front.sepia.ceph.com 2026-02-26T03:36:59.522 DEBUG:teuthology.orchestra.run.trial134:> rm -- /home/ubuntu/.ssh/id_rsa 2026-02-26T03:36:59.560 DEBUG:teuthology.orchestra.run.trial134:> rm -- /home/ubuntu/.ssh/id_rsa.pub 2026-02-26T03:36:59.617 DEBUG:teuthology.orchestra.run.trial134:> stat -c '"%a"' /home/ubuntu/.ssh/authorized_keys 2026-02-26T03:36:59.675 INFO:teuthology.orchestra.run.trial134.stdout:"600" 2026-02-26T03:36:59.675 DEBUG:teuthology.orchestra.run.trial134:> mv -- /home/ubuntu/.ssh/authorized_keys_2026-02-26_03:33:27:459389 /home/ubuntu/.ssh/authorized_keys 2026-02-26T03:36:59.733 DEBUG:teuthology.orchestra.run.trial134:> chmod 600 /home/ubuntu/.ssh/authorized_keys 2026-02-26T03:36:59.789 DEBUG:teuthology.orchestra.run.trial134:> rm /home/ubuntu/.ssh/config 2026-02-26T03:36:59.846 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2026-02-26T03:36:59.861 DEBUG:teuthology.orchestra.run.trial134:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2026-02-26T03:36:59.962 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-26T03:36:59.957+0000 7fddfed0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:36:59.963 INFO:teuthology.orchestra.run.trial134.stderr:2026-02-26T03:36:59.962+0000 7fddfed0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:37:00.085 INFO:teuthology.misc:Shutting down mds daemons... 2026-02-26T03:37:00.085 INFO:teuthology.misc:Shutting down osd daemons... 2026-02-26T03:37:00.085 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2026-02-26T03:37:00.085 INFO:teuthology.orchestra.run:waiting for 300 2026-02-26T03:37:00.158 INFO:tasks.ceph.osd.0:Stopped 2026-02-26T03:37:00.158 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2026-02-26T03:37:00.159 INFO:teuthology.orchestra.run:waiting for 300 2026-02-26T03:37:00.251 INFO:tasks.ceph.osd.1:Stopped 2026-02-26T03:37:00.251 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2026-02-26T03:37:00.251 INFO:teuthology.orchestra.run:waiting for 300 2026-02-26T03:37:00.339 INFO:tasks.ceph.osd.2:Stopped 2026-02-26T03:37:00.339 INFO:teuthology.misc:Shutting down mgr daemons... 2026-02-26T03:37:00.340 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2026-02-26T03:37:00.340 INFO:teuthology.orchestra.run:waiting for 300 2026-02-26T03:37:00.375 INFO:tasks.ceph.mgr.x:Stopped 2026-02-26T03:37:00.375 INFO:teuthology.misc:Shutting down mon daemons... 2026-02-26T03:37:00.375 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2026-02-26T03:37:00.376 INFO:teuthology.orchestra.run:waiting for 300 2026-02-26T03:37:00.424 INFO:tasks.ceph.mon.a:Stopped 2026-02-26T03:37:00.424 INFO:tasks.ceph:Checking cluster log for badness... 2026-02-26T03:37:00.425 DEBUG:teuthology.orchestra.run.trial134:> sudo grep -E '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | grep -E -v '\(MDS_ALL_DOWN\)' | grep -E -v '\(MDS_UP_LESS_THAN_MAX\)' | grep -E -v '\(POOL_APP_NOT_ENABLED\)' | grep -E -v '\(PG_' | grep -E -v '\(OSD_' | grep -E -v '\(OBJECT_' | grep -E -v 'overall HEALTH' | head -n 1 2026-02-26T03:37:00.490 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@trial134.front.sepia.ceph.com 2026-02-26T03:37:00.490 DEBUG:teuthology.orchestra.run.trial134:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2026-02-26T03:37:00.561 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@trial134.front.sepia.ceph.com 2026-02-26T03:37:00.561 DEBUG:teuthology.orchestra.run.trial134:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2026-02-26T03:37:00.635 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@trial134.front.sepia.ceph.com 2026-02-26T03:37:00.636 DEBUG:teuthology.orchestra.run.trial134:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2026-02-26T03:37:00.722 INFO:tasks.ceph:Archiving mon data... 2026-02-26T03:37:00.724 DEBUG:teuthology.misc:Transferring archived files from trial134:/var/lib/ceph/mon/ceph-a to /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-26_02:22:56-crimson-rados-main-distro-debug-trial/71162/data/mon.a.tgz 2026-02-26T03:37:00.725 DEBUG:teuthology.orchestra.run.trial134:> mktemp 2026-02-26T03:37:00.778 INFO:teuthology.orchestra.run.trial134.stdout:/tmp/tmp.Umv7P49OSI 2026-02-26T03:37:00.779 DEBUG:teuthology.orchestra.run.trial134:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.Umv7P49OSI 2026-02-26T03:37:00.870 DEBUG:teuthology.orchestra.run.trial134:> sudo chmod 0666 /tmp/tmp.Umv7P49OSI 2026-02-26T03:37:00.950 DEBUG:teuthology.orchestra.remote:trial134:/tmp/tmp.Umv7P49OSI is 206KB 2026-02-26T03:37:01.044 DEBUG:teuthology.orchestra.run.trial134:> rm -fr /tmp/tmp.Umv7P49OSI 2026-02-26T03:37:01.062 INFO:tasks.ceph:Cleaning ceph cluster... 2026-02-26T03:37:01.062 DEBUG:teuthology.orchestra.run.trial134:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2026-02-26T03:37:01.182 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2026-02-26T03:37:01.183 INFO:tasks.ceph:Archiving crash dumps... 2026-02-26T03:37:01.186 DEBUG:teuthology.misc:Transferring archived files from trial134:/var/lib/ceph/crash to /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-26_02:22:56-crimson-rados-main-distro-debug-trial/71162/remote/trial134/crash 2026-02-26T03:37:01.187 DEBUG:teuthology.orchestra.run.trial134:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2026-02-26T03:37:01.246 INFO:tasks.ceph:Compressing logs... 2026-02-26T03:37:01.247 DEBUG:teuthology.orchestra.run.trial134:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2026-02-26T03:37:01.311 INFO:teuthology.orchestra.run.trial134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2026-02-26T03:37:01.312 INFO:teuthology.orchestra.run.trial134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47246.log 2026-02-26T03:37:01.312 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2026-02-26T03:37:01.325 INFO:teuthology.orchestra.run.trial134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42092.log 2026-02-26T03:37:01.325 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.47246.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47246.log.gz 2026-02-26T03:37:01.325 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42239.log 2026-02-26T03:37:01.325 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.42092.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2026-02-26T03:37:01.325 INFO:teuthology.orchestra.run.trial134.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42092.log.gz 2026-02-26T03:37:01.325 INFO:teuthology.orchestra.run.trial134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46569.log 2026-02-26T03:37:01.325 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.42239.log: /var/log/ceph/ceph-osd.0.log: gzip -5 91.3% --verbose -- -- replaced with /var/log/ceph/ceph-mon.a.log.gz /var/log/ceph/ceph-client.admin.40967.log 2026-02-26T03:37:01.325 INFO:teuthology.orchestra.run.trial134.stderr: 2026-02-26T03:37:01.326 INFO:teuthology.orchestra.run.trial134.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42239.log.gz 2026-02-26T03:37:01.326 INFO:teuthology.orchestra.run.trial134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41215.log 2026-02-26T03:37:01.326 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.46569.log: 66.7% -- replaced with /var/log/ceph/ceph-client.admin.46569.log.gz 2026-02-26T03:37:01.326 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.40967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41749.log 2026-02-26T03:37:01.326 INFO:teuthology.orchestra.run.trial134.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40967.log.gz 2026-02-26T03:37:01.326 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.41215.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41997.log 2026-02-26T03:37:01.326 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.41749.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41215.log.gz 2026-02-26T03:37:01.326 INFO:teuthology.orchestra.run.trial134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41852.log 2026-02-26T03:37:01.326 INFO:teuthology.orchestra.run.trial134.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41749.log.gz 2026-02-26T03:37:01.326 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.41997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41442.log 2026-02-26T03:37:01.326 INFO:teuthology.orchestra.run.trial134.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41997.log.gz 2026-02-26T03:37:01.326 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.41852.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41999.log 2026-02-26T03:37:01.326 INFO:teuthology.orchestra.run.trial134.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41852.log.gz 2026-02-26T03:37:01.327 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.41442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45283.log 2026-02-26T03:37:01.327 INFO:teuthology.orchestra.run.trial134.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41442.log.gz 2026-02-26T03:37:01.327 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.41999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45888.log 2026-02-26T03:37:01.327 INFO:teuthology.orchestra.run.trial134.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41999.log.gz 2026-02-26T03:37:01.327 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.45283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45337.log 2026-02-26T03:37:01.327 INFO:teuthology.orchestra.run.trial134.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45283.log.gz 2026-02-26T03:37:01.327 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.45888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41602.log 2026-02-26T03:37:01.327 INFO:teuthology.orchestra.run.trial134.stderr: 66.4% -- replaced with /var/log/ceph/ceph-client.admin.45888.log.gz 2026-02-26T03:37:01.327 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.45337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42141.log 2026-02-26T03:37:01.327 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.41602.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45337.log.gz 2026-02-26T03:37:01.327 INFO:teuthology.orchestra.run.trial134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40689.log 2026-02-26T03:37:01.327 INFO:teuthology.orchestra.run.trial134.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41602.log.gz 2026-02-26T03:37:01.328 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.42141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41651.log 2026-02-26T03:37:01.328 INFO:teuthology.orchestra.run.trial134.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42141.log.gz 2026-02-26T03:37:01.328 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.40689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40594.log 2026-02-26T03:37:01.328 INFO:teuthology.orchestra.run.trial134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2026-02-26T03:37:01.328 INFO:teuthology.orchestra.run.trial134.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40689.log.gz 2026-02-26T03:37:01.328 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.41651.log: /var/log/ceph/ceph-client.admin.40594.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41651.log.gz 2026-02-26T03:37:01.328 INFO:teuthology.orchestra.run.trial134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41998.log 2026-02-26T03:37:01.328 INFO:teuthology.orchestra.run.trial134.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40594.log.gz 2026-02-26T03:37:01.328 INFO:teuthology.orchestra.run.trial134.stderr:gzip/var/log/ceph/ceph.audit.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.40736.log 2026-02-26T03:37:01.328 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.41998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41537.log 2026-02-26T03:37:01.328 INFO:teuthology.orchestra.run.trial134.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41998.log.gz 2026-02-26T03:37:01.328 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.40736.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2026-02-26T03:37:01.328 INFO:teuthology.orchestra.run.trial134.stderr: 87.7% -- replaced with /var/log/ceph/ceph.audit.log.gz 2026-02-26T03:37:01.329 INFO:teuthology.orchestra.run.trial134.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40736.log.gz 2026-02-26T03:37:01.329 INFO:teuthology.orchestra.run.trial134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45123.log 2026-02-26T03:37:01.329 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.41537.log: /var/log/ceph/ceph-osd.1.log: 66.9% -- replaced with /var/log/ceph/ceph-client.admin.41537.log.gz 2026-02-26T03:37:01.329 INFO:teuthology.orchestra.run.trial134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45179.log 2026-02-26T03:37:01.329 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.45123.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42190.log 2026-02-26T03:37:01.329 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.45179.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.45123.log.gz 2026-02-26T03:37:01.329 INFO:teuthology.orchestra.run.trial134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41512.log 2026-02-26T03:37:01.329 INFO:teuthology.orchestra.run.trial134.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45179.log.gz 2026-02-26T03:37:01.329 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.42190.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.39298.log 2026-02-26T03:37:01.329 INFO:teuthology.orchestra.run.trial134.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42190.log.gz 2026-02-26T03:37:01.329 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.41512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45237.log 2026-02-26T03:37:01.329 INFO:teuthology.orchestra.run.trial134.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41512.log.gz 2026-02-26T03:37:01.329 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph.tmp-client.admin.39298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46567.log 2026-02-26T03:37:01.330 INFO:teuthology.orchestra.run.trial134.stderr: 67.7% -- replaced with /var/log/ceph/ceph.tmp-client.admin.39298.log.gz 2026-02-26T03:37:01.330 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.45237.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41372.log 2026-02-26T03:37:01.330 INFO:teuthology.orchestra.run.trial134.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45237.log.gz 2026-02-26T03:37:01.330 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.46567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41186.log 2026-02-26T03:37:01.330 INFO:teuthology.orchestra.run.trial134.stderr: 66.4% -- replaced with /var/log/ceph/ceph-client.admin.46567.log.gz 2026-02-26T03:37:01.330 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.41372.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2026-02-26T03:37:01.330 INFO:teuthology.orchestra.run.trial134.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41372.log.gz 2026-02-26T03:37:01.330 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.41186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40920.log 2026-02-26T03:37:01.330 INFO:teuthology.orchestra.run.trial134.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41186.log.gz 2026-02-26T03:37:01.330 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40873.log 2026-02-26T03:37:01.330 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.40920.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2026-02-26T03:37:01.330 INFO:teuthology.orchestra.run.trial134.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.40920.log.gz 2026-02-26T03:37:01.330 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.40873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41850.log 2026-02-26T03:37:01.331 INFO:teuthology.orchestra.run.trial134.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.40873.log.gz 2026-02-26T03:37:01.331 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45892.log 2026-02-26T03:37:01.331 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.41850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41700.log 2026-02-26T03:37:01.331 INFO:teuthology.orchestra.run.trial134.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41850.log.gz/var/log/ceph/ceph-client.admin.45892.log: 2026-02-26T03:37:01.331 INFO:teuthology.orchestra.run.trial134.stderr: 85.8% -- replaced with /var/log/ceph/ceph.log.gz 2026-02-26T03:37:01.331 INFO:teuthology.orchestra.run.trial134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41851.log 2026-02-26T03:37:01.331 INFO:teuthology.orchestra.run.trial134.stderr: 66.4% -- replaced with /var/log/ceph/ceph-client.admin.45892.log.gz 2026-02-26T03:37:01.331 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.41700.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41700.log.gz 2026-02-26T03:37:01.331 INFO:teuthology.orchestra.run.trial134.stderr:/var/log/ceph/ceph-client.admin.41851.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41851.log.gz 2026-02-26T03:37:01.332 INFO:teuthology.orchestra.run.trial134.stderr: 92.2% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2026-02-26T03:37:07.505 INFO:teuthology.orchestra.run.trial134.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2026-02-26T03:37:07.521 INFO:teuthology.orchestra.run.trial134.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2026-02-26T03:37:08.095 INFO:teuthology.orchestra.run.trial134.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2026-02-26T03:37:08.096 INFO:teuthology.orchestra.run.trial134.stderr: 2026-02-26T03:37:08.096 INFO:teuthology.orchestra.run.trial134.stderr:real 0m6.794s 2026-02-26T03:37:08.096 INFO:teuthology.orchestra.run.trial134.stderr:user 0m17.884s 2026-02-26T03:37:08.096 INFO:teuthology.orchestra.run.trial134.stderr:sys 0m1.268s 2026-02-26T03:37:08.097 INFO:tasks.ceph:Archiving logs... 2026-02-26T03:37:08.098 DEBUG:teuthology.misc:Transferring archived files from trial134:/var/log/ceph to /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-26_02:22:56-crimson-rados-main-distro-debug-trial/71162/remote/trial134/log 2026-02-26T03:37:08.099 DEBUG:teuthology.orchestra.run.trial134:> sudo tar c -f - -C /var/log/ceph -- . 2026-02-26T03:37:09.728 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-26T03:37:09.795 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-26T03:37:09.806 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2026-02-26T03:37:09.806 DEBUG:teuthology.orchestra.run.trial134:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-26T03:37:09.832 DEBUG:teuthology.run_tasks:Unwinding manager clock 2026-02-26T03:37:09.843 INFO:teuthology.task.clock:Checking final clock skew... 2026-02-26T03:37:09.843 DEBUG:teuthology.orchestra.run.trial134:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-26T03:37:09.887 INFO:teuthology.orchestra.run.trial134.stderr:bash: line 1: ntpq: command not found 2026-02-26T03:37:09.906 INFO:teuthology.orchestra.run.trial134.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-26T03:37:09.906 INFO:teuthology.orchestra.run.trial134.stdout:=============================================================================== 2026-02-26T03:37:09.906 INFO:teuthology.orchestra.run.trial134.stdout:^* nyc3.us.ntp.li 2 7 377 119 -27us[ -36us] +/- 2276us 2026-02-26T03:37:09.906 INFO:teuthology.orchestra.run.trial134.stdout:^- rn-02.koehn.com 2 8 377 52 -624us[ -624us] +/- 12ms 2026-02-26T03:37:09.906 INFO:teuthology.orchestra.run.trial134.stdout:^- ip229.ip-51-81-226.us 2 8 377 48 +1147us[+1147us] +/- 108ms 2026-02-26T03:37:09.906 INFO:teuthology.orchestra.run.trial134.stdout:^- wdc1.us.ntp.li 3 7 377 51 -1982us[-1982us] +/- 32ms 2026-02-26T03:37:09.908 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2026-02-26T03:37:09.920 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2026-02-26T03:37:09.920 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2026-02-26T03:37:09.932 DEBUG:teuthology.orchestra.run.trial134:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2026-02-26T03:37:10.008 DEBUG:teuthology.orchestra.run.trial134:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-26T03:37:10.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-26T03:37:10.034 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2026-02-26T03:37:10.044 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2026-02-26T03:37:10.058 INFO:teuthology.task.internal:Duration was 1841.503095 seconds 2026-02-26T03:37:10.059 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2026-02-26T03:37:10.070 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2026-02-26T03:37:10.070 DEBUG:teuthology.orchestra.run.trial134:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-26T03:37:10.111 INFO:teuthology.orchestra.run.trial134.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-26T03:37:10.296 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2026-02-26T03:37:10.296 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial134.front.sepia.ceph.com 2026-02-26T03:37:10.296 DEBUG:teuthology.orchestra.run.trial134:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-26T03:37:10.354 INFO:teuthology.task.internal.syslog:Gathering journactl... 2026-02-26T03:37:10.354 DEBUG:teuthology.orchestra.run.trial134:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-26T03:37:10.461 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2026-02-26T03:37:10.461 DEBUG:teuthology.orchestra.run.trial134:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-26T03:37:10.484 INFO:teuthology.orchestra.run.trial134.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-26T03:37:10.485 INFO:teuthology.orchestra.run.trial134.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-26T03:37:10.485 INFO:teuthology.orchestra.run.trial134.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-26T03:37:10.485 INFO:teuthology.orchestra.run.trial134.stderr: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-02-26T03:37:10.485 INFO:teuthology.orchestra.run.trial134.stderr:/home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-26T03:37:10.495 INFO:teuthology.orchestra.run.trial134.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: 82.8% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-26T03:37:10.497 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2026-02-26T03:37:10.514 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2026-02-26T03:37:10.514 DEBUG:teuthology.orchestra.run.trial134:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-26T03:37:10.562 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2026-02-26T03:37:10.579 DEBUG:teuthology.orchestra.run.trial134:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-26T03:37:10.627 INFO:teuthology.orchestra.run.trial134.stdout:kernel.core_pattern = core 2026-02-26T03:37:10.641 DEBUG:teuthology.orchestra.run.trial134:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-26T03:37:10.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-26T03:37:10.696 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2026-02-26T03:37:10.714 INFO:teuthology.task.internal:Transferring archived files... 2026-02-26T03:37:10.715 DEBUG:teuthology.misc:Transferring archived files from trial134:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-26_02:22:56-crimson-rados-main-distro-debug-trial/71162/remote/trial134 2026-02-26T03:37:10.715 DEBUG:teuthology.orchestra.run.trial134:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-26T03:37:10.813 INFO:teuthology.task.internal:Removing archive directory... 2026-02-26T03:37:10.813 DEBUG:teuthology.orchestra.run.trial134:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-26T03:37:10.833 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2026-02-26T03:37:10.853 INFO:teuthology.task.internal:Not uploading archives. 2026-02-26T03:37:10.854 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2026-02-26T03:37:10.870 INFO:teuthology.task.internal:Tidying up after the test... 2026-02-26T03:37:10.870 DEBUG:teuthology.orchestra.run.trial134:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-26T03:37:10.889 INFO:teuthology.orchestra.run.trial134.stdout: 920834 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 26 03:37 /home/ubuntu/cephtest 2026-02-26T03:37:10.890 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2026-02-26T03:37:10.908 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2026-02-26T03:37:10.948 INFO:teuthology.run:Summary data: description: crimson-rados/perf/{0-crimson_install clusters/{crimson-fixed-1} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/segmented$/{crimson_seastore_segmented} perf_ignorelist settings/optimized workloads/radosbench_4K_rand_read} duration: 1841.503095149994 flavor: debug owner: scheduled_shraddhaag@soko04.front.sepia.ceph.com success: true 2026-02-26T03:37:10.948 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-26T03:37:11.070 INFO:teuthology.run:pass