2026-02-26T03:15:14.792 INFO:root:teuthology version: 1.2.4.dev11+g6e3d0419c 2026-02-26T03:15:14.809 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-26T03:15:14.895 INFO:teuthology.run:Config: archive_path: /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-26_02:22:56-crimson-rados-main-distro-debug-trial/71217 branch: main description: crimson-rados/perf/{0-crimson_install clusters/{crimson-fixed-1} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/segmented$/{crimson_seastore_segmented} perf_ignorelist settings/optimized workloads/radosbench_4M_rand_read} email: null first_in_suite: false flavor: debug job_id: '71217' kernel: branch: distro kdb: true sha1: distro last_in_suite: false machine_type: trial name: shraddhaag-2026-02-26_02:22:56-crimson-rados-main-distro-debug-trial no_nested_subset: false os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: global: auth client required: none auth cluster required: none auth service required: none auth supported: none debug asok: 0/0 debug auth: 0/0 debug bdev: 0/0 debug bluefs: 0/0 debug bluestore: 0/0 debug buffer: 0/0 debug civetweb: 0/0 debug client: 0/0 debug compressor: 0/0 debug context: 0/0 debug crush: 0/0 debug crypto: 0/0 debug dpdk: 0/0 debug eventtrace: 0/0 debug filer: 0/0 debug filestore: 0/0 debug finisher: 0/0 debug fuse: 0/0 debug heartbeatmap: 0/0 debug javaclient: 0/0 debug journal: 0/0 debug journaler: 0/0 debug kstore: 0/0 debug leveldb: 0/0 debug lockdep: 0/0 debug mds: 0/0 debug mds balancer: 0/0 debug mds locker: 0/0 debug mds log: 0/0 debug mds log expire: 0/0 debug mds migrator: 0/0 debug memdb: 0/0 debug mgr: 0/0 debug mgrc: 0/0 debug mon: 0/0 debug monc: 0/0 debug ms: 0/0 debug objclass: 0/0 debug objectcacher: 0/0 debug objecter: 0/0 debug optracker: 0/0 debug osd: 0/0 debug paxos: 0/0 debug perfcounter: 0/0 debug rados: 0/0 debug rbd: 0/0 debug rbd mirror: 0/0 debug rbd replay: 0/0 debug refs: 0/0 debug rgw: 0/0 debug rgw lifecycle: 0/0 debug rgw notification: 0/0 debug rgw sync: 0/0 debug rocksdb: 0/0 debug striper: 0/0 debug throttle: 0/0 debug timer: 0/0 debug tp: 0/0 enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 0/0 debug ms: 0/0 debug paxos: 0/0 osd pool default crimson: true osd: crimson cpu num: 4 crimson osd obc lru size: 10 debug filestore: 0/0 debug journal: 0/0 debug ms: 0/0 debug osd: 0/0 debug seastore: 20 debug seastore backref: 20 debug seastore cache: 20 debug seastore cleaner: 20 debug seastore device: 20 debug seastore epm: 20 debug seastore fixedkv tree: 20 debug seastore journal: 20 debug seastore lba: 20 debug seastore odata: 20 debug seastore omap: 20 debug seastore onode: 20 debug seastore t: 20 debug seastore tm: 20 osd objectstore: seastore osd shutdown pgref assert: true seastore cachepin size pershard: 64M seastore max concurrent transactions: 8 crimson_compat: true flavor: debug log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) - \(PG_ - \(OSD_ - \(OBJECT_ - overall HEALTH sha1: abffb617683951511fccf3c3cab6e219f2ec83a8 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log global: osd crush chooseleaf type: 0 osd pool default pg num: 128 osd pool default pgp num: 128 osd pool default size: 2 mon: {} install: ceph: flavor: debug sha1: abffb617683951511fccf3c3cab6e219f2ec83a8 workunit: branch: main env: CRIMSON_COMPAT: '1' sha1: 1f87b69b1a8041112e26ae0ea3c0a2a0fd14d996 owner: scheduled_shraddhaag@soko04.front.sepia.ceph.com priority: 101 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 5117 sha1: abffb617683951511fccf3c3cab6e219f2ec83a8 sleep_before_teardown: 0 suite: crimson-rados suite_branch: main suite_path: /home/teuthworker/src/github.com_ceph_ceph_1f87b69b1a8041112e26ae0ea3c0a2a0fd14d996/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph.git suite_sha1: 1f87b69b1a8041112e26ae0ea3c0a2a0fd14d996 targets: trial104.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD8htrLs1THJsH+RKU+45pysJq9LGLBCRf12EeT+nRDZAGy1UJQ6AqUqVnNJ3KxkNRB0bVXpMsj9h/U/SL7iGNw= tasks: - install: extra_packages: deb: - ceph-osd-crimson - ceph-osd-crimson-dbg rpm: - ceph-osd-crimson - ceph-osd-crimson-debuginfo - pexec: all: - sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson - install: null - ceph: conf: osd: debug monc: 20 wait-for-scrub: false - ssh_keys: null - cbt: benchmarks: radosbench: concurrent_ops: 4 concurrent_procs: 2 op_size: - 4194304 pool_monitoring_list: - collectl pool_profile: replicated readmode: rand run_monitoring_list: - collectl time: 60 write_only: false cluster: iterations: 1 osds_per_node: 3 pool_profiles: replicated: pg_size: 256 pgp_size: 256 replication: replicated user: ubuntu monitoring_profiles: perf: args: stat -p {pid} -o {perf_dir}/perf_stat.{pid} nodes: - osds perf_cmd: perf pid_glob: crimson-osd.*.pid teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6e3d0419ccca0e75a03e08feb708946e198236f4 timestamp: 2026-02-26_02:22:56 tube: trial user: shraddhaag verbose: false worker_log: /home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.904699 2026-02-26T03:15:14.895 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph_1f87b69b1a8041112e26ae0ea3c0a2a0fd14d996/qa; will attempt to use it 2026-02-26T03:15:14.896 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph_1f87b69b1a8041112e26ae0ea3c0a2a0fd14d996/qa/tasks 2026-02-26T03:15:14.896 INFO:teuthology.run_tasks:Running task internal.check_packages... 2026-02-26T03:15:14.897 INFO:teuthology.task.internal:Checking packages... 2026-02-26T03:15:14.897 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'debug' and ceph hash 'abffb617683951511fccf3c3cab6e219f2ec83a8' 2026-02-26T03:15:14.898 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2026-02-26T03:15:14.898 INFO:teuthology.packaging:ref: None 2026-02-26T03:15:14.898 INFO:teuthology.packaging:tag: None 2026-02-26T03:15:14.898 INFO:teuthology.packaging:branch: main 2026-02-26T03:15:14.898 INFO:teuthology.packaging:sha1: abffb617683951511fccf3c3cab6e219f2ec83a8 2026-02-26T03:15:14.898 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&ref=main 2026-02-26T03:15:15.025 INFO:teuthology.task.internal:Found packages for ceph version 20.3.0-5593.g63656871 2026-02-26T03:15:15.027 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2026-02-26T03:15:15.038 INFO:teuthology.task.internal:no buildpackages task found 2026-02-26T03:15:15.039 INFO:teuthology.run_tasks:Running task internal.save_config... 2026-02-26T03:15:15.052 INFO:teuthology.task.internal:Saving configuration 2026-02-26T03:15:15.068 INFO:teuthology.run_tasks:Running task internal.check_lock... 2026-02-26T03:15:15.082 INFO:teuthology.task.internal.check_lock:Checking locks... 2026-02-26T03:15:15.168 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial104.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/shraddhaag-2026-02-26_02:22:56-crimson-rados-main-distro-debug-trial/71217', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-26 03:11:44.969278', 'locked_by': 'scheduled_shraddhaag@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD8htrLs1THJsH+RKU+45pysJq9LGLBCRf12EeT+nRDZAGy1UJQ6AqUqVnNJ3KxkNRB0bVXpMsj9h/U/SL7iGNw='} 2026-02-26T03:15:15.168 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2026-02-26T03:15:15.180 INFO:teuthology.task.internal:roles: ubuntu@trial104.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2026-02-26T03:15:15.180 INFO:teuthology.run_tasks:Running task console_log... 2026-02-26T03:15:15.264 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x70f718162d40>, signals=[15]) 2026-02-26T03:15:15.264 INFO:teuthology.run_tasks:Running task internal.connect... 2026-02-26T03:15:15.278 INFO:teuthology.task.internal:Opening connections... 2026-02-26T03:15:15.278 DEBUG:teuthology.task.internal:connecting to ubuntu@trial104.front.sepia.ceph.com 2026-02-26T03:15:15.279 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-26T03:15:15.346 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2026-02-26T03:15:15.360 DEBUG:teuthology.orchestra.run.trial104:> uname -m 2026-02-26T03:15:15.424 INFO:teuthology.orchestra.run.trial104.stdout:x86_64 2026-02-26T03:15:15.425 DEBUG:teuthology.orchestra.run.trial104:> cat /etc/os-release 2026-02-26T03:15:15.481 INFO:teuthology.orchestra.run.trial104.stdout:NAME="CentOS Stream" 2026-02-26T03:15:15.481 INFO:teuthology.orchestra.run.trial104.stdout:VERSION="9" 2026-02-26T03:15:15.481 INFO:teuthology.orchestra.run.trial104.stdout:ID="centos" 2026-02-26T03:15:15.481 INFO:teuthology.orchestra.run.trial104.stdout:ID_LIKE="rhel fedora" 2026-02-26T03:15:15.481 INFO:teuthology.orchestra.run.trial104.stdout:VERSION_ID="9" 2026-02-26T03:15:15.481 INFO:teuthology.orchestra.run.trial104.stdout:PLATFORM_ID="platform:el9" 2026-02-26T03:15:15.481 INFO:teuthology.orchestra.run.trial104.stdout:PRETTY_NAME="CentOS Stream 9" 2026-02-26T03:15:15.481 INFO:teuthology.orchestra.run.trial104.stdout:ANSI_COLOR="0;31" 2026-02-26T03:15:15.481 INFO:teuthology.orchestra.run.trial104.stdout:LOGO="fedora-logo-icon" 2026-02-26T03:15:15.481 INFO:teuthology.orchestra.run.trial104.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2026-02-26T03:15:15.481 INFO:teuthology.orchestra.run.trial104.stdout:HOME_URL="https://centos.org/" 2026-02-26T03:15:15.481 INFO:teuthology.orchestra.run.trial104.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2026-02-26T03:15:15.481 INFO:teuthology.orchestra.run.trial104.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2026-02-26T03:15:15.481 INFO:teuthology.orchestra.run.trial104.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2026-02-26T03:15:15.482 INFO:teuthology.lock.ops:Updating trial104.front.sepia.ceph.com on lock server 2026-02-26T03:15:15.564 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2026-02-26T03:15:15.577 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2026-02-26T03:15:15.589 INFO:teuthology.task.internal:Checking for old test directory... 2026-02-26T03:15:15.589 DEBUG:teuthology.orchestra.run.trial104:> test '!' -e /home/ubuntu/cephtest 2026-02-26T03:15:15.603 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2026-02-26T03:15:15.614 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2026-02-26T03:15:15.615 DEBUG:teuthology.orchestra.run.trial104:> test -z $(ls -A /var/lib/ceph) 2026-02-26T03:15:15.660 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2026-02-26T03:15:15.746 INFO:teuthology.run_tasks:Running task kernel... 2026-02-26T03:15:15.766 INFO:teuthology.task.kernel:normalize config orig: {'branch': 'distro', 'kdb': True, 'sha1': 'distro'} 2026-02-26T03:15:15.766 INFO:teuthology.task.kernel:config {'mon.a': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}}, timeout 300 2026-02-26T03:15:15.766 DEBUG:teuthology.orchestra.run.trial104:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-26T03:15:15.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-26T03:15:15.781 DEBUG:teuthology.orchestra.run.trial104:> uname -r 2026-02-26T03:15:15.837 INFO:teuthology.orchestra.run.trial104.stdout:5.14.0-681.el9.x86_64 2026-02-26T03:15:15.837 INFO:teuthology.task.kernel:Running kernel on trial104: 5.14.0-681.el9.x86_64 2026-02-26T03:15:15.837 DEBUG:teuthology.orchestra.run.trial104:> sudo yum install -y kernel 2026-02-26T03:15:17.602 INFO:teuthology.orchestra.run.trial104.stdout:CentOS Stream 9 - BaseOS 6.5 MB/s | 8.9 MB 00:01 2026-02-26T03:15:20.790 INFO:teuthology.orchestra.run.trial104.stdout:CentOS Stream 9 - AppStream 10 MB/s | 27 MB 00:02 2026-02-26T03:15:24.397 INFO:teuthology.orchestra.run.trial104.stdout:CentOS Stream 9 - CRB 6.5 MB/s | 8.0 MB 00:01 2026-02-26T03:15:25.593 INFO:teuthology.orchestra.run.trial104.stdout:CentOS Stream 9 - Extras packages 45 kB/s | 20 kB 00:00 2026-02-26T03:15:26.082 INFO:teuthology.orchestra.run.trial104.stdout:Extra Packages for Enterprise Linux 48 MB/s | 20 MB 00:00 2026-02-26T03:15:29.238 INFO:teuthology.orchestra.run.trial104.stdout:lab-extras 2.4 MB/s | 50 kB 00:00 2026-02-26T03:15:30.258 INFO:teuthology.orchestra.run.trial104.stdout:Package kernel-5.14.0-645.el9.x86_64 is already installed. 2026-02-26T03:15:30.258 INFO:teuthology.orchestra.run.trial104.stdout:Package kernel-5.14.0-681.el9.x86_64 is already installed. 2026-02-26T03:15:30.279 INFO:teuthology.orchestra.run.trial104.stdout:Dependencies resolved. 2026-02-26T03:15:30.280 INFO:teuthology.orchestra.run.trial104.stdout:================================================================================ 2026-02-26T03:15:30.280 INFO:teuthology.orchestra.run.trial104.stdout: Package Architecture Version Repository Size 2026-02-26T03:15:30.280 INFO:teuthology.orchestra.run.trial104.stdout:================================================================================ 2026-02-26T03:15:30.280 INFO:teuthology.orchestra.run.trial104.stdout:Installing: 2026-02-26T03:15:30.280 INFO:teuthology.orchestra.run.trial104.stdout: kernel x86_64 5.14.0-686.el9 baseos 901 k 2026-02-26T03:15:30.280 INFO:teuthology.orchestra.run.trial104.stdout:Installing dependencies: 2026-02-26T03:15:30.280 INFO:teuthology.orchestra.run.trial104.stdout: kernel-core x86_64 5.14.0-686.el9 baseos 17 M 2026-02-26T03:15:30.281 INFO:teuthology.orchestra.run.trial104.stdout: kernel-modules x86_64 5.14.0-686.el9 baseos 40 M 2026-02-26T03:15:30.281 INFO:teuthology.orchestra.run.trial104.stdout: kernel-modules-core x86_64 5.14.0-686.el9 baseos 31 M 2026-02-26T03:15:30.281 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:15:30.281 INFO:teuthology.orchestra.run.trial104.stdout:Transaction Summary 2026-02-26T03:15:30.281 INFO:teuthology.orchestra.run.trial104.stdout:================================================================================ 2026-02-26T03:15:30.281 INFO:teuthology.orchestra.run.trial104.stdout:Install 4 Packages 2026-02-26T03:15:30.281 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:15:30.281 INFO:teuthology.orchestra.run.trial104.stdout:Total download size: 89 M 2026-02-26T03:15:30.281 INFO:teuthology.orchestra.run.trial104.stdout:Installed size: 135 M 2026-02-26T03:15:30.281 INFO:teuthology.orchestra.run.trial104.stdout:Downloading Packages: 2026-02-26T03:15:31.312 INFO:teuthology.orchestra.run.trial104.stdout:(1/4): kernel-5.14.0-686.el9.x86_64.rpm 951 kB/s | 901 kB 00:00 2026-02-26T03:15:31.836 INFO:teuthology.orchestra.run.trial104.stdout:(2/4): kernel-core-5.14.0-686.el9.x86_64.rpm 12 MB/s | 17 MB 00:01 2026-02-26T03:15:32.326 INFO:teuthology.orchestra.run.trial104.stdout:(3/4): kernel-modules-5.14.0-686.el9.x86_64.rpm 20 MB/s | 40 MB 00:01 2026-02-26T03:15:33.125 INFO:teuthology.orchestra.run.trial104.stdout:(4/4): kernel-modules-core-5.14.0-686.el9.x86_6 17 MB/s | 31 MB 00:01 2026-02-26T03:15:33.125 INFO:teuthology.orchestra.run.trial104.stdout:-------------------------------------------------------------------------------- 2026-02-26T03:15:33.126 INFO:teuthology.orchestra.run.trial104.stdout:Total 31 MB/s | 89 MB 00:02 2026-02-26T03:15:33.316 INFO:teuthology.orchestra.run.trial104.stdout:Running transaction check 2026-02-26T03:15:33.521 INFO:teuthology.orchestra.run.trial104.stdout:Transaction check succeeded. 2026-02-26T03:15:33.522 INFO:teuthology.orchestra.run.trial104.stdout:Running transaction test 2026-02-26T03:15:33.903 INFO:teuthology.orchestra.run.trial104.stdout:Transaction test succeeded. 2026-02-26T03:15:33.903 INFO:teuthology.orchestra.run.trial104.stdout:Running transaction 2026-02-26T03:15:35.014 INFO:teuthology.orchestra.run.trial104.stdout: Preparing : 1/1 2026-02-26T03:15:35.303 INFO:teuthology.orchestra.run.trial104.stdout: Installing : kernel-modules-core-5.14.0-686.el9.x86_64 1/4 2026-02-26T03:15:35.325 INFO:teuthology.orchestra.run.trial104.stdout: Installing : kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-26T03:15:37.900 INFO:teuthology.orchestra.run.trial104.stdout: Running scriptlet: kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-26T03:15:37.934 INFO:teuthology.orchestra.run.trial104.stdout: Installing : kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-26T03:15:40.968 INFO:teuthology.orchestra.run.trial104.stdout: Running scriptlet: kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-26T03:15:40.975 INFO:teuthology.orchestra.run.trial104.stdout: Installing : kernel-5.14.0-686.el9.x86_64 4/4 2026-02-26T03:15:44.024 INFO:teuthology.orchestra.run.trial104.stdout: Running scriptlet: kernel-modules-core-5.14.0-686.el9.x86_64 4/4 2026-02-26T03:16:01.599 INFO:teuthology.orchestra.run.trial104.stdout: Running scriptlet: kernel-core-5.14.0-686.el9.x86_64 4/4 2026-02-26T03:16:01.620 INFO:teuthology.orchestra.run.trial104.stdout: Running scriptlet: kernel-modules-5.14.0-686.el9.x86_64 4/4 2026-02-26T03:16:01.809 INFO:teuthology.orchestra.run.trial104.stdout: Running scriptlet: kernel-5.14.0-686.el9.x86_64 4/4 2026-02-26T03:16:01.809 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : kernel-5.14.0-686.el9.x86_64 1/4 2026-02-26T03:16:01.810 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-26T03:16:01.810 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-26T03:16:01.866 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : kernel-modules-core-5.14.0-686.el9.x86_64 4/4 2026-02-26T03:16:01.866 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:16:01.866 INFO:teuthology.orchestra.run.trial104.stdout:Installed: 2026-02-26T03:16:01.866 INFO:teuthology.orchestra.run.trial104.stdout: kernel-5.14.0-686.el9.x86_64 2026-02-26T03:16:01.867 INFO:teuthology.orchestra.run.trial104.stdout: kernel-core-5.14.0-686.el9.x86_64 2026-02-26T03:16:01.867 INFO:teuthology.orchestra.run.trial104.stdout: kernel-modules-5.14.0-686.el9.x86_64 2026-02-26T03:16:01.867 INFO:teuthology.orchestra.run.trial104.stdout: kernel-modules-core-5.14.0-686.el9.x86_64 2026-02-26T03:16:01.867 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:16:01.867 INFO:teuthology.orchestra.run.trial104.stdout:Complete! 2026-02-26T03:16:01.925 DEBUG:teuthology.orchestra.run.trial104:> rpm -q kernel | sort -rV | head -n 1 2026-02-26T03:16:01.953 INFO:teuthology.orchestra.run.trial104.stdout:kernel-5.14.0-686.el9.x86_64 2026-02-26T03:16:01.955 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-686.el9.x86_64 2026-02-26T03:16:01.955 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-681.el9.x86_64 Expected: 5.14.0-686.el9.x86_64 2026-02-26T03:16:01.955 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2026-02-26T03:16:01.955 DEBUG:teuthology.task.kernel:src is distro, skipping download 2026-02-26T03:16:01.955 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2026-02-26T03:16:01.956 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@trial104.front.sepia.ceph.com, path=None, version=distro) 2026-02-26T03:16:01.956 DEBUG:teuthology.orchestra.run.trial104:> rpm -q kernel | sort -rV | head -n 1 2026-02-26T03:16:02.022 INFO:teuthology.orchestra.run.trial104.stdout:kernel-5.14.0-686.el9.x86_64 2026-02-26T03:16:02.022 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-686.el9.x86_64 2026-02-26T03:16:02.023 DEBUG:teuthology.orchestra.run.trial104:> sudo rpm -qi grub2-tools 2026-02-26T03:16:02.094 INFO:teuthology.orchestra.run.trial104.stdout:Name : grub2-tools 2026-02-26T03:16:02.094 INFO:teuthology.orchestra.run.trial104.stdout:Epoch : 1 2026-02-26T03:16:02.094 INFO:teuthology.orchestra.run.trial104.stdout:Version : 2.06 2026-02-26T03:16:02.094 INFO:teuthology.orchestra.run.trial104.stdout:Release : 120.el9 2026-02-26T03:16:02.094 INFO:teuthology.orchestra.run.trial104.stdout:Architecture: x86_64 2026-02-26T03:16:02.094 INFO:teuthology.orchestra.run.trial104.stdout:Install Date: Wed 18 Feb 2026 05:18:42 PM UTC 2026-02-26T03:16:02.094 INFO:teuthology.orchestra.run.trial104.stdout:Group : Unspecified 2026-02-26T03:16:02.094 INFO:teuthology.orchestra.run.trial104.stdout:Size : 9108611 2026-02-26T03:16:02.094 INFO:teuthology.orchestra.run.trial104.stdout:License : GPLv3+ 2026-02-26T03:16:02.095 INFO:teuthology.orchestra.run.trial104.stdout:Signature : RSA/SHA256, Thu 04 Dec 2025 09:42:59 AM UTC, Key ID 05b555b38483c65d 2026-02-26T03:16:02.095 INFO:teuthology.orchestra.run.trial104.stdout:Source RPM : grub2-2.06-120.el9.src.rpm 2026-02-26T03:16:02.095 INFO:teuthology.orchestra.run.trial104.stdout:Build Date : Wed 03 Dec 2025 12:10:38 PM UTC 2026-02-26T03:16:02.095 INFO:teuthology.orchestra.run.trial104.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2026-02-26T03:16:02.095 INFO:teuthology.orchestra.run.trial104.stdout:Packager : builder@centos.org 2026-02-26T03:16:02.095 INFO:teuthology.orchestra.run.trial104.stdout:Vendor : CentOS 2026-02-26T03:16:02.095 INFO:teuthology.orchestra.run.trial104.stdout:URL : http://www.gnu.org/software/grub/ 2026-02-26T03:16:02.095 INFO:teuthology.orchestra.run.trial104.stdout:Summary : Support tools for GRUB. 2026-02-26T03:16:02.095 INFO:teuthology.orchestra.run.trial104.stdout:Description : 2026-02-26T03:16:02.095 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:16:02.095 INFO:teuthology.orchestra.run.trial104.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2026-02-26T03:16:02.095 INFO:teuthology.orchestra.run.trial104.stdout:customizable bootloader with modular architecture. It supports a rich 2026-02-26T03:16:02.096 INFO:teuthology.orchestra.run.trial104.stdout:variety of kernel formats, file systems, computer architectures and 2026-02-26T03:16:02.096 INFO:teuthology.orchestra.run.trial104.stdout:hardware devices. 2026-02-26T03:16:02.096 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:16:02.096 INFO:teuthology.orchestra.run.trial104.stdout:This subpackage provides tools for support of all platforms. 2026-02-26T03:16:02.097 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2026-02-26T03:16:02.097 INFO:teuthology.task.kernel:Updating grub on trial104 to boot 5.14.0-686.el9.x86_64 2026-02-26T03:16:02.097 DEBUG:teuthology.orchestra.run.trial104:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f -print -quit 2026-02-26T03:16:02.163 INFO:teuthology.orchestra.run.trial104.stdout:/boot/loader/entries/b0373be789df49cdafedf423703d2566-5.14.0-681.el9.x86_64.conf 2026-02-26T03:16:02.165 DEBUG:teuthology.orchestra.run.trial104:> sudo command -v grubby && echo yes || echo no 2026-02-26T03:16:02.227 INFO:teuthology.orchestra.run.trial104.stdout:/sbin/grubby 2026-02-26T03:16:02.228 INFO:teuthology.orchestra.run.trial104.stdout:yes 2026-02-26T03:16:02.228 DEBUG:teuthology.orchestra.run.trial104:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f | grep -F "5.14.0-686.el9.x86_64" | head -n 1 2026-02-26T03:16:02.293 INFO:teuthology.orchestra.run.trial104.stdout:/boot/loader/entries/b0373be789df49cdafedf423703d2566-5.14.0-686.el9.x86_64.conf 2026-02-26T03:16:02.293 DEBUG:teuthology.orchestra.run.trial104:> sudo grub2-set-default b0373be789df49cdafedf423703d2566-5.14.0-686.el9.x86_64 2026-02-26T03:16:02.364 DEBUG:teuthology.orchestra.run.trial104:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2026-02-26T03:16:02.572 INFO:teuthology.orchestra.run.trial104.stderr:Generating grub configuration file ... 2026-02-26T03:16:02.640 INFO:teuthology.orchestra.run.trial104.stderr:Found linux image: /boot/vmlinuz-5.14.0-686.el9.x86_64 2026-02-26T03:16:02.647 INFO:teuthology.orchestra.run.trial104.stderr:Found initrd image: /boot/initramfs-5.14.0-686.el9.x86_64.img 2026-02-26T03:16:02.893 INFO:teuthology.orchestra.run.trial104.stderr:Found linux image: /boot/vmlinuz-5.14.0-681.el9.x86_64 2026-02-26T03:16:02.900 INFO:teuthology.orchestra.run.trial104.stderr:Found initrd image: /boot/initramfs-5.14.0-681.el9.x86_64.img 2026-02-26T03:16:02.935 INFO:teuthology.orchestra.run.trial104.stderr:Found linux image: /boot/vmlinuz-5.14.0-645.el9.x86_64 2026-02-26T03:16:02.942 INFO:teuthology.orchestra.run.trial104.stderr:Found initrd image: /boot/initramfs-5.14.0-645.el9.x86_64.img 2026-02-26T03:16:02.974 INFO:teuthology.orchestra.run.trial104.stderr:Found linux image: /boot/vmlinuz-0-rescue-b0373be789df49cdafedf423703d2566 2026-02-26T03:16:02.980 INFO:teuthology.orchestra.run.trial104.stderr:Found initrd image: /boot/initramfs-0-rescue-b0373be789df49cdafedf423703d2566.img 2026-02-26T03:16:03.003 INFO:teuthology.orchestra.run.trial104.stderr:Found linux image: /boot/vmlinuz-0-rescue-2734456005d743dca7f3dc77c35a2d0d 2026-02-26T03:16:03.010 INFO:teuthology.orchestra.run.trial104.stderr:Found initrd image: /boot/initramfs-0-rescue-2734456005d743dca7f3dc77c35a2d0d.img 2026-02-26T03:16:03.095 INFO:teuthology.orchestra.run.trial104.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2026-02-26T03:16:03.104 INFO:teuthology.orchestra.run.trial104.stderr:done 2026-02-26T03:16:03.105 DEBUG:teuthology.orchestra.run.trial104:> test -d /sys/firmware/efi 2026-02-26T03:16:03.121 DEBUG:teuthology.orchestra.run.trial104:> sudo find /boot/efi/EFI -mindepth 1 -maxdepth 1 -type d || true 2026-02-26T03:16:03.185 INFO:teuthology.orchestra.run.trial104.stdout:/boot/efi/EFI/BOOT 2026-02-26T03:16:03.186 INFO:teuthology.orchestra.run.trial104.stdout:/boot/efi/EFI/centos 2026-02-26T03:16:03.187 DEBUG:teuthology.orchestra.run.trial104:> sudo test -f /boot/efi/EFI/BOOT/grub.cfg && echo yes || echo no 2026-02-26T03:16:03.252 INFO:teuthology.orchestra.run.trial104.stdout:no 2026-02-26T03:16:03.252 DEBUG:teuthology.orchestra.run.trial104:> sudo test -f /boot/efi/EFI/centos/grub.cfg && echo yes || echo no 2026-02-26T03:16:03.316 INFO:teuthology.orchestra.run.trial104.stdout:yes 2026-02-26T03:16:03.317 DEBUG:teuthology.orchestra.run.trial104:> sudo cp -f /boot/grub2/grub.cfg /boot/efi/EFI/centos/grub.cfg 2026-02-26T03:16:03.380 INFO:teuthology.orchestra.remote:Triggering safe hard reboot via SysRq on trial104 2026-02-26T03:16:03.380 DEBUG:teuthology.orchestra.run.trial104:> sudo bash -c ' 2026-02-26T03:16:03.381 DEBUG:teuthology.orchestra.run.trial104:> echo pci > /sys/kernel/reboot/type && echo 1 > /sys/kernel/reboot/force && sync && echo s > /proc/sysrq-trigger && echo u > /proc/sysrq-trigger && sleep 2 && echo b > /proc/sysrq-trigger 2026-02-26T03:16:03.381 DEBUG:teuthology.orchestra.run.trial104:> ' 2026-02-26T03:16:33.423 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2026-02-26T03:16:33.424 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial104.front.sepia.ceph.com' 2026-02-26T03:16:33.425 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-26T03:17:33.488 DEBUG:teuthology.orchestra.remote:timed out 2026-02-26T03:17:42.497 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial104.front.sepia.ceph.com' 2026-02-26T03:17:42.498 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-26T03:17:42.586 DEBUG:teuthology.orchestra.run.trial104:> true 2026-02-26T03:17:42.742 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial104.front.sepia.ceph.com' 2026-02-26T03:17:42.742 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-686.el9.x86_64"... 2026-02-26T03:17:42.742 DEBUG:teuthology.orchestra.run.trial104:> uname -r 2026-02-26T03:17:42.797 INFO:teuthology.orchestra.run.trial104.stdout:5.14.0-686.el9.x86_64 2026-02-26T03:17:42.797 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-686.el9.x86_64 vs 5.14.0-686.el9.x86_64 2026-02-26T03:17:42.798 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2026-02-26T03:17:42.798 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2026-02-26T03:17:43.798 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2026-02-26T03:17:43.799 DEBUG:teuthology.orchestra.run.trial104:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-26T03:17:43.829 INFO:teuthology.orchestra.run.trial104.stdout:ttyS1 2026-02-26T03:17:43.860 DEBUG:teuthology.parallel:result is None 2026-02-26T03:17:43.860 INFO:teuthology.run_tasks:Running task internal.base... 2026-02-26T03:17:43.871 INFO:teuthology.task.internal:Creating test directory... 2026-02-26T03:17:43.872 DEBUG:teuthology.orchestra.run.trial104:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-26T03:17:43.916 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2026-02-26T03:17:43.932 INFO:teuthology.run_tasks:Running task internal.archive... 2026-02-26T03:17:43.957 INFO:teuthology.task.internal:Creating archive directory... 2026-02-26T03:17:43.957 DEBUG:teuthology.orchestra.run.trial104:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-26T03:17:43.982 INFO:teuthology.run_tasks:Running task internal.coredump... 2026-02-26T03:17:43.996 INFO:teuthology.task.internal:Enabling coredump saving... 2026-02-26T03:17:43.996 DEBUG:teuthology.orchestra.run.trial104:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-26T03:17:44.045 INFO:teuthology.orchestra.run.trial104.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-26T03:17:44.054 INFO:teuthology.orchestra.run.trial104.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-26T03:17:44.056 INFO:teuthology.run_tasks:Running task internal.sudo... 2026-02-26T03:17:44.087 INFO:teuthology.task.internal:Configuring sudo... 2026-02-26T03:17:44.088 DEBUG:teuthology.orchestra.run.trial104:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-26T03:17:44.120 INFO:teuthology.run_tasks:Running task internal.syslog... 2026-02-26T03:17:44.137 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2026-02-26T03:17:44.138 DEBUG:teuthology.orchestra.run.trial104:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-26T03:17:44.177 DEBUG:teuthology.orchestra.run.trial104:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-26T03:17:44.336 DEBUG:teuthology.orchestra.run.trial104:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-26T03:17:44.361 DEBUG:teuthology.orchestra.run.trial104:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-26T03:17:44.481 DEBUG:teuthology.orchestra.run.trial104:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-26T03:17:44.504 DEBUG:teuthology.orchestra.run.trial104:> set -ex 2026-02-26T03:17:44.504 DEBUG:teuthology.orchestra.run.trial104:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-26T03:17:44.568 DEBUG:teuthology.orchestra.run.trial104:> sudo service rsyslog restart 2026-02-26T03:17:44.635 INFO:teuthology.orchestra.run.trial104.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-26T03:17:44.823 INFO:teuthology.run_tasks:Running task internal.timer... 2026-02-26T03:17:44.838 INFO:teuthology.task.internal:Starting timer... 2026-02-26T03:17:44.838 INFO:teuthology.run_tasks:Running task pcp... 2026-02-26T03:17:44.860 INFO:teuthology.run_tasks:Running task selinux... 2026-02-26T03:17:44.879 DEBUG:teuthology.orchestra.run.trial104:> sudo service auditd rotate 2026-02-26T03:17:44.908 INFO:teuthology.orchestra.run.trial104.stdout:Rotating logs: 2026-02-26T03:17:44.910 DEBUG:teuthology.task.selinux:Getting current SELinux state 2026-02-26T03:17:44.911 DEBUG:teuthology.orchestra.run.trial104:> /usr/sbin/getenforce 2026-02-26T03:17:44.969 INFO:teuthology.orchestra.run.trial104.stdout:Permissive 2026-02-26T03:17:44.969 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@trial104.front.sepia.ceph.com': 'permissive'} 2026-02-26T03:17:44.969 DEBUG:teuthology.orchestra.run.trial104:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-26T03:17:45.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-26T03:17:45.035 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2026-02-26T03:17:45.035 DEBUG:teuthology.orchestra.run.trial104:> sudo /usr/sbin/setenforce permissive 2026-02-26T03:17:45.098 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2026-02-26T03:17:45.114 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2026-02-26T03:17:45.129 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2026-02-26T03:17:45.131 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit trial104.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2026-02-26T03:20:32.580 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@trial104.front.sepia.ceph.com')] 2026-02-26T03:20:32.582 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial104.front.sepia.ceph.com' 2026-02-26T03:20:32.584 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-26T03:20:32.694 DEBUG:teuthology.orchestra.run.trial104:> true 2026-02-26T03:20:32.767 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial104.front.sepia.ceph.com' 2026-02-26T03:20:32.767 INFO:teuthology.run_tasks:Running task clock... 2026-02-26T03:20:32.781 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2026-02-26T03:20:32.782 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-26T03:20:32.782 DEBUG:teuthology.orchestra.run.trial104:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-26T03:20:32.834 INFO:teuthology.orchestra.run.trial104.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2026-02-26T03:20:32.846 INFO:teuthology.orchestra.run.trial104.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2026-02-26T03:20:32.866 INFO:teuthology.orchestra.run.trial104.stderr:sudo: ntpd: command not found 2026-02-26T03:20:32.876 INFO:teuthology.orchestra.run.trial104.stdout:506 Cannot talk to daemon 2026-02-26T03:20:32.887 INFO:teuthology.orchestra.run.trial104.stderr:Failed to start ntp.service: Unit ntp.service not found. 2026-02-26T03:20:32.899 INFO:teuthology.orchestra.run.trial104.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2026-02-26T03:20:32.948 INFO:teuthology.orchestra.run.trial104.stderr:bash: line 1: ntpq: command not found 2026-02-26T03:20:32.970 INFO:teuthology.orchestra.run.trial104.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-26T03:20:32.971 INFO:teuthology.orchestra.run.trial104.stdout:=============================================================================== 2026-02-26T03:20:32.971 INFO:teuthology.orchestra.run.trial104.stdout:^? triangle.kansas.net 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-26T03:20:32.971 INFO:teuthology.orchestra.run.trial104.stdout:^? lax1.us.ntp.li 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-26T03:20:32.971 INFO:teuthology.orchestra.run.trial104.stdout:^? time.gslb.gravinoc.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-26T03:20:32.971 INFO:teuthology.orchestra.run.trial104.stdout:^? 23.186.168.129 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-26T03:20:32.971 INFO:teuthology.run_tasks:Running task install... 2026-02-26T03:20:32.988 DEBUG:teuthology.task.install:project ceph 2026-02-26T03:20:32.989 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'debug', 'sha1': 'abffb617683951511fccf3c3cab6e219f2ec83a8'}} 2026-02-26T03:20:32.989 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'flavor': 'debug', 'sha1': 'abffb617683951511fccf3c3cab6e219f2ec83a8'} 2026-02-26T03:20:32.989 INFO:teuthology.task.install:Using flavor: debug 2026-02-26T03:20:32.999 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-osd', 'ceph-osd-classic', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-osd', 'ceph-osd-classic', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-26T03:20:32.999 INFO:teuthology.task.install:extra packages: {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']} 2026-02-26T03:20:33.000 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'debug', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'abffb617683951511fccf3c3cab6e219f2ec83a8', 'tag': None, 'wait_for_package': False} 2026-02-26T03:20:33.000 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=abffb617683951511fccf3c3cab6e219f2ec83a8 2026-02-26T03:20:33.174 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-shraddhaag-peering-bug/abffb617683951511fccf3c3cab6e219f2ec83a8/centos/9/flavors/debug/ 2026-02-26T03:20:33.174 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5312.gabffb617 2026-02-26T03:20:33.530 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-shraddhaag-peering-bug/abffb617683951511fccf3c3cab6e219f2ec83a8/centos/9/flavors/debug/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-shraddhaag-peering-bug/abffb617683951511fccf3c3cab6e219f2ec83a8/centos/9/flavors/debug/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-shraddhaag-peering-bug/abffb617683951511fccf3c3cab6e219f2ec83a8/centos/9/flavors/debug/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-26T03:20:33.531 DEBUG:teuthology.orchestra.run.trial104:> set -ex 2026-02-26T03:20:33.531 DEBUG:teuthology.orchestra.run.trial104:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-26T03:20:33.557 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-osd-crimson, ceph-osd-crimson-debuginfo on remote rpm x86_64 2026-02-26T03:20:33.557 DEBUG:teuthology.orchestra.run.trial104:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-26T03:20:33.622 DEBUG:teuthology.orchestra.run.trial104:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-26T03:20:33.695 DEBUG:teuthology.orchestra.run.trial104:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-26T03:20:33.759 INFO:teuthology.orchestra.run.trial104.stdout:check_obsoletes = 1 2026-02-26T03:20:33.761 DEBUG:teuthology.orchestra.run.trial104:> sudo yum clean all 2026-02-26T03:20:33.913 INFO:teuthology.orchestra.run.trial104.stdout:41 files removed 2026-02-26T03:20:33.928 DEBUG:teuthology.orchestra.run.trial104:> sudo yum -y install ceph-radosgw 2026-02-26T03:20:34.780 INFO:teuthology.orchestra.run.trial104.stdout:ceph packages for x86_64 133 kB/s | 94 kB 00:00 2026-02-26T03:20:35.451 INFO:teuthology.orchestra.run.trial104.stdout:ceph noarch packages 38 kB/s | 24 kB 00:00 2026-02-26T03:20:36.044 INFO:teuthology.orchestra.run.trial104.stdout:ceph source packages 3.6 kB/s | 2.1 kB 00:00 2026-02-26T03:20:36.877 INFO:teuthology.orchestra.run.trial104.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.9 MB 00:00 2026-02-26T03:20:40.183 INFO:teuthology.orchestra.run.trial104.stdout:CentOS Stream 9 - AppStream 9.7 MB/s | 27 MB 00:02 2026-02-26T03:20:43.753 INFO:teuthology.orchestra.run.trial104.stdout:CentOS Stream 9 - CRB 6.7 MB/s | 8.0 MB 00:01 2026-02-26T03:20:44.983 INFO:teuthology.orchestra.run.trial104.stdout:CentOS Stream 9 - Extras packages 42 kB/s | 20 kB 00:00 2026-02-26T03:20:45.648 INFO:teuthology.orchestra.run.trial104.stdout:Extra Packages for Enterprise Linux 34 MB/s | 20 MB 00:00 2026-02-26T03:20:48.825 INFO:teuthology.orchestra.run.trial104.stdout:lab-extras 1.9 MB/s | 50 kB 00:00 2026-02-26T03:20:50.064 INFO:teuthology.orchestra.run.trial104.stdout:Dependencies resolved. 2026-02-26T03:20:50.066 INFO:teuthology.orchestra.run.trial104.stdout:================================================================================ 2026-02-26T03:20:50.066 INFO:teuthology.orchestra.run.trial104.stdout: Package Arch Version Repository Size 2026-02-26T03:20:50.066 INFO:teuthology.orchestra.run.trial104.stdout:================================================================================ 2026-02-26T03:20:50.066 INFO:teuthology.orchestra.run.trial104.stdout:Installing: 2026-02-26T03:20:50.066 INFO:teuthology.orchestra.run.trial104.stdout: ceph-radosgw x86_64 2:20.3.0-5312.gabffb617.el9 ceph 11 M 2026-02-26T03:20:50.066 INFO:teuthology.orchestra.run.trial104.stdout:Upgrading: 2026-02-26T03:20:50.066 INFO:teuthology.orchestra.run.trial104.stdout: librados2 x86_64 2:20.3.0-5312.gabffb617.el9 ceph 3.3 M 2026-02-26T03:20:50.066 INFO:teuthology.orchestra.run.trial104.stdout: librbd1 x86_64 2:20.3.0-5312.gabffb617.el9 ceph 2.7 M 2026-02-26T03:20:50.067 INFO:teuthology.orchestra.run.trial104.stdout:Installing dependencies: 2026-02-26T03:20:50.067 INFO:teuthology.orchestra.run.trial104.stdout: boost-program-options x86_64 1.75.0-13.el9 appstream 104 k 2026-02-26T03:20:50.067 INFO:teuthology.orchestra.run.trial104.stdout: ceph-base x86_64 2:20.3.0-5312.gabffb617.el9 ceph 5.1 M 2026-02-26T03:20:50.067 INFO:teuthology.orchestra.run.trial104.stdout: ceph-common x86_64 2:20.3.0-5312.gabffb617.el9 ceph 21 M 2026-02-26T03:20:50.067 INFO:teuthology.orchestra.run.trial104.stdout: ceph-selinux x86_64 2:20.3.0-5312.gabffb617.el9 ceph 25 k 2026-02-26T03:20:50.067 INFO:teuthology.orchestra.run.trial104.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2026-02-26T03:20:50.067 INFO:teuthology.orchestra.run.trial104.stdout: libarrow x86_64 9.0.0-15.el9 epel 4.4 M 2026-02-26T03:20:50.067 INFO:teuthology.orchestra.run.trial104.stdout: libarrow-doc noarch 9.0.0-15.el9 epel 25 k 2026-02-26T03:20:50.067 INFO:teuthology.orchestra.run.trial104.stdout: libcephfs2 x86_64 2:20.3.0-5312.gabffb617.el9 ceph 859 k 2026-02-26T03:20:50.067 INFO:teuthology.orchestra.run.trial104.stdout: libnbd x86_64 1.20.3-4.el9 appstream 164 k 2026-02-26T03:20:50.067 INFO:teuthology.orchestra.run.trial104.stdout: liboath x86_64 2.6.12-1.el9 epel 49 k 2026-02-26T03:20:50.067 INFO:teuthology.orchestra.run.trial104.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2026-02-26T03:20:50.068 INFO:teuthology.orchestra.run.trial104.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2026-02-26T03:20:50.068 INFO:teuthology.orchestra.run.trial104.stdout: libradosstriper1 x86_64 2:20.3.0-5312.gabffb617.el9 ceph 263 k 2026-02-26T03:20:50.068 INFO:teuthology.orchestra.run.trial104.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2026-02-26T03:20:50.068 INFO:teuthology.orchestra.run.trial104.stdout: librgw2 x86_64 2:20.3.0-5312.gabffb617.el9 ceph 5.6 M 2026-02-26T03:20:50.068 INFO:teuthology.orchestra.run.trial104.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2026-02-26T03:20:50.068 INFO:teuthology.orchestra.run.trial104.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2026-02-26T03:20:50.068 INFO:teuthology.orchestra.run.trial104.stdout: mailcap noarch 2.1.49-5.el9 baseos 33 k 2026-02-26T03:20:50.068 INFO:teuthology.orchestra.run.trial104.stdout: parquet-libs x86_64 9.0.0-15.el9 epel 838 k 2026-02-26T03:20:50.068 INFO:teuthology.orchestra.run.trial104.stdout: pciutils x86_64 3.7.0-7.el9 baseos 93 k 2026-02-26T03:20:50.068 INFO:teuthology.orchestra.run.trial104.stdout: python3-ceph-argparse x86_64 2:20.3.0-5312.gabffb617.el9 ceph 45 k 2026-02-26T03:20:50.068 INFO:teuthology.orchestra.run.trial104.stdout: python3-ceph-common x86_64 2:20.3.0-5312.gabffb617.el9 ceph 185 k 2026-02-26T03:20:50.068 INFO:teuthology.orchestra.run.trial104.stdout: python3-cephfs x86_64 2:20.3.0-5312.gabffb617.el9 ceph 203 k 2026-02-26T03:20:50.069 INFO:teuthology.orchestra.run.trial104.stdout: python3-rados x86_64 2:20.3.0-5312.gabffb617.el9 ceph 324 k 2026-02-26T03:20:50.069 INFO:teuthology.orchestra.run.trial104.stdout: python3-rbd x86_64 2:20.3.0-5312.gabffb617.el9 ceph 305 k 2026-02-26T03:20:50.069 INFO:teuthology.orchestra.run.trial104.stdout: python3-rgw x86_64 2:20.3.0-5312.gabffb617.el9 ceph 100 k 2026-02-26T03:20:50.069 INFO:teuthology.orchestra.run.trial104.stdout: qatlib x86_64 25.08.0-2.el9 appstream 240 k 2026-02-26T03:20:50.069 INFO:teuthology.orchestra.run.trial104.stdout: qatzip-libs x86_64 1.3.1-1.el9 appstream 66 k 2026-02-26T03:20:50.069 INFO:teuthology.orchestra.run.trial104.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2026-02-26T03:20:50.069 INFO:teuthology.orchestra.run.trial104.stdout: thrift x86_64 0.15.0-4.el9 epel 1.6 M 2026-02-26T03:20:50.069 INFO:teuthology.orchestra.run.trial104.stdout:Installing weak dependencies: 2026-02-26T03:20:50.069 INFO:teuthology.orchestra.run.trial104.stdout: libcephfs-daemon x86_64 2:20.3.0-5312.gabffb617.el9 ceph 37 k 2026-02-26T03:20:50.069 INFO:teuthology.orchestra.run.trial104.stdout: libcephfs-proxy2 x86_64 2:20.3.0-5312.gabffb617.el9 ceph 26 k 2026-02-26T03:20:50.069 INFO:teuthology.orchestra.run.trial104.stdout: nvme-cli x86_64 2.16-1.el9 baseos 1.2 M 2026-02-26T03:20:50.069 INFO:teuthology.orchestra.run.trial104.stdout: qatlib-service x86_64 25.08.0-2.el9 appstream 37 k 2026-02-26T03:20:50.070 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:20:50.070 INFO:teuthology.orchestra.run.trial104.stdout:Transaction Summary 2026-02-26T03:20:50.070 INFO:teuthology.orchestra.run.trial104.stdout:================================================================================ 2026-02-26T03:20:50.070 INFO:teuthology.orchestra.run.trial104.stdout:Install 35 Packages 2026-02-26T03:20:50.070 INFO:teuthology.orchestra.run.trial104.stdout:Upgrade 2 Packages 2026-02-26T03:20:50.070 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:20:50.071 INFO:teuthology.orchestra.run.trial104.stdout:Total download size: 61 M 2026-02-26T03:20:50.071 INFO:teuthology.orchestra.run.trial104.stdout:Downloading Packages: 2026-02-26T03:20:54.934 INFO:teuthology.orchestra.run.trial104.stdout:(1/37): ceph-base-20.3.0-5312.gabffb617.el9.x86 1.1 MB/s | 5.1 MB 00:04 2026-02-26T03:20:55.004 INFO:teuthology.orchestra.run.trial104.stdout:(2/37): ceph-selinux-20.3.0-5312.gabffb617.el9. 358 kB/s | 25 kB 00:00 2026-02-26T03:20:55.075 INFO:teuthology.orchestra.run.trial104.stdout:(3/37): libcephfs-daemon-20.3.0-5312.gabffb617. 526 kB/s | 37 kB 00:00 2026-02-26T03:20:55.147 INFO:teuthology.orchestra.run.trial104.stdout:(4/37): libcephfs-proxy2-20.3.0-5312.gabffb617. 356 kB/s | 26 kB 00:00 2026-02-26T03:20:56.311 INFO:teuthology.orchestra.run.trial104.stdout:(5/37): libcephfs2-20.3.0-5312.gabffb617.el9.x8 738 kB/s | 859 kB 00:01 2026-02-26T03:20:56.659 INFO:teuthology.orchestra.run.trial104.stdout:(6/37): libradosstriper1-20.3.0-5312.gabffb617. 755 kB/s | 263 kB 00:00 2026-02-26T03:21:04.074 INFO:teuthology.orchestra.run.trial104.stdout:(7/37): librgw2-20.3.0-5312.gabffb617.el9.x86_6 777 kB/s | 5.6 MB 00:07 2026-02-26T03:21:04.145 INFO:teuthology.orchestra.run.trial104.stdout:(8/37): python3-ceph-argparse-20.3.0-5312.gabff 629 kB/s | 45 kB 00:00 2026-02-26T03:21:04.420 INFO:teuthology.orchestra.run.trial104.stdout:(9/37): python3-ceph-common-20.3.0-5312.gabffb6 674 kB/s | 185 kB 00:00 2026-02-26T03:21:04.769 INFO:teuthology.orchestra.run.trial104.stdout:(10/37): python3-cephfs-20.3.0-5312.gabffb617.e 583 kB/s | 203 kB 00:00 2026-02-26T03:21:05.320 INFO:teuthology.orchestra.run.trial104.stdout:(11/37): python3-rados-20.3.0-5312.gabffb617.el 588 kB/s | 324 kB 00:00 2026-02-26T03:21:05.942 INFO:teuthology.orchestra.run.trial104.stdout:(12/37): python3-rbd-20.3.0-5312.gabffb617.el9. 490 kB/s | 305 kB 00:00 2026-02-26T03:21:06.148 INFO:teuthology.orchestra.run.trial104.stdout:(13/37): python3-rgw-20.3.0-5312.gabffb617.el9. 486 kB/s | 100 kB 00:00 2026-02-26T03:21:06.176 INFO:teuthology.orchestra.run.trial104.stdout:(14/37): mailcap-2.1.49-5.el9.noarch.rpm 1.2 MB/s | 33 kB 00:00 2026-02-26T03:21:06.224 INFO:teuthology.orchestra.run.trial104.stdout:(15/37): nvme-cli-2.16-1.el9.x86_64.rpm 24 MB/s | 1.2 MB 00:00 2026-02-26T03:21:06.234 INFO:teuthology.orchestra.run.trial104.stdout:(16/37): pciutils-3.7.0-7.el9.x86_64.rpm 9.3 MB/s | 93 kB 00:00 2026-02-26T03:21:06.401 INFO:teuthology.orchestra.run.trial104.stdout:(17/37): boost-program-options-1.75.0-13.el9.x8 621 kB/s | 104 kB 00:00 2026-02-26T03:21:06.445 INFO:teuthology.orchestra.run.trial104.stdout:(18/37): libnbd-1.20.3-4.el9.x86_64.rpm 3.6 MB/s | 164 kB 00:00 2026-02-26T03:21:06.472 INFO:teuthology.orchestra.run.trial104.stdout:(19/37): libpmemobj-1.12.1-1.el9.x86_64.rpm 5.8 MB/s | 160 kB 00:00 2026-02-26T03:21:06.493 INFO:teuthology.orchestra.run.trial104.stdout:(20/37): librabbitmq-0.11.0-7.el9.x86_64.rpm 2.1 MB/s | 45 kB 00:00 2026-02-26T03:21:06.546 INFO:teuthology.orchestra.run.trial104.stdout:(21/37): librdkafka-1.6.1-102.el9.x86_64.rpm 12 MB/s | 662 kB 00:00 2026-02-26T03:21:06.571 INFO:teuthology.orchestra.run.trial104.stdout:(22/37): lttng-ust-2.12.0-6.el9.x86_64.rpm 12 MB/s | 292 kB 00:00 2026-02-26T03:21:06.594 INFO:teuthology.orchestra.run.trial104.stdout:(23/37): qatlib-25.08.0-2.el9.x86_64.rpm 10 MB/s | 240 kB 00:00 2026-02-26T03:21:06.613 INFO:teuthology.orchestra.run.trial104.stdout:(24/37): qatlib-service-25.08.0-2.el9.x86_64.rp 1.8 MB/s | 37 kB 00:00 2026-02-26T03:21:06.633 INFO:teuthology.orchestra.run.trial104.stdout:(25/37): qatzip-libs-1.3.1-1.el9.x86_64.rpm 3.2 MB/s | 66 kB 00:00 2026-02-26T03:21:06.753 INFO:teuthology.orchestra.run.trial104.stdout:(26/37): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.5 MB/s | 308 kB 00:00 2026-02-26T03:21:06.864 INFO:teuthology.orchestra.run.trial104.stdout:(27/37): libarrow-9.0.0-15.el9.x86_64.rpm 40 MB/s | 4.4 MB 00:00 2026-02-26T03:21:06.879 INFO:teuthology.orchestra.run.trial104.stdout:(28/37): libarrow-doc-9.0.0-15.el9.noarch.rpm 1.7 MB/s | 25 kB 00:00 2026-02-26T03:21:06.908 INFO:teuthology.orchestra.run.trial104.stdout:(29/37): liboath-2.6.12-1.el9.x86_64.rpm 1.6 MB/s | 49 kB 00:00 2026-02-26T03:21:06.935 INFO:teuthology.orchestra.run.trial104.stdout:(30/37): libunwind-1.6.2-1.el9.x86_64.rpm 2.4 MB/s | 67 kB 00:00 2026-02-26T03:21:06.969 INFO:teuthology.orchestra.run.trial104.stdout:(31/37): parquet-libs-9.0.0-15.el9.x86_64.rpm 24 MB/s | 838 kB 00:00 2026-02-26T03:21:07.005 INFO:teuthology.orchestra.run.trial104.stdout:(32/37): re2-20211101-20.el9.x86_64.rpm 5.1 MB/s | 191 kB 00:00 2026-02-26T03:21:07.040 INFO:teuthology.orchestra.run.trial104.stdout:(33/37): thrift-0.15.0-4.el9.x86_64.rpm 46 MB/s | 1.6 MB 00:00 2026-02-26T03:21:11.638 INFO:teuthology.orchestra.run.trial104.stdout:(34/37): ceph-radosgw-20.3.0-5312.gabffb617.el9 537 kB/s | 11 MB 00:21 2026-02-26T03:21:12.946 INFO:teuthology.orchestra.run.trial104.stdout:(35/37): librados2-20.3.0-5312.gabffb617.el9.x8 569 kB/s | 3.3 MB 00:05 2026-02-26T03:21:18.108 INFO:teuthology.orchestra.run.trial104.stdout:(36/37): librbd1-20.3.0-5312.gabffb617.el9.x86_ 426 kB/s | 2.7 MB 00:06 2026-02-26T03:21:19.534 INFO:teuthology.orchestra.run.trial104.stdout:(37/37): ceph-common-20.3.0-5312.gabffb617.el9. 729 kB/s | 21 MB 00:29 2026-02-26T03:21:19.536 INFO:teuthology.orchestra.run.trial104.stdout:-------------------------------------------------------------------------------- 2026-02-26T03:21:19.536 INFO:teuthology.orchestra.run.trial104.stdout:Total 2.1 MB/s | 61 MB 00:29 2026-02-26T03:21:19.633 INFO:teuthology.orchestra.run.trial104.stdout:Running transaction check 2026-02-26T03:21:19.653 INFO:teuthology.orchestra.run.trial104.stdout:Transaction check succeeded. 2026-02-26T03:21:19.653 INFO:teuthology.orchestra.run.trial104.stdout:Running transaction test 2026-02-26T03:21:19.898 INFO:teuthology.orchestra.run.trial104.stdout:Transaction test succeeded. 2026-02-26T03:21:19.898 INFO:teuthology.orchestra.run.trial104.stdout:Running transaction 2026-02-26T03:21:20.191 INFO:teuthology.orchestra.run.trial104.stdout: Preparing : 1/1 2026-02-26T03:21:20.331 INFO:teuthology.orchestra.run.trial104.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/39 2026-02-26T03:21:20.336 INFO:teuthology.orchestra.run.trial104.stdout: Upgrading : librados2-2:20.3.0-5312.gabffb617.el9.x86_64 2/39 2026-02-26T03:21:20.385 INFO:teuthology.orchestra.run.trial104.stdout: Running scriptlet: librados2-2:20.3.0-5312.gabffb617.el9.x86_64 2/39 2026-02-26T03:21:20.386 INFO:teuthology.orchestra.run.trial104.stdout: Installing : libcephfs2-2:20.3.0-5312.gabffb617.el9.x86_64 3/39 2026-02-26T03:21:20.398 INFO:teuthology.orchestra.run.trial104.stdout: Running scriptlet: libcephfs2-2:20.3.0-5312.gabffb617.el9.x86_64 3/39 2026-02-26T03:21:20.400 INFO:teuthology.orchestra.run.trial104.stdout: Installing : libcephfs-daemon-2:20.3.0-5312.gabffb617.el9.x86_6 4/39 2026-02-26T03:21:20.401 INFO:teuthology.orchestra.run.trial104.stdout: Installing : libcephfs-proxy2-2:20.3.0-5312.gabffb617.el9.x86_6 5/39 2026-02-26T03:21:20.428 INFO:teuthology.orchestra.run.trial104.stdout: Running scriptlet: libcephfs-proxy2-2:20.3.0-5312.gabffb617.el9.x86_6 5/39 2026-02-26T03:21:20.431 INFO:teuthology.orchestra.run.trial104.stdout: Installing : python3-rados-2:20.3.0-5312.gabffb617.el9.x86_64 6/39 2026-02-26T03:21:20.438 INFO:teuthology.orchestra.run.trial104.stdout: Installing : liboath-2.6.12-1.el9.x86_64 7/39 2026-02-26T03:21:20.441 INFO:teuthology.orchestra.run.trial104.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/39 2026-02-26T03:21:20.444 INFO:teuthology.orchestra.run.trial104.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/39 2026-02-26T03:21:20.448 INFO:teuthology.orchestra.run.trial104.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/39 2026-02-26T03:21:20.459 INFO:teuthology.orchestra.run.trial104.stdout: Installing : python3-ceph-argparse-2:20.3.0-5312.gabffb617.el9. 11/39 2026-02-26T03:21:20.472 INFO:teuthology.orchestra.run.trial104.stdout: Installing : python3-cephfs-2:20.3.0-5312.gabffb617.el9.x86_64 12/39 2026-02-26T03:21:20.473 INFO:teuthology.orchestra.run.trial104.stdout: Installing : libradosstriper1-2:20.3.0-5312.gabffb617.el9.x86_6 13/39 2026-02-26T03:21:20.494 INFO:teuthology.orchestra.run.trial104.stdout: Running scriptlet: libradosstriper1-2:20.3.0-5312.gabffb617.el9.x86_6 13/39 2026-02-26T03:21:20.498 INFO:teuthology.orchestra.run.trial104.stdout: Installing : thrift-0.15.0-4.el9.x86_64 14/39 2026-02-26T03:21:20.530 INFO:teuthology.orchestra.run.trial104.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/39 2026-02-26T03:21:20.533 INFO:teuthology.orchestra.run.trial104.stdout: Installing : libarrow-9.0.0-15.el9.x86_64 16/39 2026-02-26T03:21:20.540 INFO:teuthology.orchestra.run.trial104.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/39 2026-02-26T03:21:20.543 INFO:teuthology.orchestra.run.trial104.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/39 2026-02-26T03:21:20.547 INFO:teuthology.orchestra.run.trial104.stdout: Installing : libarrow-doc-9.0.0-15.el9.noarch 19/39 2026-02-26T03:21:20.660 INFO:teuthology.orchestra.run.trial104.stdout: Installing : libnbd-1.20.3-4.el9.x86_64 20/39 2026-02-26T03:21:20.661 INFO:teuthology.orchestra.run.trial104.stdout: Upgrading : librbd1-2:20.3.0-5312.gabffb617.el9.x86_64 21/39 2026-02-26T03:21:20.687 INFO:teuthology.orchestra.run.trial104.stdout: Running scriptlet: librbd1-2:20.3.0-5312.gabffb617.el9.x86_64 21/39 2026-02-26T03:21:20.690 INFO:teuthology.orchestra.run.trial104.stdout: Installing : python3-rbd-2:20.3.0-5312.gabffb617.el9.x86_64 22/39 2026-02-26T03:21:20.696 INFO:teuthology.orchestra.run.trial104.stdout: Installing : boost-program-options-1.75.0-13.el9.x86_64 23/39 2026-02-26T03:21:20.945 INFO:teuthology.orchestra.run.trial104.stdout: Installing : parquet-libs-9.0.0-15.el9.x86_64 24/39 2026-02-26T03:21:20.946 INFO:teuthology.orchestra.run.trial104.stdout: Installing : librgw2-2:20.3.0-5312.gabffb617.el9.x86_64 25/39 2026-02-26T03:21:20.962 INFO:teuthology.orchestra.run.trial104.stdout: Running scriptlet: librgw2-2:20.3.0-5312.gabffb617.el9.x86_64 25/39 2026-02-26T03:21:20.968 INFO:teuthology.orchestra.run.trial104.stdout: Installing : python3-rgw-2:20.3.0-5312.gabffb617.el9.x86_64 26/39 2026-02-26T03:21:20.972 INFO:teuthology.orchestra.run.trial104.stdout: Installing : pciutils-3.7.0-7.el9.x86_64 27/39 2026-02-26T03:21:20.975 INFO:teuthology.orchestra.run.trial104.stdout: Installing : qatlib-25.08.0-2.el9.x86_64 28/39 2026-02-26T03:21:20.975 INFO:teuthology.orchestra.run.trial104.stdout: Installing : qatlib-service-25.08.0-2.el9.x86_64 29/39 2026-02-26T03:21:20.993 INFO:teuthology.orchestra.run.trial104.stdout: Running scriptlet: qatlib-service-25.08.0-2.el9.x86_64 29/39 2026-02-26T03:21:21.109 INFO:teuthology.orchestra.run.trial104.stdout: Installing : qatzip-libs-1.3.1-1.el9.x86_64 30/39 2026-02-26T03:21:21.111 INFO:teuthology.orchestra.run.trial104.stdout: Installing : nvme-cli-2.16-1.el9.x86_64 31/39 2026-02-26T03:21:21.407 INFO:teuthology.orchestra.run.trial104.stdout: Running scriptlet: nvme-cli-2.16-1.el9.x86_64 31/39 2026-02-26T03:21:21.407 INFO:teuthology.orchestra.run.trial104.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service → /usr/lib/systemd/system/nvmefc-boot-connections.service. 2026-02-26T03:21:21.408 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:21:21.454 INFO:teuthology.orchestra.run.trial104.stdout: Installing : mailcap-2.1.49-5.el9.noarch 32/39 2026-02-26T03:21:21.456 INFO:teuthology.orchestra.run.trial104.stdout: Installing : python3-ceph-common-2:20.3.0-5312.gabffb617.el9.x8 33/39 2026-02-26T03:21:22.399 INFO:teuthology.orchestra.run.trial104.stdout: Running scriptlet: ceph-common-2:20.3.0-5312.gabffb617.el9.x86_64 34/39 2026-02-26T03:21:22.402 INFO:teuthology.orchestra.run.trial104.stdout: Installing : ceph-common-2:20.3.0-5312.gabffb617.el9.x86_64 34/39 2026-02-26T03:21:22.662 INFO:teuthology.orchestra.run.trial104.stdout: Running scriptlet: ceph-common-2:20.3.0-5312.gabffb617.el9.x86_64 34/39 2026-02-26T03:21:22.665 INFO:teuthology.orchestra.run.trial104.stdout: Installing : ceph-base-2:20.3.0-5312.gabffb617.el9.x86_64 35/39 2026-02-26T03:21:22.702 INFO:teuthology.orchestra.run.trial104.stdout: Running scriptlet: ceph-base-2:20.3.0-5312.gabffb617.el9.x86_64 35/39 2026-02-26T03:21:22.702 INFO:teuthology.orchestra.run.trial104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2026-02-26T03:21:22.702 INFO:teuthology.orchestra.run.trial104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2026-02-26T03:21:22.703 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:21:22.705 INFO:teuthology.orchestra.run.trial104.stdout: Installing : ceph-selinux-2:20.3.0-5312.gabffb617.el9.x86_64 36/39 2026-02-26T03:21:27.164 INFO:teuthology.orchestra.run.trial104.stdout: Running scriptlet: ceph-selinux-2:20.3.0-5312.gabffb617.el9.x86_64 36/39 2026-02-26T03:21:27.164 INFO:teuthology.orchestra.run.trial104.stdout:skipping the directory /sys 2026-02-26T03:21:27.164 INFO:teuthology.orchestra.run.trial104.stdout:skipping the directory /proc 2026-02-26T03:21:27.164 INFO:teuthology.orchestra.run.trial104.stdout:skipping the directory /mnt 2026-02-26T03:21:27.164 INFO:teuthology.orchestra.run.trial104.stdout:skipping the directory /var/tmp 2026-02-26T03:21:27.164 INFO:teuthology.orchestra.run.trial104.stdout:skipping the directory /home 2026-02-26T03:21:27.164 INFO:teuthology.orchestra.run.trial104.stdout:skipping the directory /root 2026-02-26T03:21:27.165 INFO:teuthology.orchestra.run.trial104.stdout:skipping the directory /tmp 2026-02-26T03:21:27.165 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:21:27.657 INFO:teuthology.orchestra.run.trial104.stdout: Installing : ceph-radosgw-2:20.3.0-5312.gabffb617.el9.x86_64 37/39 2026-02-26T03:21:27.680 INFO:teuthology.orchestra.run.trial104.stdout: Running scriptlet: ceph-radosgw-2:20.3.0-5312.gabffb617.el9.x86_64 37/39 2026-02-26T03:21:27.680 INFO:teuthology.orchestra.run.trial104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-26T03:21:27.680 INFO:teuthology.orchestra.run.trial104.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2026-02-26T03:21:27.680 INFO:teuthology.orchestra.run.trial104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-26T03:21:27.680 INFO:teuthology.orchestra.run.trial104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-26T03:21:27.681 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:21:27.682 INFO:teuthology.orchestra.run.trial104.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 38/39 2026-02-26T03:21:27.695 INFO:teuthology.orchestra.run.trial104.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 38/39 2026-02-26T03:21:27.695 INFO:teuthology.orchestra.run.trial104.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 39/39 2026-02-26T03:21:27.750 INFO:teuthology.orchestra.run.trial104.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 39/39 2026-02-26T03:21:27.750 INFO:teuthology.orchestra.run.trial104.stdout:Creating group 'qat' with GID 992. 2026-02-26T03:21:27.750 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:21:28.042 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : ceph-base-2:20.3.0-5312.gabffb617.el9.x86_64 1/39 2026-02-26T03:21:28.042 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : ceph-common-2:20.3.0-5312.gabffb617.el9.x86_64 2/39 2026-02-26T03:21:28.042 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : ceph-radosgw-2:20.3.0-5312.gabffb617.el9.x86_64 3/39 2026-02-26T03:21:28.042 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : ceph-selinux-2:20.3.0-5312.gabffb617.el9.x86_64 4/39 2026-02-26T03:21:28.043 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : libcephfs-daemon-2:20.3.0-5312.gabffb617.el9.x86_6 5/39 2026-02-26T03:21:28.043 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : libcephfs-proxy2-2:20.3.0-5312.gabffb617.el9.x86_6 6/39 2026-02-26T03:21:28.043 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : libcephfs2-2:20.3.0-5312.gabffb617.el9.x86_64 7/39 2026-02-26T03:21:28.043 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : libradosstriper1-2:20.3.0-5312.gabffb617.el9.x86_6 8/39 2026-02-26T03:21:28.043 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : librgw2-2:20.3.0-5312.gabffb617.el9.x86_64 9/39 2026-02-26T03:21:28.043 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : python3-ceph-argparse-2:20.3.0-5312.gabffb617.el9. 10/39 2026-02-26T03:21:28.043 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : python3-ceph-common-2:20.3.0-5312.gabffb617.el9.x8 11/39 2026-02-26T03:21:28.043 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : python3-cephfs-2:20.3.0-5312.gabffb617.el9.x86_64 12/39 2026-02-26T03:21:28.043 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : python3-rados-2:20.3.0-5312.gabffb617.el9.x86_64 13/39 2026-02-26T03:21:28.043 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : python3-rbd-2:20.3.0-5312.gabffb617.el9.x86_64 14/39 2026-02-26T03:21:28.043 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : python3-rgw-2:20.3.0-5312.gabffb617.el9.x86_64 15/39 2026-02-26T03:21:28.043 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : mailcap-2.1.49-5.el9.noarch 16/39 2026-02-26T03:21:28.044 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : nvme-cli-2.16-1.el9.x86_64 17/39 2026-02-26T03:21:28.044 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : pciutils-3.7.0-7.el9.x86_64 18/39 2026-02-26T03:21:28.044 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : boost-program-options-1.75.0-13.el9.x86_64 19/39 2026-02-26T03:21:28.044 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : libnbd-1.20.3-4.el9.x86_64 20/39 2026-02-26T03:21:28.044 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 21/39 2026-02-26T03:21:28.044 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 22/39 2026-02-26T03:21:28.044 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 23/39 2026-02-26T03:21:28.044 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 24/39 2026-02-26T03:21:28.044 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : qatlib-25.08.0-2.el9.x86_64 25/39 2026-02-26T03:21:28.044 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : qatlib-service-25.08.0-2.el9.x86_64 26/39 2026-02-26T03:21:28.044 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : qatzip-libs-1.3.1-1.el9.x86_64 27/39 2026-02-26T03:21:28.044 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 28/39 2026-02-26T03:21:28.044 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : libarrow-9.0.0-15.el9.x86_64 29/39 2026-02-26T03:21:28.045 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : libarrow-doc-9.0.0-15.el9.noarch 30/39 2026-02-26T03:21:28.045 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : liboath-2.6.12-1.el9.x86_64 31/39 2026-02-26T03:21:28.045 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 32/39 2026-02-26T03:21:28.045 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : parquet-libs-9.0.0-15.el9.x86_64 33/39 2026-02-26T03:21:28.045 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : re2-1:20211101-20.el9.x86_64 34/39 2026-02-26T03:21:28.045 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : thrift-0.15.0-4.el9.x86_64 35/39 2026-02-26T03:21:28.045 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : librados2-2:20.3.0-5312.gabffb617.el9.x86_64 36/39 2026-02-26T03:21:28.045 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 37/39 2026-02-26T03:21:28.045 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : librbd1-2:20.3.0-5312.gabffb617.el9.x86_64 38/39 2026-02-26T03:21:28.110 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 39/39 2026-02-26T03:21:28.110 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:21:28.110 INFO:teuthology.orchestra.run.trial104.stdout:Upgraded: 2026-02-26T03:21:28.110 INFO:teuthology.orchestra.run.trial104.stdout: librados2-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:21:28.110 INFO:teuthology.orchestra.run.trial104.stdout: librbd1-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:21:28.110 INFO:teuthology.orchestra.run.trial104.stdout:Installed: 2026-02-26T03:21:28.111 INFO:teuthology.orchestra.run.trial104.stdout: boost-program-options-1.75.0-13.el9.x86_64 2026-02-26T03:21:28.111 INFO:teuthology.orchestra.run.trial104.stdout: ceph-base-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:21:28.111 INFO:teuthology.orchestra.run.trial104.stdout: ceph-common-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:21:28.111 INFO:teuthology.orchestra.run.trial104.stdout: ceph-radosgw-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:21:28.111 INFO:teuthology.orchestra.run.trial104.stdout: ceph-selinux-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:21:28.111 INFO:teuthology.orchestra.run.trial104.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2026-02-26T03:21:28.111 INFO:teuthology.orchestra.run.trial104.stdout: libarrow-9.0.0-15.el9.x86_64 2026-02-26T03:21:28.111 INFO:teuthology.orchestra.run.trial104.stdout: libarrow-doc-9.0.0-15.el9.noarch 2026-02-26T03:21:28.111 INFO:teuthology.orchestra.run.trial104.stdout: libcephfs-daemon-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:21:28.111 INFO:teuthology.orchestra.run.trial104.stdout: libcephfs-proxy2-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:21:28.111 INFO:teuthology.orchestra.run.trial104.stdout: libcephfs2-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:21:28.111 INFO:teuthology.orchestra.run.trial104.stdout: libnbd-1.20.3-4.el9.x86_64 2026-02-26T03:21:28.112 INFO:teuthology.orchestra.run.trial104.stdout: liboath-2.6.12-1.el9.x86_64 2026-02-26T03:21:28.112 INFO:teuthology.orchestra.run.trial104.stdout: libpmemobj-1.12.1-1.el9.x86_64 2026-02-26T03:21:28.112 INFO:teuthology.orchestra.run.trial104.stdout: librabbitmq-0.11.0-7.el9.x86_64 2026-02-26T03:21:28.112 INFO:teuthology.orchestra.run.trial104.stdout: libradosstriper1-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:21:28.112 INFO:teuthology.orchestra.run.trial104.stdout: librdkafka-1.6.1-102.el9.x86_64 2026-02-26T03:21:28.112 INFO:teuthology.orchestra.run.trial104.stdout: librgw2-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:21:28.112 INFO:teuthology.orchestra.run.trial104.stdout: libunwind-1.6.2-1.el9.x86_64 2026-02-26T03:21:28.112 INFO:teuthology.orchestra.run.trial104.stdout: lttng-ust-2.12.0-6.el9.x86_64 2026-02-26T03:21:28.112 INFO:teuthology.orchestra.run.trial104.stdout: mailcap-2.1.49-5.el9.noarch 2026-02-26T03:21:28.112 INFO:teuthology.orchestra.run.trial104.stdout: nvme-cli-2.16-1.el9.x86_64 2026-02-26T03:21:28.112 INFO:teuthology.orchestra.run.trial104.stdout: parquet-libs-9.0.0-15.el9.x86_64 2026-02-26T03:21:28.112 INFO:teuthology.orchestra.run.trial104.stdout: pciutils-3.7.0-7.el9.x86_64 2026-02-26T03:21:28.112 INFO:teuthology.orchestra.run.trial104.stdout: python3-ceph-argparse-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:21:28.113 INFO:teuthology.orchestra.run.trial104.stdout: python3-ceph-common-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:21:28.113 INFO:teuthology.orchestra.run.trial104.stdout: python3-cephfs-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:21:28.113 INFO:teuthology.orchestra.run.trial104.stdout: python3-rados-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:21:28.113 INFO:teuthology.orchestra.run.trial104.stdout: python3-rbd-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:21:28.113 INFO:teuthology.orchestra.run.trial104.stdout: python3-rgw-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:21:28.113 INFO:teuthology.orchestra.run.trial104.stdout: qatlib-25.08.0-2.el9.x86_64 2026-02-26T03:21:28.113 INFO:teuthology.orchestra.run.trial104.stdout: qatlib-service-25.08.0-2.el9.x86_64 2026-02-26T03:21:28.113 INFO:teuthology.orchestra.run.trial104.stdout: qatzip-libs-1.3.1-1.el9.x86_64 2026-02-26T03:21:28.113 INFO:teuthology.orchestra.run.trial104.stdout: re2-1:20211101-20.el9.x86_64 2026-02-26T03:21:28.113 INFO:teuthology.orchestra.run.trial104.stdout: thrift-0.15.0-4.el9.x86_64 2026-02-26T03:21:28.113 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:21:28.113 INFO:teuthology.orchestra.run.trial104.stdout:Complete! 2026-02-26T03:21:28.152 DEBUG:teuthology.orchestra.run.trial104:> sudo yum -y install ceph-test 2026-02-26T03:21:28.473 INFO:teuthology.orchestra.run.trial104.stdout:Last metadata expiration check: 0:00:40 ago on Thu 26 Feb 2026 03:20:48 AM UTC. 2026-02-26T03:21:28.558 INFO:teuthology.orchestra.run.trial104.stdout:Dependencies resolved. 2026-02-26T03:21:28.558 INFO:teuthology.orchestra.run.trial104.stdout:================================================================================ 2026-02-26T03:21:28.558 INFO:teuthology.orchestra.run.trial104.stdout: Package Arch Version Repository Size 2026-02-26T03:21:28.558 INFO:teuthology.orchestra.run.trial104.stdout:================================================================================ 2026-02-26T03:21:28.559 INFO:teuthology.orchestra.run.trial104.stdout:Installing: 2026-02-26T03:21:28.559 INFO:teuthology.orchestra.run.trial104.stdout: ceph-test x86_64 2:20.3.0-5312.gabffb617.el9 ceph 126 M 2026-02-26T03:21:28.559 INFO:teuthology.orchestra.run.trial104.stdout:Installing dependencies: 2026-02-26T03:21:28.559 INFO:teuthology.orchestra.run.trial104.stdout: hwloc-libs x86_64 2.4.1-6.el9 baseos 2.1 M 2026-02-26T03:21:28.559 INFO:teuthology.orchestra.run.trial104.stdout: libasan8 x86_64 15.2.1-7.el9 appstream 525 k 2026-02-26T03:21:28.559 INFO:teuthology.orchestra.run.trial104.stdout: libatomic x86_64 11.5.0-14.el9 baseos 23 k 2026-02-26T03:21:28.559 INFO:teuthology.orchestra.run.trial104.stdout: libcephsqlite x86_64 2:20.3.0-5312.gabffb617.el9 ceph 153 k 2026-02-26T03:21:28.559 INFO:teuthology.orchestra.run.trial104.stdout: libubsan x86_64 11.5.0-14.el9 appstream 176 k 2026-02-26T03:21:28.559 INFO:teuthology.orchestra.run.trial104.stdout: libxslt x86_64 1.1.34-12.el9 appstream 233 k 2026-02-26T03:21:28.559 INFO:teuthology.orchestra.run.trial104.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2026-02-26T03:21:28.559 INFO:teuthology.orchestra.run.trial104.stdout: protobuf x86_64 3.14.0-17.el9 appstream 1.0 M 2026-02-26T03:21:28.559 INFO:teuthology.orchestra.run.trial104.stdout: socat x86_64 1.7.4.1-8.el9 appstream 303 k 2026-02-26T03:21:28.559 INFO:teuthology.orchestra.run.trial104.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2026-02-26T03:21:28.559 INFO:teuthology.orchestra.run.trial104.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2026-02-26T03:21:28.559 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:21:28.559 INFO:teuthology.orchestra.run.trial104.stdout:Transaction Summary 2026-02-26T03:21:28.559 INFO:teuthology.orchestra.run.trial104.stdout:================================================================================ 2026-02-26T03:21:28.559 INFO:teuthology.orchestra.run.trial104.stdout:Install 12 Packages 2026-02-26T03:21:28.559 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:21:28.559 INFO:teuthology.orchestra.run.trial104.stdout:Total download size: 131 M 2026-02-26T03:21:28.559 INFO:teuthology.orchestra.run.trial104.stdout:Installed size: 976 M 2026-02-26T03:21:28.559 INFO:teuthology.orchestra.run.trial104.stdout:Downloading Packages: 2026-02-26T03:21:29.792 INFO:teuthology.orchestra.run.trial104.stdout:(1/12): libcephsqlite-20.3.0-5312.gabffb617.el9 228 kB/s | 153 kB 00:00 2026-02-26T03:21:30.272 INFO:teuthology.orchestra.run.trial104.stdout:(2/12): libatomic-11.5.0-14.el9.x86_64.rpm 48 kB/s | 23 kB 00:00 2026-02-26T03:21:30.587 INFO:teuthology.orchestra.run.trial104.stdout:(3/12): lksctp-tools-1.0.19-2.el9.x86_64.rpm 298 kB/s | 94 kB 00:00 2026-02-26T03:21:31.226 INFO:teuthology.orchestra.run.trial104.stdout:(4/12): libasan8-15.2.1-7.el9.x86_64.rpm 822 kB/s | 525 kB 00:00 2026-02-26T03:21:31.467 INFO:teuthology.orchestra.run.trial104.stdout:(5/12): libubsan-11.5.0-14.el9.x86_64.rpm 731 kB/s | 176 kB 00:00 2026-02-26T03:21:31.787 INFO:teuthology.orchestra.run.trial104.stdout:(6/12): libxslt-1.1.34-12.el9.x86_64.rpm 728 kB/s | 233 kB 00:00 2026-02-26T03:21:32.079 INFO:teuthology.orchestra.run.trial104.stdout:(7/12): hwloc-libs-2.4.1-6.el9.x86_64.rpm 726 kB/s | 2.1 MB 00:02 2026-02-26T03:21:32.472 INFO:teuthology.orchestra.run.trial104.stdout:(8/12): socat-1.7.4.1-8.el9.x86_64.rpm 773 kB/s | 303 kB 00:00 2026-02-26T03:21:32.555 INFO:teuthology.orchestra.run.trial104.stdout:(9/12): xmlstarlet-1.6.1-20.el9.x86_64.rpm 770 kB/s | 64 kB 00:00 2026-02-26T03:21:32.596 INFO:teuthology.orchestra.run.trial104.stdout:(10/12): yaml-cpp-0.6.3-5.el9.x86_64.rpm 2.9 MB/s | 122 kB 00:00 2026-02-26T03:21:32.973 INFO:teuthology.orchestra.run.trial104.stdout:(11/12): protobuf-3.14.0-17.el9.x86_64.rpm 867 kB/s | 1.0 MB 00:01 2026-02-26T03:26:48.267 INFO:teuthology.orchestra.run.trial104.stdout:(12/12): ceph-test-20.3.0-5312.gabffb617.el9.x8 404 kB/s | 126 MB 05:19 2026-02-26T03:26:48.274 INFO:teuthology.orchestra.run.trial104.stdout:-------------------------------------------------------------------------------- 2026-02-26T03:26:48.274 INFO:teuthology.orchestra.run.trial104.stdout:Total 419 kB/s | 131 MB 05:19 2026-02-26T03:26:48.339 INFO:teuthology.orchestra.run.trial104.stdout:Running transaction check 2026-02-26T03:26:48.350 INFO:teuthology.orchestra.run.trial104.stdout:Transaction check succeeded. 2026-02-26T03:26:48.350 INFO:teuthology.orchestra.run.trial104.stdout:Running transaction test 2026-02-26T03:26:48.781 INFO:teuthology.orchestra.run.trial104.stdout:Transaction test succeeded. 2026-02-26T03:26:48.782 INFO:teuthology.orchestra.run.trial104.stdout:Running transaction 2026-02-26T03:26:49.239 INFO:teuthology.orchestra.run.trial104.stdout: Preparing : 1/1 2026-02-26T03:26:49.248 INFO:teuthology.orchestra.run.trial104.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/12 2026-02-26T03:26:49.257 INFO:teuthology.orchestra.run.trial104.stdout: Installing : socat-1.7.4.1-8.el9.x86_64 2/12 2026-02-26T03:26:49.262 INFO:teuthology.orchestra.run.trial104.stdout: Installing : protobuf-3.14.0-17.el9.x86_64 3/12 2026-02-26T03:26:49.266 INFO:teuthology.orchestra.run.trial104.stdout: Installing : libxslt-1.1.34-12.el9.x86_64 4/12 2026-02-26T03:26:49.268 INFO:teuthology.orchestra.run.trial104.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 5/12 2026-02-26T03:26:49.273 INFO:teuthology.orchestra.run.trial104.stdout: Installing : libubsan-11.5.0-14.el9.x86_64 6/12 2026-02-26T03:26:49.273 INFO:teuthology.orchestra.run.trial104.stdout: Installing : libasan8-15.2.1-7.el9.x86_64 7/12 2026-02-26T03:26:49.289 INFO:teuthology.orchestra.run.trial104.stdout: Running scriptlet: libasan8-15.2.1-7.el9.x86_64 7/12 2026-02-26T03:26:49.292 INFO:teuthology.orchestra.run.trial104.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 8/12 2026-02-26T03:26:49.303 INFO:teuthology.orchestra.run.trial104.stdout: Installing : libatomic-11.5.0-14.el9.x86_64 9/12 2026-02-26T03:26:49.310 INFO:teuthology.orchestra.run.trial104.stdout: Installing : hwloc-libs-2.4.1-6.el9.x86_64 10/12 2026-02-26T03:26:49.311 INFO:teuthology.orchestra.run.trial104.stdout: Installing : libcephsqlite-2:20.3.0-5312.gabffb617.el9.x86_64 11/12 2026-02-26T03:26:56.389 INFO:teuthology.orchestra.run.trial104.stdout: Running scriptlet: libcephsqlite-2:20.3.0-5312.gabffb617.el9.x86_64 11/12 2026-02-26T03:26:56.408 INFO:teuthology.orchestra.run.trial104.stdout: Installing : ceph-test-2:20.3.0-5312.gabffb617.el9.x86_64 12/12 2026-02-26T03:26:57.200 INFO:teuthology.orchestra.run.trial104.stdout: Running scriptlet: ceph-test-2:20.3.0-5312.gabffb617.el9.x86_64 12/12 2026-02-26T03:26:57.200 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : ceph-test-2:20.3.0-5312.gabffb617.el9.x86_64 1/12 2026-02-26T03:26:57.201 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : libcephsqlite-2:20.3.0-5312.gabffb617.el9.x86_64 2/12 2026-02-26T03:26:57.201 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : hwloc-libs-2.4.1-6.el9.x86_64 3/12 2026-02-26T03:26:57.201 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : libatomic-11.5.0-14.el9.x86_64 4/12 2026-02-26T03:26:57.201 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 5/12 2026-02-26T03:26:57.201 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : libasan8-15.2.1-7.el9.x86_64 6/12 2026-02-26T03:26:57.201 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : libubsan-11.5.0-14.el9.x86_64 7/12 2026-02-26T03:26:57.201 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : libxslt-1.1.34-12.el9.x86_64 8/12 2026-02-26T03:26:57.201 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : protobuf-3.14.0-17.el9.x86_64 9/12 2026-02-26T03:26:57.201 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : socat-1.7.4.1-8.el9.x86_64 10/12 2026-02-26T03:26:57.201 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 11/12 2026-02-26T03:26:57.259 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 12/12 2026-02-26T03:26:57.259 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:26:57.259 INFO:teuthology.orchestra.run.trial104.stdout:Installed: 2026-02-26T03:26:57.259 INFO:teuthology.orchestra.run.trial104.stdout: ceph-test-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:26:57.259 INFO:teuthology.orchestra.run.trial104.stdout: hwloc-libs-2.4.1-6.el9.x86_64 2026-02-26T03:26:57.259 INFO:teuthology.orchestra.run.trial104.stdout: libasan8-15.2.1-7.el9.x86_64 2026-02-26T03:26:57.259 INFO:teuthology.orchestra.run.trial104.stdout: libatomic-11.5.0-14.el9.x86_64 2026-02-26T03:26:57.260 INFO:teuthology.orchestra.run.trial104.stdout: libcephsqlite-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:26:57.260 INFO:teuthology.orchestra.run.trial104.stdout: libubsan-11.5.0-14.el9.x86_64 2026-02-26T03:26:57.260 INFO:teuthology.orchestra.run.trial104.stdout: libxslt-1.1.34-12.el9.x86_64 2026-02-26T03:26:57.260 INFO:teuthology.orchestra.run.trial104.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2026-02-26T03:26:57.260 INFO:teuthology.orchestra.run.trial104.stdout: protobuf-3.14.0-17.el9.x86_64 2026-02-26T03:26:57.260 INFO:teuthology.orchestra.run.trial104.stdout: socat-1.7.4.1-8.el9.x86_64 2026-02-26T03:26:57.260 INFO:teuthology.orchestra.run.trial104.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2026-02-26T03:26:57.260 INFO:teuthology.orchestra.run.trial104.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2026-02-26T03:26:57.260 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:26:57.260 INFO:teuthology.orchestra.run.trial104.stdout:Complete! 2026-02-26T03:26:57.305 DEBUG:teuthology.orchestra.run.trial104:> sudo yum -y install ceph 2026-02-26T03:26:57.623 INFO:teuthology.orchestra.run.trial104.stdout:Last metadata expiration check: 0:06:09 ago on Thu 26 Feb 2026 03:20:48 AM UTC. 2026-02-26T03:26:57.711 INFO:teuthology.orchestra.run.trial104.stdout:Dependencies resolved. 2026-02-26T03:26:57.714 INFO:teuthology.orchestra.run.trial104.stdout:======================================================================================= 2026-02-26T03:26:57.714 INFO:teuthology.orchestra.run.trial104.stdout: Package Arch Version Repository Size 2026-02-26T03:26:57.715 INFO:teuthology.orchestra.run.trial104.stdout:======================================================================================= 2026-02-26T03:26:57.715 INFO:teuthology.orchestra.run.trial104.stdout:Installing: 2026-02-26T03:26:57.715 INFO:teuthology.orchestra.run.trial104.stdout: ceph x86_64 2:20.3.0-5312.gabffb617.el9 ceph 6.5 k 2026-02-26T03:26:57.715 INFO:teuthology.orchestra.run.trial104.stdout:Installing dependencies: 2026-02-26T03:26:57.715 INFO:teuthology.orchestra.run.trial104.stdout: abseil-cpp x86_64 20211102.0-4.el9 epel 551 k 2026-02-26T03:26:57.715 INFO:teuthology.orchestra.run.trial104.stdout: ceph-grafana-dashboards noarch 2:20.3.0-5312.gabffb617.el9 ceph-noarch 57 k 2026-02-26T03:26:57.715 INFO:teuthology.orchestra.run.trial104.stdout: ceph-mds x86_64 2:20.3.0-5312.gabffb617.el9 ceph 2.0 M 2026-02-26T03:26:57.715 INFO:teuthology.orchestra.run.trial104.stdout: ceph-mgr x86_64 2:20.3.0-5312.gabffb617.el9 ceph 937 k 2026-02-26T03:26:57.715 INFO:teuthology.orchestra.run.trial104.stdout: ceph-mgr-modules-core noarch 2:20.3.0-5312.gabffb617.el9 ceph-noarch 297 k 2026-02-26T03:26:57.715 INFO:teuthology.orchestra.run.trial104.stdout: ceph-mon x86_64 2:20.3.0-5312.gabffb617.el9 ceph 4.9 M 2026-02-26T03:26:57.715 INFO:teuthology.orchestra.run.trial104.stdout: ceph-osd x86_64 2:20.3.0-5312.gabffb617.el9 ceph 163 k 2026-02-26T03:26:57.716 INFO:teuthology.orchestra.run.trial104.stdout: ceph-osd-classic x86_64 2:20.3.0-5312.gabffb617.el9 ceph 15 M 2026-02-26T03:26:57.716 INFO:teuthology.orchestra.run.trial104.stdout: ceph-osd-crimson x86_64 2:20.3.0-5312.gabffb617.el9 ceph 87 M 2026-02-26T03:26:57.716 INFO:teuthology.orchestra.run.trial104.stdout: ceph-prometheus-alerts noarch 2:20.3.0-5312.gabffb617.el9 ceph-noarch 17 k 2026-02-26T03:26:57.716 INFO:teuthology.orchestra.run.trial104.stdout: cephadm noarch 2:20.3.0-5312.gabffb617.el9 ceph-noarch 1.0 M 2026-02-26T03:26:57.716 INFO:teuthology.orchestra.run.trial104.stdout: cryptsetup x86_64 2.8.1-3.el9 baseos 351 k 2026-02-26T03:26:57.716 INFO:teuthology.orchestra.run.trial104.stdout: flexiblas x86_64 3.0.4-9.el9 appstream 30 k 2026-02-26T03:26:57.716 INFO:teuthology.orchestra.run.trial104.stdout: flexiblas-netlib x86_64 3.0.4-9.el9 appstream 3.0 M 2026-02-26T03:26:57.716 INFO:teuthology.orchestra.run.trial104.stdout: flexiblas-openblas-openmp x86_64 3.0.4-9.el9 appstream 15 k 2026-02-26T03:26:57.716 INFO:teuthology.orchestra.run.trial104.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2026-02-26T03:26:57.716 INFO:teuthology.orchestra.run.trial104.stdout: ledmon-libs x86_64 1.1.0-3.el9 baseos 40 k 2026-02-26T03:26:57.716 INFO:teuthology.orchestra.run.trial104.stdout: libconfig x86_64 1.7.2-9.el9 baseos 72 k 2026-02-26T03:26:57.717 INFO:teuthology.orchestra.run.trial104.stdout: libgfortran x86_64 11.5.0-14.el9 baseos 794 k 2026-02-26T03:26:57.717 INFO:teuthology.orchestra.run.trial104.stdout: libquadmath x86_64 11.5.0-14.el9 baseos 184 k 2026-02-26T03:26:57.717 INFO:teuthology.orchestra.run.trial104.stdout: libstoragemgmt x86_64 1.10.1-1.el9 appstream 246 k 2026-02-26T03:26:57.717 INFO:teuthology.orchestra.run.trial104.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2026-02-26T03:26:57.717 INFO:teuthology.orchestra.run.trial104.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2026-02-26T03:26:57.717 INFO:teuthology.orchestra.run.trial104.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2026-02-26T03:26:57.717 INFO:teuthology.orchestra.run.trial104.stdout: openblas x86_64 0.3.29-1.el9 appstream 42 k 2026-02-26T03:26:57.717 INFO:teuthology.orchestra.run.trial104.stdout: openblas-openmp x86_64 0.3.29-1.el9 appstream 5.3 M 2026-02-26T03:26:57.717 INFO:teuthology.orchestra.run.trial104.stdout: protobuf-compiler x86_64 3.14.0-17.el9 crb 862 k 2026-02-26T03:26:57.717 INFO:teuthology.orchestra.run.trial104.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2026-02-26T03:26:57.717 INFO:teuthology.orchestra.run.trial104.stdout: python3-autocommand noarch 2.2.2-8.el9 epel 29 k 2026-02-26T03:26:57.717 INFO:teuthology.orchestra.run.trial104.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2026-02-26T03:26:57.718 INFO:teuthology.orchestra.run.trial104.stdout: python3-backports-tarfile noarch 1.2.0-1.el9 epel 60 k 2026-02-26T03:26:57.718 INFO:teuthology.orchestra.run.trial104.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2026-02-26T03:26:57.718 INFO:teuthology.orchestra.run.trial104.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2026-02-26T03:26:57.718 INFO:teuthology.orchestra.run.trial104.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2026-02-26T03:26:57.718 INFO:teuthology.orchestra.run.trial104.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2026-02-26T03:26:57.718 INFO:teuthology.orchestra.run.trial104.stdout: python3-cheroot noarch 10.0.1-4.el9 epel 173 k 2026-02-26T03:26:57.718 INFO:teuthology.orchestra.run.trial104.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2026-02-26T03:26:57.718 INFO:teuthology.orchestra.run.trial104.stdout: python3-cryptography x86_64 36.0.1-5.el9 baseos 1.2 M 2026-02-26T03:26:57.718 INFO:teuthology.orchestra.run.trial104.stdout: python3-devel x86_64 3.9.25-3.el9 appstream 244 k 2026-02-26T03:26:57.718 INFO:teuthology.orchestra.run.trial104.stdout: python3-google-auth noarch 1:2.45.0-1.el9 epel 254 k 2026-02-26T03:26:57.718 INFO:teuthology.orchestra.run.trial104.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2026-02-26T03:26:57.718 INFO:teuthology.orchestra.run.trial104.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2026-02-26T03:26:57.719 INFO:teuthology.orchestra.run.trial104.stdout: python3-isodate noarch 0.6.1-3.el9 epel 56 k 2026-02-26T03:26:57.719 INFO:teuthology.orchestra.run.trial104.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2026-02-26T03:26:57.719 INFO:teuthology.orchestra.run.trial104.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2026-02-26T03:26:57.719 INFO:teuthology.orchestra.run.trial104.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2026-02-26T03:26:57.719 INFO:teuthology.orchestra.run.trial104.stdout: python3-jaraco-context noarch 6.0.1-3.el9 epel 20 k 2026-02-26T03:26:57.719 INFO:teuthology.orchestra.run.trial104.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2026-02-26T03:26:57.719 INFO:teuthology.orchestra.run.trial104.stdout: python3-jaraco-text noarch 4.0.0-2.el9 epel 26 k 2026-02-26T03:26:57.719 INFO:teuthology.orchestra.run.trial104.stdout: python3-jinja2 noarch 2.11.3-8.el9 appstream 249 k 2026-02-26T03:26:57.719 INFO:teuthology.orchestra.run.trial104.stdout: python3-jmespath noarch 1.0.1-1.el9 appstream 48 k 2026-02-26T03:26:57.719 INFO:teuthology.orchestra.run.trial104.stdout: python3-kubernetes noarch 1:26.1.0-3.el9 epel 1.0 M 2026-02-26T03:26:57.719 INFO:teuthology.orchestra.run.trial104.stdout: python3-libstoragemgmt x86_64 1.10.1-1.el9 appstream 177 k 2026-02-26T03:26:57.719 INFO:teuthology.orchestra.run.trial104.stdout: python3-lxml x86_64 4.6.5-3.el9 appstream 1.2 M 2026-02-26T03:26:57.719 INFO:teuthology.orchestra.run.trial104.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2026-02-26T03:26:57.720 INFO:teuthology.orchestra.run.trial104.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2026-02-26T03:26:57.720 INFO:teuthology.orchestra.run.trial104.stdout: python3-msgpack x86_64 1.0.3-2.el9 epel 86 k 2026-02-26T03:26:57.720 INFO:teuthology.orchestra.run.trial104.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2026-02-26T03:26:57.720 INFO:teuthology.orchestra.run.trial104.stdout: python3-numpy x86_64 1:1.23.5-2.el9 appstream 6.1 M 2026-02-26T03:26:57.720 INFO:teuthology.orchestra.run.trial104.stdout: python3-numpy-f2py x86_64 1:1.23.5-2.el9 appstream 442 k 2026-02-26T03:26:57.720 INFO:teuthology.orchestra.run.trial104.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2026-02-26T03:26:57.720 INFO:teuthology.orchestra.run.trial104.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2026-02-26T03:26:57.720 INFO:teuthology.orchestra.run.trial104.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2026-02-26T03:26:57.720 INFO:teuthology.orchestra.run.trial104.stdout: python3-protobuf noarch 3.14.0-17.el9 appstream 267 k 2026-02-26T03:26:57.720 INFO:teuthology.orchestra.run.trial104.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2026-02-26T03:26:57.720 INFO:teuthology.orchestra.run.trial104.stdout: python3-pyasn1 noarch 0.4.8-7.el9 appstream 157 k 2026-02-26T03:26:57.720 INFO:teuthology.orchestra.run.trial104.stdout: python3-pyasn1-modules noarch 0.4.8-7.el9 appstream 277 k 2026-02-26T03:26:57.721 INFO:teuthology.orchestra.run.trial104.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2026-02-26T03:26:57.721 INFO:teuthology.orchestra.run.trial104.stdout: python3-pyparsing noarch 2.4.7-9.el9 baseos 150 k 2026-02-26T03:26:57.721 INFO:teuthology.orchestra.run.trial104.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2026-02-26T03:26:57.721 INFO:teuthology.orchestra.run.trial104.stdout: python3-requests noarch 2.25.1-10.el9 baseos 126 k 2026-02-26T03:26:57.721 INFO:teuthology.orchestra.run.trial104.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2026-02-26T03:26:57.721 INFO:teuthology.orchestra.run.trial104.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2026-02-26T03:26:57.721 INFO:teuthology.orchestra.run.trial104.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2026-02-26T03:26:57.721 INFO:teuthology.orchestra.run.trial104.stdout: python3-scipy x86_64 1.9.3-2.el9 appstream 19 M 2026-02-26T03:26:57.721 INFO:teuthology.orchestra.run.trial104.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2026-02-26T03:26:57.721 INFO:teuthology.orchestra.run.trial104.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2026-02-26T03:26:57.721 INFO:teuthology.orchestra.run.trial104.stdout: python3-typing-extensions noarch 4.15.0-1.el9 epel 86 k 2026-02-26T03:26:57.721 INFO:teuthology.orchestra.run.trial104.stdout: python3-urllib3 noarch 1.26.5-7.el9 baseos 218 k 2026-02-26T03:26:57.721 INFO:teuthology.orchestra.run.trial104.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2026-02-26T03:26:57.722 INFO:teuthology.orchestra.run.trial104.stdout: python3-xmlsec x86_64 1.3.13-1.el9 epel 48 k 2026-02-26T03:26:57.722 INFO:teuthology.orchestra.run.trial104.stdout: python3-xmltodict noarch 0.12.0-15.el9 epel 22 k 2026-02-26T03:26:57.722 INFO:teuthology.orchestra.run.trial104.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2026-02-26T03:26:57.722 INFO:teuthology.orchestra.run.trial104.stdout: unzip x86_64 6.0-59.el9 baseos 182 k 2026-02-26T03:26:57.722 INFO:teuthology.orchestra.run.trial104.stdout: xmlsec1 x86_64 1.2.29-13.el9 appstream 189 k 2026-02-26T03:26:57.722 INFO:teuthology.orchestra.run.trial104.stdout: xmlsec1-openssl x86_64 1.2.29-13.el9 appstream 90 k 2026-02-26T03:26:57.722 INFO:teuthology.orchestra.run.trial104.stdout: zip x86_64 3.0-35.el9 baseos 266 k 2026-02-26T03:26:57.722 INFO:teuthology.orchestra.run.trial104.stdout:Installing weak dependencies: 2026-02-26T03:26:57.722 INFO:teuthology.orchestra.run.trial104.stdout: ceph-mgr-cephadm noarch 2:20.3.0-5312.gabffb617.el9 ceph-noarch 184 k 2026-02-26T03:26:57.722 INFO:teuthology.orchestra.run.trial104.stdout: ceph-mgr-dashboard noarch 2:20.3.0-5312.gabffb617.el9 ceph-noarch 51 M 2026-02-26T03:26:57.722 INFO:teuthology.orchestra.run.trial104.stdout: ceph-mgr-diskprediction-local noarch 2:20.3.0-5312.gabffb617.el9 ceph-noarch 7.4 M 2026-02-26T03:26:57.722 INFO:teuthology.orchestra.run.trial104.stdout: ceph-mgr-k8sevents noarch 2:20.3.0-5312.gabffb617.el9 ceph-noarch 22 k 2026-02-26T03:26:57.723 INFO:teuthology.orchestra.run.trial104.stdout: ceph-mgr-rook noarch 2:20.3.0-5312.gabffb617.el9 ceph-noarch 50 k 2026-02-26T03:26:57.723 INFO:teuthology.orchestra.run.trial104.stdout: ceph-volume noarch 2:20.3.0-5312.gabffb617.el9 ceph-noarch 301 k 2026-02-26T03:26:57.723 INFO:teuthology.orchestra.run.trial104.stdout: python3-influxdb noarch 5.3.1-1.el9 epel 139 k 2026-02-26T03:26:57.723 INFO:teuthology.orchestra.run.trial104.stdout: python3-saml noarch 1.16.0-1.el9 epel 125 k 2026-02-26T03:26:57.723 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:26:57.723 INFO:teuthology.orchestra.run.trial104.stdout:Transaction Summary 2026-02-26T03:26:57.723 INFO:teuthology.orchestra.run.trial104.stdout:======================================================================================= 2026-02-26T03:26:57.723 INFO:teuthology.orchestra.run.trial104.stdout:Install 96 Packages 2026-02-26T03:26:57.723 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:26:57.724 INFO:teuthology.orchestra.run.trial104.stdout:Total download size: 225 M 2026-02-26T03:26:57.724 INFO:teuthology.orchestra.run.trial104.stdout:Installed size: 1.5 G 2026-02-26T03:26:57.724 INFO:teuthology.orchestra.run.trial104.stdout:Downloading Packages: 2026-02-26T03:26:58.366 INFO:teuthology.orchestra.run.trial104.stdout:(1/96): ceph-20.3.0-5312.gabffb617.el9.x86_64.r 22 kB/s | 6.5 kB 00:00 2026-02-26T03:27:00.436 INFO:teuthology.orchestra.run.trial104.stdout:(2/96): ceph-mds-20.3.0-5312.gabffb617.el9.x86_ 878 kB/s | 2.0 MB 00:02 2026-02-26T03:27:00.648 INFO:teuthology.orchestra.run.trial104.stdout:(3/96): ceph-osd-20.3.0-5312.gabffb617.el9.x86_ 770 kB/s | 163 kB 00:00 2026-02-26T03:27:01.118 INFO:teuthology.orchestra.run.trial104.stdout:(4/96): ceph-mgr-20.3.0-5312.gabffb617.el9.x86_ 308 kB/s | 937 kB 00:03 2026-02-26T03:27:10.125 INFO:teuthology.orchestra.run.trial104.stdout:(5/96): ceph-mon-20.3.0-5312.gabffb617.el9.x86_ 424 kB/s | 4.9 MB 00:11 2026-02-26T03:27:10.333 INFO:teuthology.orchestra.run.trial104.stdout:(6/96): ceph-grafana-dashboards-20.3.0-5312.gab 275 kB/s | 57 kB 00:00 2026-02-26T03:27:11.020 INFO:teuthology.orchestra.run.trial104.stdout:(7/96): ceph-mgr-cephadm-20.3.0-5312.gabffb617. 268 kB/s | 184 kB 00:00 2026-02-26T03:27:24.972 INFO:teuthology.orchestra.run.trial104.stdout:(8/96): ceph-osd-classic-20.3.0-5312.gabffb617. 622 kB/s | 15 MB 00:24 2026-02-26T03:27:39.331 INFO:teuthology.orchestra.run.trial104.stdout:(9/96): ceph-mgr-diskprediction-local-20.3.0-53 527 kB/s | 7.4 MB 00:14 2026-02-26T03:27:39.399 INFO:teuthology.orchestra.run.trial104.stdout:(10/96): ceph-mgr-k8sevents-20.3.0-5312.gabffb6 330 kB/s | 22 kB 00:00 2026-02-26T03:27:39.886 INFO:teuthology.orchestra.run.trial104.stdout:(11/96): ceph-mgr-modules-core-20.3.0-5312.gabf 610 kB/s | 297 kB 00:00 2026-02-26T03:27:40.190 INFO:teuthology.orchestra.run.trial104.stdout:(12/96): ceph-mgr-rook-20.3.0-5312.gabffb617.el 165 kB/s | 50 kB 00:00 2026-02-26T03:27:40.261 INFO:teuthology.orchestra.run.trial104.stdout:(13/96): ceph-prometheus-alerts-20.3.0-5312.gab 244 kB/s | 17 kB 00:00 2026-02-26T03:27:41.255 INFO:teuthology.orchestra.run.trial104.stdout:(14/96): ceph-volume-20.3.0-5312.gabffb617.el9. 304 kB/s | 301 kB 00:00 2026-02-26T03:27:43.686 INFO:teuthology.orchestra.run.trial104.stdout:(15/96): cephadm-20.3.0-5312.gabffb617.el9.noar 428 kB/s | 1.0 MB 00:02 2026-02-26T03:27:44.483 INFO:teuthology.orchestra.run.trial104.stdout:(16/96): cryptsetup-2.8.1-3.el9.x86_64.rpm 440 kB/s | 351 kB 00:00 2026-02-26T03:27:44.566 INFO:teuthology.orchestra.run.trial104.stdout:(17/96): ledmon-libs-1.1.0-3.el9.x86_64.rpm 489 kB/s | 40 kB 00:00 2026-02-26T03:27:44.727 INFO:teuthology.orchestra.run.trial104.stdout:(18/96): libconfig-1.7.2-9.el9.x86_64.rpm 447 kB/s | 72 kB 00:00 2026-02-26T03:27:45.669 INFO:teuthology.orchestra.run.trial104.stdout:(19/96): libgfortran-11.5.0-14.el9.x86_64.rpm 843 kB/s | 794 kB 00:00 2026-02-26T03:27:45.908 INFO:teuthology.orchestra.run.trial104.stdout:(20/96): libquadmath-11.5.0-14.el9.x86_64.rpm 773 kB/s | 184 kB 00:00 2026-02-26T03:27:46.111 INFO:teuthology.orchestra.run.trial104.stdout:(21/96): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.2 MB/s | 253 kB 00:00 2026-02-26T03:27:46.352 INFO:teuthology.orchestra.run.trial104.stdout:(22/96): python3-cryptography-36.0.1-5.el9.x86_ 5.2 MB/s | 1.2 MB 00:00 2026-02-26T03:27:46.436 INFO:teuthology.orchestra.run.trial104.stdout:(23/96): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 106 kB 00:00 2026-02-26T03:27:46.520 INFO:teuthology.orchestra.run.trial104.stdout:(24/96): python3-pycparser-2.20-6.el9.noarch.rp 1.6 MB/s | 135 kB 00:00 2026-02-26T03:27:46.605 INFO:teuthology.orchestra.run.trial104.stdout:(25/96): python3-pyparsing-2.4.7-9.el9.noarch.r 1.7 MB/s | 150 kB 00:00 2026-02-26T03:27:46.689 INFO:teuthology.orchestra.run.trial104.stdout:(26/96): python3-requests-2.25.1-10.el9.noarch. 1.5 MB/s | 126 kB 00:00 2026-02-26T03:27:46.773 INFO:teuthology.orchestra.run.trial104.stdout:(27/96): python3-urllib3-1.26.5-7.el9.noarch.rp 2.5 MB/s | 218 kB 00:00 2026-02-26T03:27:46.857 INFO:teuthology.orchestra.run.trial104.stdout:(28/96): unzip-6.0-59.el9.x86_64.rpm 2.1 MB/s | 182 kB 00:00 2026-02-26T03:27:46.941 INFO:teuthology.orchestra.run.trial104.stdout:(29/96): zip-3.0-35.el9.x86_64.rpm 3.1 MB/s | 266 kB 00:00 2026-02-26T03:27:46.968 INFO:teuthology.orchestra.run.trial104.stdout:(30/96): flexiblas-3.0.4-9.el9.x86_64.rpm 1.1 MB/s | 30 kB 00:00 2026-02-26T03:27:47.028 INFO:teuthology.orchestra.run.trial104.stdout:(31/96): flexiblas-netlib-3.0.4-9.el9.x86_64.rp 50 MB/s | 3.0 MB 00:00 2026-02-26T03:27:47.037 INFO:teuthology.orchestra.run.trial104.stdout:(32/96): flexiblas-openblas-openmp-3.0.4-9.el9. 1.6 MB/s | 15 kB 00:00 2026-02-26T03:27:47.048 INFO:teuthology.orchestra.run.trial104.stdout:(33/96): libstoragemgmt-1.10.1-1.el9.x86_64.rpm 23 MB/s | 246 kB 00:00 2026-02-26T03:27:47.058 INFO:teuthology.orchestra.run.trial104.stdout:(34/96): lua-5.4.4-4.el9.x86_64.rpm 17 MB/s | 188 kB 00:00 2026-02-26T03:27:47.067 INFO:teuthology.orchestra.run.trial104.stdout:(35/96): openblas-0.3.29-1.el9.x86_64.rpm 4.6 MB/s | 42 kB 00:00 2026-02-26T03:27:47.095 INFO:teuthology.orchestra.run.trial104.stdout:(36/96): openblas-openmp-0.3.29-1.el9.x86_64.rp 195 MB/s | 5.3 MB 00:00 2026-02-26T03:27:47.129 INFO:teuthology.orchestra.run.trial104.stdout:(37/96): python3-babel-2.9.1-2.el9.noarch.rpm 174 MB/s | 6.0 MB 00:00 2026-02-26T03:27:47.141 INFO:teuthology.orchestra.run.trial104.stdout:(38/96): python3-devel-3.9.25-3.el9.x86_64.rpm 20 MB/s | 244 kB 00:00 2026-02-26T03:27:47.152 INFO:teuthology.orchestra.run.trial104.stdout:(39/96): python3-jinja2-2.11.3-8.el9.noarch.rpm 22 MB/s | 249 kB 00:00 2026-02-26T03:27:47.162 INFO:teuthology.orchestra.run.trial104.stdout:(40/96): python3-jmespath-1.0.1-1.el9.noarch.rp 4.9 MB/s | 48 kB 00:00 2026-02-26T03:27:47.172 INFO:teuthology.orchestra.run.trial104.stdout:(41/96): python3-libstoragemgmt-1.10.1-1.el9.x8 17 MB/s | 177 kB 00:00 2026-02-26T03:27:47.187 INFO:teuthology.orchestra.run.trial104.stdout:(42/96): python3-lxml-4.6.5-3.el9.x86_64.rpm 87 MB/s | 1.2 MB 00:00 2026-02-26T03:27:47.196 INFO:teuthology.orchestra.run.trial104.stdout:(43/96): python3-markupsafe-1.1.1-12.el9.x86_64 3.7 MB/s | 35 kB 00:00 2026-02-26T03:27:47.228 INFO:teuthology.orchestra.run.trial104.stdout:(44/96): python3-numpy-1.23.5-2.el9.x86_64.rpm 190 MB/s | 6.1 MB 00:00 2026-02-26T03:27:47.241 INFO:teuthology.orchestra.run.trial104.stdout:(45/96): python3-numpy-f2py-1.23.5-2.el9.x86_64 35 MB/s | 442 kB 00:00 2026-02-26T03:27:47.251 INFO:teuthology.orchestra.run.trial104.stdout:(46/96): python3-packaging-20.9-5.el9.noarch.rp 7.5 MB/s | 77 kB 00:00 2026-02-26T03:27:47.261 INFO:teuthology.orchestra.run.trial104.stdout:(47/96): python3-protobuf-3.14.0-17.el9.noarch. 25 MB/s | 267 kB 00:00 2026-02-26T03:27:47.272 INFO:teuthology.orchestra.run.trial104.stdout:(48/96): python3-pyasn1-0.4.8-7.el9.noarch.rpm 14 MB/s | 157 kB 00:00 2026-02-26T03:27:47.284 INFO:teuthology.orchestra.run.trial104.stdout:(49/96): python3-pyasn1-modules-0.4.8-7.el9.noa 23 MB/s | 277 kB 00:00 2026-02-26T03:27:47.294 INFO:teuthology.orchestra.run.trial104.stdout:(50/96): python3-requests-oauthlib-1.3.0-12.el9 5.4 MB/s | 54 kB 00:00 2026-02-26T03:27:47.387 INFO:teuthology.orchestra.run.trial104.stdout:(51/96): python3-scipy-1.9.3-2.el9.x86_64.rpm 207 MB/s | 19 MB 00:00 2026-02-26T03:27:47.395 INFO:teuthology.orchestra.run.trial104.stdout:(52/96): python3-toml-0.10.2-6.el9.noarch.rpm 5.0 MB/s | 42 kB 00:00 2026-02-26T03:27:47.404 INFO:teuthology.orchestra.run.trial104.stdout:(53/96): xmlsec1-1.2.29-13.el9.x86_64.rpm 21 MB/s | 189 kB 00:00 2026-02-26T03:27:47.413 INFO:teuthology.orchestra.run.trial104.stdout:(54/96): xmlsec1-openssl-1.2.29-13.el9.x86_64.r 11 MB/s | 90 kB 00:00 2026-02-26T03:27:47.489 INFO:teuthology.orchestra.run.trial104.stdout:(55/96): lua-devel-5.4.4-4.el9.x86_64.rpm 295 kB/s | 22 kB 00:00 2026-02-26T03:27:47.635 INFO:teuthology.orchestra.run.trial104.stdout:(56/96): protobuf-compiler-3.14.0-17.el9.x86_64 5.7 MB/s | 862 kB 00:00 2026-02-26T03:27:47.712 INFO:teuthology.orchestra.run.trial104.stdout:(57/96): abseil-cpp-20211102.0-4.el9.x86_64.rpm 7.0 MB/s | 551 kB 00:00 2026-02-26T03:27:47.722 INFO:teuthology.orchestra.run.trial104.stdout:(58/96): grpc-data-1.46.7-10.el9.noarch.rpm 2.0 MB/s | 19 kB 00:00 2026-02-26T03:27:47.733 INFO:teuthology.orchestra.run.trial104.stdout:(59/96): luarocks-3.9.2-5.el9.noarch.rpm 14 MB/s | 151 kB 00:00 2026-02-26T03:27:47.749 INFO:teuthology.orchestra.run.trial104.stdout:(60/96): python3-asyncssh-2.13.2-5.el9.noarch.r 34 MB/s | 548 kB 00:00 2026-02-26T03:27:47.759 INFO:teuthology.orchestra.run.trial104.stdout:(61/96): python3-autocommand-2.2.2-8.el9.noarch 2.8 MB/s | 29 kB 00:00 2026-02-26T03:27:47.770 INFO:teuthology.orchestra.run.trial104.stdout:(62/96): python3-backports-tarfile-1.2.0-1.el9. 5.5 MB/s | 60 kB 00:00 2026-02-26T03:27:47.780 INFO:teuthology.orchestra.run.trial104.stdout:(63/96): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 4.2 MB/s | 43 kB 00:00 2026-02-26T03:27:47.790 INFO:teuthology.orchestra.run.trial104.stdout:(64/96): python3-cachetools-4.2.4-1.el9.noarch. 3.0 MB/s | 32 kB 00:00 2026-02-26T03:27:47.800 INFO:teuthology.orchestra.run.trial104.stdout:(65/96): python3-certifi-2023.05.07-4.el9.noarc 1.4 MB/s | 14 kB 00:00 2026-02-26T03:27:47.812 INFO:teuthology.orchestra.run.trial104.stdout:(66/96): python3-cheroot-10.0.1-4.el9.noarch.rp 14 MB/s | 173 kB 00:00 2026-02-26T03:27:47.829 INFO:teuthology.orchestra.run.trial104.stdout:(67/96): python3-cherrypy-18.6.1-2.el9.noarch.r 21 MB/s | 358 kB 00:00 2026-02-26T03:27:47.845 INFO:teuthology.orchestra.run.trial104.stdout:(68/96): python3-google-auth-2.45.0-1.el9.noarc 16 MB/s | 254 kB 00:00 2026-02-26T03:27:47.897 INFO:teuthology.orchestra.run.trial104.stdout:(69/96): python3-grpcio-1.46.7-10.el9.x86_64.rp 39 MB/s | 2.0 MB 00:00 2026-02-26T03:27:47.909 INFO:teuthology.orchestra.run.trial104.stdout:(70/96): python3-grpcio-tools-1.46.7-10.el9.x86 12 MB/s | 144 kB 00:00 2026-02-26T03:27:47.921 INFO:teuthology.orchestra.run.trial104.stdout:(71/96): python3-influxdb-5.3.1-1.el9.noarch.rp 11 MB/s | 139 kB 00:00 2026-02-26T03:27:47.931 INFO:teuthology.orchestra.run.trial104.stdout:(72/96): python3-isodate-0.6.1-3.el9.noarch.rpm 5.4 MB/s | 56 kB 00:00 2026-02-26T03:27:47.941 INFO:teuthology.orchestra.run.trial104.stdout:(73/96): python3-jaraco-8.2.1-3.el9.noarch.rpm 1.1 MB/s | 11 kB 00:00 2026-02-26T03:27:47.951 INFO:teuthology.orchestra.run.trial104.stdout:(74/96): python3-jaraco-classes-3.2.1-5.el9.noa 1.8 MB/s | 18 kB 00:00 2026-02-26T03:27:47.961 INFO:teuthology.orchestra.run.trial104.stdout:(75/96): python3-jaraco-collections-3.0.0-8.el9 2.3 MB/s | 23 kB 00:00 2026-02-26T03:27:47.971 INFO:teuthology.orchestra.run.trial104.stdout:(76/96): python3-jaraco-context-6.0.1-3.el9.noa 1.9 MB/s | 20 kB 00:00 2026-02-26T03:27:47.981 INFO:teuthology.orchestra.run.trial104.stdout:(77/96): python3-jaraco-functools-3.5.0-2.el9.n 1.9 MB/s | 19 kB 00:00 2026-02-26T03:27:47.992 INFO:teuthology.orchestra.run.trial104.stdout:(78/96): python3-jaraco-text-4.0.0-2.el9.noarch 2.3 MB/s | 26 kB 00:00 2026-02-26T03:27:48.009 INFO:teuthology.orchestra.run.trial104.stdout:(79/96): python3-kubernetes-26.1.0-3.el9.noarch 61 MB/s | 1.0 MB 00:00 2026-02-26T03:27:48.020 INFO:teuthology.orchestra.run.trial104.stdout:(80/96): python3-more-itertools-8.12.0-2.el9.no 7.2 MB/s | 79 kB 00:00 2026-02-26T03:27:48.030 INFO:teuthology.orchestra.run.trial104.stdout:(81/96): python3-msgpack-1.0.3-2.el9.x86_64.rpm 8.2 MB/s | 86 kB 00:00 2026-02-26T03:27:48.041 INFO:teuthology.orchestra.run.trial104.stdout:(82/96): python3-natsort-7.1.1-5.el9.noarch.rpm 5.3 MB/s | 58 kB 00:00 2026-02-26T03:27:48.050 INFO:teuthology.orchestra.run.trial104.stdout:(83/96): python3-portend-3.1.0-2.el9.noarch.rpm 1.7 MB/s | 16 kB 00:00 2026-02-26T03:27:48.061 INFO:teuthology.orchestra.run.trial104.stdout:(84/96): python3-pyOpenSSL-21.0.0-1.el9.noarch. 8.5 MB/s | 90 kB 00:00 2026-02-26T03:27:48.071 INFO:teuthology.orchestra.run.trial104.stdout:(85/96): python3-repoze-lru-0.7-16.el9.noarch.r 3.0 MB/s | 31 kB 00:00 2026-02-26T03:27:48.082 INFO:teuthology.orchestra.run.trial104.stdout:(86/96): python3-routes-2.5.1-5.el9.noarch.rpm 16 MB/s | 188 kB 00:00 2026-02-26T03:27:48.093 INFO:teuthology.orchestra.run.trial104.stdout:(87/96): python3-rsa-4.9-2.el9.noarch.rpm 5.5 MB/s | 59 kB 00:00 2026-02-26T03:27:48.105 INFO:teuthology.orchestra.run.trial104.stdout:(88/96): python3-saml-1.16.0-1.el9.noarch.rpm 10 MB/s | 125 kB 00:00 2026-02-26T03:27:48.115 INFO:teuthology.orchestra.run.trial104.stdout:(89/96): python3-tempora-5.0.0-2.el9.noarch.rpm 3.4 MB/s | 36 kB 00:00 2026-02-26T03:27:48.126 INFO:teuthology.orchestra.run.trial104.stdout:(90/96): python3-typing-extensions-4.15.0-1.el9 7.9 MB/s | 86 kB 00:00 2026-02-26T03:27:48.137 INFO:teuthology.orchestra.run.trial104.stdout:(91/96): python3-websocket-client-1.2.3-2.el9.n 8.4 MB/s | 90 kB 00:00 2026-02-26T03:27:48.148 INFO:teuthology.orchestra.run.trial104.stdout:(92/96): python3-xmlsec-1.3.13-1.el9.x86_64.rpm 4.2 MB/s | 48 kB 00:00 2026-02-26T03:27:48.158 INFO:teuthology.orchestra.run.trial104.stdout:(93/96): python3-xmltodict-0.12.0-15.el9.noarch 2.3 MB/s | 22 kB 00:00 2026-02-26T03:27:48.167 INFO:teuthology.orchestra.run.trial104.stdout:(94/96): python3-zc-lockfile-2.0-10.el9.noarch. 2.0 MB/s | 20 kB 00:00 2026-02-26T03:29:12.221 INFO:teuthology.orchestra.run.trial104.stdout:(95/96): ceph-mgr-dashboard-20.3.0-5312.gabffb6 428 kB/s | 51 MB 02:01 2026-02-26T03:30:50.743 INFO:teuthology.orchestra.run.trial104.stdout:(96/96): ceph-osd-crimson-20.3.0-5312.gabffb617 386 kB/s | 87 MB 03:49 2026-02-26T03:30:50.745 INFO:teuthology.orchestra.run.trial104.stdout:-------------------------------------------------------------------------------- 2026-02-26T03:30:50.745 INFO:teuthology.orchestra.run.trial104.stdout:Total 987 kB/s | 225 MB 03:53 2026-02-26T03:30:51.115 INFO:teuthology.orchestra.run.trial104.stdout:Running transaction check 2026-02-26T03:30:51.156 INFO:teuthology.orchestra.run.trial104.stdout:Transaction check succeeded. 2026-02-26T03:30:51.156 INFO:teuthology.orchestra.run.trial104.stdout:Running transaction test 2026-02-26T03:30:51.957 INFO:teuthology.orchestra.run.trial104.stdout:Transaction test succeeded. 2026-02-26T03:30:51.957 INFO:teuthology.orchestra.run.trial104.stdout:Running transaction 2026-02-26T03:30:52.786 INFO:teuthology.orchestra.run.trial104.stdout: Preparing : 1/1 2026-02-26T03:30:52.788 INFO:teuthology.orchestra.run.trial104.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/96 2026-02-26T03:30:52.792 INFO:teuthology.orchestra.run.trial104.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/96 2026-02-26T03:30:52.813 INFO:teuthology.orchestra.run.trial104.stdout: Installing : xmlsec1-1.2.29-13.el9.x86_64 3/96 2026-02-26T03:30:52.847 INFO:teuthology.orchestra.run.trial104.stdout: Installing : python3-pyasn1-0.4.8-7.el9.noarch 4/96 2026-02-26T03:30:52.854 INFO:teuthology.orchestra.run.trial104.stdout: Installing : python3-lxml-4.6.5-3.el9.x86_64 5/96 2026-02-26T03:30:52.856 INFO:teuthology.orchestra.run.trial104.stdout: Installing : lua-5.4.4-4.el9.x86_64 6/96 2026-02-26T03:30:52.861 INFO:teuthology.orchestra.run.trial104.stdout: Installing : flexiblas-3.0.4-9.el9.x86_64 7/96 2026-02-26T03:30:52.884 INFO:teuthology.orchestra.run.trial104.stdout: Installing : unzip-6.0-59.el9.x86_64 8/96 2026-02-26T03:30:52.897 INFO:teuthology.orchestra.run.trial104.stdout: Installing : python3-urllib3-1.26.5-7.el9.noarch 9/96 2026-02-26T03:30:52.901 INFO:teuthology.orchestra.run.trial104.stdout: Installing : python3-requests-2.25.1-10.el9.noarch 10/96 2026-02-26T03:30:52.906 INFO:teuthology.orchestra.run.trial104.stdout: Installing : libquadmath-11.5.0-14.el9.x86_64 11/96 2026-02-26T03:30:52.908 INFO:teuthology.orchestra.run.trial104.stdout: Installing : libgfortran-11.5.0-14.el9.x86_64 12/96 2026-02-26T03:30:52.919 INFO:teuthology.orchestra.run.trial104.stdout: Installing : ledmon-libs-1.1.0-3.el9.x86_64 13/96 2026-02-26T03:30:52.925 INFO:teuthology.orchestra.run.trial104.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 14/96 2026-02-26T03:30:52.949 INFO:teuthology.orchestra.run.trial104.stdout: Installing : zip-3.0-35.el9.x86_64 15/96 2026-02-26T03:30:52.953 INFO:teuthology.orchestra.run.trial104.stdout: Installing : luarocks-3.9.2-5.el9.noarch 16/96 2026-02-26T03:30:53.003 INFO:teuthology.orchestra.run.trial104.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 17/96 2026-02-26T03:30:53.016 INFO:teuthology.orchestra.run.trial104.stdout: Installing : python3-pyasn1-modules-0.4.8-7.el9.noarch 18/96 2026-02-26T03:30:53.018 INFO:teuthology.orchestra.run.trial104.stdout: Installing : python3-rsa-4.9-2.el9.noarch 19/96 2026-02-26T03:30:53.023 INFO:teuthology.orchestra.run.trial104.stdout: Installing : xmlsec1-openssl-1.2.29-13.el9.x86_64 20/96 2026-02-26T03:30:53.028 INFO:teuthology.orchestra.run.trial104.stdout: Installing : python3-xmlsec-1.3.13-1.el9.x86_64 21/96 2026-02-26T03:30:53.032 INFO:teuthology.orchestra.run.trial104.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 22/96 2026-02-26T03:30:53.035 INFO:teuthology.orchestra.run.trial104.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 23/96 2026-02-26T03:30:53.050 INFO:teuthology.orchestra.run.trial104.stdout: Installing : python3-xmltodict-0.12.0-15.el9.noarch 24/96 2026-02-26T03:30:53.054 INFO:teuthology.orchestra.run.trial104.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 25/96 2026-02-26T03:30:53.059 INFO:teuthology.orchestra.run.trial104.stdout: Installing : python3-typing-extensions-4.15.0-1.el9.noarch 26/96 2026-02-26T03:30:53.070 INFO:teuthology.orchestra.run.trial104.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 27/96 2026-02-26T03:30:53.079 INFO:teuthology.orchestra.run.trial104.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 28/96 2026-02-26T03:30:53.085 INFO:teuthology.orchestra.run.trial104.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 29/96 2026-02-26T03:30:53.105 INFO:teuthology.orchestra.run.trial104.stdout: Installing : python3-msgpack-1.0.3-2.el9.x86_64 30/96 2026-02-26T03:30:53.116 INFO:teuthology.orchestra.run.trial104.stdout: Installing : python3-influxdb-5.3.1-1.el9.noarch 31/96 2026-02-26T03:30:53.131 INFO:teuthology.orchestra.run.trial104.stdout: Installing : python3-isodate-0.6.1-3.el9.noarch 32/96 2026-02-26T03:30:53.136 INFO:teuthology.orchestra.run.trial104.stdout: Installing : python3-saml-1.16.0-1.el9.noarch 33/96 2026-02-26T03:30:53.143 INFO:teuthology.orchestra.run.trial104.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 34/96 2026-02-26T03:30:53.183 INFO:teuthology.orchestra.run.trial104.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 35/96 2026-02-26T03:30:53.504 INFO:teuthology.orchestra.run.trial104.stdout: Installing : python3-google-auth-1:2.45.0-1.el9.noarch 36/96 2026-02-26T03:30:53.513 INFO:teuthology.orchestra.run.trial104.stdout: Installing : python3-kubernetes-1:26.1.0-3.el9.noarch 37/96 2026-02-26T03:30:53.517 INFO:teuthology.orchestra.run.trial104.stdout: Installing : python3-backports-tarfile-1.2.0-1.el9.noarch 38/96 2026-02-26T03:30:53.523 INFO:teuthology.orchestra.run.trial104.stdout: Installing : python3-jaraco-context-6.0.1-3.el9.noarch 39/96 2026-02-26T03:30:53.525 INFO:teuthology.orchestra.run.trial104.stdout: Installing : python3-autocommand-2.2.2-8.el9.noarch 40/96 2026-02-26T03:30:53.552 INFO:teuthology.orchestra.run.trial104.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 41/96 2026-02-26T03:30:53.596 INFO:teuthology.orchestra.run.trial104.stdout: Installing : abseil-cpp-20211102.0-4.el9.x86_64 42/96 2026-02-26T03:30:53.604 INFO:teuthology.orchestra.run.trial104.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 43/96 2026-02-26T03:30:53.610 INFO:teuthology.orchestra.run.trial104.stdout: Installing : protobuf-compiler-3.14.0-17.el9.x86_64 44/96 2026-02-26T03:30:53.614 INFO:teuthology.orchestra.run.trial104.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 45/96 2026-02-26T03:30:53.619 INFO:teuthology.orchestra.run.trial104.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 46/96 2026-02-26T03:30:53.623 INFO:teuthology.orchestra.run.trial104.stdout: Installing : python3-jaraco-text-4.0.0-2.el9.noarch 47/96 2026-02-26T03:30:53.630 INFO:teuthology.orchestra.run.trial104.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 48/96 2026-02-26T03:30:53.633 INFO:teuthology.orchestra.run.trial104.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 49/96 2026-02-26T03:30:53.661 INFO:teuthology.orchestra.run.trial104.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 50/96 2026-02-26T03:30:53.671 INFO:teuthology.orchestra.run.trial104.stdout: Installing : python3-protobuf-3.14.0-17.el9.noarch 51/96 2026-02-26T03:30:53.677 INFO:teuthology.orchestra.run.trial104.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 52/96 2026-02-26T03:30:53.683 INFO:teuthology.orchestra.run.trial104.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 53/96 2026-02-26T03:30:53.718 INFO:teuthology.orchestra.run.trial104.stdout: Installing : python3-jmespath-1.0.1-1.el9.noarch 54/96 2026-02-26T03:30:53.940 INFO:teuthology.orchestra.run.trial104.stdout: Installing : python3-devel-3.9.25-3.el9.x86_64 55/96 2026-02-26T03:30:53.963 INFO:teuthology.orchestra.run.trial104.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 56/96 2026-02-26T03:30:53.967 INFO:teuthology.orchestra.run.trial104.stdout: Installing : python3-jinja2-2.11.3-8.el9.noarch 57/96 2026-02-26T03:30:54.026 INFO:teuthology.orchestra.run.trial104.stdout: Installing : openblas-0.3.29-1.el9.x86_64 58/96 2026-02-26T03:30:54.027 INFO:teuthology.orchestra.run.trial104.stdout: Installing : openblas-openmp-0.3.29-1.el9.x86_64 59/96 2026-02-26T03:30:54.049 INFO:teuthology.orchestra.run.trial104.stdout: Installing : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 60/96 2026-02-26T03:30:54.372 INFO:teuthology.orchestra.run.trial104.stdout: Installing : flexiblas-netlib-3.0.4-9.el9.x86_64 61/96 2026-02-26T03:30:54.445 INFO:teuthology.orchestra.run.trial104.stdout: Installing : python3-numpy-1:1.23.5-2.el9.x86_64 62/96 2026-02-26T03:30:55.114 INFO:teuthology.orchestra.run.trial104.stdout: Installing : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 63/96 2026-02-26T03:30:55.131 INFO:teuthology.orchestra.run.trial104.stdout: Installing : python3-scipy-1.9.3-2.el9.x86_64 64/96 2026-02-26T03:30:55.141 INFO:teuthology.orchestra.run.trial104.stdout: Installing : python3-pyparsing-2.4.7-9.el9.noarch 65/96 2026-02-26T03:30:55.147 INFO:teuthology.orchestra.run.trial104.stdout: Installing : python3-packaging-20.9-5.el9.noarch 66/96 2026-02-26T03:30:55.161 INFO:teuthology.orchestra.run.trial104.stdout: Installing : python3-ply-3.11-14.el9.noarch 67/96 2026-02-26T03:30:55.176 INFO:teuthology.orchestra.run.trial104.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 68/96 2026-02-26T03:30:55.252 INFO:teuthology.orchestra.run.trial104.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 69/96 2026-02-26T03:30:55.260 INFO:teuthology.orchestra.run.trial104.stdout: Installing : python3-cryptography-36.0.1-5.el9.x86_64 70/96 2026-02-26T03:30:55.284 INFO:teuthology.orchestra.run.trial104.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 71/96 2026-02-26T03:30:55.313 INFO:teuthology.orchestra.run.trial104.stdout: Installing : python3-cheroot-10.0.1-4.el9.noarch 72/96 2026-02-26T03:30:55.363 INFO:teuthology.orchestra.run.trial104.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 73/96 2026-02-26T03:30:55.369 INFO:teuthology.orchestra.run.trial104.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 74/96 2026-02-26T03:30:55.373 INFO:teuthology.orchestra.run.trial104.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 75/96 2026-02-26T03:30:55.374 INFO:teuthology.orchestra.run.trial104.stdout: Installing : libconfig-1.7.2-9.el9.x86_64 76/96 2026-02-26T03:30:55.392 INFO:teuthology.orchestra.run.trial104.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-26T03:30:55.392 INFO:teuthology.orchestra.run.trial104.stdout:Creating group 'libstoragemgmt' with GID 991. 2026-02-26T03:30:55.392 INFO:teuthology.orchestra.run.trial104.stdout:Creating user 'libstoragemgmt' (daemon account for libstoragemgmt) with UID 991 and GID 991. 2026-02-26T03:30:55.393 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:30:55.400 INFO:teuthology.orchestra.run.trial104.stdout: Installing : libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-26T03:30:55.426 INFO:teuthology.orchestra.run.trial104.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-26T03:30:55.426 INFO:teuthology.orchestra.run.trial104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/libstoragemgmt.service → /usr/lib/systemd/system/libstoragemgmt.service. 2026-02-26T03:30:55.427 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:30:55.462 INFO:teuthology.orchestra.run.trial104.stdout: Installing : python3-libstoragemgmt-1.10.1-1.el9.x86_64 78/96 2026-02-26T03:30:56.073 INFO:teuthology.orchestra.run.trial104.stdout: Installing : cryptsetup-2.8.1-3.el9.x86_64 79/96 2026-02-26T03:31:01.546 INFO:teuthology.orchestra.run.trial104.stdout: Installing : ceph-osd-classic-2:20.3.0-5312.gabffb617.el9.x86_6 80/96 2026-02-26T03:31:01.609 INFO:teuthology.orchestra.run.trial104.stdout: Installing : ceph-osd-crimson-2:20.3.0-5312.gabffb617.el9.x86_6 81/96 2026-02-26T03:31:01.611 INFO:teuthology.orchestra.run.trial104.stdout: Installing : ceph-volume-2:20.3.0-5312.gabffb617.el9.noarch 82/96 2026-02-26T03:31:01.625 INFO:teuthology.orchestra.run.trial104.stdout: Running scriptlet: ceph-volume-2:20.3.0-5312.gabffb617.el9.noarch 82/96 2026-02-26T03:31:01.625 INFO:teuthology.orchestra.run.trial104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-26T03:31:01.625 INFO:teuthology.orchestra.run.trial104.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2026-02-26T03:31:01.625 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:31:01.635 INFO:teuthology.orchestra.run.trial104.stdout: Installing : ceph-osd-2:20.3.0-5312.gabffb617.el9.x86_64 83/96 2026-02-26T03:31:01.659 INFO:teuthology.orchestra.run.trial104.stdout: Running scriptlet: ceph-osd-2:20.3.0-5312.gabffb617.el9.x86_64 83/96 2026-02-26T03:31:01.659 INFO:teuthology.orchestra.run.trial104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-26T03:31:01.660 INFO:teuthology.orchestra.run.trial104.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2026-02-26T03:31:01.660 INFO:teuthology.orchestra.run.trial104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-26T03:31:01.660 INFO:teuthology.orchestra.run.trial104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-26T03:31:01.660 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:31:01.718 INFO:teuthology.orchestra.run.trial104.stdout: Running scriptlet: cephadm-2:20.3.0-5312.gabffb617.el9.noarch 84/96 2026-02-26T03:31:01.720 INFO:teuthology.orchestra.run.trial104.stdout: Installing : cephadm-2:20.3.0-5312.gabffb617.el9.noarch 84/96 2026-02-26T03:31:01.729 INFO:teuthology.orchestra.run.trial104.stdout: Installing : ceph-prometheus-alerts-2:20.3.0-5312.gabffb617.el9 85/96 2026-02-26T03:31:01.752 INFO:teuthology.orchestra.run.trial104.stdout: Installing : ceph-grafana-dashboards-2:20.3.0-5312.gabffb617.el 86/96 2026-02-26T03:31:01.754 INFO:teuthology.orchestra.run.trial104.stdout: Installing : ceph-mgr-cephadm-2:20.3.0-5312.gabffb617.el9.noarc 87/96 2026-02-26T03:31:04.756 INFO:teuthology.orchestra.run.trial104.stdout: Running scriptlet: ceph-mgr-cephadm-2:20.3.0-5312.gabffb617.el9.noarc 87/96 2026-02-26T03:31:04.764 INFO:teuthology.orchestra.run.trial104.stdout: Installing : ceph-mgr-dashboard-2:20.3.0-5312.gabffb617.el9.noa 88/96 2026-02-26T03:31:05.230 INFO:teuthology.orchestra.run.trial104.stdout: Running scriptlet: ceph-mgr-dashboard-2:20.3.0-5312.gabffb617.el9.noa 88/96 2026-02-26T03:31:05.231 INFO:teuthology.orchestra.run.trial104.stdout: Installing : ceph-mgr-diskprediction-local-2:20.3.0-5312.gabffb 89/96 2026-02-26T03:31:05.242 INFO:teuthology.orchestra.run.trial104.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:20.3.0-5312.gabffb 89/96 2026-02-26T03:31:05.243 INFO:teuthology.orchestra.run.trial104.stdout: Installing : ceph-mgr-k8sevents-2:20.3.0-5312.gabffb617.el9.noa 90/96 2026-02-26T03:31:05.305 INFO:teuthology.orchestra.run.trial104.stdout: Running scriptlet: ceph-mgr-k8sevents-2:20.3.0-5312.gabffb617.el9.noa 90/96 2026-02-26T03:31:05.347 INFO:teuthology.orchestra.run.trial104.stdout: Installing : ceph-mgr-modules-core-2:20.3.0-5312.gabffb617.el9. 91/96 2026-02-26T03:31:05.348 INFO:teuthology.orchestra.run.trial104.stdout: Installing : ceph-mgr-2:20.3.0-5312.gabffb617.el9.x86_64 92/96 2026-02-26T03:31:05.369 INFO:teuthology.orchestra.run.trial104.stdout: Running scriptlet: ceph-mgr-2:20.3.0-5312.gabffb617.el9.x86_64 92/96 2026-02-26T03:31:05.369 INFO:teuthology.orchestra.run.trial104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-26T03:31:05.369 INFO:teuthology.orchestra.run.trial104.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2026-02-26T03:31:05.369 INFO:teuthology.orchestra.run.trial104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-26T03:31:05.370 INFO:teuthology.orchestra.run.trial104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-26T03:31:05.370 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:31:05.382 INFO:teuthology.orchestra.run.trial104.stdout: Installing : ceph-mgr-rook-2:20.3.0-5312.gabffb617.el9.noarch 93/96 2026-02-26T03:31:05.599 INFO:teuthology.orchestra.run.trial104.stdout: Running scriptlet: ceph-mgr-rook-2:20.3.0-5312.gabffb617.el9.noarch 93/96 2026-02-26T03:31:05.600 INFO:teuthology.orchestra.run.trial104.stdout: Installing : ceph-mon-2:20.3.0-5312.gabffb617.el9.x86_64 94/96 2026-02-26T03:31:05.621 INFO:teuthology.orchestra.run.trial104.stdout: Running scriptlet: ceph-mon-2:20.3.0-5312.gabffb617.el9.x86_64 94/96 2026-02-26T03:31:05.621 INFO:teuthology.orchestra.run.trial104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-26T03:31:05.621 INFO:teuthology.orchestra.run.trial104.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2026-02-26T03:31:05.621 INFO:teuthology.orchestra.run.trial104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-26T03:31:05.621 INFO:teuthology.orchestra.run.trial104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-26T03:31:05.621 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:31:05.707 INFO:teuthology.orchestra.run.trial104.stdout: Installing : ceph-mds-2:20.3.0-5312.gabffb617.el9.x86_64 95/96 2026-02-26T03:31:05.727 INFO:teuthology.orchestra.run.trial104.stdout: Running scriptlet: ceph-mds-2:20.3.0-5312.gabffb617.el9.x86_64 95/96 2026-02-26T03:31:05.727 INFO:teuthology.orchestra.run.trial104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-26T03:31:05.728 INFO:teuthology.orchestra.run.trial104.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2026-02-26T03:31:05.728 INFO:teuthology.orchestra.run.trial104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-26T03:31:05.728 INFO:teuthology.orchestra.run.trial104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-26T03:31:05.728 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:31:05.729 INFO:teuthology.orchestra.run.trial104.stdout: Installing : ceph-2:20.3.0-5312.gabffb617.el9.x86_64 96/96 2026-02-26T03:31:05.734 INFO:teuthology.orchestra.run.trial104.stdout: Running scriptlet: ceph-osd-classic-2:20.3.0-5312.gabffb617.el9.x86_6 96/96 2026-02-26T03:31:05.759 INFO:teuthology.orchestra.run.trial104.stdout: Running scriptlet: ceph-osd-crimson-2:20.3.0-5312.gabffb617.el9.x86_6 96/96 2026-02-26T03:31:07.205 INFO:teuthology.orchestra.run.trial104.stdout: Running scriptlet: ceph-2:20.3.0-5312.gabffb617.el9.x86_64 96/96 2026-02-26T03:31:07.206 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : ceph-2:20.3.0-5312.gabffb617.el9.x86_64 1/96 2026-02-26T03:31:07.206 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : ceph-mds-2:20.3.0-5312.gabffb617.el9.x86_64 2/96 2026-02-26T03:31:07.206 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : ceph-mgr-2:20.3.0-5312.gabffb617.el9.x86_64 3/96 2026-02-26T03:31:07.206 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : ceph-mon-2:20.3.0-5312.gabffb617.el9.x86_64 4/96 2026-02-26T03:31:07.206 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : ceph-osd-2:20.3.0-5312.gabffb617.el9.x86_64 5/96 2026-02-26T03:31:07.206 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : ceph-osd-classic-2:20.3.0-5312.gabffb617.el9.x86_6 6/96 2026-02-26T03:31:07.206 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : ceph-osd-crimson-2:20.3.0-5312.gabffb617.el9.x86_6 7/96 2026-02-26T03:31:07.206 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : ceph-grafana-dashboards-2:20.3.0-5312.gabffb617.el 8/96 2026-02-26T03:31:07.206 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : ceph-mgr-cephadm-2:20.3.0-5312.gabffb617.el9.noarc 9/96 2026-02-26T03:31:07.206 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : ceph-mgr-dashboard-2:20.3.0-5312.gabffb617.el9.noa 10/96 2026-02-26T03:31:07.207 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : ceph-mgr-diskprediction-local-2:20.3.0-5312.gabffb 11/96 2026-02-26T03:31:07.207 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : ceph-mgr-k8sevents-2:20.3.0-5312.gabffb617.el9.noa 12/96 2026-02-26T03:31:07.207 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : ceph-mgr-modules-core-2:20.3.0-5312.gabffb617.el9. 13/96 2026-02-26T03:31:07.207 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : ceph-mgr-rook-2:20.3.0-5312.gabffb617.el9.noarch 14/96 2026-02-26T03:31:07.207 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : ceph-prometheus-alerts-2:20.3.0-5312.gabffb617.el9 15/96 2026-02-26T03:31:07.207 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : ceph-volume-2:20.3.0-5312.gabffb617.el9.noarch 16/96 2026-02-26T03:31:07.207 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : cephadm-2:20.3.0-5312.gabffb617.el9.noarch 17/96 2026-02-26T03:31:07.207 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : cryptsetup-2.8.1-3.el9.x86_64 18/96 2026-02-26T03:31:07.207 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : ledmon-libs-1.1.0-3.el9.x86_64 19/96 2026-02-26T03:31:07.207 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : libconfig-1.7.2-9.el9.x86_64 20/96 2026-02-26T03:31:07.207 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : libgfortran-11.5.0-14.el9.x86_64 21/96 2026-02-26T03:31:07.207 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : libquadmath-11.5.0-14.el9.x86_64 22/96 2026-02-26T03:31:07.208 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 23/96 2026-02-26T03:31:07.208 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : python3-cryptography-36.0.1-5.el9.x86_64 24/96 2026-02-26T03:31:07.208 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : python3-ply-3.11-14.el9.noarch 25/96 2026-02-26T03:31:07.208 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 26/96 2026-02-26T03:31:07.208 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : python3-pyparsing-2.4.7-9.el9.noarch 27/96 2026-02-26T03:31:07.208 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : python3-requests-2.25.1-10.el9.noarch 28/96 2026-02-26T03:31:07.208 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : python3-urllib3-1.26.5-7.el9.noarch 29/96 2026-02-26T03:31:07.208 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : unzip-6.0-59.el9.x86_64 30/96 2026-02-26T03:31:07.208 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : zip-3.0-35.el9.x86_64 31/96 2026-02-26T03:31:07.208 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : flexiblas-3.0.4-9.el9.x86_64 32/96 2026-02-26T03:31:07.208 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : flexiblas-netlib-3.0.4-9.el9.x86_64 33/96 2026-02-26T03:31:07.208 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 34/96 2026-02-26T03:31:07.208 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : libstoragemgmt-1.10.1-1.el9.x86_64 35/96 2026-02-26T03:31:07.209 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : lua-5.4.4-4.el9.x86_64 36/96 2026-02-26T03:31:07.209 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : openblas-0.3.29-1.el9.x86_64 37/96 2026-02-26T03:31:07.209 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : openblas-openmp-0.3.29-1.el9.x86_64 38/96 2026-02-26T03:31:07.209 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 39/96 2026-02-26T03:31:07.209 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : python3-devel-3.9.25-3.el9.x86_64 40/96 2026-02-26T03:31:07.209 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : python3-jinja2-2.11.3-8.el9.noarch 41/96 2026-02-26T03:31:07.209 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : python3-jmespath-1.0.1-1.el9.noarch 42/96 2026-02-26T03:31:07.209 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : python3-libstoragemgmt-1.10.1-1.el9.x86_64 43/96 2026-02-26T03:31:07.209 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : python3-lxml-4.6.5-3.el9.x86_64 44/96 2026-02-26T03:31:07.209 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 45/96 2026-02-26T03:31:07.209 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : python3-numpy-1:1.23.5-2.el9.x86_64 46/96 2026-02-26T03:31:07.209 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 47/96 2026-02-26T03:31:07.210 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 48/96 2026-02-26T03:31:07.210 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : python3-protobuf-3.14.0-17.el9.noarch 49/96 2026-02-26T03:31:07.210 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : python3-pyasn1-0.4.8-7.el9.noarch 50/96 2026-02-26T03:31:07.210 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : python3-pyasn1-modules-0.4.8-7.el9.noarch 51/96 2026-02-26T03:31:07.210 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 52/96 2026-02-26T03:31:07.210 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : python3-scipy-1.9.3-2.el9.x86_64 53/96 2026-02-26T03:31:07.210 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 54/96 2026-02-26T03:31:07.210 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : xmlsec1-1.2.29-13.el9.x86_64 55/96 2026-02-26T03:31:07.210 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : xmlsec1-openssl-1.2.29-13.el9.x86_64 56/96 2026-02-26T03:31:07.210 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 57/96 2026-02-26T03:31:07.210 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : protobuf-compiler-3.14.0-17.el9.x86_64 58/96 2026-02-26T03:31:07.210 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : abseil-cpp-20211102.0-4.el9.x86_64 59/96 2026-02-26T03:31:07.210 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 60/96 2026-02-26T03:31:07.210 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 61/96 2026-02-26T03:31:07.211 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 62/96 2026-02-26T03:31:07.211 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : python3-autocommand-2.2.2-8.el9.noarch 63/96 2026-02-26T03:31:07.211 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : python3-backports-tarfile-1.2.0-1.el9.noarch 64/96 2026-02-26T03:31:07.211 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 65/96 2026-02-26T03:31:07.211 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 66/96 2026-02-26T03:31:07.211 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 67/96 2026-02-26T03:31:07.211 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : python3-cheroot-10.0.1-4.el9.noarch 68/96 2026-02-26T03:31:07.211 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 69/96 2026-02-26T03:31:07.211 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : python3-google-auth-1:2.45.0-1.el9.noarch 70/96 2026-02-26T03:31:07.211 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 71/96 2026-02-26T03:31:07.211 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 72/96 2026-02-26T03:31:07.211 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : python3-influxdb-5.3.1-1.el9.noarch 73/96 2026-02-26T03:31:07.211 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : python3-isodate-0.6.1-3.el9.noarch 74/96 2026-02-26T03:31:07.212 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 75/96 2026-02-26T03:31:07.212 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 76/96 2026-02-26T03:31:07.212 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 77/96 2026-02-26T03:31:07.212 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : python3-jaraco-context-6.0.1-3.el9.noarch 78/96 2026-02-26T03:31:07.212 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 79/96 2026-02-26T03:31:07.212 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : python3-jaraco-text-4.0.0-2.el9.noarch 80/96 2026-02-26T03:31:07.212 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : python3-kubernetes-1:26.1.0-3.el9.noarch 81/96 2026-02-26T03:31:07.212 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 82/96 2026-02-26T03:31:07.212 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : python3-msgpack-1.0.3-2.el9.x86_64 83/96 2026-02-26T03:31:07.212 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 84/96 2026-02-26T03:31:07.212 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 85/96 2026-02-26T03:31:07.212 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/96 2026-02-26T03:31:07.212 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 87/96 2026-02-26T03:31:07.213 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 88/96 2026-02-26T03:31:07.213 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 89/96 2026-02-26T03:31:07.213 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : python3-saml-1.16.0-1.el9.noarch 90/96 2026-02-26T03:31:07.213 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 91/96 2026-02-26T03:31:07.213 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : python3-typing-extensions-4.15.0-1.el9.noarch 92/96 2026-02-26T03:31:07.213 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 93/96 2026-02-26T03:31:07.213 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : python3-xmlsec-1.3.13-1.el9.x86_64 94/96 2026-02-26T03:31:07.213 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : python3-xmltodict-0.12.0-15.el9.noarch 95/96 2026-02-26T03:31:07.273 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 96/96 2026-02-26T03:31:07.274 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:31:07.274 INFO:teuthology.orchestra.run.trial104.stdout:Installed: 2026-02-26T03:31:07.274 INFO:teuthology.orchestra.run.trial104.stdout: abseil-cpp-20211102.0-4.el9.x86_64 2026-02-26T03:31:07.274 INFO:teuthology.orchestra.run.trial104.stdout: ceph-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:31:07.274 INFO:teuthology.orchestra.run.trial104.stdout: ceph-grafana-dashboards-2:20.3.0-5312.gabffb617.el9.noarch 2026-02-26T03:31:07.274 INFO:teuthology.orchestra.run.trial104.stdout: ceph-mds-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:31:07.274 INFO:teuthology.orchestra.run.trial104.stdout: ceph-mgr-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:31:07.274 INFO:teuthology.orchestra.run.trial104.stdout: ceph-mgr-cephadm-2:20.3.0-5312.gabffb617.el9.noarch 2026-02-26T03:31:07.274 INFO:teuthology.orchestra.run.trial104.stdout: ceph-mgr-dashboard-2:20.3.0-5312.gabffb617.el9.noarch 2026-02-26T03:31:07.275 INFO:teuthology.orchestra.run.trial104.stdout: ceph-mgr-diskprediction-local-2:20.3.0-5312.gabffb617.el9.noarch 2026-02-26T03:31:07.275 INFO:teuthology.orchestra.run.trial104.stdout: ceph-mgr-k8sevents-2:20.3.0-5312.gabffb617.el9.noarch 2026-02-26T03:31:07.275 INFO:teuthology.orchestra.run.trial104.stdout: ceph-mgr-modules-core-2:20.3.0-5312.gabffb617.el9.noarch 2026-02-26T03:31:07.275 INFO:teuthology.orchestra.run.trial104.stdout: ceph-mgr-rook-2:20.3.0-5312.gabffb617.el9.noarch 2026-02-26T03:31:07.275 INFO:teuthology.orchestra.run.trial104.stdout: ceph-mon-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:31:07.275 INFO:teuthology.orchestra.run.trial104.stdout: ceph-osd-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:31:07.275 INFO:teuthology.orchestra.run.trial104.stdout: ceph-osd-classic-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:31:07.275 INFO:teuthology.orchestra.run.trial104.stdout: ceph-osd-crimson-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:31:07.275 INFO:teuthology.orchestra.run.trial104.stdout: ceph-prometheus-alerts-2:20.3.0-5312.gabffb617.el9.noarch 2026-02-26T03:31:07.275 INFO:teuthology.orchestra.run.trial104.stdout: ceph-volume-2:20.3.0-5312.gabffb617.el9.noarch 2026-02-26T03:31:07.275 INFO:teuthology.orchestra.run.trial104.stdout: cephadm-2:20.3.0-5312.gabffb617.el9.noarch 2026-02-26T03:31:07.275 INFO:teuthology.orchestra.run.trial104.stdout: cryptsetup-2.8.1-3.el9.x86_64 2026-02-26T03:31:07.276 INFO:teuthology.orchestra.run.trial104.stdout: flexiblas-3.0.4-9.el9.x86_64 2026-02-26T03:31:07.276 INFO:teuthology.orchestra.run.trial104.stdout: flexiblas-netlib-3.0.4-9.el9.x86_64 2026-02-26T03:31:07.276 INFO:teuthology.orchestra.run.trial104.stdout: flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 2026-02-26T03:31:07.276 INFO:teuthology.orchestra.run.trial104.stdout: grpc-data-1.46.7-10.el9.noarch 2026-02-26T03:31:07.276 INFO:teuthology.orchestra.run.trial104.stdout: ledmon-libs-1.1.0-3.el9.x86_64 2026-02-26T03:31:07.276 INFO:teuthology.orchestra.run.trial104.stdout: libconfig-1.7.2-9.el9.x86_64 2026-02-26T03:31:07.276 INFO:teuthology.orchestra.run.trial104.stdout: libgfortran-11.5.0-14.el9.x86_64 2026-02-26T03:31:07.276 INFO:teuthology.orchestra.run.trial104.stdout: libquadmath-11.5.0-14.el9.x86_64 2026-02-26T03:31:07.276 INFO:teuthology.orchestra.run.trial104.stdout: libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-26T03:31:07.276 INFO:teuthology.orchestra.run.trial104.stdout: lua-5.4.4-4.el9.x86_64 2026-02-26T03:31:07.276 INFO:teuthology.orchestra.run.trial104.stdout: lua-devel-5.4.4-4.el9.x86_64 2026-02-26T03:31:07.276 INFO:teuthology.orchestra.run.trial104.stdout: luarocks-3.9.2-5.el9.noarch 2026-02-26T03:31:07.277 INFO:teuthology.orchestra.run.trial104.stdout: openblas-0.3.29-1.el9.x86_64 2026-02-26T03:31:07.277 INFO:teuthology.orchestra.run.trial104.stdout: openblas-openmp-0.3.29-1.el9.x86_64 2026-02-26T03:31:07.277 INFO:teuthology.orchestra.run.trial104.stdout: protobuf-compiler-3.14.0-17.el9.x86_64 2026-02-26T03:31:07.277 INFO:teuthology.orchestra.run.trial104.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2026-02-26T03:31:07.277 INFO:teuthology.orchestra.run.trial104.stdout: python3-autocommand-2.2.2-8.el9.noarch 2026-02-26T03:31:07.277 INFO:teuthology.orchestra.run.trial104.stdout: python3-babel-2.9.1-2.el9.noarch 2026-02-26T03:31:07.277 INFO:teuthology.orchestra.run.trial104.stdout: python3-backports-tarfile-1.2.0-1.el9.noarch 2026-02-26T03:31:07.277 INFO:teuthology.orchestra.run.trial104.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2026-02-26T03:31:07.277 INFO:teuthology.orchestra.run.trial104.stdout: python3-cachetools-4.2.4-1.el9.noarch 2026-02-26T03:31:07.277 INFO:teuthology.orchestra.run.trial104.stdout: python3-certifi-2023.05.07-4.el9.noarch 2026-02-26T03:31:07.277 INFO:teuthology.orchestra.run.trial104.stdout: python3-cffi-1.14.5-5.el9.x86_64 2026-02-26T03:31:07.277 INFO:teuthology.orchestra.run.trial104.stdout: python3-cheroot-10.0.1-4.el9.noarch 2026-02-26T03:31:07.278 INFO:teuthology.orchestra.run.trial104.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2026-02-26T03:31:07.278 INFO:teuthology.orchestra.run.trial104.stdout: python3-cryptography-36.0.1-5.el9.x86_64 2026-02-26T03:31:07.278 INFO:teuthology.orchestra.run.trial104.stdout: python3-devel-3.9.25-3.el9.x86_64 2026-02-26T03:31:07.278 INFO:teuthology.orchestra.run.trial104.stdout: python3-google-auth-1:2.45.0-1.el9.noarch 2026-02-26T03:31:07.278 INFO:teuthology.orchestra.run.trial104.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2026-02-26T03:31:07.278 INFO:teuthology.orchestra.run.trial104.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2026-02-26T03:31:07.278 INFO:teuthology.orchestra.run.trial104.stdout: python3-influxdb-5.3.1-1.el9.noarch 2026-02-26T03:31:07.278 INFO:teuthology.orchestra.run.trial104.stdout: python3-isodate-0.6.1-3.el9.noarch 2026-02-26T03:31:07.278 INFO:teuthology.orchestra.run.trial104.stdout: python3-jaraco-8.2.1-3.el9.noarch 2026-02-26T03:31:07.278 INFO:teuthology.orchestra.run.trial104.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2026-02-26T03:31:07.278 INFO:teuthology.orchestra.run.trial104.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2026-02-26T03:31:07.278 INFO:teuthology.orchestra.run.trial104.stdout: python3-jaraco-context-6.0.1-3.el9.noarch 2026-02-26T03:31:07.278 INFO:teuthology.orchestra.run.trial104.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2026-02-26T03:31:07.279 INFO:teuthology.orchestra.run.trial104.stdout: python3-jaraco-text-4.0.0-2.el9.noarch 2026-02-26T03:31:07.279 INFO:teuthology.orchestra.run.trial104.stdout: python3-jinja2-2.11.3-8.el9.noarch 2026-02-26T03:31:07.279 INFO:teuthology.orchestra.run.trial104.stdout: python3-jmespath-1.0.1-1.el9.noarch 2026-02-26T03:31:07.279 INFO:teuthology.orchestra.run.trial104.stdout: python3-kubernetes-1:26.1.0-3.el9.noarch 2026-02-26T03:31:07.279 INFO:teuthology.orchestra.run.trial104.stdout: python3-libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-26T03:31:07.279 INFO:teuthology.orchestra.run.trial104.stdout: python3-lxml-4.6.5-3.el9.x86_64 2026-02-26T03:31:07.279 INFO:teuthology.orchestra.run.trial104.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2026-02-26T03:31:07.279 INFO:teuthology.orchestra.run.trial104.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2026-02-26T03:31:07.279 INFO:teuthology.orchestra.run.trial104.stdout: python3-msgpack-1.0.3-2.el9.x86_64 2026-02-26T03:31:07.279 INFO:teuthology.orchestra.run.trial104.stdout: python3-natsort-7.1.1-5.el9.noarch 2026-02-26T03:31:07.279 INFO:teuthology.orchestra.run.trial104.stdout: python3-numpy-1:1.23.5-2.el9.x86_64 2026-02-26T03:31:07.279 INFO:teuthology.orchestra.run.trial104.stdout: python3-numpy-f2py-1:1.23.5-2.el9.x86_64 2026-02-26T03:31:07.280 INFO:teuthology.orchestra.run.trial104.stdout: python3-packaging-20.9-5.el9.noarch 2026-02-26T03:31:07.280 INFO:teuthology.orchestra.run.trial104.stdout: python3-ply-3.11-14.el9.noarch 2026-02-26T03:31:07.280 INFO:teuthology.orchestra.run.trial104.stdout: python3-portend-3.1.0-2.el9.noarch 2026-02-26T03:31:07.280 INFO:teuthology.orchestra.run.trial104.stdout: python3-protobuf-3.14.0-17.el9.noarch 2026-02-26T03:31:07.280 INFO:teuthology.orchestra.run.trial104.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2026-02-26T03:31:07.280 INFO:teuthology.orchestra.run.trial104.stdout: python3-pyasn1-0.4.8-7.el9.noarch 2026-02-26T03:31:07.280 INFO:teuthology.orchestra.run.trial104.stdout: python3-pyasn1-modules-0.4.8-7.el9.noarch 2026-02-26T03:31:07.280 INFO:teuthology.orchestra.run.trial104.stdout: python3-pycparser-2.20-6.el9.noarch 2026-02-26T03:31:07.280 INFO:teuthology.orchestra.run.trial104.stdout: python3-pyparsing-2.4.7-9.el9.noarch 2026-02-26T03:31:07.280 INFO:teuthology.orchestra.run.trial104.stdout: python3-repoze-lru-0.7-16.el9.noarch 2026-02-26T03:31:07.280 INFO:teuthology.orchestra.run.trial104.stdout: python3-requests-2.25.1-10.el9.noarch 2026-02-26T03:31:07.280 INFO:teuthology.orchestra.run.trial104.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2026-02-26T03:31:07.280 INFO:teuthology.orchestra.run.trial104.stdout: python3-routes-2.5.1-5.el9.noarch 2026-02-26T03:31:07.281 INFO:teuthology.orchestra.run.trial104.stdout: python3-rsa-4.9-2.el9.noarch 2026-02-26T03:31:07.281 INFO:teuthology.orchestra.run.trial104.stdout: python3-saml-1.16.0-1.el9.noarch 2026-02-26T03:31:07.281 INFO:teuthology.orchestra.run.trial104.stdout: python3-scipy-1.9.3-2.el9.x86_64 2026-02-26T03:31:07.281 INFO:teuthology.orchestra.run.trial104.stdout: python3-tempora-5.0.0-2.el9.noarch 2026-02-26T03:31:07.281 INFO:teuthology.orchestra.run.trial104.stdout: python3-toml-0.10.2-6.el9.noarch 2026-02-26T03:31:07.281 INFO:teuthology.orchestra.run.trial104.stdout: python3-typing-extensions-4.15.0-1.el9.noarch 2026-02-26T03:31:07.281 INFO:teuthology.orchestra.run.trial104.stdout: python3-urllib3-1.26.5-7.el9.noarch 2026-02-26T03:31:07.281 INFO:teuthology.orchestra.run.trial104.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2026-02-26T03:31:07.281 INFO:teuthology.orchestra.run.trial104.stdout: python3-xmlsec-1.3.13-1.el9.x86_64 2026-02-26T03:31:07.281 INFO:teuthology.orchestra.run.trial104.stdout: python3-xmltodict-0.12.0-15.el9.noarch 2026-02-26T03:31:07.281 INFO:teuthology.orchestra.run.trial104.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2026-02-26T03:31:07.281 INFO:teuthology.orchestra.run.trial104.stdout: unzip-6.0-59.el9.x86_64 2026-02-26T03:31:07.282 INFO:teuthology.orchestra.run.trial104.stdout: xmlsec1-1.2.29-13.el9.x86_64 2026-02-26T03:31:07.282 INFO:teuthology.orchestra.run.trial104.stdout: xmlsec1-openssl-1.2.29-13.el9.x86_64 2026-02-26T03:31:07.282 INFO:teuthology.orchestra.run.trial104.stdout: zip-3.0-35.el9.x86_64 2026-02-26T03:31:07.282 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:31:07.282 INFO:teuthology.orchestra.run.trial104.stdout:Complete! 2026-02-26T03:31:07.342 DEBUG:teuthology.orchestra.run.trial104:> sudo yum -y install ceph-base 2026-02-26T03:31:07.664 INFO:teuthology.orchestra.run.trial104.stdout:Last metadata expiration check: 0:10:19 ago on Thu 26 Feb 2026 03:20:48 AM UTC. 2026-02-26T03:31:07.731 INFO:teuthology.orchestra.run.trial104.stdout:Package ceph-base-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:31:07.750 INFO:teuthology.orchestra.run.trial104.stdout:Dependencies resolved. 2026-02-26T03:31:07.751 INFO:teuthology.orchestra.run.trial104.stdout:Nothing to do. 2026-02-26T03:31:07.751 INFO:teuthology.orchestra.run.trial104.stdout:Complete! 2026-02-26T03:31:07.773 DEBUG:teuthology.orchestra.run.trial104:> sudo yum -y install cephadm 2026-02-26T03:31:08.093 INFO:teuthology.orchestra.run.trial104.stdout:Last metadata expiration check: 0:10:20 ago on Thu 26 Feb 2026 03:20:48 AM UTC. 2026-02-26T03:31:08.158 INFO:teuthology.orchestra.run.trial104.stdout:Package cephadm-2:20.3.0-5312.gabffb617.el9.noarch is already installed. 2026-02-26T03:31:08.177 INFO:teuthology.orchestra.run.trial104.stdout:Dependencies resolved. 2026-02-26T03:31:08.177 INFO:teuthology.orchestra.run.trial104.stdout:Nothing to do. 2026-02-26T03:31:08.178 INFO:teuthology.orchestra.run.trial104.stdout:Complete! 2026-02-26T03:31:08.200 DEBUG:teuthology.orchestra.run.trial104:> sudo yum -y install ceph-immutable-object-cache 2026-02-26T03:31:08.516 INFO:teuthology.orchestra.run.trial104.stdout:Last metadata expiration check: 0:10:20 ago on Thu 26 Feb 2026 03:20:48 AM UTC. 2026-02-26T03:31:08.600 INFO:teuthology.orchestra.run.trial104.stdout:Dependencies resolved. 2026-02-26T03:31:08.601 INFO:teuthology.orchestra.run.trial104.stdout:================================================================================ 2026-02-26T03:31:08.601 INFO:teuthology.orchestra.run.trial104.stdout: Package Arch Version Repo Size 2026-02-26T03:31:08.601 INFO:teuthology.orchestra.run.trial104.stdout:================================================================================ 2026-02-26T03:31:08.601 INFO:teuthology.orchestra.run.trial104.stdout:Installing: 2026-02-26T03:31:08.601 INFO:teuthology.orchestra.run.trial104.stdout: ceph-immutable-object-cache x86_64 2:20.3.0-5312.gabffb617.el9 ceph 149 k 2026-02-26T03:31:08.601 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:31:08.601 INFO:teuthology.orchestra.run.trial104.stdout:Transaction Summary 2026-02-26T03:31:08.601 INFO:teuthology.orchestra.run.trial104.stdout:================================================================================ 2026-02-26T03:31:08.601 INFO:teuthology.orchestra.run.trial104.stdout:Install 1 Package 2026-02-26T03:31:08.602 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:31:08.602 INFO:teuthology.orchestra.run.trial104.stdout:Total download size: 149 k 2026-02-26T03:31:08.602 INFO:teuthology.orchestra.run.trial104.stdout:Installed size: 455 k 2026-02-26T03:31:08.602 INFO:teuthology.orchestra.run.trial104.stdout:Downloading Packages: 2026-02-26T03:31:09.166 INFO:teuthology.orchestra.run.trial104.stdout:ceph-immutable-object-cache-20.3.0-5312.gabffb6 264 kB/s | 149 kB 00:00 2026-02-26T03:31:09.167 INFO:teuthology.orchestra.run.trial104.stdout:-------------------------------------------------------------------------------- 2026-02-26T03:31:09.167 INFO:teuthology.orchestra.run.trial104.stdout:Total 264 kB/s | 149 kB 00:00 2026-02-26T03:31:09.167 INFO:teuthology.orchestra.run.trial104.stdout:Running transaction check 2026-02-26T03:31:09.171 INFO:teuthology.orchestra.run.trial104.stdout:Transaction check succeeded. 2026-02-26T03:31:09.172 INFO:teuthology.orchestra.run.trial104.stdout:Running transaction test 2026-02-26T03:31:09.202 INFO:teuthology.orchestra.run.trial104.stdout:Transaction test succeeded. 2026-02-26T03:31:09.203 INFO:teuthology.orchestra.run.trial104.stdout:Running transaction 2026-02-26T03:31:09.258 INFO:teuthology.orchestra.run.trial104.stdout: Preparing : 1/1 2026-02-26T03:31:09.260 INFO:teuthology.orchestra.run.trial104.stdout: Installing : ceph-immutable-object-cache-2:20.3.0-5312.gabffb617. 1/1 2026-02-26T03:31:09.283 INFO:teuthology.orchestra.run.trial104.stdout: Running scriptlet: ceph-immutable-object-cache-2:20.3.0-5312.gabffb617. 1/1 2026-02-26T03:31:09.283 INFO:teuthology.orchestra.run.trial104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-26T03:31:09.284 INFO:teuthology.orchestra.run.trial104.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2026-02-26T03:31:09.284 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:31:09.601 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : ceph-immutable-object-cache-2:20.3.0-5312.gabffb617. 1/1 2026-02-26T03:31:09.601 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:31:09.602 INFO:teuthology.orchestra.run.trial104.stdout:Installed: 2026-02-26T03:31:09.602 INFO:teuthology.orchestra.run.trial104.stdout: ceph-immutable-object-cache-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:31:09.602 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:31:09.602 INFO:teuthology.orchestra.run.trial104.stdout:Complete! 2026-02-26T03:31:09.629 DEBUG:teuthology.orchestra.run.trial104:> sudo yum -y install ceph-mgr 2026-02-26T03:31:09.944 INFO:teuthology.orchestra.run.trial104.stdout:Last metadata expiration check: 0:10:21 ago on Thu 26 Feb 2026 03:20:48 AM UTC. 2026-02-26T03:31:10.010 INFO:teuthology.orchestra.run.trial104.stdout:Package ceph-mgr-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:31:10.029 INFO:teuthology.orchestra.run.trial104.stdout:Dependencies resolved. 2026-02-26T03:31:10.030 INFO:teuthology.orchestra.run.trial104.stdout:Nothing to do. 2026-02-26T03:31:10.030 INFO:teuthology.orchestra.run.trial104.stdout:Complete! 2026-02-26T03:31:10.050 DEBUG:teuthology.orchestra.run.trial104:> sudo yum -y install ceph-mgr-dashboard 2026-02-26T03:31:10.372 INFO:teuthology.orchestra.run.trial104.stdout:Last metadata expiration check: 0:10:22 ago on Thu 26 Feb 2026 03:20:48 AM UTC. 2026-02-26T03:31:10.439 INFO:teuthology.orchestra.run.trial104.stdout:Package ceph-mgr-dashboard-2:20.3.0-5312.gabffb617.el9.noarch is already installed. 2026-02-26T03:31:10.458 INFO:teuthology.orchestra.run.trial104.stdout:Dependencies resolved. 2026-02-26T03:31:10.458 INFO:teuthology.orchestra.run.trial104.stdout:Nothing to do. 2026-02-26T03:31:10.458 INFO:teuthology.orchestra.run.trial104.stdout:Complete! 2026-02-26T03:31:10.479 DEBUG:teuthology.orchestra.run.trial104:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-26T03:31:10.795 INFO:teuthology.orchestra.run.trial104.stdout:Last metadata expiration check: 0:10:22 ago on Thu 26 Feb 2026 03:20:48 AM UTC. 2026-02-26T03:31:10.860 INFO:teuthology.orchestra.run.trial104.stdout:Package ceph-mgr-diskprediction-local-2:20.3.0-5312.gabffb617.el9.noarch is already installed. 2026-02-26T03:31:10.879 INFO:teuthology.orchestra.run.trial104.stdout:Dependencies resolved. 2026-02-26T03:31:10.880 INFO:teuthology.orchestra.run.trial104.stdout:Nothing to do. 2026-02-26T03:31:10.880 INFO:teuthology.orchestra.run.trial104.stdout:Complete! 2026-02-26T03:31:10.900 DEBUG:teuthology.orchestra.run.trial104:> sudo yum -y install ceph-mgr-rook 2026-02-26T03:31:11.220 INFO:teuthology.orchestra.run.trial104.stdout:Last metadata expiration check: 0:10:23 ago on Thu 26 Feb 2026 03:20:48 AM UTC. 2026-02-26T03:31:11.287 INFO:teuthology.orchestra.run.trial104.stdout:Package ceph-mgr-rook-2:20.3.0-5312.gabffb617.el9.noarch is already installed. 2026-02-26T03:31:11.305 INFO:teuthology.orchestra.run.trial104.stdout:Dependencies resolved. 2026-02-26T03:31:11.306 INFO:teuthology.orchestra.run.trial104.stdout:Nothing to do. 2026-02-26T03:31:11.306 INFO:teuthology.orchestra.run.trial104.stdout:Complete! 2026-02-26T03:31:11.328 DEBUG:teuthology.orchestra.run.trial104:> sudo yum -y install ceph-mgr-cephadm 2026-02-26T03:31:11.648 INFO:teuthology.orchestra.run.trial104.stdout:Last metadata expiration check: 0:10:23 ago on Thu 26 Feb 2026 03:20:48 AM UTC. 2026-02-26T03:31:11.714 INFO:teuthology.orchestra.run.trial104.stdout:Package ceph-mgr-cephadm-2:20.3.0-5312.gabffb617.el9.noarch is already installed. 2026-02-26T03:31:11.733 INFO:teuthology.orchestra.run.trial104.stdout:Dependencies resolved. 2026-02-26T03:31:11.734 INFO:teuthology.orchestra.run.trial104.stdout:Nothing to do. 2026-02-26T03:31:11.734 INFO:teuthology.orchestra.run.trial104.stdout:Complete! 2026-02-26T03:31:11.756 DEBUG:teuthology.orchestra.run.trial104:> sudo yum -y install ceph-osd 2026-02-26T03:31:12.070 INFO:teuthology.orchestra.run.trial104.stdout:Last metadata expiration check: 0:10:24 ago on Thu 26 Feb 2026 03:20:48 AM UTC. 2026-02-26T03:31:12.136 INFO:teuthology.orchestra.run.trial104.stdout:Package ceph-osd-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:31:12.155 INFO:teuthology.orchestra.run.trial104.stdout:Dependencies resolved. 2026-02-26T03:31:12.156 INFO:teuthology.orchestra.run.trial104.stdout:Nothing to do. 2026-02-26T03:31:12.156 INFO:teuthology.orchestra.run.trial104.stdout:Complete! 2026-02-26T03:31:12.178 DEBUG:teuthology.orchestra.run.trial104:> sudo yum -y install ceph-osd-classic 2026-02-26T03:31:12.498 INFO:teuthology.orchestra.run.trial104.stdout:Last metadata expiration check: 0:10:24 ago on Thu 26 Feb 2026 03:20:48 AM UTC. 2026-02-26T03:31:12.565 INFO:teuthology.orchestra.run.trial104.stdout:Package ceph-osd-classic-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:31:12.584 INFO:teuthology.orchestra.run.trial104.stdout:Dependencies resolved. 2026-02-26T03:31:12.584 INFO:teuthology.orchestra.run.trial104.stdout:Nothing to do. 2026-02-26T03:31:12.584 INFO:teuthology.orchestra.run.trial104.stdout:Complete! 2026-02-26T03:31:12.606 DEBUG:teuthology.orchestra.run.trial104:> sudo yum -y install ceph-fuse 2026-02-26T03:31:12.923 INFO:teuthology.orchestra.run.trial104.stdout:Last metadata expiration check: 0:10:24 ago on Thu 26 Feb 2026 03:20:48 AM UTC. 2026-02-26T03:31:13.008 INFO:teuthology.orchestra.run.trial104.stdout:Dependencies resolved. 2026-02-26T03:31:13.008 INFO:teuthology.orchestra.run.trial104.stdout:================================================================================ 2026-02-26T03:31:13.008 INFO:teuthology.orchestra.run.trial104.stdout: Package Arch Version Repo Size 2026-02-26T03:31:13.008 INFO:teuthology.orchestra.run.trial104.stdout:================================================================================ 2026-02-26T03:31:13.009 INFO:teuthology.orchestra.run.trial104.stdout:Installing: 2026-02-26T03:31:13.009 INFO:teuthology.orchestra.run.trial104.stdout: ceph-fuse x86_64 2:20.3.0-5312.gabffb617.el9 ceph 914 k 2026-02-26T03:31:13.009 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:31:13.009 INFO:teuthology.orchestra.run.trial104.stdout:Transaction Summary 2026-02-26T03:31:13.009 INFO:teuthology.orchestra.run.trial104.stdout:================================================================================ 2026-02-26T03:31:13.009 INFO:teuthology.orchestra.run.trial104.stdout:Install 1 Package 2026-02-26T03:31:13.009 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:31:13.009 INFO:teuthology.orchestra.run.trial104.stdout:Total download size: 914 k 2026-02-26T03:31:13.009 INFO:teuthology.orchestra.run.trial104.stdout:Installed size: 2.9 M 2026-02-26T03:31:13.009 INFO:teuthology.orchestra.run.trial104.stdout:Downloading Packages: 2026-02-26T03:31:14.014 INFO:teuthology.orchestra.run.trial104.stdout:ceph-fuse-20.3.0-5312.gabffb617.el9.x86_64.rpm 909 kB/s | 914 kB 00:01 2026-02-26T03:31:14.015 INFO:teuthology.orchestra.run.trial104.stdout:-------------------------------------------------------------------------------- 2026-02-26T03:31:14.015 INFO:teuthology.orchestra.run.trial104.stdout:Total 908 kB/s | 914 kB 00:01 2026-02-26T03:31:14.015 INFO:teuthology.orchestra.run.trial104.stdout:Running transaction check 2026-02-26T03:31:14.020 INFO:teuthology.orchestra.run.trial104.stdout:Transaction check succeeded. 2026-02-26T03:31:14.020 INFO:teuthology.orchestra.run.trial104.stdout:Running transaction test 2026-02-26T03:31:14.053 INFO:teuthology.orchestra.run.trial104.stdout:Transaction test succeeded. 2026-02-26T03:31:14.053 INFO:teuthology.orchestra.run.trial104.stdout:Running transaction 2026-02-26T03:31:14.144 INFO:teuthology.orchestra.run.trial104.stdout: Preparing : 1/1 2026-02-26T03:31:14.162 INFO:teuthology.orchestra.run.trial104.stdout: Installing : ceph-fuse-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:31:14.406 INFO:teuthology.orchestra.run.trial104.stdout: Running scriptlet: ceph-fuse-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:31:14.469 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : ceph-fuse-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:31:14.469 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:31:14.469 INFO:teuthology.orchestra.run.trial104.stdout:Installed: 2026-02-26T03:31:14.469 INFO:teuthology.orchestra.run.trial104.stdout: ceph-fuse-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:31:14.469 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:31:14.469 INFO:teuthology.orchestra.run.trial104.stdout:Complete! 2026-02-26T03:31:14.500 DEBUG:teuthology.orchestra.run.trial104:> sudo yum -y install ceph-volume 2026-02-26T03:31:14.814 INFO:teuthology.orchestra.run.trial104.stdout:Last metadata expiration check: 0:10:26 ago on Thu 26 Feb 2026 03:20:48 AM UTC. 2026-02-26T03:31:14.879 INFO:teuthology.orchestra.run.trial104.stdout:Package ceph-volume-2:20.3.0-5312.gabffb617.el9.noarch is already installed. 2026-02-26T03:31:14.898 INFO:teuthology.orchestra.run.trial104.stdout:Dependencies resolved. 2026-02-26T03:31:14.899 INFO:teuthology.orchestra.run.trial104.stdout:Nothing to do. 2026-02-26T03:31:14.899 INFO:teuthology.orchestra.run.trial104.stdout:Complete! 2026-02-26T03:31:14.921 DEBUG:teuthology.orchestra.run.trial104:> sudo yum -y install librados-devel 2026-02-26T03:31:15.245 INFO:teuthology.orchestra.run.trial104.stdout:Last metadata expiration check: 0:10:27 ago on Thu 26 Feb 2026 03:20:48 AM UTC. 2026-02-26T03:31:15.331 INFO:teuthology.orchestra.run.trial104.stdout:Dependencies resolved. 2026-02-26T03:31:15.331 INFO:teuthology.orchestra.run.trial104.stdout:================================================================================ 2026-02-26T03:31:15.331 INFO:teuthology.orchestra.run.trial104.stdout: Package Arch Version Repo Size 2026-02-26T03:31:15.332 INFO:teuthology.orchestra.run.trial104.stdout:================================================================================ 2026-02-26T03:31:15.332 INFO:teuthology.orchestra.run.trial104.stdout:Installing: 2026-02-26T03:31:15.332 INFO:teuthology.orchestra.run.trial104.stdout: librados-devel x86_64 2:20.3.0-5312.gabffb617.el9 ceph 124 k 2026-02-26T03:31:15.332 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:31:15.332 INFO:teuthology.orchestra.run.trial104.stdout:Transaction Summary 2026-02-26T03:31:15.332 INFO:teuthology.orchestra.run.trial104.stdout:================================================================================ 2026-02-26T03:31:15.332 INFO:teuthology.orchestra.run.trial104.stdout:Install 1 Package 2026-02-26T03:31:15.332 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:31:15.332 INFO:teuthology.orchestra.run.trial104.stdout:Total download size: 124 k 2026-02-26T03:31:15.332 INFO:teuthology.orchestra.run.trial104.stdout:Installed size: 434 k 2026-02-26T03:31:15.333 INFO:teuthology.orchestra.run.trial104.stdout:Downloading Packages: 2026-02-26T03:31:15.908 INFO:teuthology.orchestra.run.trial104.stdout:librados-devel-20.3.0-5312.gabffb617.el9.x86_64 215 kB/s | 124 kB 00:00 2026-02-26T03:31:15.909 INFO:teuthology.orchestra.run.trial104.stdout:-------------------------------------------------------------------------------- 2026-02-26T03:31:15.909 INFO:teuthology.orchestra.run.trial104.stdout:Total 214 kB/s | 124 kB 00:00 2026-02-26T03:31:15.909 INFO:teuthology.orchestra.run.trial104.stdout:Running transaction check 2026-02-26T03:31:15.911 INFO:teuthology.orchestra.run.trial104.stdout:Transaction check succeeded. 2026-02-26T03:31:15.912 INFO:teuthology.orchestra.run.trial104.stdout:Running transaction test 2026-02-26T03:31:15.943 INFO:teuthology.orchestra.run.trial104.stdout:Transaction test succeeded. 2026-02-26T03:31:15.943 INFO:teuthology.orchestra.run.trial104.stdout:Running transaction 2026-02-26T03:31:15.998 INFO:teuthology.orchestra.run.trial104.stdout: Preparing : 1/1 2026-02-26T03:31:16.015 INFO:teuthology.orchestra.run.trial104.stdout: Installing : librados-devel-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:31:16.109 INFO:teuthology.orchestra.run.trial104.stdout: Running scriptlet: librados-devel-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:31:16.178 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : librados-devel-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:31:16.178 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:31:16.179 INFO:teuthology.orchestra.run.trial104.stdout:Installed: 2026-02-26T03:31:16.179 INFO:teuthology.orchestra.run.trial104.stdout: librados-devel-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:31:16.179 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:31:16.179 INFO:teuthology.orchestra.run.trial104.stdout:Complete! 2026-02-26T03:31:16.206 DEBUG:teuthology.orchestra.run.trial104:> sudo yum -y install libcephfs2 2026-02-26T03:31:16.525 INFO:teuthology.orchestra.run.trial104.stdout:Last metadata expiration check: 0:10:28 ago on Thu 26 Feb 2026 03:20:48 AM UTC. 2026-02-26T03:31:16.590 INFO:teuthology.orchestra.run.trial104.stdout:Package libcephfs2-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:31:16.609 INFO:teuthology.orchestra.run.trial104.stdout:Dependencies resolved. 2026-02-26T03:31:16.610 INFO:teuthology.orchestra.run.trial104.stdout:Nothing to do. 2026-02-26T03:31:16.610 INFO:teuthology.orchestra.run.trial104.stdout:Complete! 2026-02-26T03:31:16.632 DEBUG:teuthology.orchestra.run.trial104:> sudo yum -y install libcephfs-devel 2026-02-26T03:31:16.947 INFO:teuthology.orchestra.run.trial104.stdout:Last metadata expiration check: 0:10:28 ago on Thu 26 Feb 2026 03:20:48 AM UTC. 2026-02-26T03:31:17.032 INFO:teuthology.orchestra.run.trial104.stdout:Dependencies resolved. 2026-02-26T03:31:17.032 INFO:teuthology.orchestra.run.trial104.stdout:================================================================================ 2026-02-26T03:31:17.032 INFO:teuthology.orchestra.run.trial104.stdout: Package Arch Version Repo Size 2026-02-26T03:31:17.032 INFO:teuthology.orchestra.run.trial104.stdout:================================================================================ 2026-02-26T03:31:17.032 INFO:teuthology.orchestra.run.trial104.stdout:Installing: 2026-02-26T03:31:17.032 INFO:teuthology.orchestra.run.trial104.stdout: libcephfs-devel x86_64 2:20.3.0-5312.gabffb617.el9 ceph 37 k 2026-02-26T03:31:17.032 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:31:17.033 INFO:teuthology.orchestra.run.trial104.stdout:Transaction Summary 2026-02-26T03:31:17.033 INFO:teuthology.orchestra.run.trial104.stdout:================================================================================ 2026-02-26T03:31:17.033 INFO:teuthology.orchestra.run.trial104.stdout:Install 1 Package 2026-02-26T03:31:17.033 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:31:17.033 INFO:teuthology.orchestra.run.trial104.stdout:Total download size: 37 k 2026-02-26T03:31:17.033 INFO:teuthology.orchestra.run.trial104.stdout:Installed size: 171 k 2026-02-26T03:31:17.033 INFO:teuthology.orchestra.run.trial104.stdout:Downloading Packages: 2026-02-26T03:31:17.386 INFO:teuthology.orchestra.run.trial104.stdout:libcephfs-devel-20.3.0-5312.gabffb617.el9.x86_6 106 kB/s | 37 kB 00:00 2026-02-26T03:31:17.386 INFO:teuthology.orchestra.run.trial104.stdout:-------------------------------------------------------------------------------- 2026-02-26T03:31:17.386 INFO:teuthology.orchestra.run.trial104.stdout:Total 105 kB/s | 37 kB 00:00 2026-02-26T03:31:17.387 INFO:teuthology.orchestra.run.trial104.stdout:Running transaction check 2026-02-26T03:31:17.388 INFO:teuthology.orchestra.run.trial104.stdout:Transaction check succeeded. 2026-02-26T03:31:17.389 INFO:teuthology.orchestra.run.trial104.stdout:Running transaction test 2026-02-26T03:31:17.391 INFO:teuthology.orchestra.run.trial104.stdout:Transaction test succeeded. 2026-02-26T03:31:17.391 INFO:teuthology.orchestra.run.trial104.stdout:Running transaction 2026-02-26T03:31:17.413 INFO:teuthology.orchestra.run.trial104.stdout: Preparing : 1/1 2026-02-26T03:31:17.430 INFO:teuthology.orchestra.run.trial104.stdout: Installing : libcephfs-devel-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:31:17.492 INFO:teuthology.orchestra.run.trial104.stdout: Running scriptlet: libcephfs-devel-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:31:17.553 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : libcephfs-devel-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:31:17.554 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:31:17.554 INFO:teuthology.orchestra.run.trial104.stdout:Installed: 2026-02-26T03:31:17.554 INFO:teuthology.orchestra.run.trial104.stdout: libcephfs-devel-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:31:17.554 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:31:17.554 INFO:teuthology.orchestra.run.trial104.stdout:Complete! 2026-02-26T03:31:17.583 DEBUG:teuthology.orchestra.run.trial104:> sudo yum -y install librados2 2026-02-26T03:31:17.899 INFO:teuthology.orchestra.run.trial104.stdout:Last metadata expiration check: 0:10:29 ago on Thu 26 Feb 2026 03:20:48 AM UTC. 2026-02-26T03:31:17.965 INFO:teuthology.orchestra.run.trial104.stdout:Package librados2-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:31:17.982 INFO:teuthology.orchestra.run.trial104.stdout:Dependencies resolved. 2026-02-26T03:31:17.983 INFO:teuthology.orchestra.run.trial104.stdout:Nothing to do. 2026-02-26T03:31:17.983 INFO:teuthology.orchestra.run.trial104.stdout:Complete! 2026-02-26T03:31:18.005 DEBUG:teuthology.orchestra.run.trial104:> sudo yum -y install librbd1 2026-02-26T03:31:18.319 INFO:teuthology.orchestra.run.trial104.stdout:Last metadata expiration check: 0:10:30 ago on Thu 26 Feb 2026 03:20:48 AM UTC. 2026-02-26T03:31:18.385 INFO:teuthology.orchestra.run.trial104.stdout:Package librbd1-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:31:18.403 INFO:teuthology.orchestra.run.trial104.stdout:Dependencies resolved. 2026-02-26T03:31:18.404 INFO:teuthology.orchestra.run.trial104.stdout:Nothing to do. 2026-02-26T03:31:18.404 INFO:teuthology.orchestra.run.trial104.stdout:Complete! 2026-02-26T03:31:18.425 DEBUG:teuthology.orchestra.run.trial104:> sudo yum -y install python3-rados 2026-02-26T03:31:18.744 INFO:teuthology.orchestra.run.trial104.stdout:Last metadata expiration check: 0:10:30 ago on Thu 26 Feb 2026 03:20:48 AM UTC. 2026-02-26T03:31:18.809 INFO:teuthology.orchestra.run.trial104.stdout:Package python3-rados-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:31:18.827 INFO:teuthology.orchestra.run.trial104.stdout:Dependencies resolved. 2026-02-26T03:31:18.828 INFO:teuthology.orchestra.run.trial104.stdout:Nothing to do. 2026-02-26T03:31:18.828 INFO:teuthology.orchestra.run.trial104.stdout:Complete! 2026-02-26T03:31:18.849 DEBUG:teuthology.orchestra.run.trial104:> sudo yum -y install python3-rgw 2026-02-26T03:31:19.164 INFO:teuthology.orchestra.run.trial104.stdout:Last metadata expiration check: 0:10:31 ago on Thu 26 Feb 2026 03:20:48 AM UTC. 2026-02-26T03:31:19.230 INFO:teuthology.orchestra.run.trial104.stdout:Package python3-rgw-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:31:19.247 INFO:teuthology.orchestra.run.trial104.stdout:Dependencies resolved. 2026-02-26T03:31:19.248 INFO:teuthology.orchestra.run.trial104.stdout:Nothing to do. 2026-02-26T03:31:19.248 INFO:teuthology.orchestra.run.trial104.stdout:Complete! 2026-02-26T03:31:19.270 DEBUG:teuthology.orchestra.run.trial104:> sudo yum -y install python3-cephfs 2026-02-26T03:31:19.584 INFO:teuthology.orchestra.run.trial104.stdout:Last metadata expiration check: 0:10:31 ago on Thu 26 Feb 2026 03:20:48 AM UTC. 2026-02-26T03:31:19.650 INFO:teuthology.orchestra.run.trial104.stdout:Package python3-cephfs-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:31:19.668 INFO:teuthology.orchestra.run.trial104.stdout:Dependencies resolved. 2026-02-26T03:31:19.669 INFO:teuthology.orchestra.run.trial104.stdout:Nothing to do. 2026-02-26T03:31:19.669 INFO:teuthology.orchestra.run.trial104.stdout:Complete! 2026-02-26T03:31:19.691 DEBUG:teuthology.orchestra.run.trial104:> sudo yum -y install python3-rbd 2026-02-26T03:31:20.003 INFO:teuthology.orchestra.run.trial104.stdout:Last metadata expiration check: 0:10:32 ago on Thu 26 Feb 2026 03:20:48 AM UTC. 2026-02-26T03:31:20.068 INFO:teuthology.orchestra.run.trial104.stdout:Package python3-rbd-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:31:20.086 INFO:teuthology.orchestra.run.trial104.stdout:Dependencies resolved. 2026-02-26T03:31:20.086 INFO:teuthology.orchestra.run.trial104.stdout:Nothing to do. 2026-02-26T03:31:20.087 INFO:teuthology.orchestra.run.trial104.stdout:Complete! 2026-02-26T03:31:20.107 DEBUG:teuthology.orchestra.run.trial104:> sudo yum -y install rbd-fuse 2026-02-26T03:31:20.423 INFO:teuthology.orchestra.run.trial104.stdout:Last metadata expiration check: 0:10:32 ago on Thu 26 Feb 2026 03:20:48 AM UTC. 2026-02-26T03:31:20.506 INFO:teuthology.orchestra.run.trial104.stdout:Dependencies resolved. 2026-02-26T03:31:20.507 INFO:teuthology.orchestra.run.trial104.stdout:================================================================================ 2026-02-26T03:31:20.507 INFO:teuthology.orchestra.run.trial104.stdout: Package Architecture Version Repository Size 2026-02-26T03:31:20.507 INFO:teuthology.orchestra.run.trial104.stdout:================================================================================ 2026-02-26T03:31:20.508 INFO:teuthology.orchestra.run.trial104.stdout:Installing: 2026-02-26T03:31:20.508 INFO:teuthology.orchestra.run.trial104.stdout: rbd-fuse x86_64 2:20.3.0-5312.gabffb617.el9 ceph 87 k 2026-02-26T03:31:20.508 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:31:20.508 INFO:teuthology.orchestra.run.trial104.stdout:Transaction Summary 2026-02-26T03:31:20.508 INFO:teuthology.orchestra.run.trial104.stdout:================================================================================ 2026-02-26T03:31:20.508 INFO:teuthology.orchestra.run.trial104.stdout:Install 1 Package 2026-02-26T03:31:20.508 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:31:20.508 INFO:teuthology.orchestra.run.trial104.stdout:Total download size: 87 k 2026-02-26T03:31:20.508 INFO:teuthology.orchestra.run.trial104.stdout:Installed size: 236 k 2026-02-26T03:31:20.508 INFO:teuthology.orchestra.run.trial104.stdout:Downloading Packages: 2026-02-26T03:31:21.300 INFO:teuthology.orchestra.run.trial104.stdout:rbd-fuse-20.3.0-5312.gabffb617.el9.x86_64.rpm 109 kB/s | 87 kB 00:00 2026-02-26T03:31:21.301 INFO:teuthology.orchestra.run.trial104.stdout:-------------------------------------------------------------------------------- 2026-02-26T03:31:21.301 INFO:teuthology.orchestra.run.trial104.stdout:Total 109 kB/s | 87 kB 00:00 2026-02-26T03:31:21.301 INFO:teuthology.orchestra.run.trial104.stdout:Running transaction check 2026-02-26T03:31:21.306 INFO:teuthology.orchestra.run.trial104.stdout:Transaction check succeeded. 2026-02-26T03:31:21.306 INFO:teuthology.orchestra.run.trial104.stdout:Running transaction test 2026-02-26T03:31:21.337 INFO:teuthology.orchestra.run.trial104.stdout:Transaction test succeeded. 2026-02-26T03:31:21.337 INFO:teuthology.orchestra.run.trial104.stdout:Running transaction 2026-02-26T03:31:21.389 INFO:teuthology.orchestra.run.trial104.stdout: Preparing : 1/1 2026-02-26T03:31:21.407 INFO:teuthology.orchestra.run.trial104.stdout: Installing : rbd-fuse-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:31:21.493 INFO:teuthology.orchestra.run.trial104.stdout: Running scriptlet: rbd-fuse-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:31:21.555 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : rbd-fuse-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:31:21.555 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:31:21.555 INFO:teuthology.orchestra.run.trial104.stdout:Installed: 2026-02-26T03:31:21.555 INFO:teuthology.orchestra.run.trial104.stdout: rbd-fuse-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:31:21.555 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:31:21.555 INFO:teuthology.orchestra.run.trial104.stdout:Complete! 2026-02-26T03:31:21.583 DEBUG:teuthology.orchestra.run.trial104:> sudo yum -y install rbd-mirror 2026-02-26T03:31:21.899 INFO:teuthology.orchestra.run.trial104.stdout:Last metadata expiration check: 0:10:33 ago on Thu 26 Feb 2026 03:20:48 AM UTC. 2026-02-26T03:31:21.984 INFO:teuthology.orchestra.run.trial104.stdout:Dependencies resolved. 2026-02-26T03:31:21.985 INFO:teuthology.orchestra.run.trial104.stdout:================================================================================ 2026-02-26T03:31:21.985 INFO:teuthology.orchestra.run.trial104.stdout: Package Arch Version Repo Size 2026-02-26T03:31:21.985 INFO:teuthology.orchestra.run.trial104.stdout:================================================================================ 2026-02-26T03:31:21.985 INFO:teuthology.orchestra.run.trial104.stdout:Installing: 2026-02-26T03:31:21.985 INFO:teuthology.orchestra.run.trial104.stdout: rbd-mirror x86_64 2:20.3.0-5312.gabffb617.el9 ceph 2.6 M 2026-02-26T03:31:21.985 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:31:21.985 INFO:teuthology.orchestra.run.trial104.stdout:Transaction Summary 2026-02-26T03:31:21.985 INFO:teuthology.orchestra.run.trial104.stdout:================================================================================ 2026-02-26T03:31:21.985 INFO:teuthology.orchestra.run.trial104.stdout:Install 1 Package 2026-02-26T03:31:21.985 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:31:21.985 INFO:teuthology.orchestra.run.trial104.stdout:Total download size: 2.6 M 2026-02-26T03:31:21.986 INFO:teuthology.orchestra.run.trial104.stdout:Installed size: 11 M 2026-02-26T03:31:21.986 INFO:teuthology.orchestra.run.trial104.stdout:Downloading Packages: 2026-02-26T03:31:24.386 INFO:teuthology.orchestra.run.trial104.stdout:rbd-mirror-20.3.0-5312.gabffb617.el9.x86_64.rpm 1.1 MB/s | 2.6 MB 00:02 2026-02-26T03:31:24.386 INFO:teuthology.orchestra.run.trial104.stdout:-------------------------------------------------------------------------------- 2026-02-26T03:31:24.386 INFO:teuthology.orchestra.run.trial104.stdout:Total 1.1 MB/s | 2.6 MB 00:02 2026-02-26T03:31:24.386 INFO:teuthology.orchestra.run.trial104.stdout:Running transaction check 2026-02-26T03:31:24.391 INFO:teuthology.orchestra.run.trial104.stdout:Transaction check succeeded. 2026-02-26T03:31:24.391 INFO:teuthology.orchestra.run.trial104.stdout:Running transaction test 2026-02-26T03:31:24.430 INFO:teuthology.orchestra.run.trial104.stdout:Transaction test succeeded. 2026-02-26T03:31:24.430 INFO:teuthology.orchestra.run.trial104.stdout:Running transaction 2026-02-26T03:31:24.593 INFO:teuthology.orchestra.run.trial104.stdout: Preparing : 1/1 2026-02-26T03:31:24.595 INFO:teuthology.orchestra.run.trial104.stdout: Installing : rbd-mirror-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:31:24.619 INFO:teuthology.orchestra.run.trial104.stdout: Running scriptlet: rbd-mirror-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:31:24.619 INFO:teuthology.orchestra.run.trial104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-26T03:31:24.619 INFO:teuthology.orchestra.run.trial104.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2026-02-26T03:31:24.619 INFO:teuthology.orchestra.run.trial104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-26T03:31:24.619 INFO:teuthology.orchestra.run.trial104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-26T03:31:24.619 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:31:24.951 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : rbd-mirror-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:31:24.951 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:31:24.951 INFO:teuthology.orchestra.run.trial104.stdout:Installed: 2026-02-26T03:31:24.951 INFO:teuthology.orchestra.run.trial104.stdout: rbd-mirror-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:31:24.951 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:31:24.951 INFO:teuthology.orchestra.run.trial104.stdout:Complete! 2026-02-26T03:31:24.980 DEBUG:teuthology.orchestra.run.trial104:> sudo yum -y install rbd-nbd 2026-02-26T03:31:25.300 INFO:teuthology.orchestra.run.trial104.stdout:Last metadata expiration check: 0:10:37 ago on Thu 26 Feb 2026 03:20:48 AM UTC. 2026-02-26T03:31:25.385 INFO:teuthology.orchestra.run.trial104.stdout:Dependencies resolved. 2026-02-26T03:31:25.385 INFO:teuthology.orchestra.run.trial104.stdout:================================================================================ 2026-02-26T03:31:25.385 INFO:teuthology.orchestra.run.trial104.stdout: Package Architecture Version Repository Size 2026-02-26T03:31:25.386 INFO:teuthology.orchestra.run.trial104.stdout:================================================================================ 2026-02-26T03:31:25.386 INFO:teuthology.orchestra.run.trial104.stdout:Installing: 2026-02-26T03:31:25.386 INFO:teuthology.orchestra.run.trial104.stdout: rbd-nbd x86_64 2:20.3.0-5312.gabffb617.el9 ceph 166 k 2026-02-26T03:31:25.386 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:31:25.386 INFO:teuthology.orchestra.run.trial104.stdout:Transaction Summary 2026-02-26T03:31:25.386 INFO:teuthology.orchestra.run.trial104.stdout:================================================================================ 2026-02-26T03:31:25.386 INFO:teuthology.orchestra.run.trial104.stdout:Install 1 Package 2026-02-26T03:31:25.386 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:31:25.386 INFO:teuthology.orchestra.run.trial104.stdout:Total download size: 166 k 2026-02-26T03:31:25.386 INFO:teuthology.orchestra.run.trial104.stdout:Installed size: 496 k 2026-02-26T03:31:25.387 INFO:teuthology.orchestra.run.trial104.stdout:Downloading Packages: 2026-02-26T03:31:25.993 INFO:teuthology.orchestra.run.trial104.stdout:rbd-nbd-20.3.0-5312.gabffb617.el9.x86_64.rpm 274 kB/s | 166 kB 00:00 2026-02-26T03:31:25.994 INFO:teuthology.orchestra.run.trial104.stdout:-------------------------------------------------------------------------------- 2026-02-26T03:31:25.994 INFO:teuthology.orchestra.run.trial104.stdout:Total 273 kB/s | 166 kB 00:00 2026-02-26T03:31:25.994 INFO:teuthology.orchestra.run.trial104.stdout:Running transaction check 2026-02-26T03:31:25.998 INFO:teuthology.orchestra.run.trial104.stdout:Transaction check succeeded. 2026-02-26T03:31:25.999 INFO:teuthology.orchestra.run.trial104.stdout:Running transaction test 2026-02-26T03:31:26.030 INFO:teuthology.orchestra.run.trial104.stdout:Transaction test succeeded. 2026-02-26T03:31:26.030 INFO:teuthology.orchestra.run.trial104.stdout:Running transaction 2026-02-26T03:31:26.086 INFO:teuthology.orchestra.run.trial104.stdout: Preparing : 1/1 2026-02-26T03:31:26.103 INFO:teuthology.orchestra.run.trial104.stdout: Installing : rbd-nbd-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:31:26.194 INFO:teuthology.orchestra.run.trial104.stdout: Running scriptlet: rbd-nbd-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:31:26.257 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : rbd-nbd-2:20.3.0-5312.gabffb617.el9.x86_64 1/1 2026-02-26T03:31:26.258 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:31:26.258 INFO:teuthology.orchestra.run.trial104.stdout:Installed: 2026-02-26T03:31:26.258 INFO:teuthology.orchestra.run.trial104.stdout: rbd-nbd-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:31:26.258 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:31:26.258 INFO:teuthology.orchestra.run.trial104.stdout:Complete! 2026-02-26T03:31:26.287 DEBUG:teuthology.orchestra.run.trial104:> sudo yum -y install ceph-osd-crimson 2026-02-26T03:31:26.607 INFO:teuthology.orchestra.run.trial104.stdout:Last metadata expiration check: 0:10:38 ago on Thu 26 Feb 2026 03:20:48 AM UTC. 2026-02-26T03:31:26.674 INFO:teuthology.orchestra.run.trial104.stdout:Package ceph-osd-crimson-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:31:26.692 INFO:teuthology.orchestra.run.trial104.stdout:Dependencies resolved. 2026-02-26T03:31:26.692 INFO:teuthology.orchestra.run.trial104.stdout:Nothing to do. 2026-02-26T03:31:26.692 INFO:teuthology.orchestra.run.trial104.stdout:Complete! 2026-02-26T03:31:26.715 DEBUG:teuthology.orchestra.run.trial104:> sudo yum -y install ceph-osd-crimson-debuginfo 2026-02-26T03:31:27.030 INFO:teuthology.orchestra.run.trial104.stdout:Last metadata expiration check: 0:10:39 ago on Thu 26 Feb 2026 03:20:48 AM UTC. 2026-02-26T03:31:27.114 INFO:teuthology.orchestra.run.trial104.stdout:Dependencies resolved. 2026-02-26T03:31:27.115 INFO:teuthology.orchestra.run.trial104.stdout:================================================================================ 2026-02-26T03:31:27.115 INFO:teuthology.orchestra.run.trial104.stdout: Package Arch Version Repo Size 2026-02-26T03:31:27.115 INFO:teuthology.orchestra.run.trial104.stdout:================================================================================ 2026-02-26T03:31:27.115 INFO:teuthology.orchestra.run.trial104.stdout:Installing: 2026-02-26T03:31:27.115 INFO:teuthology.orchestra.run.trial104.stdout: ceph-osd-crimson-debuginfo x86_64 2:20.3.0-5312.gabffb617.el9 ceph 429 M 2026-02-26T03:31:27.115 INFO:teuthology.orchestra.run.trial104.stdout:Installing weak dependencies: 2026-02-26T03:31:27.115 INFO:teuthology.orchestra.run.trial104.stdout: ceph-debugsource x86_64 2:20.3.0-5312.gabffb617.el9 ceph 12 M 2026-02-26T03:31:27.115 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:31:27.115 INFO:teuthology.orchestra.run.trial104.stdout:Transaction Summary 2026-02-26T03:31:27.116 INFO:teuthology.orchestra.run.trial104.stdout:================================================================================ 2026-02-26T03:31:27.116 INFO:teuthology.orchestra.run.trial104.stdout:Install 2 Packages 2026-02-26T03:31:27.116 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:31:27.116 INFO:teuthology.orchestra.run.trial104.stdout:Total download size: 441 M 2026-02-26T03:31:27.116 INFO:teuthology.orchestra.run.trial104.stdout:Installed size: 2.7 G 2026-02-26T03:31:27.116 INFO:teuthology.orchestra.run.trial104.stdout:Downloading Packages: 2026-02-26T03:31:39.381 INFO:teuthology.orchestra.run.trial104.stdout:(1/2): ceph-debugsource-20.3.0-5312.gabffb617.e 1.0 MB/s | 12 MB 00:12 2026-02-26T03:40:20.188 INFO:teuthology.orchestra.run.trial104.stdout:(2/2): ceph-osd-crimson-debuginfo-20.3.0-5312.g 824 kB/s | 429 MB 08:53 2026-02-26T03:40:20.198 INFO:teuthology.orchestra.run.trial104.stdout:-------------------------------------------------------------------------------- 2026-02-26T03:40:20.198 INFO:teuthology.orchestra.run.trial104.stdout:Total 847 kB/s | 441 MB 08:53 2026-02-26T03:40:20.198 INFO:teuthology.orchestra.run.trial104.stdout:Running transaction check 2026-02-26T03:40:20.198 INFO:teuthology.orchestra.run.trial104.stdout:Transaction check succeeded. 2026-02-26T03:40:20.198 INFO:teuthology.orchestra.run.trial104.stdout:Running transaction test 2026-02-26T03:40:21.563 INFO:teuthology.orchestra.run.trial104.stdout:Transaction test succeeded. 2026-02-26T03:40:21.563 INFO:teuthology.orchestra.run.trial104.stdout:Running transaction 2026-02-26T03:40:25.127 INFO:teuthology.orchestra.run.trial104.stdout: Preparing : 1/1 2026-02-26T03:40:47.163 INFO:teuthology.orchestra.run.trial104.stdout: Installing : ceph-debugsource-2:20.3.0-5312.gabffb617.el9.x86_64 1/2 2026-02-26T03:40:47.181 INFO:teuthology.orchestra.run.trial104.stdout: Installing : ceph-osd-crimson-debuginfo-2:20.3.0-5312.gabffb617.e 2/2 2026-02-26T03:40:47.514 INFO:teuthology.orchestra.run.trial104.stdout: Running scriptlet: ceph-osd-crimson-debuginfo-2:20.3.0-5312.gabffb617.e 2/2 2026-02-26T03:40:47.514 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : ceph-debugsource-2:20.3.0-5312.gabffb617.el9.x86_64 1/2 2026-02-26T03:40:47.577 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : ceph-osd-crimson-debuginfo-2:20.3.0-5312.gabffb617.e 2/2 2026-02-26T03:40:47.577 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:40:47.577 INFO:teuthology.orchestra.run.trial104.stdout:Installed: 2026-02-26T03:40:47.577 INFO:teuthology.orchestra.run.trial104.stdout: ceph-debugsource-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:40:47.577 INFO:teuthology.orchestra.run.trial104.stdout: ceph-osd-crimson-debuginfo-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:40:47.577 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:40:47.577 INFO:teuthology.orchestra.run.trial104.stdout:Complete! 2026-02-26T03:40:47.657 DEBUG:teuthology.parallel:result is None 2026-02-26T03:40:47.657 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=abffb617683951511fccf3c3cab6e219f2ec83a8 2026-02-26T03:40:47.799 DEBUG:teuthology.orchestra.run.trial104:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-26T03:40:47.819 INFO:teuthology.orchestra.run.trial104.stdout:20.3.0-5312.gabffb617.el9 2026-02-26T03:40:47.820 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5312.gabffb617.el9 2026-02-26T03:40:47.820 INFO:teuthology.task.install:The correct ceph version 20.3.0-5312.gabffb617 is installed. 2026-02-26T03:40:47.821 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2026-02-26T03:40:47.821 DEBUG:teuthology.orchestra.run.trial104:> set -ex 2026-02-26T03:40:47.822 DEBUG:teuthology.orchestra.run.trial104:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-26T03:40:47.884 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2026-02-26T03:40:47.884 DEBUG:teuthology.orchestra.run.trial104:> set -ex 2026-02-26T03:40:47.884 DEBUG:teuthology.orchestra.run.trial104:> sudo dd of=/usr/bin/daemon-helper 2026-02-26T03:40:47.947 DEBUG:teuthology.orchestra.run.trial104:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-26T03:40:48.010 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2026-02-26T03:40:48.010 DEBUG:teuthology.orchestra.run.trial104:> set -ex 2026-02-26T03:40:48.010 DEBUG:teuthology.orchestra.run.trial104:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-26T03:40:48.074 DEBUG:teuthology.orchestra.run.trial104:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-26T03:40:48.136 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2026-02-26T03:40:48.137 DEBUG:teuthology.orchestra.run.trial104:> set -ex 2026-02-26T03:40:48.137 DEBUG:teuthology.orchestra.run.trial104:> sudo dd of=/usr/bin/stdin-killer 2026-02-26T03:40:48.199 DEBUG:teuthology.orchestra.run.trial104:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-26T03:40:48.262 INFO:teuthology.run_tasks:Running task pexec... 2026-02-26T03:40:48.274 INFO:teuthology.task.pexec:Executing custom commands... 2026-02-26T03:40:48.275 DEBUG:teuthology.orchestra.run.trial104:> TESTDIR=/home/ubuntu/cephtest bash -s 2026-02-26T03:40:48.303 INFO:teuthology.task.pexec:Running commands on host ubuntu@trial104.front.sepia.ceph.com 2026-02-26T03:40:48.303 INFO:teuthology.task.pexec:sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson 2026-02-26T03:40:48.354 DEBUG:teuthology.parallel:result is None 2026-02-26T03:40:48.354 INFO:teuthology.run_tasks:Running task install... 2026-02-26T03:40:48.366 DEBUG:teuthology.task.install:project ceph 2026-02-26T03:40:48.366 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'debug', 'sha1': 'abffb617683951511fccf3c3cab6e219f2ec83a8'}} 2026-02-26T03:40:48.366 DEBUG:teuthology.task.install:config {'flavor': 'debug', 'sha1': 'abffb617683951511fccf3c3cab6e219f2ec83a8'} 2026-02-26T03:40:48.366 INFO:teuthology.task.install:Using flavor: debug 2026-02-26T03:40:48.369 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-osd', 'ceph-osd-classic', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-osd', 'ceph-osd-classic', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-26T03:40:48.369 INFO:teuthology.task.install:extra packages: [] 2026-02-26T03:40:48.369 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'debug', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'abffb617683951511fccf3c3cab6e219f2ec83a8', 'tag': None, 'wait_for_package': False} 2026-02-26T03:40:48.370 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=abffb617683951511fccf3c3cab6e219f2ec83a8 2026-02-26T03:40:48.496 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-shraddhaag-peering-bug/abffb617683951511fccf3c3cab6e219f2ec83a8/centos/9/flavors/debug/ 2026-02-26T03:40:48.496 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5312.gabffb617 2026-02-26T03:40:48.842 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-shraddhaag-peering-bug/abffb617683951511fccf3c3cab6e219f2ec83a8/centos/9/flavors/debug/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-shraddhaag-peering-bug/abffb617683951511fccf3c3cab6e219f2ec83a8/centos/9/flavors/debug/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-shraddhaag-peering-bug/abffb617683951511fccf3c3cab6e219f2ec83a8/centos/9/flavors/debug/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-26T03:40:48.842 DEBUG:teuthology.orchestra.run.trial104:> set -ex 2026-02-26T03:40:48.842 DEBUG:teuthology.orchestra.run.trial104:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-26T03:40:48.867 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2026-02-26T03:40:48.867 DEBUG:teuthology.orchestra.run.trial104:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-26T03:40:48.930 DEBUG:teuthology.orchestra.run.trial104:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-26T03:40:48.996 DEBUG:teuthology.orchestra.run.trial104:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-26T03:40:49.055 INFO:teuthology.orchestra.run.trial104.stdout:check_obsoletes = 1 2026-02-26T03:40:49.065 DEBUG:teuthology.orchestra.run.trial104:> sudo yum clean all 2026-02-26T03:40:49.229 INFO:teuthology.orchestra.run.trial104.stdout:56 files removed 2026-02-26T03:40:49.245 DEBUG:teuthology.orchestra.run.trial104:> sudo yum -y install ceph-radosgw 2026-02-26T03:40:50.123 INFO:teuthology.orchestra.run.trial104.stdout:ceph packages for x86_64 130 kB/s | 94 kB 00:00 2026-02-26T03:40:50.793 INFO:teuthology.orchestra.run.trial104.stdout:ceph noarch packages 38 kB/s | 24 kB 00:00 2026-02-26T03:40:51.372 INFO:teuthology.orchestra.run.trial104.stdout:ceph source packages 3.7 kB/s | 2.1 kB 00:00 2026-02-26T03:40:52.113 INFO:teuthology.orchestra.run.trial104.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.9 MB 00:00 2026-02-26T03:41:59.601 INFO:teuthology.orchestra.run.trial104.stdout:CentOS Stream 9 - AppStream 413 kB/s | 27 MB 01:06 2026-02-26T03:42:03.127 INFO:teuthology.orchestra.run.trial104.stdout:CentOS Stream 9 - CRB 6.9 MB/s | 8.0 MB 00:01 2026-02-26T03:42:04.048 INFO:teuthology.orchestra.run.trial104.stdout:CentOS Stream 9 - Extras packages 110 kB/s | 20 kB 00:00 2026-02-26T03:42:05.085 INFO:teuthology.orchestra.run.trial104.stdout:Extra Packages for Enterprise Linux 21 MB/s | 20 MB 00:00 2026-02-26T03:42:08.271 INFO:teuthology.orchestra.run.trial104.stdout:lab-extras 2.5 MB/s | 50 kB 00:00 2026-02-26T03:42:09.468 INFO:teuthology.orchestra.run.trial104.stdout:Package ceph-radosgw-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:42:09.486 INFO:teuthology.orchestra.run.trial104.stdout:Dependencies resolved. 2026-02-26T03:42:09.486 INFO:teuthology.orchestra.run.trial104.stdout:Nothing to do. 2026-02-26T03:42:09.486 INFO:teuthology.orchestra.run.trial104.stdout:Complete! 2026-02-26T03:42:09.511 DEBUG:teuthology.orchestra.run.trial104:> sudo yum -y install ceph-test 2026-02-26T03:42:09.831 INFO:teuthology.orchestra.run.trial104.stdout:Last metadata expiration check: 0:00:01 ago on Thu 26 Feb 2026 03:42:08 AM UTC. 2026-02-26T03:42:09.895 INFO:teuthology.orchestra.run.trial104.stdout:Package ceph-test-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:42:09.913 INFO:teuthology.orchestra.run.trial104.stdout:Dependencies resolved. 2026-02-26T03:42:09.913 INFO:teuthology.orchestra.run.trial104.stdout:Nothing to do. 2026-02-26T03:42:09.914 INFO:teuthology.orchestra.run.trial104.stdout:Complete! 2026-02-26T03:42:09.935 DEBUG:teuthology.orchestra.run.trial104:> sudo yum -y install ceph 2026-02-26T03:42:10.255 INFO:teuthology.orchestra.run.trial104.stdout:Last metadata expiration check: 0:00:02 ago on Thu 26 Feb 2026 03:42:08 AM UTC. 2026-02-26T03:42:10.322 INFO:teuthology.orchestra.run.trial104.stdout:Package ceph-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:42:10.340 INFO:teuthology.orchestra.run.trial104.stdout:Dependencies resolved. 2026-02-26T03:42:10.341 INFO:teuthology.orchestra.run.trial104.stdout:Nothing to do. 2026-02-26T03:42:10.341 INFO:teuthology.orchestra.run.trial104.stdout:Complete! 2026-02-26T03:42:10.362 DEBUG:teuthology.orchestra.run.trial104:> sudo yum -y install ceph-base 2026-02-26T03:42:10.677 INFO:teuthology.orchestra.run.trial104.stdout:Last metadata expiration check: 0:00:02 ago on Thu 26 Feb 2026 03:42:08 AM UTC. 2026-02-26T03:42:10.743 INFO:teuthology.orchestra.run.trial104.stdout:Package ceph-base-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:42:10.761 INFO:teuthology.orchestra.run.trial104.stdout:Dependencies resolved. 2026-02-26T03:42:10.762 INFO:teuthology.orchestra.run.trial104.stdout:Nothing to do. 2026-02-26T03:42:10.762 INFO:teuthology.orchestra.run.trial104.stdout:Complete! 2026-02-26T03:42:10.783 DEBUG:teuthology.orchestra.run.trial104:> sudo yum -y install cephadm 2026-02-26T03:42:11.099 INFO:teuthology.orchestra.run.trial104.stdout:Last metadata expiration check: 0:00:03 ago on Thu 26 Feb 2026 03:42:08 AM UTC. 2026-02-26T03:42:11.164 INFO:teuthology.orchestra.run.trial104.stdout:Package cephadm-2:20.3.0-5312.gabffb617.el9.noarch is already installed. 2026-02-26T03:42:11.182 INFO:teuthology.orchestra.run.trial104.stdout:Dependencies resolved. 2026-02-26T03:42:11.182 INFO:teuthology.orchestra.run.trial104.stdout:Nothing to do. 2026-02-26T03:42:11.182 INFO:teuthology.orchestra.run.trial104.stdout:Complete! 2026-02-26T03:42:11.204 DEBUG:teuthology.orchestra.run.trial104:> sudo yum -y install ceph-immutable-object-cache 2026-02-26T03:42:11.521 INFO:teuthology.orchestra.run.trial104.stdout:Last metadata expiration check: 0:00:03 ago on Thu 26 Feb 2026 03:42:08 AM UTC. 2026-02-26T03:42:11.587 INFO:teuthology.orchestra.run.trial104.stdout:Package ceph-immutable-object-cache-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:42:11.605 INFO:teuthology.orchestra.run.trial104.stdout:Dependencies resolved. 2026-02-26T03:42:11.606 INFO:teuthology.orchestra.run.trial104.stdout:Nothing to do. 2026-02-26T03:42:11.606 INFO:teuthology.orchestra.run.trial104.stdout:Complete! 2026-02-26T03:42:11.628 DEBUG:teuthology.orchestra.run.trial104:> sudo yum -y install ceph-mgr 2026-02-26T03:42:11.952 INFO:teuthology.orchestra.run.trial104.stdout:Last metadata expiration check: 0:00:03 ago on Thu 26 Feb 2026 03:42:08 AM UTC. 2026-02-26T03:42:12.018 INFO:teuthology.orchestra.run.trial104.stdout:Package ceph-mgr-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:42:12.036 INFO:teuthology.orchestra.run.trial104.stdout:Dependencies resolved. 2026-02-26T03:42:12.036 INFO:teuthology.orchestra.run.trial104.stdout:Nothing to do. 2026-02-26T03:42:12.036 INFO:teuthology.orchestra.run.trial104.stdout:Complete! 2026-02-26T03:42:12.058 DEBUG:teuthology.orchestra.run.trial104:> sudo yum -y install ceph-mgr-dashboard 2026-02-26T03:42:12.375 INFO:teuthology.orchestra.run.trial104.stdout:Last metadata expiration check: 0:00:04 ago on Thu 26 Feb 2026 03:42:08 AM UTC. 2026-02-26T03:42:12.440 INFO:teuthology.orchestra.run.trial104.stdout:Package ceph-mgr-dashboard-2:20.3.0-5312.gabffb617.el9.noarch is already installed. 2026-02-26T03:42:12.458 INFO:teuthology.orchestra.run.trial104.stdout:Dependencies resolved. 2026-02-26T03:42:12.458 INFO:teuthology.orchestra.run.trial104.stdout:Nothing to do. 2026-02-26T03:42:12.458 INFO:teuthology.orchestra.run.trial104.stdout:Complete! 2026-02-26T03:42:12.481 DEBUG:teuthology.orchestra.run.trial104:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-26T03:42:12.804 INFO:teuthology.orchestra.run.trial104.stdout:Last metadata expiration check: 0:00:04 ago on Thu 26 Feb 2026 03:42:08 AM UTC. 2026-02-26T03:42:12.869 INFO:teuthology.orchestra.run.trial104.stdout:Package ceph-mgr-diskprediction-local-2:20.3.0-5312.gabffb617.el9.noarch is already installed. 2026-02-26T03:42:12.886 INFO:teuthology.orchestra.run.trial104.stdout:Dependencies resolved. 2026-02-26T03:42:12.887 INFO:teuthology.orchestra.run.trial104.stdout:Nothing to do. 2026-02-26T03:42:12.887 INFO:teuthology.orchestra.run.trial104.stdout:Complete! 2026-02-26T03:42:12.909 DEBUG:teuthology.orchestra.run.trial104:> sudo yum -y install ceph-mgr-rook 2026-02-26T03:42:13.230 INFO:teuthology.orchestra.run.trial104.stdout:Last metadata expiration check: 0:00:05 ago on Thu 26 Feb 2026 03:42:08 AM UTC. 2026-02-26T03:42:13.295 INFO:teuthology.orchestra.run.trial104.stdout:Package ceph-mgr-rook-2:20.3.0-5312.gabffb617.el9.noarch is already installed. 2026-02-26T03:42:13.313 INFO:teuthology.orchestra.run.trial104.stdout:Dependencies resolved. 2026-02-26T03:42:13.313 INFO:teuthology.orchestra.run.trial104.stdout:Nothing to do. 2026-02-26T03:42:13.314 INFO:teuthology.orchestra.run.trial104.stdout:Complete! 2026-02-26T03:42:13.335 DEBUG:teuthology.orchestra.run.trial104:> sudo yum -y install ceph-mgr-cephadm 2026-02-26T03:42:13.651 INFO:teuthology.orchestra.run.trial104.stdout:Last metadata expiration check: 0:00:05 ago on Thu 26 Feb 2026 03:42:08 AM UTC. 2026-02-26T03:42:13.718 INFO:teuthology.orchestra.run.trial104.stdout:Package ceph-mgr-cephadm-2:20.3.0-5312.gabffb617.el9.noarch is already installed. 2026-02-26T03:42:13.736 INFO:teuthology.orchestra.run.trial104.stdout:Dependencies resolved. 2026-02-26T03:42:13.736 INFO:teuthology.orchestra.run.trial104.stdout:Nothing to do. 2026-02-26T03:42:13.736 INFO:teuthology.orchestra.run.trial104.stdout:Complete! 2026-02-26T03:42:13.758 DEBUG:teuthology.orchestra.run.trial104:> sudo yum -y install ceph-osd 2026-02-26T03:42:14.078 INFO:teuthology.orchestra.run.trial104.stdout:Last metadata expiration check: 0:00:06 ago on Thu 26 Feb 2026 03:42:08 AM UTC. 2026-02-26T03:42:14.143 INFO:teuthology.orchestra.run.trial104.stdout:Package ceph-osd-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:42:14.160 INFO:teuthology.orchestra.run.trial104.stdout:Dependencies resolved. 2026-02-26T03:42:14.161 INFO:teuthology.orchestra.run.trial104.stdout:Nothing to do. 2026-02-26T03:42:14.161 INFO:teuthology.orchestra.run.trial104.stdout:Complete! 2026-02-26T03:42:14.183 DEBUG:teuthology.orchestra.run.trial104:> sudo yum -y install ceph-osd-classic 2026-02-26T03:42:14.500 INFO:teuthology.orchestra.run.trial104.stdout:Last metadata expiration check: 0:00:06 ago on Thu 26 Feb 2026 03:42:08 AM UTC. 2026-02-26T03:42:14.567 INFO:teuthology.orchestra.run.trial104.stdout:Package ceph-osd-classic-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:42:14.584 INFO:teuthology.orchestra.run.trial104.stdout:Dependencies resolved. 2026-02-26T03:42:14.585 INFO:teuthology.orchestra.run.trial104.stdout:Nothing to do. 2026-02-26T03:42:14.585 INFO:teuthology.orchestra.run.trial104.stdout:Complete! 2026-02-26T03:42:14.607 DEBUG:teuthology.orchestra.run.trial104:> sudo yum -y install ceph-fuse 2026-02-26T03:42:14.922 INFO:teuthology.orchestra.run.trial104.stdout:Last metadata expiration check: 0:00:06 ago on Thu 26 Feb 2026 03:42:08 AM UTC. 2026-02-26T03:42:14.986 INFO:teuthology.orchestra.run.trial104.stdout:Package ceph-fuse-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:42:15.004 INFO:teuthology.orchestra.run.trial104.stdout:Dependencies resolved. 2026-02-26T03:42:15.005 INFO:teuthology.orchestra.run.trial104.stdout:Nothing to do. 2026-02-26T03:42:15.005 INFO:teuthology.orchestra.run.trial104.stdout:Complete! 2026-02-26T03:42:15.026 DEBUG:teuthology.orchestra.run.trial104:> sudo yum -y install ceph-volume 2026-02-26T03:42:15.342 INFO:teuthology.orchestra.run.trial104.stdout:Last metadata expiration check: 0:00:07 ago on Thu 26 Feb 2026 03:42:08 AM UTC. 2026-02-26T03:42:15.407 INFO:teuthology.orchestra.run.trial104.stdout:Package ceph-volume-2:20.3.0-5312.gabffb617.el9.noarch is already installed. 2026-02-26T03:42:15.425 INFO:teuthology.orchestra.run.trial104.stdout:Dependencies resolved. 2026-02-26T03:42:15.425 INFO:teuthology.orchestra.run.trial104.stdout:Nothing to do. 2026-02-26T03:42:15.425 INFO:teuthology.orchestra.run.trial104.stdout:Complete! 2026-02-26T03:42:15.447 DEBUG:teuthology.orchestra.run.trial104:> sudo yum -y install librados-devel 2026-02-26T03:42:15.765 INFO:teuthology.orchestra.run.trial104.stdout:Last metadata expiration check: 0:00:07 ago on Thu 26 Feb 2026 03:42:08 AM UTC. 2026-02-26T03:42:15.832 INFO:teuthology.orchestra.run.trial104.stdout:Package librados-devel-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:42:15.850 INFO:teuthology.orchestra.run.trial104.stdout:Dependencies resolved. 2026-02-26T03:42:15.851 INFO:teuthology.orchestra.run.trial104.stdout:Nothing to do. 2026-02-26T03:42:15.851 INFO:teuthology.orchestra.run.trial104.stdout:Complete! 2026-02-26T03:42:15.873 DEBUG:teuthology.orchestra.run.trial104:> sudo yum -y install libcephfs2 2026-02-26T03:42:16.191 INFO:teuthology.orchestra.run.trial104.stdout:Last metadata expiration check: 0:00:08 ago on Thu 26 Feb 2026 03:42:08 AM UTC. 2026-02-26T03:42:16.259 INFO:teuthology.orchestra.run.trial104.stdout:Package libcephfs2-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:42:16.277 INFO:teuthology.orchestra.run.trial104.stdout:Dependencies resolved. 2026-02-26T03:42:16.277 INFO:teuthology.orchestra.run.trial104.stdout:Nothing to do. 2026-02-26T03:42:16.277 INFO:teuthology.orchestra.run.trial104.stdout:Complete! 2026-02-26T03:42:16.299 DEBUG:teuthology.orchestra.run.trial104:> sudo yum -y install libcephfs-devel 2026-02-26T03:42:16.621 INFO:teuthology.orchestra.run.trial104.stdout:Last metadata expiration check: 0:00:08 ago on Thu 26 Feb 2026 03:42:08 AM UTC. 2026-02-26T03:42:16.687 INFO:teuthology.orchestra.run.trial104.stdout:Package libcephfs-devel-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:42:16.704 INFO:teuthology.orchestra.run.trial104.stdout:Dependencies resolved. 2026-02-26T03:42:16.705 INFO:teuthology.orchestra.run.trial104.stdout:Nothing to do. 2026-02-26T03:42:16.705 INFO:teuthology.orchestra.run.trial104.stdout:Complete! 2026-02-26T03:42:16.728 DEBUG:teuthology.orchestra.run.trial104:> sudo yum -y install librados2 2026-02-26T03:42:17.052 INFO:teuthology.orchestra.run.trial104.stdout:Last metadata expiration check: 0:00:09 ago on Thu 26 Feb 2026 03:42:08 AM UTC. 2026-02-26T03:42:17.116 INFO:teuthology.orchestra.run.trial104.stdout:Package librados2-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:42:17.134 INFO:teuthology.orchestra.run.trial104.stdout:Dependencies resolved. 2026-02-26T03:42:17.135 INFO:teuthology.orchestra.run.trial104.stdout:Nothing to do. 2026-02-26T03:42:17.135 INFO:teuthology.orchestra.run.trial104.stdout:Complete! 2026-02-26T03:42:17.157 DEBUG:teuthology.orchestra.run.trial104:> sudo yum -y install librbd1 2026-02-26T03:42:17.477 INFO:teuthology.orchestra.run.trial104.stdout:Last metadata expiration check: 0:00:09 ago on Thu 26 Feb 2026 03:42:08 AM UTC. 2026-02-26T03:42:17.542 INFO:teuthology.orchestra.run.trial104.stdout:Package librbd1-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:42:17.560 INFO:teuthology.orchestra.run.trial104.stdout:Dependencies resolved. 2026-02-26T03:42:17.560 INFO:teuthology.orchestra.run.trial104.stdout:Nothing to do. 2026-02-26T03:42:17.561 INFO:teuthology.orchestra.run.trial104.stdout:Complete! 2026-02-26T03:42:17.582 DEBUG:teuthology.orchestra.run.trial104:> sudo yum -y install python3-rados 2026-02-26T03:42:17.899 INFO:teuthology.orchestra.run.trial104.stdout:Last metadata expiration check: 0:00:09 ago on Thu 26 Feb 2026 03:42:08 AM UTC. 2026-02-26T03:42:17.966 INFO:teuthology.orchestra.run.trial104.stdout:Package python3-rados-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:42:17.984 INFO:teuthology.orchestra.run.trial104.stdout:Dependencies resolved. 2026-02-26T03:42:17.985 INFO:teuthology.orchestra.run.trial104.stdout:Nothing to do. 2026-02-26T03:42:17.985 INFO:teuthology.orchestra.run.trial104.stdout:Complete! 2026-02-26T03:42:18.007 DEBUG:teuthology.orchestra.run.trial104:> sudo yum -y install python3-rgw 2026-02-26T03:42:18.328 INFO:teuthology.orchestra.run.trial104.stdout:Last metadata expiration check: 0:00:10 ago on Thu 26 Feb 2026 03:42:08 AM UTC. 2026-02-26T03:42:18.395 INFO:teuthology.orchestra.run.trial104.stdout:Package python3-rgw-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:42:18.414 INFO:teuthology.orchestra.run.trial104.stdout:Dependencies resolved. 2026-02-26T03:42:18.415 INFO:teuthology.orchestra.run.trial104.stdout:Nothing to do. 2026-02-26T03:42:18.415 INFO:teuthology.orchestra.run.trial104.stdout:Complete! 2026-02-26T03:42:18.438 DEBUG:teuthology.orchestra.run.trial104:> sudo yum -y install python3-cephfs 2026-02-26T03:42:18.761 INFO:teuthology.orchestra.run.trial104.stdout:Last metadata expiration check: 0:00:10 ago on Thu 26 Feb 2026 03:42:08 AM UTC. 2026-02-26T03:42:18.828 INFO:teuthology.orchestra.run.trial104.stdout:Package python3-cephfs-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:42:18.846 INFO:teuthology.orchestra.run.trial104.stdout:Dependencies resolved. 2026-02-26T03:42:18.847 INFO:teuthology.orchestra.run.trial104.stdout:Nothing to do. 2026-02-26T03:42:18.847 INFO:teuthology.orchestra.run.trial104.stdout:Complete! 2026-02-26T03:42:18.869 DEBUG:teuthology.orchestra.run.trial104:> sudo yum -y install python3-rbd 2026-02-26T03:42:19.185 INFO:teuthology.orchestra.run.trial104.stdout:Last metadata expiration check: 0:00:11 ago on Thu 26 Feb 2026 03:42:08 AM UTC. 2026-02-26T03:42:19.251 INFO:teuthology.orchestra.run.trial104.stdout:Package python3-rbd-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:42:19.269 INFO:teuthology.orchestra.run.trial104.stdout:Dependencies resolved. 2026-02-26T03:42:19.270 INFO:teuthology.orchestra.run.trial104.stdout:Nothing to do. 2026-02-26T03:42:19.270 INFO:teuthology.orchestra.run.trial104.stdout:Complete! 2026-02-26T03:42:19.291 DEBUG:teuthology.orchestra.run.trial104:> sudo yum -y install rbd-fuse 2026-02-26T03:42:19.610 INFO:teuthology.orchestra.run.trial104.stdout:Last metadata expiration check: 0:00:11 ago on Thu 26 Feb 2026 03:42:08 AM UTC. 2026-02-26T03:42:19.677 INFO:teuthology.orchestra.run.trial104.stdout:Package rbd-fuse-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:42:19.694 INFO:teuthology.orchestra.run.trial104.stdout:Dependencies resolved. 2026-02-26T03:42:19.695 INFO:teuthology.orchestra.run.trial104.stdout:Nothing to do. 2026-02-26T03:42:19.695 INFO:teuthology.orchestra.run.trial104.stdout:Complete! 2026-02-26T03:42:19.717 DEBUG:teuthology.orchestra.run.trial104:> sudo yum -y install rbd-mirror 2026-02-26T03:42:20.037 INFO:teuthology.orchestra.run.trial104.stdout:Last metadata expiration check: 0:00:12 ago on Thu 26 Feb 2026 03:42:08 AM UTC. 2026-02-26T03:42:20.103 INFO:teuthology.orchestra.run.trial104.stdout:Package rbd-mirror-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:42:20.120 INFO:teuthology.orchestra.run.trial104.stdout:Dependencies resolved. 2026-02-26T03:42:20.121 INFO:teuthology.orchestra.run.trial104.stdout:Nothing to do. 2026-02-26T03:42:20.121 INFO:teuthology.orchestra.run.trial104.stdout:Complete! 2026-02-26T03:42:20.143 DEBUG:teuthology.orchestra.run.trial104:> sudo yum -y install rbd-nbd 2026-02-26T03:42:20.458 INFO:teuthology.orchestra.run.trial104.stdout:Last metadata expiration check: 0:00:12 ago on Thu 26 Feb 2026 03:42:08 AM UTC. 2026-02-26T03:42:20.522 INFO:teuthology.orchestra.run.trial104.stdout:Package rbd-nbd-2:20.3.0-5312.gabffb617.el9.x86_64 is already installed. 2026-02-26T03:42:20.540 INFO:teuthology.orchestra.run.trial104.stdout:Dependencies resolved. 2026-02-26T03:42:20.540 INFO:teuthology.orchestra.run.trial104.stdout:Nothing to do. 2026-02-26T03:42:20.541 INFO:teuthology.orchestra.run.trial104.stdout:Complete! 2026-02-26T03:42:20.565 DEBUG:teuthology.parallel:result is None 2026-02-26T03:42:20.566 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=abffb617683951511fccf3c3cab6e219f2ec83a8 2026-02-26T03:42:20.732 DEBUG:teuthology.orchestra.run.trial104:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-26T03:42:20.753 INFO:teuthology.orchestra.run.trial104.stdout:20.3.0-5312.gabffb617.el9 2026-02-26T03:42:20.753 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5312.gabffb617.el9 2026-02-26T03:42:20.754 INFO:teuthology.task.install:The correct ceph version 20.3.0-5312.gabffb617 is installed. 2026-02-26T03:42:20.756 INFO:teuthology.task.install.util:Utilities already shipped, skip it... 2026-02-26T03:42:20.756 INFO:teuthology.run_tasks:Running task ceph... 2026-02-26T03:42:20.861 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2026-02-26T03:42:20.861 DEBUG:teuthology.orchestra.run.trial104:> sudo chmod 777 /var/log/ceph 2026-02-26T03:42:20.886 INFO:tasks.ceph:Disabling ceph logrotate... 2026-02-26T03:42:20.886 DEBUG:teuthology.orchestra.run.trial104:> sudo rm -f -- /etc/logrotate.d/ceph 2026-02-26T03:42:20.950 INFO:tasks.ceph:Creating extra log directories... 2026-02-26T03:42:20.950 DEBUG:teuthology.orchestra.run.trial104:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2026-02-26T03:42:21.022 INFO:tasks.ceph:Creating ceph cluster ceph... 2026-02-26T03:42:21.023 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore max concurrent transactions': 8}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2026-02-26T03:42:21.023 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/mnt/teuthology/shraddhaag-2026-02-26_02:22:56-crimson-rados-main-distro-debug-trial/71217', 'branch': 'main', 'description': 'crimson-rados/perf/{0-crimson_install clusters/{crimson-fixed-1} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/segmented$/{crimson_seastore_segmented} perf_ignorelist settings/optimized workloads/radosbench_4M_rand_read}', 'email': None, 'first_in_suite': False, 'flavor': 'debug', 'job_id': '71217', 'kernel': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'trial', 'name': 'shraddhaag-2026-02-26_02:22:56-crimson-rados-main-distro-debug-trial', 'no_nested_subset': False, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd': {'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore max concurrent transactions': 8}}, 'crimson_compat': True, 'flavor': 'debug', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH'], 'sha1': 'abffb617683951511fccf3c3cab6e219f2ec83a8'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'global': {'osd crush chooseleaf type': 0, 'osd pool default pg num': 128, 'osd pool default pgp num': 128, 'osd pool default size': 2}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'debug', 'sha1': 'abffb617683951511fccf3c3cab6e219f2ec83a8'}}, 'workunit': {'branch': 'main', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '1f87b69b1a8041112e26ae0ea3c0a2a0fd14d996'}}, 'owner': 'scheduled_shraddhaag@soko04.front.sepia.ceph.com', 'priority': 101, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 5117, 'sha1': 'abffb617683951511fccf3c3cab6e219f2ec83a8', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph_1f87b69b1a8041112e26ae0ea3c0a2a0fd14d996/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph.git', 'suite_sha1': '1f87b69b1a8041112e26ae0ea3c0a2a0fd14d996', 'targets': {'trial104.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD8htrLs1THJsH+RKU+45pysJq9LGLBCRf12EeT+nRDZAGy1UJQ6AqUqVnNJ3KxkNRB0bVXpMsj9h/U/SL7iGNw='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'flavor': 'debug', 'sha1': 'abffb617683951511fccf3c3cab6e219f2ec83a8'}}, {'pexec': {'all': ['sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson']}}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore max concurrent transactions': 8}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}}, 'wait-for-scrub': False, 'crimson_compat': True, 'flavor': 'debug', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH'], 'sha1': 'abffb617683951511fccf3c3cab6e219f2ec83a8', 'cluster': 'ceph'}}, {'ssh_keys': None}, {'cbt': {'benchmarks': {'radosbench': {'concurrent_ops': 4, 'concurrent_procs': 2, 'op_size': [4194304], 'pool_monitoring_list': ['collectl'], 'pool_profile': 'replicated', 'readmode': 'rand', 'run_monitoring_list': ['collectl'], 'time': 60, 'write_only': False}}, 'cluster': {'iterations': 1, 'osds_per_node': 3, 'pool_profiles': {'replicated': {'pg_size': 256, 'pgp_size': 256, 'replication': 'replicated'}}, 'user': 'ubuntu'}, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '6e3d0419ccca0e75a03e08feb708946e198236f4', 'timestamp': '2026-02-26_02:22:56', 'tube': 'trial', 'user': 'shraddhaag', 'verbose': False, 'worker_log': '/home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.904699'} 2026-02-26T03:42:21.023 DEBUG:teuthology.orchestra.run.trial104:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2026-02-26T03:42:21.069 DEBUG:teuthology.orchestra.run.trial104:> sudo install -d -m0777 -- /var/run/ceph 2026-02-26T03:42:21.135 DEBUG:teuthology.orchestra.run.trial104:> set -ex 2026-02-26T03:42:21.135 DEBUG:teuthology.orchestra.run.trial104:> dd if=/scratch_devs of=/dev/stdout 2026-02-26T03:42:21.192 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2026-02-26T03:42:21.192 DEBUG:teuthology.orchestra.run.trial104:> stat /dev/vg_nvme/lv_1 2026-02-26T03:42:21.248 INFO:teuthology.orchestra.run.trial104.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2026-02-26T03:42:21.248 INFO:teuthology.orchestra.run.trial104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-26T03:42:21.248 INFO:teuthology.orchestra.run.trial104.stdout:Device: 6h/6d Inode: 1292 Links: 1 2026-02-26T03:42:21.249 INFO:teuthology.orchestra.run.trial104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-26T03:42:21.249 INFO:teuthology.orchestra.run.trial104.stdout:Context: system_u:object_r:device_t:s0 2026-02-26T03:42:21.249 INFO:teuthology.orchestra.run.trial104.stdout:Access: 2026-02-26 03:31:24.761122804 +0000 2026-02-26T03:42:21.249 INFO:teuthology.orchestra.run.trial104.stdout:Modify: 2026-02-26 03:21:21.430077012 +0000 2026-02-26T03:42:21.249 INFO:teuthology.orchestra.run.trial104.stdout:Change: 2026-02-26 03:21:21.430077012 +0000 2026-02-26T03:42:21.249 INFO:teuthology.orchestra.run.trial104.stdout: Birth: 2026-02-26 03:21:21.430077012 +0000 2026-02-26T03:42:21.249 DEBUG:teuthology.orchestra.run.trial104:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2026-02-26T03:42:21.311 INFO:teuthology.orchestra.run.trial104.stderr:1+0 records in 2026-02-26T03:42:21.311 INFO:teuthology.orchestra.run.trial104.stderr:1+0 records out 2026-02-26T03:42:21.311 INFO:teuthology.orchestra.run.trial104.stderr:512 bytes copied, 0.000157973 s, 3.2 MB/s 2026-02-26T03:42:21.312 DEBUG:teuthology.orchestra.run.trial104:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2026-02-26T03:42:21.369 DEBUG:teuthology.orchestra.run.trial104:> stat /dev/vg_nvme/lv_2 2026-02-26T03:42:21.426 INFO:teuthology.orchestra.run.trial104.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2026-02-26T03:42:21.426 INFO:teuthology.orchestra.run.trial104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-26T03:42:21.426 INFO:teuthology.orchestra.run.trial104.stdout:Device: 6h/6d Inode: 1301 Links: 1 2026-02-26T03:42:21.426 INFO:teuthology.orchestra.run.trial104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-26T03:42:21.427 INFO:teuthology.orchestra.run.trial104.stdout:Context: system_u:object_r:device_t:s0 2026-02-26T03:42:21.427 INFO:teuthology.orchestra.run.trial104.stdout:Access: 2026-02-26 03:31:24.761122804 +0000 2026-02-26T03:42:21.427 INFO:teuthology.orchestra.run.trial104.stdout:Modify: 2026-02-26 03:21:21.430077012 +0000 2026-02-26T03:42:21.427 INFO:teuthology.orchestra.run.trial104.stdout:Change: 2026-02-26 03:21:21.430077012 +0000 2026-02-26T03:42:21.427 INFO:teuthology.orchestra.run.trial104.stdout: Birth: 2026-02-26 03:21:21.430077012 +0000 2026-02-26T03:42:21.427 DEBUG:teuthology.orchestra.run.trial104:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2026-02-26T03:42:21.489 INFO:teuthology.orchestra.run.trial104.stderr:1+0 records in 2026-02-26T03:42:21.489 INFO:teuthology.orchestra.run.trial104.stderr:1+0 records out 2026-02-26T03:42:21.489 INFO:teuthology.orchestra.run.trial104.stderr:512 bytes copied, 0.000150999 s, 3.4 MB/s 2026-02-26T03:42:21.490 DEBUG:teuthology.orchestra.run.trial104:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2026-02-26T03:42:21.546 DEBUG:teuthology.orchestra.run.trial104:> stat /dev/vg_nvme/lv_3 2026-02-26T03:42:21.602 INFO:teuthology.orchestra.run.trial104.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2026-02-26T03:42:21.602 INFO:teuthology.orchestra.run.trial104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-26T03:42:21.603 INFO:teuthology.orchestra.run.trial104.stdout:Device: 6h/6d Inode: 1310 Links: 1 2026-02-26T03:42:21.603 INFO:teuthology.orchestra.run.trial104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-26T03:42:21.603 INFO:teuthology.orchestra.run.trial104.stdout:Context: system_u:object_r:device_t:s0 2026-02-26T03:42:21.603 INFO:teuthology.orchestra.run.trial104.stdout:Access: 2026-02-26 03:31:24.762122778 +0000 2026-02-26T03:42:21.603 INFO:teuthology.orchestra.run.trial104.stdout:Modify: 2026-02-26 03:21:21.431076973 +0000 2026-02-26T03:42:21.603 INFO:teuthology.orchestra.run.trial104.stdout:Change: 2026-02-26 03:21:21.431076973 +0000 2026-02-26T03:42:21.603 INFO:teuthology.orchestra.run.trial104.stdout: Birth: 2026-02-26 03:21:21.431076973 +0000 2026-02-26T03:42:21.604 DEBUG:teuthology.orchestra.run.trial104:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2026-02-26T03:42:21.665 INFO:teuthology.orchestra.run.trial104.stderr:1+0 records in 2026-02-26T03:42:21.665 INFO:teuthology.orchestra.run.trial104.stderr:1+0 records out 2026-02-26T03:42:21.665 INFO:teuthology.orchestra.run.trial104.stderr:512 bytes copied, 0.00014578 s, 3.5 MB/s 2026-02-26T03:42:21.667 DEBUG:teuthology.orchestra.run.trial104:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2026-02-26T03:42:21.724 DEBUG:teuthology.orchestra.run.trial104:> stat /dev/vg_nvme/lv_4 2026-02-26T03:42:21.780 INFO:teuthology.orchestra.run.trial104.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2026-02-26T03:42:21.780 INFO:teuthology.orchestra.run.trial104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-26T03:42:21.781 INFO:teuthology.orchestra.run.trial104.stdout:Device: 6h/6d Inode: 1305 Links: 1 2026-02-26T03:42:21.781 INFO:teuthology.orchestra.run.trial104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-26T03:42:21.781 INFO:teuthology.orchestra.run.trial104.stdout:Context: system_u:object_r:device_t:s0 2026-02-26T03:42:21.781 INFO:teuthology.orchestra.run.trial104.stdout:Access: 2026-02-26 03:31:24.762122778 +0000 2026-02-26T03:42:21.781 INFO:teuthology.orchestra.run.trial104.stdout:Modify: 2026-02-26 03:21:21.431076973 +0000 2026-02-26T03:42:21.781 INFO:teuthology.orchestra.run.trial104.stdout:Change: 2026-02-26 03:21:21.431076973 +0000 2026-02-26T03:42:21.781 INFO:teuthology.orchestra.run.trial104.stdout: Birth: 2026-02-26 03:21:21.431076973 +0000 2026-02-26T03:42:21.782 DEBUG:teuthology.orchestra.run.trial104:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2026-02-26T03:42:21.843 INFO:teuthology.orchestra.run.trial104.stderr:1+0 records in 2026-02-26T03:42:21.843 INFO:teuthology.orchestra.run.trial104.stderr:1+0 records out 2026-02-26T03:42:21.843 INFO:teuthology.orchestra.run.trial104.stderr:512 bytes copied, 0.000143406 s, 3.6 MB/s 2026-02-26T03:42:21.844 DEBUG:teuthology.orchestra.run.trial104:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2026-02-26T03:42:21.901 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-26T03:42:21.902 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@trial104.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2026-02-26T03:42:21.902 INFO:tasks.ceph:Generating config... 2026-02-26T03:42:21.903 INFO:tasks.ceph:[osd] debug monc = 20 2026-02-26T03:42:21.904 INFO:tasks.ceph:[osd] crimson cpu num = 4 2026-02-26T03:42:21.904 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2026-02-26T03:42:21.904 INFO:tasks.ceph:[osd] debug filestore = 0/0 2026-02-26T03:42:21.904 INFO:tasks.ceph:[osd] debug journal = 0/0 2026-02-26T03:42:21.904 INFO:tasks.ceph:[osd] debug ms = 0/0 2026-02-26T03:42:21.904 INFO:tasks.ceph:[osd] debug osd = 0/0 2026-02-26T03:42:21.904 INFO:tasks.ceph:[osd] debug seastore = 20 2026-02-26T03:42:21.904 INFO:tasks.ceph:[osd] debug seastore backref = 20 2026-02-26T03:42:21.904 INFO:tasks.ceph:[osd] debug seastore cache = 20 2026-02-26T03:42:21.904 INFO:tasks.ceph:[osd] debug seastore cleaner = 20 2026-02-26T03:42:21.904 INFO:tasks.ceph:[osd] debug seastore device = 20 2026-02-26T03:42:21.905 INFO:tasks.ceph:[osd] debug seastore epm = 20 2026-02-26T03:42:21.905 INFO:tasks.ceph:[osd] debug seastore fixedkv tree = 20 2026-02-26T03:42:21.905 INFO:tasks.ceph:[osd] debug seastore journal = 20 2026-02-26T03:42:21.905 INFO:tasks.ceph:[osd] debug seastore lba = 20 2026-02-26T03:42:21.905 INFO:tasks.ceph:[osd] debug seastore odata = 20 2026-02-26T03:42:21.905 INFO:tasks.ceph:[osd] debug seastore omap = 20 2026-02-26T03:42:21.905 INFO:tasks.ceph:[osd] debug seastore onode = 20 2026-02-26T03:42:21.905 INFO:tasks.ceph:[osd] debug seastore t = 20 2026-02-26T03:42:21.905 INFO:tasks.ceph:[osd] debug seastore tm = 20 2026-02-26T03:42:21.905 INFO:tasks.ceph:[osd] osd objectstore = seastore 2026-02-26T03:42:21.905 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2026-02-26T03:42:21.905 INFO:tasks.ceph:[osd] seastore cachepin size pershard = 64M 2026-02-26T03:42:21.905 INFO:tasks.ceph:[osd] seastore max concurrent transactions = 8 2026-02-26T03:42:21.906 INFO:tasks.ceph:[global] auth client required = none 2026-02-26T03:42:21.906 INFO:tasks.ceph:[global] auth cluster required = none 2026-02-26T03:42:21.906 INFO:tasks.ceph:[global] auth service required = none 2026-02-26T03:42:21.906 INFO:tasks.ceph:[global] auth supported = none 2026-02-26T03:42:21.906 INFO:tasks.ceph:[global] debug asok = 0/0 2026-02-26T03:42:21.906 INFO:tasks.ceph:[global] debug auth = 0/0 2026-02-26T03:42:21.906 INFO:tasks.ceph:[global] debug bdev = 0/0 2026-02-26T03:42:21.906 INFO:tasks.ceph:[global] debug bluefs = 0/0 2026-02-26T03:42:21.906 INFO:tasks.ceph:[global] debug bluestore = 0/0 2026-02-26T03:42:21.906 INFO:tasks.ceph:[global] debug buffer = 0/0 2026-02-26T03:42:21.906 INFO:tasks.ceph:[global] debug civetweb = 0/0 2026-02-26T03:42:21.906 INFO:tasks.ceph:[global] debug client = 0/0 2026-02-26T03:42:21.906 INFO:tasks.ceph:[global] debug compressor = 0/0 2026-02-26T03:42:21.907 INFO:tasks.ceph:[global] debug context = 0/0 2026-02-26T03:42:21.907 INFO:tasks.ceph:[global] debug crush = 0/0 2026-02-26T03:42:21.907 INFO:tasks.ceph:[global] debug crypto = 0/0 2026-02-26T03:42:21.907 INFO:tasks.ceph:[global] debug dpdk = 0/0 2026-02-26T03:42:21.907 INFO:tasks.ceph:[global] debug eventtrace = 0/0 2026-02-26T03:42:21.907 INFO:tasks.ceph:[global] debug filer = 0/0 2026-02-26T03:42:21.907 INFO:tasks.ceph:[global] debug filestore = 0/0 2026-02-26T03:42:21.907 INFO:tasks.ceph:[global] debug finisher = 0/0 2026-02-26T03:42:21.907 INFO:tasks.ceph:[global] debug fuse = 0/0 2026-02-26T03:42:21.907 INFO:tasks.ceph:[global] debug heartbeatmap = 0/0 2026-02-26T03:42:21.907 INFO:tasks.ceph:[global] debug javaclient = 0/0 2026-02-26T03:42:21.907 INFO:tasks.ceph:[global] debug journal = 0/0 2026-02-26T03:42:21.908 INFO:tasks.ceph:[global] debug journaler = 0/0 2026-02-26T03:42:21.908 INFO:tasks.ceph:[global] debug kstore = 0/0 2026-02-26T03:42:21.908 INFO:tasks.ceph:[global] debug leveldb = 0/0 2026-02-26T03:42:21.908 INFO:tasks.ceph:[global] debug lockdep = 0/0 2026-02-26T03:42:21.908 INFO:tasks.ceph:[global] debug mds = 0/0 2026-02-26T03:42:21.908 INFO:tasks.ceph:[global] debug mds balancer = 0/0 2026-02-26T03:42:21.908 INFO:tasks.ceph:[global] debug mds locker = 0/0 2026-02-26T03:42:21.908 INFO:tasks.ceph:[global] debug mds log = 0/0 2026-02-26T03:42:21.908 INFO:tasks.ceph:[global] debug mds log expire = 0/0 2026-02-26T03:42:21.908 INFO:tasks.ceph:[global] debug mds migrator = 0/0 2026-02-26T03:42:21.908 INFO:tasks.ceph:[global] debug memdb = 0/0 2026-02-26T03:42:21.908 INFO:tasks.ceph:[global] debug mgr = 0/0 2026-02-26T03:42:21.908 INFO:tasks.ceph:[global] debug mgrc = 0/0 2026-02-26T03:42:21.908 INFO:tasks.ceph:[global] debug mon = 0/0 2026-02-26T03:42:21.909 INFO:tasks.ceph:[global] debug monc = 0/0 2026-02-26T03:42:21.909 INFO:tasks.ceph:[global] debug ms = 0/0 2026-02-26T03:42:21.909 INFO:tasks.ceph:[global] debug objclass = 0/0 2026-02-26T03:42:21.909 INFO:tasks.ceph:[global] debug objectcacher = 0/0 2026-02-26T03:42:21.909 INFO:tasks.ceph:[global] debug objecter = 0/0 2026-02-26T03:42:21.909 INFO:tasks.ceph:[global] debug optracker = 0/0 2026-02-26T03:42:21.909 INFO:tasks.ceph:[global] debug osd = 0/0 2026-02-26T03:42:21.909 INFO:tasks.ceph:[global] debug paxos = 0/0 2026-02-26T03:42:21.909 INFO:tasks.ceph:[global] debug perfcounter = 0/0 2026-02-26T03:42:21.909 INFO:tasks.ceph:[global] debug rados = 0/0 2026-02-26T03:42:21.909 INFO:tasks.ceph:[global] debug rbd = 0/0 2026-02-26T03:42:21.909 INFO:tasks.ceph:[global] debug rbd mirror = 0/0 2026-02-26T03:42:21.909 INFO:tasks.ceph:[global] debug rbd replay = 0/0 2026-02-26T03:42:21.910 INFO:tasks.ceph:[global] debug refs = 0/0 2026-02-26T03:42:21.910 INFO:tasks.ceph:[global] debug rgw = 0/0 2026-02-26T03:42:21.910 INFO:tasks.ceph:[global] debug rgw lifecycle = 0/0 2026-02-26T03:42:21.910 INFO:tasks.ceph:[global] debug rgw notification = 0/0 2026-02-26T03:42:21.910 INFO:tasks.ceph:[global] debug rgw sync = 0/0 2026-02-26T03:42:21.910 INFO:tasks.ceph:[global] debug rocksdb = 0/0 2026-02-26T03:42:21.910 INFO:tasks.ceph:[global] debug striper = 0/0 2026-02-26T03:42:21.910 INFO:tasks.ceph:[global] debug throttle = 0/0 2026-02-26T03:42:21.910 INFO:tasks.ceph:[global] debug timer = 0/0 2026-02-26T03:42:21.910 INFO:tasks.ceph:[global] debug tp = 0/0 2026-02-26T03:42:21.910 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2026-02-26T03:42:21.910 INFO:tasks.ceph:[global] ms client mode = crc 2026-02-26T03:42:21.910 INFO:tasks.ceph:[global] ms cluster mode = crc 2026-02-26T03:42:21.911 INFO:tasks.ceph:[global] ms mon client mode = crc 2026-02-26T03:42:21.911 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2026-02-26T03:42:21.911 INFO:tasks.ceph:[global] ms mon service mode = crc 2026-02-26T03:42:21.911 INFO:tasks.ceph:[global] ms service mode = crc 2026-02-26T03:42:21.911 INFO:tasks.ceph:[mgr] debug mgr = 20 2026-02-26T03:42:21.911 INFO:tasks.ceph:[mgr] debug ms = 1 2026-02-26T03:42:21.911 INFO:tasks.ceph:[mon] debug mon = 0/0 2026-02-26T03:42:21.911 INFO:tasks.ceph:[mon] debug ms = 0/0 2026-02-26T03:42:21.911 INFO:tasks.ceph:[mon] debug paxos = 0/0 2026-02-26T03:42:21.911 INFO:tasks.ceph:[mon] osd pool default crimson = True 2026-02-26T03:42:21.912 INFO:tasks.ceph:Setting up mon.a... 2026-02-26T03:42:21.912 DEBUG:teuthology.orchestra.run.trial104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2026-02-26T03:42:21.982 INFO:teuthology.orchestra.run.trial104.stdout:creating /etc/ceph/ceph.keyring 2026-02-26T03:42:21.985 DEBUG:teuthology.orchestra.run.trial104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2026-02-26T03:42:22.068 DEBUG:teuthology.orchestra.run.trial104:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-26T03:42:22.133 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '10.20.193.104')] 2026-02-26T03:42:22.134 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'none', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '10.20.193.104', 'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': 'true', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore max concurrent transactions': 8}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': '0/0', 'debug mon': '0/0', 'debug paxos': '0/0', 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}} 2026-02-26T03:42:22.136 DEBUG:teuthology.orchestra.run.trial104:> set -ex 2026-02-26T03:42:22.136 DEBUG:teuthology.orchestra.run.trial104:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2026-02-26T03:42:22.191 DEBUG:teuthology.orchestra.run.trial104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 10.20.193.104 --print /home/ubuntu/cephtest/ceph.monmap 2026-02-26T03:42:22.271 INFO:teuthology.orchestra.run.trial104.stderr:2026-02-26T03:42:22.269+0000 7f0309c2cf40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:42:22.272 INFO:teuthology.orchestra.run.trial104.stderr:2026-02-26T03:42:22.269+0000 7f0309c2cf40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:42:22.272 INFO:teuthology.orchestra.run.trial104.stderr:2026-02-26T03:42:22.269+0000 7f0309c2cf40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:42:22.273 INFO:teuthology.orchestra.run.trial104.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2026-02-26T03:42:22.273 INFO:teuthology.orchestra.run.trial104.stdout:monmaptool: generated fsid 9e3b9093-cd16-4cf3-9d35-5ccecd63a325 2026-02-26T03:42:22.273 INFO:teuthology.orchestra.run.trial104.stdout:setting min_mon_release = tentacle 2026-02-26T03:42:22.273 INFO:teuthology.orchestra.run.trial104.stdout:epoch 0 2026-02-26T03:42:22.274 INFO:teuthology.orchestra.run.trial104.stdout:fsid 9e3b9093-cd16-4cf3-9d35-5ccecd63a325 2026-02-26T03:42:22.274 INFO:teuthology.orchestra.run.trial104.stdout:last_changed 2026-02-26T03:42:22.270784+0000 2026-02-26T03:42:22.274 INFO:teuthology.orchestra.run.trial104.stdout:created 2026-02-26T03:42:22.270784+0000 2026-02-26T03:42:22.274 INFO:teuthology.orchestra.run.trial104.stdout:min_mon_release 20 (tentacle) 2026-02-26T03:42:22.274 INFO:teuthology.orchestra.run.trial104.stdout:election_strategy: 1 2026-02-26T03:42:22.274 INFO:teuthology.orchestra.run.trial104.stdout:0: [v2:10.20.193.104:3300/0,v1:10.20.193.104:6789/0] mon.a 2026-02-26T03:42:22.274 INFO:teuthology.orchestra.run.trial104.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2026-02-26T03:42:22.275 DEBUG:teuthology.orchestra.run.trial104:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2026-02-26T03:42:22.330 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 9e3b9093-cd16-4cf3-9d35-5ccecd63a325... 2026-02-26T03:42:22.332 DEBUG:teuthology.orchestra.run.trial104:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2026-02-26T03:42:22.409 INFO:teuthology.orchestra.run.trial104.stdout:[global] 2026-02-26T03:42:22.409 INFO:teuthology.orchestra.run.trial104.stdout: chdir = "" 2026-02-26T03:42:22.410 INFO:teuthology.orchestra.run.trial104.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2026-02-26T03:42:22.410 INFO:teuthology.orchestra.run.trial104.stdout: auth supported = none 2026-02-26T03:42:22.410 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:42:22.410 INFO:teuthology.orchestra.run.trial104.stdout: filestore xattr use omap = true 2026-02-26T03:42:22.410 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:42:22.410 INFO:teuthology.orchestra.run.trial104.stdout: mon clock drift allowed = 1.000 2026-02-26T03:42:22.410 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:42:22.410 INFO:teuthology.orchestra.run.trial104.stdout: osd crush chooseleaf type = 0 2026-02-26T03:42:22.410 INFO:teuthology.orchestra.run.trial104.stdout: auth debug = true 2026-02-26T03:42:22.410 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:42:22.411 INFO:teuthology.orchestra.run.trial104.stdout: ms die on old message = true 2026-02-26T03:42:22.411 INFO:teuthology.orchestra.run.trial104.stdout: ms die on bug = true 2026-02-26T03:42:22.411 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:42:22.411 INFO:teuthology.orchestra.run.trial104.stdout: mon max pg per osd = 10000 # >= luminous 2026-02-26T03:42:22.411 INFO:teuthology.orchestra.run.trial104.stdout: mon pg warn max object skew = 0 2026-02-26T03:42:22.411 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:42:22.411 INFO:teuthology.orchestra.run.trial104.stdout: # disable pg_autoscaler by default for new pools 2026-02-26T03:42:22.411 INFO:teuthology.orchestra.run.trial104.stdout: osd_pool_default_pg_autoscale_mode = off 2026-02-26T03:42:22.411 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:42:22.411 INFO:teuthology.orchestra.run.trial104.stdout: osd pool default size = 2 2026-02-26T03:42:22.411 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:42:22.412 INFO:teuthology.orchestra.run.trial104.stdout: mon osd allow primary affinity = true 2026-02-26T03:42:22.412 INFO:teuthology.orchestra.run.trial104.stdout: mon osd allow pg remap = true 2026-02-26T03:42:22.412 INFO:teuthology.orchestra.run.trial104.stdout: mon warn on legacy crush tunables = false 2026-02-26T03:42:22.412 INFO:teuthology.orchestra.run.trial104.stdout: mon warn on crush straw calc version zero = false 2026-02-26T03:42:22.412 INFO:teuthology.orchestra.run.trial104.stdout: mon warn on no sortbitwise = false 2026-02-26T03:42:22.412 INFO:teuthology.orchestra.run.trial104.stdout: mon warn on osd down out interval zero = false 2026-02-26T03:42:22.412 INFO:teuthology.orchestra.run.trial104.stdout: mon warn on too few osds = false 2026-02-26T03:42:22.412 INFO:teuthology.orchestra.run.trial104.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2026-02-26T03:42:22.412 INFO:teuthology.orchestra.run.trial104.stdout: mon_warn_on_pool_no_redundancy = false 2026-02-26T03:42:22.412 INFO:teuthology.orchestra.run.trial104.stdout: mon_allow_pool_size_one = true 2026-02-26T03:42:22.413 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:42:22.413 INFO:teuthology.orchestra.run.trial104.stdout: osd pool default erasure code profile = plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2026-02-26T03:42:22.413 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:42:22.413 INFO:teuthology.orchestra.run.trial104.stdout: osd default data pool replay window = 5 2026-02-26T03:42:22.413 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:42:22.413 INFO:teuthology.orchestra.run.trial104.stdout: mon allow pool delete = true 2026-02-26T03:42:22.413 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:42:22.413 INFO:teuthology.orchestra.run.trial104.stdout: mon cluster log file level = debug 2026-02-26T03:42:22.413 INFO:teuthology.orchestra.run.trial104.stdout: debug asserts on shutdown = true 2026-02-26T03:42:22.413 INFO:teuthology.orchestra.run.trial104.stdout: mon health detail to clog = false 2026-02-26T03:42:22.414 INFO:teuthology.orchestra.run.trial104.stdout: mon host = 10.20.193.104 2026-02-26T03:42:22.414 INFO:teuthology.orchestra.run.trial104.stdout: auth client required = none 2026-02-26T03:42:22.414 INFO:teuthology.orchestra.run.trial104.stdout: auth cluster required = none 2026-02-26T03:42:22.414 INFO:teuthology.orchestra.run.trial104.stdout: auth service required = none 2026-02-26T03:42:22.414 INFO:teuthology.orchestra.run.trial104.stdout: debug asok = 0/0 2026-02-26T03:42:22.414 INFO:teuthology.orchestra.run.trial104.stdout: debug auth = 0/0 2026-02-26T03:42:22.414 INFO:teuthology.orchestra.run.trial104.stdout: debug bdev = 0/0 2026-02-26T03:42:22.414 INFO:teuthology.orchestra.run.trial104.stdout: debug bluefs = 0/0 2026-02-26T03:42:22.414 INFO:teuthology.orchestra.run.trial104.stdout: debug bluestore = 0/0 2026-02-26T03:42:22.414 INFO:teuthology.orchestra.run.trial104.stdout: debug buffer = 0/0 2026-02-26T03:42:22.414 INFO:teuthology.orchestra.run.trial104.stdout: debug civetweb = 0/0 2026-02-26T03:42:22.415 INFO:teuthology.orchestra.run.trial104.stdout: debug client = 0/0 2026-02-26T03:42:22.415 INFO:teuthology.orchestra.run.trial104.stdout: debug compressor = 0/0 2026-02-26T03:42:22.415 INFO:teuthology.orchestra.run.trial104.stdout: debug context = 0/0 2026-02-26T03:42:22.415 INFO:teuthology.orchestra.run.trial104.stdout: debug crush = 0/0 2026-02-26T03:42:22.415 INFO:teuthology.orchestra.run.trial104.stdout: debug crypto = 0/0 2026-02-26T03:42:22.415 INFO:teuthology.orchestra.run.trial104.stdout: debug dpdk = 0/0 2026-02-26T03:42:22.415 INFO:teuthology.orchestra.run.trial104.stdout: debug eventtrace = 0/0 2026-02-26T03:42:22.415 INFO:teuthology.orchestra.run.trial104.stdout: debug filer = 0/0 2026-02-26T03:42:22.415 INFO:teuthology.orchestra.run.trial104.stdout: debug filestore = 0/0 2026-02-26T03:42:22.415 INFO:teuthology.orchestra.run.trial104.stdout: debug finisher = 0/0 2026-02-26T03:42:22.415 INFO:teuthology.orchestra.run.trial104.stdout: debug fuse = 0/0 2026-02-26T03:42:22.415 INFO:teuthology.orchestra.run.trial104.stdout: debug heartbeatmap = 0/0 2026-02-26T03:42:22.416 INFO:teuthology.orchestra.run.trial104.stdout: debug javaclient = 0/0 2026-02-26T03:42:22.416 INFO:teuthology.orchestra.run.trial104.stdout: debug journal = 0/0 2026-02-26T03:42:22.416 INFO:teuthology.orchestra.run.trial104.stdout: debug journaler = 0/0 2026-02-26T03:42:22.416 INFO:teuthology.orchestra.run.trial104.stdout: debug kstore = 0/0 2026-02-26T03:42:22.416 INFO:teuthology.orchestra.run.trial104.stdout: debug leveldb = 0/0 2026-02-26T03:42:22.416 INFO:teuthology.orchestra.run.trial104.stdout: debug lockdep = 0/0 2026-02-26T03:42:22.416 INFO:teuthology.orchestra.run.trial104.stdout: debug mds = 0/0 2026-02-26T03:42:22.416 INFO:teuthology.orchestra.run.trial104.stdout: debug mds balancer = 0/0 2026-02-26T03:42:22.416 INFO:teuthology.orchestra.run.trial104.stdout: debug mds locker = 0/0 2026-02-26T03:42:22.416 INFO:teuthology.orchestra.run.trial104.stdout: debug mds log = 0/0 2026-02-26T03:42:22.417 INFO:teuthology.orchestra.run.trial104.stdout: debug mds log expire = 0/0 2026-02-26T03:42:22.417 INFO:teuthology.orchestra.run.trial104.stdout: debug mds migrator = 0/0 2026-02-26T03:42:22.417 INFO:teuthology.orchestra.run.trial104.stdout: debug memdb = 0/0 2026-02-26T03:42:22.417 INFO:teuthology.orchestra.run.trial104.stdout: debug mgr = 0/0 2026-02-26T03:42:22.417 INFO:teuthology.orchestra.run.trial104.stdout: debug mgrc = 0/0 2026-02-26T03:42:22.417 INFO:teuthology.orchestra.run.trial104.stdout: debug mon = 0/0 2026-02-26T03:42:22.417 INFO:teuthology.orchestra.run.trial104.stdout: debug monc = 0/0 2026-02-26T03:42:22.417 INFO:teuthology.orchestra.run.trial104.stdout: debug ms = 0/0 2026-02-26T03:42:22.417 INFO:teuthology.orchestra.run.trial104.stdout: debug objclass = 0/0 2026-02-26T03:42:22.417 INFO:teuthology.orchestra.run.trial104.stdout: debug objectcacher = 0/0 2026-02-26T03:42:22.417 INFO:teuthology.orchestra.run.trial104.stdout: debug objecter = 0/0 2026-02-26T03:42:22.418 INFO:teuthology.orchestra.run.trial104.stdout: debug optracker = 0/0 2026-02-26T03:42:22.418 INFO:teuthology.orchestra.run.trial104.stdout: debug osd = 0/0 2026-02-26T03:42:22.418 INFO:teuthology.orchestra.run.trial104.stdout: debug paxos = 0/0 2026-02-26T03:42:22.418 INFO:teuthology.orchestra.run.trial104.stdout: debug perfcounter = 0/0 2026-02-26T03:42:22.418 INFO:teuthology.orchestra.run.trial104.stdout: debug rados = 0/0 2026-02-26T03:42:22.418 INFO:teuthology.orchestra.run.trial104.stdout: debug rbd = 0/0 2026-02-26T03:42:22.418 INFO:teuthology.orchestra.run.trial104.stdout: debug rbd mirror = 0/0 2026-02-26T03:42:22.418 INFO:teuthology.orchestra.run.trial104.stdout: debug rbd replay = 0/0 2026-02-26T03:42:22.418 INFO:teuthology.orchestra.run.trial104.stdout: debug refs = 0/0 2026-02-26T03:42:22.418 INFO:teuthology.orchestra.run.trial104.stdout: debug rgw = 0/0 2026-02-26T03:42:22.418 INFO:teuthology.orchestra.run.trial104.stdout: debug rgw lifecycle = 0/0 2026-02-26T03:42:22.418 INFO:teuthology.orchestra.run.trial104.stdout: debug rgw notification = 0/0 2026-02-26T03:42:22.419 INFO:teuthology.orchestra.run.trial104.stdout: debug rgw sync = 0/0 2026-02-26T03:42:22.419 INFO:teuthology.orchestra.run.trial104.stdout: debug rocksdb = 0/0 2026-02-26T03:42:22.419 INFO:teuthology.orchestra.run.trial104.stdout: debug striper = 0/0 2026-02-26T03:42:22.419 INFO:teuthology.orchestra.run.trial104.stdout: debug throttle = 0/0 2026-02-26T03:42:22.419 INFO:teuthology.orchestra.run.trial104.stdout: debug timer = 0/0 2026-02-26T03:42:22.419 INFO:teuthology.orchestra.run.trial104.stdout: debug tp = 0/0 2026-02-26T03:42:22.419 INFO:teuthology.orchestra.run.trial104.stdout: enable experimental unrecoverable data corrupting features = crimson 2026-02-26T03:42:22.419 INFO:teuthology.orchestra.run.trial104.stdout: ms client mode = crc 2026-02-26T03:42:22.419 INFO:teuthology.orchestra.run.trial104.stdout: ms cluster mode = crc 2026-02-26T03:42:22.419 INFO:teuthology.orchestra.run.trial104.stdout: ms mon client mode = crc 2026-02-26T03:42:22.419 INFO:teuthology.orchestra.run.trial104.stdout: ms mon cluster mode = crc 2026-02-26T03:42:22.420 INFO:teuthology.orchestra.run.trial104.stdout: ms mon service mode = crc 2026-02-26T03:42:22.420 INFO:teuthology.orchestra.run.trial104.stdout: ms service mode = crc 2026-02-26T03:42:22.420 INFO:teuthology.orchestra.run.trial104.stdout: fsid = 9e3b9093-cd16-4cf3-9d35-5ccecd63a325 2026-02-26T03:42:22.420 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:42:22.420 INFO:teuthology.orchestra.run.trial104.stdout:[osd] 2026-02-26T03:42:22.420 INFO:teuthology.orchestra.run.trial104.stdout: osd journal size = 100 2026-02-26T03:42:22.420 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:42:22.420 INFO:teuthology.orchestra.run.trial104.stdout: osd scrub load threshold = 5.0 2026-02-26T03:42:22.420 INFO:teuthology.orchestra.run.trial104.stdout: osd scrub max interval = 600 2026-02-26T03:42:22.420 INFO:teuthology.orchestra.run.trial104.stdout: osd mclock profile = high_recovery_ops 2026-02-26T03:42:22.420 INFO:teuthology.orchestra.run.trial104.stdout: osd mclock skip benchmark = true 2026-02-26T03:42:22.421 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:42:22.421 INFO:teuthology.orchestra.run.trial104.stdout: osd recover clone overlap = true 2026-02-26T03:42:22.421 INFO:teuthology.orchestra.run.trial104.stdout: osd recovery max chunk = 1048576 2026-02-26T03:42:22.421 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:42:22.421 INFO:teuthology.orchestra.run.trial104.stdout: osd debug shutdown = true 2026-02-26T03:42:22.421 INFO:teuthology.orchestra.run.trial104.stdout: osd debug op order = true 2026-02-26T03:42:22.421 INFO:teuthology.orchestra.run.trial104.stdout: osd debug verify stray on activate = true 2026-02-26T03:42:22.421 INFO:teuthology.orchestra.run.trial104.stdout: osd debug trim objects = true 2026-02-26T03:42:22.421 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:42:22.421 INFO:teuthology.orchestra.run.trial104.stdout: osd open classes on start = true 2026-02-26T03:42:22.421 INFO:teuthology.orchestra.run.trial104.stdout: osd debug pg log writeout = true 2026-02-26T03:42:22.422 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:42:22.422 INFO:teuthology.orchestra.run.trial104.stdout: osd deep scrub update digest min age = 30 2026-02-26T03:42:22.422 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:42:22.422 INFO:teuthology.orchestra.run.trial104.stdout: osd map max advance = 10 2026-02-26T03:42:22.422 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:42:22.422 INFO:teuthology.orchestra.run.trial104.stdout: journal zero on create = true 2026-02-26T03:42:22.422 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:42:22.422 INFO:teuthology.orchestra.run.trial104.stdout: filestore ondisk finisher threads = 3 2026-02-26T03:42:22.422 INFO:teuthology.orchestra.run.trial104.stdout: filestore apply finisher threads = 3 2026-02-26T03:42:22.422 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:42:22.422 INFO:teuthology.orchestra.run.trial104.stdout: bdev debug aio = true 2026-02-26T03:42:22.423 INFO:teuthology.orchestra.run.trial104.stdout: osd debug misdirected ops = true 2026-02-26T03:42:22.423 INFO:teuthology.orchestra.run.trial104.stdout: debug monc = 20 2026-02-26T03:42:22.423 INFO:teuthology.orchestra.run.trial104.stdout: crimson cpu num = 4 2026-02-26T03:42:22.423 INFO:teuthology.orchestra.run.trial104.stdout: crimson osd obc lru size = 10 2026-02-26T03:42:22.423 INFO:teuthology.orchestra.run.trial104.stdout: debug filestore = 0/0 2026-02-26T03:42:22.423 INFO:teuthology.orchestra.run.trial104.stdout: debug journal = 0/0 2026-02-26T03:42:22.423 INFO:teuthology.orchestra.run.trial104.stdout: debug ms = 0/0 2026-02-26T03:42:22.423 INFO:teuthology.orchestra.run.trial104.stdout: debug osd = 0/0 2026-02-26T03:42:22.423 INFO:teuthology.orchestra.run.trial104.stdout: debug seastore = 20 2026-02-26T03:42:22.423 INFO:teuthology.orchestra.run.trial104.stdout: debug seastore backref = 20 2026-02-26T03:42:22.423 INFO:teuthology.orchestra.run.trial104.stdout: debug seastore cache = 20 2026-02-26T03:42:22.424 INFO:teuthology.orchestra.run.trial104.stdout: debug seastore cleaner = 20 2026-02-26T03:42:22.424 INFO:teuthology.orchestra.run.trial104.stdout: debug seastore device = 20 2026-02-26T03:42:22.424 INFO:teuthology.orchestra.run.trial104.stdout: debug seastore epm = 20 2026-02-26T03:42:22.424 INFO:teuthology.orchestra.run.trial104.stdout: debug seastore fixedkv tree = 20 2026-02-26T03:42:22.424 INFO:teuthology.orchestra.run.trial104.stdout: debug seastore journal = 20 2026-02-26T03:42:22.424 INFO:teuthology.orchestra.run.trial104.stdout: debug seastore lba = 20 2026-02-26T03:42:22.424 INFO:teuthology.orchestra.run.trial104.stdout: debug seastore odata = 20 2026-02-26T03:42:22.424 INFO:teuthology.orchestra.run.trial104.stdout: debug seastore omap = 20 2026-02-26T03:42:22.424 INFO:teuthology.orchestra.run.trial104.stdout: debug seastore onode = 20 2026-02-26T03:42:22.424 INFO:teuthology.orchestra.run.trial104.stdout: debug seastore t = 20 2026-02-26T03:42:22.424 INFO:teuthology.orchestra.run.trial104.stdout: debug seastore tm = 20 2026-02-26T03:42:22.425 INFO:teuthology.orchestra.run.trial104.stdout: osd objectstore = seastore 2026-02-26T03:42:22.425 INFO:teuthology.orchestra.run.trial104.stdout: osd shutdown pgref assert = True 2026-02-26T03:42:22.425 INFO:teuthology.orchestra.run.trial104.stdout: seastore cachepin size pershard = 64M 2026-02-26T03:42:22.425 INFO:teuthology.orchestra.run.trial104.stdout: seastore max concurrent transactions = 8 2026-02-26T03:42:22.425 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:42:22.425 INFO:teuthology.orchestra.run.trial104.stdout:[mgr] 2026-02-26T03:42:22.425 INFO:teuthology.orchestra.run.trial104.stdout: debug ms = 1 2026-02-26T03:42:22.425 INFO:teuthology.orchestra.run.trial104.stdout: debug mgr = 20 2026-02-26T03:42:22.425 INFO:teuthology.orchestra.run.trial104.stdout: debug mon = 20 2026-02-26T03:42:22.425 INFO:teuthology.orchestra.run.trial104.stdout: debug auth = 20 2026-02-26T03:42:22.425 INFO:teuthology.orchestra.run.trial104.stdout: mon reweight min pgs per osd = 4 2026-02-26T03:42:22.425 INFO:teuthology.orchestra.run.trial104.stdout: mon reweight min bytes per osd = 10 2026-02-26T03:42:22.426 INFO:teuthology.orchestra.run.trial104.stdout: mgr/telemetry/nag = false 2026-02-26T03:42:22.426 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:42:22.426 INFO:teuthology.orchestra.run.trial104.stdout:[mon] 2026-02-26T03:42:22.426 INFO:teuthology.orchestra.run.trial104.stdout: debug ms = 0/0 2026-02-26T03:42:22.426 INFO:teuthology.orchestra.run.trial104.stdout: debug mon = 0/0 2026-02-26T03:42:22.426 INFO:teuthology.orchestra.run.trial104.stdout: debug paxos = 0/0 2026-02-26T03:42:22.426 INFO:teuthology.orchestra.run.trial104.stdout: debug auth = 20 2026-02-26T03:42:22.426 INFO:teuthology.orchestra.run.trial104.stdout: mon data avail warn = 5 2026-02-26T03:42:22.426 INFO:teuthology.orchestra.run.trial104.stdout: mon mgr mkfs grace = 240 2026-02-26T03:42:22.426 INFO:teuthology.orchestra.run.trial104.stdout: mon reweight min pgs per osd = 4 2026-02-26T03:42:22.426 INFO:teuthology.orchestra.run.trial104.stdout: mon osd reporter subtree level = osd 2026-02-26T03:42:22.426 INFO:teuthology.orchestra.run.trial104.stdout: mon osd prime pg temp = true 2026-02-26T03:42:22.427 INFO:teuthology.orchestra.run.trial104.stdout: mon reweight min bytes per osd = 10 2026-02-26T03:42:22.427 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:42:22.427 INFO:teuthology.orchestra.run.trial104.stdout: # rotate auth tickets quickly to exercise renewal paths 2026-02-26T03:42:22.427 INFO:teuthology.orchestra.run.trial104.stdout: auth mon ticket ttl = 660 # 11m 2026-02-26T03:42:22.427 INFO:teuthology.orchestra.run.trial104.stdout: auth service ticket ttl = 240 # 4m 2026-02-26T03:42:22.427 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:42:22.427 INFO:teuthology.orchestra.run.trial104.stdout: # don't complain about insecure global_id in the test suite 2026-02-26T03:42:22.427 INFO:teuthology.orchestra.run.trial104.stdout: mon_warn_on_insecure_global_id_reclaim = false 2026-02-26T03:42:22.427 INFO:teuthology.orchestra.run.trial104.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2026-02-26T03:42:22.427 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:42:22.428 INFO:teuthology.orchestra.run.trial104.stdout: # 1m isn't quite enough 2026-02-26T03:42:22.428 INFO:teuthology.orchestra.run.trial104.stdout: mon_down_mkfs_grace = 2m 2026-02-26T03:42:22.428 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:42:22.428 INFO:teuthology.orchestra.run.trial104.stdout: mon_warn_on_filestore_osds = false 2026-02-26T03:42:22.428 INFO:teuthology.orchestra.run.trial104.stdout: osd pool default crimson = True 2026-02-26T03:42:22.428 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:42:22.428 INFO:teuthology.orchestra.run.trial104.stdout:[client] 2026-02-26T03:42:22.428 INFO:teuthology.orchestra.run.trial104.stdout: rgw cache enabled = true 2026-02-26T03:42:22.428 INFO:teuthology.orchestra.run.trial104.stdout: rgw enable ops log = true 2026-02-26T03:42:22.428 INFO:teuthology.orchestra.run.trial104.stdout: rgw enable usage log = true 2026-02-26T03:42:22.428 INFO:teuthology.orchestra.run.trial104.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2026-02-26T03:42:22.429 INFO:teuthology.orchestra.run.trial104.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2026-02-26T03:42:22.429 INFO:teuthology.orchestra.run.trial104.stdout:[mon.a] 2026-02-26T03:42:22.430 INFO:tasks.ceph:Creating admin key on mon.a... 2026-02-26T03:42:22.430 DEBUG:teuthology.orchestra.run.trial104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2026-02-26T03:42:22.519 INFO:tasks.ceph:Copying monmap to all nodes... 2026-02-26T03:42:22.519 DEBUG:teuthology.orchestra.run.trial104:> set -ex 2026-02-26T03:42:22.520 DEBUG:teuthology.orchestra.run.trial104:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2026-02-26T03:42:22.536 DEBUG:teuthology.orchestra.run.trial104:> set -ex 2026-02-26T03:42:22.536 DEBUG:teuthology.orchestra.run.trial104:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2026-02-26T03:42:22.592 INFO:tasks.ceph:Sending monmap to node ubuntu@trial104.front.sepia.ceph.com 2026-02-26T03:42:22.593 DEBUG:teuthology.orchestra.run.trial104:> set -ex 2026-02-26T03:42:22.593 DEBUG:teuthology.orchestra.run.trial104:> sudo dd of=/etc/ceph/ceph.keyring 2026-02-26T03:42:22.593 DEBUG:teuthology.orchestra.run.trial104:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-26T03:42:22.665 DEBUG:teuthology.orchestra.run.trial104:> set -ex 2026-02-26T03:42:22.665 DEBUG:teuthology.orchestra.run.trial104:> dd of=/home/ubuntu/cephtest/ceph.monmap 2026-02-26T03:42:22.721 INFO:tasks.ceph:Setting up mon nodes... 2026-02-26T03:42:22.721 INFO:tasks.ceph:Setting up mgr nodes... 2026-02-26T03:42:22.721 DEBUG:teuthology.orchestra.run.trial104:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2026-02-26T03:42:22.812 INFO:teuthology.orchestra.run.trial104.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2026-02-26T03:42:22.815 INFO:tasks.ceph:Setting up mds nodes... 2026-02-26T03:42:22.815 INFO:tasks.ceph_client:Setting up client nodes... 2026-02-26T03:42:22.815 DEBUG:teuthology.orchestra.run.trial104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2026-02-26T03:42:22.856 INFO:teuthology.orchestra.run.trial104.stdout:creating /etc/ceph/ceph.client.0.keyring 2026-02-26T03:42:22.868 INFO:tasks.ceph:Running mkfs on osd nodes... 2026-02-26T03:42:22.868 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@trial104.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2026-02-26T03:42:22.869 DEBUG:teuthology.orchestra.run.trial104:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2026-02-26T03:42:22.933 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-26T03:42:22.933 INFO:tasks.ceph:role: osd.0 2026-02-26T03:42:22.933 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@trial104.front.sepia.ceph.com 2026-02-26T03:42:22.934 DEBUG:teuthology.orchestra.run.trial104:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2026-02-26T03:42:22.998 INFO:teuthology.orchestra.run.trial104.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=45875200 blks 2026-02-26T03:42:22.998 INFO:teuthology.orchestra.run.trial104.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-26T03:42:22.998 INFO:teuthology.orchestra.run.trial104.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-26T03:42:22.998 INFO:teuthology.orchestra.run.trial104.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-26T03:42:22.998 INFO:teuthology.orchestra.run.trial104.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-26T03:42:22.999 INFO:teuthology.orchestra.run.trial104.stdout: = sunit=0 swidth=0 blks 2026-02-26T03:42:22.999 INFO:teuthology.orchestra.run.trial104.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-26T03:42:22.999 INFO:teuthology.orchestra.run.trial104.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-26T03:42:22.999 INFO:teuthology.orchestra.run.trial104.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-26T03:42:22.999 INFO:teuthology.orchestra.run.trial104.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-26T03:42:23.569 INFO:teuthology.orchestra.run.trial104.stdout:Discarding blocks...Done. 2026-02-26T03:42:23.571 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@trial104.front.sepia.ceph.com -o noatime 2026-02-26T03:42:23.571 DEBUG:teuthology.orchestra.run.trial104:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2026-02-26T03:42:23.643 DEBUG:teuthology.orchestra.run.trial104:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2026-02-26T03:42:23.713 DEBUG:teuthology.orchestra.run.trial104:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2026-02-26T03:42:23.777 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-26T03:42:23.778 INFO:tasks.ceph:role: osd.1 2026-02-26T03:42:23.778 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@trial104.front.sepia.ceph.com 2026-02-26T03:42:23.778 DEBUG:teuthology.orchestra.run.trial104:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2026-02-26T03:42:23.843 INFO:teuthology.orchestra.run.trial104.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=45875200 blks 2026-02-26T03:42:23.844 INFO:teuthology.orchestra.run.trial104.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-26T03:42:23.844 INFO:teuthology.orchestra.run.trial104.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-26T03:42:23.844 INFO:teuthology.orchestra.run.trial104.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-26T03:42:23.844 INFO:teuthology.orchestra.run.trial104.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-26T03:42:23.844 INFO:teuthology.orchestra.run.trial104.stdout: = sunit=0 swidth=0 blks 2026-02-26T03:42:23.844 INFO:teuthology.orchestra.run.trial104.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-26T03:42:23.844 INFO:teuthology.orchestra.run.trial104.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-26T03:42:23.844 INFO:teuthology.orchestra.run.trial104.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-26T03:42:23.844 INFO:teuthology.orchestra.run.trial104.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-26T03:42:24.403 INFO:teuthology.orchestra.run.trial104.stdout:Discarding blocks...Done. 2026-02-26T03:42:24.405 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@trial104.front.sepia.ceph.com -o noatime 2026-02-26T03:42:24.405 DEBUG:teuthology.orchestra.run.trial104:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2026-02-26T03:42:24.473 DEBUG:teuthology.orchestra.run.trial104:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2026-02-26T03:42:24.540 DEBUG:teuthology.orchestra.run.trial104:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2026-02-26T03:42:24.603 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-26T03:42:24.604 INFO:tasks.ceph:role: osd.2 2026-02-26T03:42:24.604 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@trial104.front.sepia.ceph.com 2026-02-26T03:42:24.604 DEBUG:teuthology.orchestra.run.trial104:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2026-02-26T03:42:24.669 INFO:teuthology.orchestra.run.trial104.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=45875200 blks 2026-02-26T03:42:24.670 INFO:teuthology.orchestra.run.trial104.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-26T03:42:24.670 INFO:teuthology.orchestra.run.trial104.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-26T03:42:24.670 INFO:teuthology.orchestra.run.trial104.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-26T03:42:24.670 INFO:teuthology.orchestra.run.trial104.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-26T03:42:24.670 INFO:teuthology.orchestra.run.trial104.stdout: = sunit=0 swidth=0 blks 2026-02-26T03:42:24.670 INFO:teuthology.orchestra.run.trial104.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-26T03:42:24.670 INFO:teuthology.orchestra.run.trial104.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-26T03:42:24.670 INFO:teuthology.orchestra.run.trial104.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-26T03:42:24.670 INFO:teuthology.orchestra.run.trial104.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-26T03:42:25.226 INFO:teuthology.orchestra.run.trial104.stdout:Discarding blocks...Done. 2026-02-26T03:42:25.228 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@trial104.front.sepia.ceph.com -o noatime 2026-02-26T03:42:25.228 DEBUG:teuthology.orchestra.run.trial104:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2026-02-26T03:42:25.295 DEBUG:teuthology.orchestra.run.trial104:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2026-02-26T03:42:25.362 DEBUG:teuthology.orchestra.run.trial104:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-26T03:42:25.511 INFO:teuthology.orchestra.run.trial104.stderr:INFO  osd - OSD::main: parsing early config 2026-02-26T03:42:25.564 INFO:teuthology.orchestra.run.trial104.stderr:INFO 2026-02-26 03:42:25,563 seastar - Reactor backend: linux-aio 2026-02-26T03:42:25.573 INFO:teuthology.orchestra.run.trial104.stderr:INFO 2026-02-26 03:42:25,572 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:42:25.573 INFO:teuthology.orchestra.run.trial104.stderr:INFO 2026-02-26 03:42:25,572 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:42:25.575 INFO:teuthology.orchestra.run.trial104.stderr:==39365==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-26T03:42:25.635 INFO:teuthology.orchestra.run.trial104.stderr:DEBUG 2026-02-26 03:42:25,634 [shard 0:main] seastar - reactor::drain 2026-02-26T03:42:27.673 INFO:teuthology.orchestra.run.trial104.stderr:----------------------------------------------------- 2026-02-26T03:42:27.674 INFO:teuthology.orchestra.run.trial104.stderr:Suppressions used: 2026-02-26T03:42:27.674 INFO:teuthology.orchestra.run.trial104.stderr: count bytes template 2026-02-26T03:42:27.674 INFO:teuthology.orchestra.run.trial104.stderr: 1 8 MallocExtension::Register 2026-02-26T03:42:27.674 INFO:teuthology.orchestra.run.trial104.stderr:----------------------------------------------------- 2026-02-26T03:42:27.674 INFO:teuthology.orchestra.run.trial104.stderr: 2026-02-26T03:42:27.693 INFO:teuthology.orchestra.run.trial104.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-26T03:42:27.693 INFO:teuthology.orchestra.run.trial104.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-26T03:42:27.732 INFO:teuthology.orchestra.run.trial104.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-26T03:42:27.732 INFO:teuthology.orchestra.run.trial104.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-26T03:42:27.738 INFO:teuthology.orchestra.run.trial104.stderr:INFO 2026-02-26 03:42:27,737 seastar - Reactor backend: linux-aio 2026-02-26T03:42:27.749 INFO:teuthology.orchestra.run.trial104.stderr:INFO 2026-02-26 03:42:27,747 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:42:27.749 INFO:teuthology.orchestra.run.trial104.stderr:INFO 2026-02-26 03:42:27,747 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:42:27.752 INFO:teuthology.orchestra.run.trial104.stderr:==39363==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-26T03:42:27.752 INFO:teuthology.orchestra.run.trial104.stderr:INFO 2026-02-26 03:42:27,751 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-26T03:42:27.892 INFO:teuthology.orchestra.run.trial104.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x1090832 0xf2e0dc 0x9c50c5c 0x9c50f59 0x9c511cc 0x9c521cc 0x9c5273a 0x9c52a5f 0x9c52bc9 0x9d32f39 0x9d45d3c 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6fa06ea 0x6fa44eb 0x6faa095 0x7419054 0x743ba5c 0x728f369 0x728f461 0x736153a 0x73615f3 0x73617aa 0x743badd 0x728f4de 0x728f563 0x72bb301 0x7423ee4 0x7423f69 0x7424047 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:42:27.892 INFO:teuthology.orchestra.run.trial104.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x151f5 0x316f517 0x9c5110d 0x9c51482 0x9c51f31 0x9c5242f 0x9c52a5f 0x9c52bc9 0x9d32f39 0x9d45d3c 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6fa06ea 0x6fa44eb 0x6faa095 0x7419054 0x743ba5c 0x728f369 0x728f461 0x736153a 0x73615f3 0x73617aa 0x743badd 0x728f4de 0x728f563 0x72bb301 0x7423ee4 0x7423f69 0x7424047 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:42:27.893 INFO:teuthology.orchestra.run.trial104.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1297e4 /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x15197 0x9c50af8 0x9c50f59 0x9c511cc 0x9c51482 0x9c51a44 0x9c59eeb 0x9c5a5cb 0x9d43118 0x9d477c3 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6fa06ea 0x6fa44eb 0x6faa095 0x7419054 0x743ba5c 0x728f369 0x728f461 0x736153a 0x73615f3 0x73617aa 0x743badd 0x728f4de 0x728f563 0x72bb301 0x7423ee4 0x7423f69 0x7424047 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:42:27.989 INFO:teuthology.orchestra.run.trial104.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc65c2 /lib64/libasan.so.8+0x15197 0x2a55a3f 0x7d837b9 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f7a738 0x9bf83c1 0x9bf8bfc 0xecf14d /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb46f44 2026-02-26T03:42:28.028 INFO:teuthology.orchestra.run.trial104.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x151f5 0x2a55a3f 0x7d837b9 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:42:28.028 INFO:teuthology.orchestra.run.trial104.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x2a55990 0x7d837b9 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:42:28.028 INFO:teuthology.orchestra.run.trial104.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc65d9 /lib64/libasan.so.8+0x15197 0x2a55a3f 0x7d837b9 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:42:28.152 INFO:teuthology.orchestra.run.trial104.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x9d48c0f 0x78947dd 0x7be9c23 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6f43702 0x6f45a6a 0x7483f52 0x72a690f 0x72a6a11 0x72a6b09 0x7360706 0x73607bf 0x7360972 0x7360aaa 0x7360b25 0x7360baa 0x7360d05 0x7423961 0x7483fdb 0x728dda4 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:42:28.153 INFO:teuthology.orchestra.run.trial104.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc65c2 /lib64/libasan.so.8+0x15197 0x316f517 0x9c511ba 0x9c52ef3 0x9c53440 0x9c5363f 0x9d38961 0x9d390bc 0x9d4334a 0x9d477c3 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6f43702 0x6f45a6a 0x7483f52 0x72a690f 0x72a6a11 0x72a6b09 0x7360706 0x73607bf 0x7360972 0x7360aaa 0x7360b25 0x7360baa 0x7360d05 0x7423961 0x7483fdb 0x728dda4 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:42:28.153 INFO:teuthology.orchestra.run.trial104.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x9c50d5f 0x9c511cc 0x9c52ef3 0x9d368ca 0x9d3851c 0x9d38b76 0x9d390bc 0x9d4334a 0x9d477c3 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6f43702 0x6f45a6a 0x7483f52 0x72a690f 0x72a6a11 0x72a6b09 0x7360706 0x73607bf 0x7360972 0x7360aaa 0x7360b25 0x7360baa 0x7360d05 0x7423961 0x7483fdb 0x728dda4 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:42:28.234 INFO:teuthology.orchestra.run.trial104.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x129749 /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x15197 0x2a55a3f 0x7d837b9 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f7a738 0x9bf83c1 0x9bf8bfc 0xecf14d /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb46f44 2026-02-26T03:42:28.248 INFO:teuthology.orchestra.run.trial104.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/ld-linux-x86-64.so.2+0x16ebc /lib64/libasan.so.8+0xc65b1 /lib64/libasan.so.8+0x15197 0x2a55a3f 0x7d837b9 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:42:28.248 INFO:teuthology.orchestra.run.trial104.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x2a54d2f 0x7d837e7 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:42:28.252 INFO:teuthology.orchestra.run.trial104.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x151f5 0x2a55a3f 0x7d837b9 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:42:30.402 INFO:teuthology.orchestra.run.trial104.stderr:----------------------------------------------------- 2026-02-26T03:42:30.403 INFO:teuthology.orchestra.run.trial104.stderr:Suppressions used: 2026-02-26T03:42:30.403 INFO:teuthology.orchestra.run.trial104.stderr: count bytes template 2026-02-26T03:42:30.403 INFO:teuthology.orchestra.run.trial104.stderr: 1 8 MallocExtension::Register 2026-02-26T03:42:30.403 INFO:teuthology.orchestra.run.trial104.stderr:----------------------------------------------------- 2026-02-26T03:42:30.403 INFO:teuthology.orchestra.run.trial104.stderr: 2026-02-26T03:42:30.438 DEBUG:teuthology.orchestra.run.trial104:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2026-02-26T03:42:30.463 DEBUG:teuthology.orchestra.run.trial104:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-26T03:42:30.615 INFO:teuthology.orchestra.run.trial104.stderr:INFO  osd - OSD::main: parsing early config 2026-02-26T03:42:30.668 INFO:teuthology.orchestra.run.trial104.stderr:INFO 2026-02-26 03:42:30,667 seastar - Reactor backend: linux-aio 2026-02-26T03:42:30.678 INFO:teuthology.orchestra.run.trial104.stderr:INFO 2026-02-26 03:42:30,677 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:42:30.679 INFO:teuthology.orchestra.run.trial104.stderr:INFO 2026-02-26 03:42:30,677 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:42:30.681 INFO:teuthology.orchestra.run.trial104.stderr:==39430==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-26T03:42:30.741 INFO:teuthology.orchestra.run.trial104.stderr:DEBUG 2026-02-26 03:42:30,740 [shard 0:main] seastar - reactor::drain 2026-02-26T03:42:32.811 INFO:teuthology.orchestra.run.trial104.stderr:----------------------------------------------------- 2026-02-26T03:42:32.811 INFO:teuthology.orchestra.run.trial104.stderr:Suppressions used: 2026-02-26T03:42:32.812 INFO:teuthology.orchestra.run.trial104.stderr: count bytes template 2026-02-26T03:42:32.812 INFO:teuthology.orchestra.run.trial104.stderr: 1 8 MallocExtension::Register 2026-02-26T03:42:32.812 INFO:teuthology.orchestra.run.trial104.stderr:----------------------------------------------------- 2026-02-26T03:42:32.812 INFO:teuthology.orchestra.run.trial104.stderr: 2026-02-26T03:42:32.831 INFO:teuthology.orchestra.run.trial104.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-26T03:42:32.831 INFO:teuthology.orchestra.run.trial104.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-26T03:42:32.872 INFO:teuthology.orchestra.run.trial104.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-26T03:42:32.872 INFO:teuthology.orchestra.run.trial104.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-26T03:42:32.879 INFO:teuthology.orchestra.run.trial104.stderr:INFO 2026-02-26 03:42:32,878 seastar - Reactor backend: linux-aio 2026-02-26T03:42:32.892 INFO:teuthology.orchestra.run.trial104.stderr:INFO 2026-02-26 03:42:32,891 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:42:32.892 INFO:teuthology.orchestra.run.trial104.stderr:INFO 2026-02-26 03:42:32,891 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:42:32.896 INFO:teuthology.orchestra.run.trial104.stderr:==39428==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-26T03:42:32.896 INFO:teuthology.orchestra.run.trial104.stderr:INFO 2026-02-26 03:42:32,895 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-26T03:42:33.035 INFO:teuthology.orchestra.run.trial104.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x10069f /lib64/libasan.so.8+0xc65b9 /lib64/libasan.so.8+0x157c7 0x10908ef 0xf2e0dc 0x9c554b6 0x9c55763 0x9d42c55 0x9d477c3 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6fa06ea 0x6fa44eb 0x6faa095 0x7419054 0x743ba5c 0x728f369 0x728f461 0x736153a 0x73615f3 0x73617aa 0x743badd 0x728f4de 0x728f563 0x72bb301 0x7423ee4 0x7423f69 0x7424047 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:42:33.035 INFO:teuthology.orchestra.run.trial104.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1006a8 /lib64/libasan.so.8+0xc530 /lib64/libasan.so.8+0x8eea /lib64/libasan.so.8+0xedc45 0xe1d59b 0xe1d8f2 0x7be9f2c 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6fa06ea 0x6fa44eb 0x6faa095 0x7419054 0x743ba5c 0x728f369 0x728f461 0x736153a 0x73615f3 0x73617aa 0x743badd 0x728f4de 0x728f563 0x72bb301 0x7423ee4 0x7423f69 0x7424047 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:42:33.035 INFO:teuthology.orchestra.run.trial104.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x9c513a8 0x9c51a44 0x9c59eeb 0x9c5a5cb 0x9d43118 0x9d477c3 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6fa06ea 0x6fa44eb 0x6faa095 0x7419054 0x743ba5c 0x728f369 0x728f461 0x736153a 0x73615f3 0x73617aa 0x743badd 0x728f4de 0x728f563 0x72bb301 0x7423ee4 0x7423f69 0x7424047 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:42:33.133 INFO:teuthology.orchestra.run.trial104.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x2a55901 0x7d837b9 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f7a738 0x9bf83c1 0x9bf8bfc 0xecf14d /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb46f44 2026-02-26T03:42:33.139 INFO:teuthology.orchestra.run.trial104.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x2a55990 0x7d837b9 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:42:33.171 INFO:teuthology.orchestra.run.trial104.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x2a55954 0x7d837b9 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:42:33.171 INFO:teuthology.orchestra.run.trial104.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x151b5 0x2a55a3f 0x7d837b9 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:42:33.294 INFO:teuthology.orchestra.run.trial104.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x9c5139f 0x9c51a44 0x9c59eeb 0x9c5a5cb 0x9d43118 0x9d477c3 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6f43702 0x6f45a6a 0x7483f52 0x72a690f 0x72a6a11 0x72a6b09 0x7360706 0x73607bf 0x7360972 0x7360aaa 0x7360b25 0x7360baa 0x7360d05 0x7423961 0x7483fdb 0x728dda4 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:42:33.294 INFO:teuthology.orchestra.run.trial104.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc6592 /lib64/libasan.so.8+0x15197 0x360962d 0x9c63157 0x9c632ef 0x9d23007 0x9d2319c 0x9d2352e 0x9d24b15 0x9d25e30 0x9d26250 0x9d42adb 0x9d477c3 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6f43702 0x6f45a6a 0x7483f52 0x72a690f 0x72a6a11 0x72a6b09 0x7360706 0x73607bf 0x7360972 0x7360aaa 0x7360b25 0x7360baa 0x7360d05 0x7423961 0x7483fdb 0x728dda4 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:42:33.294 INFO:teuthology.orchestra.run.trial104.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1297e4 /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x157c7 0x10908ef 0xf2e0dc 0x9d33899 0x9d33e86 0x9d34209 0x9d44360 0x9d477c3 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6f43702 0x6f45a6a 0x7483f52 0x72a690f 0x72a6a11 0x72a6b09 0x7360706 0x73607bf 0x7360972 0x7360aaa 0x7360b25 0x7360baa 0x7360d05 0x7423961 0x7483fdb 0x728dda4 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:42:33.376 INFO:teuthology.orchestra.run.trial104.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x2a559ce 0x7d837b9 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f7a738 0x9bf83c1 0x9bf8bfc 0xecf14d /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb46f44 2026-02-26T03:42:33.390 INFO:teuthology.orchestra.run.trial104.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x2a55954 0x7d837b9 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:42:33.390 INFO:teuthology.orchestra.run.trial104.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x2a55875 0x7d837b9 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:42:33.391 INFO:teuthology.orchestra.run.trial104.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x151f5 0x2a55a3f 0x7d837b9 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:42:35.539 INFO:teuthology.orchestra.run.trial104.stderr:----------------------------------------------------- 2026-02-26T03:42:35.539 INFO:teuthology.orchestra.run.trial104.stderr:Suppressions used: 2026-02-26T03:42:35.539 INFO:teuthology.orchestra.run.trial104.stderr: count bytes template 2026-02-26T03:42:35.539 INFO:teuthology.orchestra.run.trial104.stderr: 1 8 MallocExtension::Register 2026-02-26T03:42:35.539 INFO:teuthology.orchestra.run.trial104.stderr:----------------------------------------------------- 2026-02-26T03:42:35.540 INFO:teuthology.orchestra.run.trial104.stderr: 2026-02-26T03:42:35.575 DEBUG:teuthology.orchestra.run.trial104:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2026-02-26T03:42:35.600 DEBUG:teuthology.orchestra.run.trial104:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-26T03:42:35.750 INFO:teuthology.orchestra.run.trial104.stderr:INFO  osd - OSD::main: parsing early config 2026-02-26T03:42:35.801 INFO:teuthology.orchestra.run.trial104.stderr:INFO 2026-02-26 03:42:35,800 seastar - Reactor backend: linux-aio 2026-02-26T03:42:35.818 INFO:teuthology.orchestra.run.trial104.stderr:INFO 2026-02-26 03:42:35,817 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:42:35.818 INFO:teuthology.orchestra.run.trial104.stderr:INFO 2026-02-26 03:42:35,817 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:42:35.820 INFO:teuthology.orchestra.run.trial104.stderr:==39498==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-26T03:42:35.880 INFO:teuthology.orchestra.run.trial104.stderr:DEBUG 2026-02-26 03:42:35,878 [shard 0:main] seastar - reactor::drain 2026-02-26T03:42:37.928 INFO:teuthology.orchestra.run.trial104.stderr:----------------------------------------------------- 2026-02-26T03:42:37.928 INFO:teuthology.orchestra.run.trial104.stderr:Suppressions used: 2026-02-26T03:42:37.928 INFO:teuthology.orchestra.run.trial104.stderr: count bytes template 2026-02-26T03:42:37.929 INFO:teuthology.orchestra.run.trial104.stderr: 1 8 MallocExtension::Register 2026-02-26T03:42:37.929 INFO:teuthology.orchestra.run.trial104.stderr:----------------------------------------------------- 2026-02-26T03:42:37.929 INFO:teuthology.orchestra.run.trial104.stderr: 2026-02-26T03:42:37.948 INFO:teuthology.orchestra.run.trial104.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-26T03:42:37.948 INFO:teuthology.orchestra.run.trial104.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-26T03:42:37.996 INFO:teuthology.orchestra.run.trial104.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-26T03:42:37.996 INFO:teuthology.orchestra.run.trial104.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-26T03:42:38.003 INFO:teuthology.orchestra.run.trial104.stderr:INFO 2026-02-26 03:42:38,002 seastar - Reactor backend: linux-aio 2026-02-26T03:42:38.017 INFO:teuthology.orchestra.run.trial104.stderr:INFO 2026-02-26 03:42:38,016 [shard 2:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:42:38.018 INFO:teuthology.orchestra.run.trial104.stderr:INFO 2026-02-26 03:42:38,016 [shard 2:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-26T03:42:38.022 INFO:teuthology.orchestra.run.trial104.stderr:==39496==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-26T03:42:38.022 INFO:teuthology.orchestra.run.trial104.stderr:INFO 2026-02-26 03:42:38,020 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-26T03:42:38.162 INFO:teuthology.orchestra.run.trial104.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1517f 0x9c50af8 0x9c50f59 0x9c511cc 0x9c52ef3 0x9d368ca 0x9d3851c 0x9d38b76 0x9d390bc 0x9d4334a 0x9d477c3 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6fa06ea 0x6fa44eb 0x6faa095 0x7419054 0x743ba5c 0x728f369 0x728f461 0x736153a 0x73615f3 0x73617aa 0x743badd 0x728f4de 0x728f563 0x72bb301 0x7423ee4 0x7423f69 0x7424047 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:42:38.162 INFO:teuthology.orchestra.run.trial104.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x316f437 0x9c511ba 0x9c51482 0x9c51f31 0x9c5242f 0x9c52a5f 0x9c52bc9 0x9d32f39 0x9d45d3c 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6fa06ea 0x6fa44eb 0x6faa095 0x7419054 0x743ba5c 0x728f369 0x728f461 0x736153a 0x73615f3 0x73617aa 0x743badd 0x728f4de 0x728f563 0x72bb301 0x7423ee4 0x7423f69 0x7424047 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:42:38.162 INFO:teuthology.orchestra.run.trial104.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc3e0 /lib64/libasan.so.8+0x91c1 /lib64/libasan.so.8+0xee7dc 0x9b2559b 0x9ba2d36 0x9ba2e4c 0x9ba2f28 0x9ba31a4 0x9ba325d 0x9ba374b 0x9ba3b4f 0x9ba3e0c 0x9d45991 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6fa06ea 0x6fa44eb 0x6faa095 0x7419054 0x743ba5c 0x728f369 0x728f461 0x736153a 0x73615f3 0x73617aa 0x743badd 0x728f4de 0x728f563 0x72bb301 0x7423ee4 0x7423f69 0x7424047 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:42:38.194 INFO:teuthology.orchestra.run.trial104.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1586f 0x9c51220 0x9c51482 0x9c51f31 0x9c5242f 0x9c52a5f 0x9c52bc9 0x9d32f39 0x9d45d3c 0x9d07d91 0x9d06d91 0x7e61695 0x7e64490 0x8f5d61d 0x8f5ff8b 0x787cd7f 0x78b3782 0x6fa18b5 0x755a4ec 0x755a640 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f7a738 0x9bf83c1 0x9bf8bfc 0xecf14d /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb46f44 2026-02-26T03:42:38.227 INFO:teuthology.orchestra.run.trial104.stderr:Reactor stalled for 32 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1521a 0x2a55a3f 0x7d837b9 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f7a738 0x9bf83c1 0x9bf8bfc 0xecf14d /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb46f44 2026-02-26T03:42:38.300 INFO:teuthology.orchestra.run.trial104.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x151be 0x2a55a3f 0x7d837b9 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:42:38.300 INFO:teuthology.orchestra.run.trial104.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x7d837f4 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:42:38.300 INFO:teuthology.orchestra.run.trial104.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x129764 /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x15197 0x2a55a3f 0x7d837b9 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:42:38.425 INFO:teuthology.orchestra.run.trial104.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1297e4 /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x157c7 0x10908ef 0xf2e0dc 0x9c50a8f 0x9c50f59 0x9c511cc 0x9c52ef3 0x9d38e4d 0x9d390bc 0x9d4334a 0x9d477c3 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6f43702 0x6f45a6a 0x7483f52 0x72a690f 0x72a6a11 0x72a6b09 0x7360706 0x73607bf 0x7360972 0x7360aaa 0x7360b25 0x7360baa 0x7360d05 0x7423961 0x7483fdb 0x728dda4 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:42:38.425 INFO:teuthology.orchestra.run.trial104.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc597 /lib64/libasan.so.8+0x91c1 /lib64/libasan.so.8+0xee7dc 0x9b2559b 0x9d0c53e 0x9d0c66c 0x9d3ea77 0x9d3f21a 0x9d496e7 0x78947dd 0x7bea228 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6f43702 0x6f45a6a 0x7483f52 0x72a690f 0x72a6a11 0x72a6b09 0x7360706 0x73607bf 0x7360972 0x7360aaa 0x7360b25 0x7360baa 0x7360d05 0x7423961 0x7483fdb 0x728dda4 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:42:38.426 INFO:teuthology.orchestra.run.trial104.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffa9362e9e 0xffffffffa89fa81b 0xffffffffa89fc7f9 0xffffffffa89fdd21 0xffffffffa8a25a53 0xffffffffa8a25c15 0xffffffffa89ca888 0xffffffffa89d4bab 0xffffffffa89d500b 0xffffffffa86910d2 0xffffffffa9350451 0xffffffffa9400ba2 2026-02-26T03:42:38.426 INFO:teuthology.orchestra.run.trial104.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x9c9c199 0x9d07f22 0x9d086a2 0x9d177b2 0x9d1a23d 0x9d1a44e 0x9d1a4fa 0x9d42619 0x9d477c3 0x9d07d91 0x9d06d91 0x7bea2f1 0x7bf3e60 0x7a431c9 0x7a807bd 0x711f05d 0x6f43702 0x6f45a6a 0x7483f52 0x72a690f 0x72a6a11 0x72a6b09 0x7360706 0x73607bf 0x7360972 0x7360aaa 0x7360b25 0x7360baa 0x7360d05 0x7423961 0x7483fdb 0x728dda4 0x72a6c86 0x73b49a4 0x743027a 0x74302f5 0x7430370 0x74f3b82 0x75ebd86 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:42:38.508 INFO:teuthology.orchestra.run.trial104.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x2a00bea 0x2a559e0 0x7d837b9 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f7a738 0x9bf83c1 0x9bf8bfc 0xecf14d /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb46f44 2026-02-26T03:42:38.523 INFO:teuthology.orchestra.run.trial104.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f 0x2a558cb 0x7d837b9 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:42:38.523 INFO:teuthology.orchestra.run.trial104.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc6592 /lib64/libasan.so.8+0x15197 0x2a55a3f 0x7d837b9 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:42:38.523 INFO:teuthology.orchestra.run.trial104.stderr:Reactor stalled for 66 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9e28178 0x9e28461 0x9e88dff 0x9e895f2 0x9e89860 0x9e899a8 0x9e89e67 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1521a 0x2a55a3f 0x7d837b9 0x7d9fd5e 0x7e419e3 0x7dc12dd 0x7877bf3 0x78788e8 0x79adc2a 0x7836cfc 0x78b1d9e 0x78fa531 0x78fa7aa 0x799b982 0x799bae1 0x79add8d 0x7868874 0x798a5f1 0x798a680 0x798a724 0x73e442a 0x7404ce9 0x740998a 0x754d7e2 0x9ee2cc7 0x9e36f9c 0x9f10737 0x9f78f1a 0x9f939aa 0x9f94026 0x26ea5d3 0xa5a1f30 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-26T03:42:40.631 INFO:teuthology.orchestra.run.trial104.stderr:----------------------------------------------------- 2026-02-26T03:42:40.631 INFO:teuthology.orchestra.run.trial104.stderr:Suppressions used: 2026-02-26T03:42:40.631 INFO:teuthology.orchestra.run.trial104.stderr: count bytes template 2026-02-26T03:42:40.631 INFO:teuthology.orchestra.run.trial104.stderr: 1 8 MallocExtension::Register 2026-02-26T03:42:40.631 INFO:teuthology.orchestra.run.trial104.stderr:----------------------------------------------------- 2026-02-26T03:42:40.631 INFO:teuthology.orchestra.run.trial104.stderr: 2026-02-26T03:42:40.665 DEBUG:teuthology.orchestra.run.trial104:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2026-02-26T03:42:40.691 INFO:tasks.ceph:Reading keys from all nodes... 2026-02-26T03:42:40.691 DEBUG:teuthology.orchestra.run.trial104:> set -ex 2026-02-26T03:42:40.691 DEBUG:teuthology.orchestra.run.trial104:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2026-02-26T03:42:40.756 DEBUG:teuthology.orchestra.run.trial104:> set -ex 2026-02-26T03:42:40.756 DEBUG:teuthology.orchestra.run.trial104:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2026-02-26T03:42:40.820 DEBUG:teuthology.orchestra.run.trial104:> set -ex 2026-02-26T03:42:40.820 DEBUG:teuthology.orchestra.run.trial104:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2026-02-26T03:42:40.883 DEBUG:teuthology.orchestra.run.trial104:> set -ex 2026-02-26T03:42:40.883 DEBUG:teuthology.orchestra.run.trial104:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2026-02-26T03:42:40.947 DEBUG:teuthology.orchestra.run.trial104:> set -ex 2026-02-26T03:42:40.947 DEBUG:teuthology.orchestra.run.trial104:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2026-02-26T03:42:41.003 INFO:tasks.ceph:Adding keys to all mons... 2026-02-26T03:42:41.003 DEBUG:teuthology.orchestra.run.trial104:> sudo tee -a /etc/ceph/ceph.keyring 2026-02-26T03:42:41.065 INFO:teuthology.orchestra.run.trial104.stdout:[mgr.x] 2026-02-26T03:42:41.066 INFO:teuthology.orchestra.run.trial104.stdout: key = AQAewZ9pHl5ZMBAAWu138LpQy7gPRR2bkq3h9g== 2026-02-26T03:42:41.066 INFO:teuthology.orchestra.run.trial104.stdout:[osd.0] 2026-02-26T03:42:41.066 INFO:teuthology.orchestra.run.trial104.stdout: key = AQAjwZ9pRauaMBAAxpoA7W2OEemcBkUYsDQdCw== 2026-02-26T03:42:41.066 INFO:teuthology.orchestra.run.trial104.stdout:[osd.1] 2026-02-26T03:42:41.066 INFO:teuthology.orchestra.run.trial104.stdout: key = AQAowZ9pGGUqORAASlEVhw91qN7thztnA8gpyQ== 2026-02-26T03:42:41.066 INFO:teuthology.orchestra.run.trial104.stdout:[osd.2] 2026-02-26T03:42:41.066 INFO:teuthology.orchestra.run.trial104.stdout: key = AQAuwZ9pjWQVBRAAVnmlWALINBJO4njg9iQt0g== 2026-02-26T03:42:41.066 INFO:teuthology.orchestra.run.trial104.stdout:[client.0] 2026-02-26T03:42:41.066 INFO:teuthology.orchestra.run.trial104.stdout: key = AQAewZ9pZD39MhAAMRlEvfvUjb2RCUDOMiRUSg== 2026-02-26T03:42:41.068 DEBUG:teuthology.orchestra.run.trial104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2026-02-26T03:42:41.155 DEBUG:teuthology.orchestra.run.trial104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-26T03:42:41.204 DEBUG:teuthology.orchestra.run.trial104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-26T03:42:41.252 DEBUG:teuthology.orchestra.run.trial104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-26T03:42:41.301 DEBUG:teuthology.orchestra.run.trial104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-26T03:42:41.351 INFO:tasks.ceph:Running mkfs on mon nodes... 2026-02-26T03:42:41.351 DEBUG:teuthology.orchestra.run.trial104:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2026-02-26T03:42:41.376 DEBUG:teuthology.orchestra.run.trial104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2026-02-26T03:42:41.463 INFO:teuthology.orchestra.run.trial104.stderr:2026-02-26T03:42:41.460+0000 7fb9c98a5c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:42:41.463 INFO:teuthology.orchestra.run.trial104.stderr:2026-02-26T03:42:41.460+0000 7fb9c98a5c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:42:41.470 DEBUG:teuthology.orchestra.run.trial104:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2026-02-26T03:42:41.495 DEBUG:teuthology.orchestra.run.trial104:> rm -- /home/ubuntu/cephtest/ceph.monmap 2026-02-26T03:42:41.551 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2026-02-26T03:42:41.589 INFO:tasks.ceph.mon.a:Restarting daemon 2026-02-26T03:42:41.589 DEBUG:teuthology.orchestra.run.trial104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2026-02-26T03:42:41.593 INFO:tasks.ceph.mon.a:Started 2026-02-26T03:42:41.594 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2026-02-26T03:42:41.594 INFO:tasks.ceph.mgr.x:Restarting daemon 2026-02-26T03:42:41.594 DEBUG:teuthology.orchestra.run.trial104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2026-02-26T03:42:41.597 INFO:tasks.ceph.mgr.x:Started 2026-02-26T03:42:41.597 DEBUG:tasks.ceph:set 0 configs 2026-02-26T03:42:41.598 DEBUG:teuthology.orchestra.run.trial104:> sudo ceph --cluster ceph config dump 2026-02-26T03:42:41.655 INFO:tasks.ceph.mon.a.trial104.stderr:2026-02-26T03:42:41.652+0000 7f7bcec39c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:42:41.655 INFO:tasks.ceph.mon.a.trial104.stderr:2026-02-26T03:42:41.652+0000 7f7bcec39c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:42:41.655 INFO:tasks.ceph.mon.a.trial104.stderr:2026-02-26T03:42:41.652+0000 7f7bcec39c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:42:41.660 INFO:tasks.ceph.mgr.x.trial104.stderr:2026-02-26T03:42:41.657+0000 7ff67a3c2fc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:42:41.675 INFO:teuthology.orchestra.run.trial104.stderr:2026-02-26T03:42:41.670+0000 7faee43dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:42:41.676 INFO:teuthology.orchestra.run.trial104.stderr:2026-02-26T03:42:41.673+0000 7faee43dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:42:41.707 INFO:teuthology.orchestra.run.trial104.stdout:WHO MASK LEVEL OPTION VALUE RO 2026-02-26T03:42:41.714 INFO:tasks.ceph:Setting crush tunables to default 2026-02-26T03:42:41.714 DEBUG:teuthology.orchestra.run.trial104:> sudo ceph --cluster ceph osd crush tunables default 2026-02-26T03:42:41.788 INFO:teuthology.orchestra.run.trial104.stderr:2026-02-26T03:42:41.782+0000 7fc3ec4b7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:42:41.790 INFO:teuthology.orchestra.run.trial104.stderr:2026-02-26T03:42:41.787+0000 7fc3ec4b7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:42:41.822 INFO:teuthology.orchestra.run.trial104.stderr:adjusted tunables profile to default 2026-02-26T03:42:41.831 INFO:tasks.ceph:check_enable_crimson: True 2026-02-26T03:42:41.831 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2026-02-26T03:42:41.832 DEBUG:teuthology.orchestra.run.trial104:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2026-02-26T03:42:41.864 INFO:tasks.ceph.mgr.x.trial104.stderr:2026-02-26T03:42:41.861+0000 7ff67a3c2fc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:42:41.864 INFO:tasks.ceph.mgr.x.trial104.stderr:2026-02-26T03:42:41.861+0000 7ff67a3c2fc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:42:41.905 INFO:teuthology.orchestra.run.trial104.stderr:2026-02-26T03:42:41.900+0000 7feb9288b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:42:41.906 INFO:teuthology.orchestra.run.trial104.stderr:2026-02-26T03:42:41.904+0000 7feb9288b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:42:41.944 INFO:tasks.ceph.mon.a.trial104.stderr:2026-02-26T03:42:41.941+0000 7f7bc5ff1640 -1 WARNING: experimental feature 'crimson' is enabled 2026-02-26T03:42:41.944 INFO:tasks.ceph.mon.a.trial104.stderr:Please be aware that this feature is experimental, untested, 2026-02-26T03:42:41.944 INFO:tasks.ceph.mon.a.trial104.stderr:unsupported, and may result in data corruption, data loss, 2026-02-26T03:42:41.944 INFO:tasks.ceph.mon.a.trial104.stderr:and/or irreparable damage to your cluster. Do not use 2026-02-26T03:42:41.944 INFO:tasks.ceph.mon.a.trial104.stderr:feature with important data. 2026-02-26T03:42:41.944 INFO:tasks.ceph.mon.a.trial104.stderr: 2026-02-26T03:42:42.711 INFO:tasks.ceph.mgr.x.trial104.stderr:/usr/lib64/python3.9/site-packages/scipy/__init__.py:73: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2026-02-26T03:42:42.711 INFO:tasks.ceph.mgr.x.trial104.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2026-02-26T03:42:42.711 INFO:tasks.ceph.mgr.x.trial104.stderr: from numpy import show_config as show_numpy_config 2026-02-26T03:42:42.834 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2026-02-26T03:42:42.834 DEBUG:teuthology.orchestra.run.trial104:> set -ex 2026-02-26T03:42:42.834 DEBUG:teuthology.orchestra.run.trial104:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2026-02-26T03:42:42.899 DEBUG:teuthology.orchestra.run.trial104:> set -ex 2026-02-26T03:42:42.900 DEBUG:teuthology.orchestra.run.trial104:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2026-02-26T03:42:42.964 DEBUG:teuthology.orchestra.run.trial104:> set -ex 2026-02-26T03:42:42.964 DEBUG:teuthology.orchestra.run.trial104:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2026-02-26T03:42:43.027 DEBUG:teuthology.orchestra.run.trial104:> sudo ceph --cluster ceph osd new 3e66bfd9-eb74-4c52-8b78-ae518d67f9e9 0 2026-02-26T03:42:43.141 INFO:teuthology.orchestra.run.trial104.stderr:2026-02-26T03:42:43.135+0000 7f5abc0cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:42:43.142 INFO:teuthology.orchestra.run.trial104.stderr:2026-02-26T03:42:43.139+0000 7f5abc0cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:42:43.179 INFO:teuthology.orchestra.run.trial104.stdout:0 2026-02-26T03:42:43.188 DEBUG:teuthology.orchestra.run.trial104:> sudo ceph --cluster ceph osd new e8bad741-de92-4000-ab1a-66ea11567f53 1 2026-02-26T03:42:43.263 INFO:teuthology.orchestra.run.trial104.stderr:2026-02-26T03:42:43.258+0000 7ff3c6acb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:42:43.265 INFO:teuthology.orchestra.run.trial104.stderr:2026-02-26T03:42:43.262+0000 7ff3c6acb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:42:43.303 INFO:teuthology.orchestra.run.trial104.stdout:1 2026-02-26T03:42:43.313 DEBUG:teuthology.orchestra.run.trial104:> sudo ceph --cluster ceph osd new 7c277300-2ed0-4414-91df-26d0f291892d 2 2026-02-26T03:42:43.388 INFO:teuthology.orchestra.run.trial104.stderr:2026-02-26T03:42:43.383+0000 7fb46a393640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:42:43.390 INFO:teuthology.orchestra.run.trial104.stderr:2026-02-26T03:42:43.387+0000 7fb46a393640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:42:43.426 INFO:teuthology.orchestra.run.trial104.stdout:2 2026-02-26T03:42:43.436 DEBUG:teuthology.orchestra.run.trial104:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2026-02-26T03:42:43.463 INFO:tasks.ceph.osd.0:Restarting daemon 2026-02-26T03:42:43.463 DEBUG:teuthology.orchestra.run.trial104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2026-02-26T03:42:43.505 INFO:tasks.ceph.osd.0:Started 2026-02-26T03:42:43.505 DEBUG:teuthology.orchestra.run.trial104:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2026-02-26T03:42:43.535 INFO:tasks.ceph.osd.1:Restarting daemon 2026-02-26T03:42:43.535 DEBUG:teuthology.orchestra.run.trial104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2026-02-26T03:42:43.578 INFO:tasks.ceph.osd.1:Started 2026-02-26T03:42:43.578 DEBUG:teuthology.orchestra.run.trial104:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2026-02-26T03:42:43.608 INFO:tasks.ceph.osd.2:Restarting daemon 2026-02-26T03:42:43.608 DEBUG:teuthology.orchestra.run.trial104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2026-02-26T03:42:43.650 INFO:tasks.ceph.osd.2:Started 2026-02-26T03:42:43.651 DEBUG:teuthology.orchestra.run.trial104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-26T03:42:43.727 INFO:teuthology.orchestra.run.trial104.stderr:2026-02-26T03:42:43.722+0000 7fc363c1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:42:43.729 INFO:teuthology.orchestra.run.trial104.stderr:2026-02-26T03:42:43.726+0000 7fc363c1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:42:43.763 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:42:43.763 INFO:teuthology.orchestra.run.trial104.stdout:{"epoch":6,"fsid":"9e3b9093-cd16-4cf3-9d35-5ccecd63a325","created":"2026-02-26T03:42:41.665224+0000","modified":"2026-02-26T03:42:43.425150+0000","last_up_change":"0.000000","last_in_change":"2026-02-26T03:42:43.425150+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"3e66bfd9-eb74-4c52-8b78-ae518d67f9e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8bad741-de92-4000-ab1a-66ea11567f53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7c277300-2ed0-4414-91df-26d0f291892d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-26T03:42:43.771 INFO:tasks.ceph.ceph_manager.ceph:[] 2026-02-26T03:42:43.771 INFO:tasks.ceph:Waiting for OSDs to come up 2026-02-26T03:42:44.073 DEBUG:teuthology.orchestra.run.trial104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-26T03:42:44.142 INFO:teuthology.misc.health.trial104.stderr:2026-02-26T03:42:44.137+0000 7fea4ec3d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:42:44.143 INFO:teuthology.misc.health.trial104.stderr:2026-02-26T03:42:44.140+0000 7fea4ec3d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:42:44.174 INFO:teuthology.misc.health.trial104.stdout: 2026-02-26T03:42:44.174 INFO:teuthology.misc.health.trial104.stdout:{"epoch":6,"fsid":"9e3b9093-cd16-4cf3-9d35-5ccecd63a325","created":"2026-02-26T03:42:41.665224+0000","modified":"2026-02-26T03:42:43.425150+0000","last_up_change":"0.000000","last_in_change":"2026-02-26T03:42:43.425150+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"3e66bfd9-eb74-4c52-8b78-ae518d67f9e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8bad741-de92-4000-ab1a-66ea11567f53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7c277300-2ed0-4414-91df-26d0f291892d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-26T03:42:44.182 DEBUG:teuthology.misc:0 of 3 OSDs are up 2026-02-26T03:42:45.125 INFO:tasks.ceph.mgr.x.trial104.stderr:2026-02-26T03:42:45.122+0000 7ff59527d640 -1 mgr.server handle_report got status from non-daemon mon.a 2026-02-26T03:42:50.484 DEBUG:teuthology.orchestra.run.trial104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-26T03:42:50.585 INFO:teuthology.misc.health.trial104.stderr:2026-02-26T03:42:50.580+0000 7f9ca50cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:42:50.586 INFO:teuthology.misc.health.trial104.stderr:2026-02-26T03:42:50.585+0000 7f9ca50cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:42:50.741 INFO:teuthology.misc.health.trial104.stdout: 2026-02-26T03:42:50.741 INFO:teuthology.misc.health.trial104.stdout:{"epoch":8,"fsid":"9e3b9093-cd16-4cf3-9d35-5ccecd63a325","created":"2026-02-26T03:42:41.665224+0000","modified":"2026-02-26T03:42:50.519898+0000","last_up_change":"0.000000","last_in_change":"2026-02-26T03:42:43.425150+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"3e66bfd9-eb74-4c52-8b78-ae518d67f9e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8bad741-de92-4000-ab1a-66ea11567f53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7c277300-2ed0-4414-91df-26d0f291892d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-26T03:42:50.757 DEBUG:teuthology.misc:0 of 3 OSDs are up 2026-02-26T03:42:57.059 DEBUG:teuthology.orchestra.run.trial104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-26T03:42:57.129 INFO:teuthology.misc.health.trial104.stderr:2026-02-26T03:42:57.126+0000 7ff386fc1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:42:57.130 INFO:teuthology.misc.health.trial104.stderr:2026-02-26T03:42:57.128+0000 7ff386fc1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:42:57.242 INFO:teuthology.misc.health.trial104.stdout: 2026-02-26T03:42:57.243 INFO:teuthology.misc.health.trial104.stdout:{"epoch":13,"fsid":"9e3b9093-cd16-4cf3-9d35-5ccecd63a325","created":"2026-02-26T03:42:41.665224+0000","modified":"2026-02-26T03:42:56.534951+0000","last_up_change":"2026-02-26T03:42:52.524772+0000","last_in_change":"2026-02-26T03:42:43.425150+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-26T03:42:54.134486+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3e66bfd9-eb74-4c52-8b78-ae518d67f9e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.104:6804","nonce":891261858}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.104:6803","nonce":891261858}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.104:6811","nonce":891261858}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.104:6810","nonce":891261858}]},"public_addr":"10.20.193.104:6804/891261858","cluster_addr":"10.20.193.104:6803/891261858","heartbeat_back_addr":"10.20.193.104:6811/891261858","heartbeat_front_addr":"10.20.193.104:6810/891261858","state":["exists","up"]},{"osd":1,"uuid":"e8bad741-de92-4000-ab1a-66ea11567f53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.104:6805","nonce":1050362739}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.104:6802","nonce":1050362739}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.104:6812","nonce":1050362739}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.104:6813","nonce":1050362739}]},"public_addr":"10.20.193.104:6805/1050362739","cluster_addr":"10.20.193.104:6802/1050362739","heartbeat_back_addr":"10.20.193.104:6812/1050362739","heartbeat_front_addr":"10.20.193.104:6813/1050362739","state":["exists","up"]},{"osd":2,"uuid":"7c277300-2ed0-4414-91df-26d0f291892d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.104:6806","nonce":654573974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.104:6807","nonce":654573974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.104:6808","nonce":654573974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.104:6814","nonce":654573974}]},"public_addr":"10.20.193.104:6806/654573974","cluster_addr":"10.20.193.104:6807/654573974","heartbeat_back_addr":"10.20.193.104:6808/654573974","heartbeat_front_addr":"10.20.193.104:6814/654573974","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-26T03:42:57.250 DEBUG:teuthology.misc:3 of 3 OSDs are up 2026-02-26T03:42:57.250 INFO:tasks.ceph:Creating RBD pool 2026-02-26T03:42:57.251 DEBUG:teuthology.orchestra.run.trial104:> sudo ceph --cluster ceph osd pool create rbd 8 2026-02-26T03:42:57.325 INFO:teuthology.orchestra.run.trial104.stderr:2026-02-26T03:42:57.320+0000 7efee0ccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:42:57.326 INFO:teuthology.orchestra.run.trial104.stderr:2026-02-26T03:42:57.324+0000 7efee0ccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:42:57.864 INFO:teuthology.orchestra.run.trial104.stderr:pool 'rbd' created 2026-02-26T03:42:57.880 DEBUG:teuthology.orchestra.run.trial104:> rbd --cluster ceph pool init rbd 2026-02-26T03:42:57.922 INFO:teuthology.orchestra.run.trial104.stderr:2026-02-26T03:42:57.920+0000 7fcb84f76240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:42:57.922 INFO:teuthology.orchestra.run.trial104.stderr:2026-02-26T03:42:57.920+0000 7fcb84f76240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:42:57.923 INFO:teuthology.orchestra.run.trial104.stderr:2026-02-26T03:42:57.922+0000 7fcb84f76240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:43:00.898 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2026-02-26T03:43:00.899 DEBUG:teuthology.orchestra.run.trial104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2026-02-26T03:43:00.900 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2026-02-26T03:43:01.016 INFO:teuthology.orchestra.run.trial104.stderr:2026-02-26T03:43:01.012+0000 7f30f701a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:43:01.017 INFO:teuthology.orchestra.run.trial104.stderr:2026-02-26T03:43:01.015+0000 7f30f701a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:43:01.126 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:43:01.135 INFO:teuthology.orchestra.run.trial104.stdout:[{"version":1,"timestamp":"0.000000","name":"","changes":[]}] 2026-02-26T03:43:01.135 INFO:tasks.ceph_manager:config epoch is 1 2026-02-26T03:43:01.135 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2026-02-26T03:43:01.136 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2026-02-26T03:43:01.136 DEBUG:teuthology.orchestra.run.trial104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2026-02-26T03:43:01.211 INFO:teuthology.orchestra.run.trial104.stderr:2026-02-26T03:43:01.207+0000 7fee6221a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:43:01.212 INFO:teuthology.orchestra.run.trial104.stderr:2026-02-26T03:43:01.211+0000 7fee6221a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:43:01.353 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:43:01.361 INFO:teuthology.orchestra.run.trial104.stdout:{"epoch":5,"flags":0,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.104:6800","nonce":4191194626},{"type":"v1","addr":"10.20.193.104:6801","nonce":4191194626}]},"active_addr":"10.20.193.104:6801/4191194626","active_change":"2026-02-26T03:42:44.118857+0000","active_mgr_features":4544132024016699391,"available":true,"standbys":[],"modules":["iostat","nfs"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_pg_upmap_activity":{"name":"update_pg_upmap_activity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Updates pg_upmap activity stats to be used in `balancer status detail`","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often the agent on each host will gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"The first TCP port the agent will try to bind to. Up to 1000 subsequent ports will be attempted if this port cannot be bound","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this option can facilitate debugging daemons that encounter runtime problems.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"How frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"Fraction of total system memory to divide among autotuned daemons. Converged systems hosting both Ceph and compute should set a lower value; systems with more than 10 GiB of memory per OSD may benefit from a slightly larger value so that the abundance of RAM can be fully utilized","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command persistent logging","long_desc":"","tags":[],"see_also":[]},"certificate_automated_rotation_enabled":{"name":"certificate_automated_rotation_enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"This flag controls whether cephadm automatically rotates certificates upon expiration.","long_desc":"","tags":[],"see_also":[]},"certificate_check_debug_mode":{"name":"certificate_check_debug_mode","type":"bool","level":"dev","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"FOR TESTING ONLY: This flag forces the certificate check instead of waiting for certificate_check_period.","long_desc":"","tags":[],"see_also":[]},"certificate_check_period":{"name":"certificate_check_period","type":"int","level":"advanced","flags":0,"default_value":"1","min":"0","max":"30","enum_allowed":[],"desc":"Specifies how often (in days) the certificate should be checked for validity.","long_desc":"","tags":[],"see_also":[]},"certificate_duration_days":{"name":"certificate_duration_days","type":"int","level":"advanced","flags":0,"default_value":"1095","min":"90","max":"3650","enum_allowed":[],"desc":"Specifies the duration of self-signed certificates generated and signed by the cephadm root CA","long_desc":"","tags":[],"see_also":[]},"certificate_renewal_threshold_days":{"name":"certificate_renewal_threshold_days","type":"int","level":"advanced","flags":0,"default_value":"30","min":"10","max":"90","enum_allowed":[],"desc":"Specifies the lead time in days to initiate certificate renewal before expiration.","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently Podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.28.1","min":"","max":"","enum_allowed":[],"desc":"Alertmanager container image","long_desc":"","tags":[],"see_also":[]},"container_image_alloy":{"name":"container_image_alloy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/alloy:latest","min":"","max":"","enum_allowed":[],"desc":"Alloy container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"Elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:12.3.1","min":"","max":"","enum_allowed":[],"desc":"Grafana container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"Haproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.9.1","min":"","max":"","enum_allowed":[],"desc":"Node exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.6","min":"","max":"","enum_allowed":[],"desc":"Nvmeof container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"Oauth2 proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v3.6.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba metrics container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"Snmp gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"quay.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"Location of dashboards to include in Grafana","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"How frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether the hardware monitoring daemon be deployed on every host?","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at the debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"Max number of daemons per service per host: guard against mistakes and runaways","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"Max number of OSDs that will be drained simultaneously when OSDs are removed","long_desc":"","tags":[],"see_also":[]},"max_parallel_osd_upgrades":{"name":"max_parallel_osd_upgrades","type":"int","level":"advanced","flags":0,"default_value":"16","min":"","max":"","enum_allowed":[],"desc":"Maximum number of OSD daemons upgraded in parallel.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"Mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default IP address for RedFish API (OOB management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"Location of alerts to include in Prometheus Alertmanager","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry URL for login purposes. This is not the default registry.","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"Cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Customized SSH config file to distribute to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many sequential SSH connection liveness check failures before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often SSH connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"stray_daemon_check_interval":{"name":"stray_daemon_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"How frequently cephadm should check for the presence of stray daemons","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use the cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest to ensure that all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROM_ALERT_CREDENTIAL_CACHE_TTL":{"name":"PROM_ALERT_CREDENTIAL_CACHE_TTL","type":"int","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_HOSTNAME_PER_DAEMON":{"name":"RGW_HOSTNAME_PER_DAEMON","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crypto_caller":{"name":"crypto_caller","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":true,"error_string":"","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"healthcheck_history_max_entries":{"name":"healthcheck_history_max_entries","type":"int","level":"advanced","flags":1,"default_value":"1000","min":"","max":"","enum_allowed":[],"desc":"Maximum number of health check history entries to keep","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"prometheus_tls_secret_name":{"name":"prometheus_tls_secret_name","type":"str","level":"advanced","flags":0,"default_value":"rook-ceph-prometheus-server-tls","min":"","max":"","enum_allowed":[],"desc":"name of tls secret in k8s for prometheus","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_cloning":{"name":"pause_cloning","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_purging":{"name":"pause_purging","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous subvolume purge threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"tentacle":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"force_disabled_modules":{},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"10.20.193.104:0","nonce":1615280471}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"10.20.193.104:0","nonce":3261835768}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"10.20.193.104:0","nonce":2125134819}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"10.20.193.104:0","nonce":1119581501}]}]} 2026-02-26T03:43:01.365 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2026-02-26T03:43:01.365 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2026-02-26T03:43:01.365 DEBUG:teuthology.orchestra.run.trial104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-26T03:43:01.442 INFO:teuthology.orchestra.run.trial104.stderr:2026-02-26T03:43:01.438+0000 7fa3c7446640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:43:01.443 INFO:teuthology.orchestra.run.trial104.stderr:2026-02-26T03:43:01.441+0000 7fa3c7446640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:43:01.553 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:43:01.553 INFO:teuthology.orchestra.run.trial104.stdout:{"epoch":17,"fsid":"9e3b9093-cd16-4cf3-9d35-5ccecd63a325","created":"2026-02-26T03:42:41.665224+0000","modified":"2026-02-26T03:43:00.870207+0000","last_up_change":"2026-02-26T03:42:52.524772+0000","last_in_change":"2026-02-26T03:42:43.425150+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-26T03:42:54.134486+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-26T03:42:57.437128+0000","flags":270337,"flags_names":"hashpspool,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3e66bfd9-eb74-4c52-8b78-ae518d67f9e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.104:6804","nonce":891261858}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.104:6803","nonce":891261858}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.104:6811","nonce":891261858}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.104:6810","nonce":891261858}]},"public_addr":"10.20.193.104:6804/891261858","cluster_addr":"10.20.193.104:6803/891261858","heartbeat_back_addr":"10.20.193.104:6811/891261858","heartbeat_front_addr":"10.20.193.104:6810/891261858","state":["exists","up"]},{"osd":1,"uuid":"e8bad741-de92-4000-ab1a-66ea11567f53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.104:6805","nonce":1050362739}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.104:6802","nonce":1050362739}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.104:6812","nonce":1050362739}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.104:6813","nonce":1050362739}]},"public_addr":"10.20.193.104:6805/1050362739","cluster_addr":"10.20.193.104:6802/1050362739","heartbeat_back_addr":"10.20.193.104:6812/1050362739","heartbeat_front_addr":"10.20.193.104:6813/1050362739","state":["exists","up"]},{"osd":2,"uuid":"7c277300-2ed0-4414-91df-26d0f291892d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.104:6806","nonce":654573974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.104:6807","nonce":654573974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.104:6808","nonce":654573974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.104:6814","nonce":654573974}]},"public_addr":"10.20.193.104:6806/654573974","cluster_addr":"10.20.193.104:6807/654573974","heartbeat_back_addr":"10.20.193.104:6808/654573974","heartbeat_front_addr":"10.20.193.104:6814/654573974","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-26T03:43:01.564 INFO:tasks.ceph.ceph_manager.ceph:all up! 2026-02-26T03:43:01.564 DEBUG:teuthology.orchestra.run.trial104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-26T03:43:01.640 INFO:teuthology.orchestra.run.trial104.stderr:2026-02-26T03:43:01.636+0000 7f2493c5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:43:01.641 INFO:teuthology.orchestra.run.trial104.stderr:2026-02-26T03:43:01.639+0000 7f2493c5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:43:01.751 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:43:01.751 INFO:teuthology.orchestra.run.trial104.stdout:{"epoch":17,"fsid":"9e3b9093-cd16-4cf3-9d35-5ccecd63a325","created":"2026-02-26T03:42:41.665224+0000","modified":"2026-02-26T03:43:00.870207+0000","last_up_change":"2026-02-26T03:42:52.524772+0000","last_in_change":"2026-02-26T03:42:43.425150+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-26T03:42:54.134486+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-26T03:42:57.437128+0000","flags":270337,"flags_names":"hashpspool,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3e66bfd9-eb74-4c52-8b78-ae518d67f9e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.104:6804","nonce":891261858}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.104:6803","nonce":891261858}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.104:6811","nonce":891261858}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.104:6810","nonce":891261858}]},"public_addr":"10.20.193.104:6804/891261858","cluster_addr":"10.20.193.104:6803/891261858","heartbeat_back_addr":"10.20.193.104:6811/891261858","heartbeat_front_addr":"10.20.193.104:6810/891261858","state":["exists","up"]},{"osd":1,"uuid":"e8bad741-de92-4000-ab1a-66ea11567f53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.104:6805","nonce":1050362739}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.104:6802","nonce":1050362739}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.104:6812","nonce":1050362739}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.104:6813","nonce":1050362739}]},"public_addr":"10.20.193.104:6805/1050362739","cluster_addr":"10.20.193.104:6802/1050362739","heartbeat_back_addr":"10.20.193.104:6812/1050362739","heartbeat_front_addr":"10.20.193.104:6813/1050362739","state":["exists","up"]},{"osd":2,"uuid":"7c277300-2ed0-4414-91df-26d0f291892d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.104:6806","nonce":654573974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.104:6807","nonce":654573974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.104:6808","nonce":654573974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.104:6814","nonce":654573974}]},"public_addr":"10.20.193.104:6806/654573974","cluster_addr":"10.20.193.104:6807/654573974","heartbeat_back_addr":"10.20.193.104:6808/654573974","heartbeat_front_addr":"10.20.193.104:6814/654573974","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-26T03:43:01.761 DEBUG:teuthology.orchestra.run.trial104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-26T03:43:01.762 DEBUG:teuthology.orchestra.run.trial104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-26T03:43:01.763 DEBUG:teuthology.orchestra.run.trial104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-26T03:43:01.845 INFO:teuthology.orchestra.run.trial104.stderr:2026-02-26T03:43:01.840+0000 7f65d985a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:43:01.845 INFO:teuthology.orchestra.run.trial104.stderr:2026-02-26T03:43:01.840+0000 7fb1c8bb8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:43:01.845 INFO:teuthology.orchestra.run.trial104.stderr:2026-02-26T03:43:01.840+0000 7f25f43dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:43:01.846 INFO:teuthology.orchestra.run.trial104.stderr:2026-02-26T03:43:01.844+0000 7fb1c8bb8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:43:01.846 INFO:teuthology.orchestra.run.trial104.stderr:2026-02-26T03:43:01.844+0000 7f65d985a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:43:01.846 INFO:teuthology.orchestra.run.trial104.stderr:2026-02-26T03:43:01.844+0000 7f25f43dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:43:01.861 INFO:teuthology.orchestra.run.trial104.stdout:42949672969 2026-02-26T03:43:01.862 DEBUG:teuthology.orchestra.run.trial104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-26T03:43:01.863 INFO:teuthology.orchestra.run.trial104.stdout:38654705674 2026-02-26T03:43:01.863 DEBUG:teuthology.orchestra.run.trial104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-26T03:43:01.864 INFO:teuthology.orchestra.run.trial104.stdout:42949672969 2026-02-26T03:43:01.865 DEBUG:teuthology.orchestra.run.trial104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-26T03:43:01.982 INFO:teuthology.orchestra.run.trial104.stderr:2026-02-26T03:43:01.978+0000 7fe469bdc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:43:01.983 INFO:teuthology.orchestra.run.trial104.stderr:2026-02-26T03:43:01.978+0000 7f28253a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:43:01.983 INFO:teuthology.orchestra.run.trial104.stderr:2026-02-26T03:43:01.978+0000 7fee1297e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:43:01.984 INFO:teuthology.orchestra.run.trial104.stderr:2026-02-26T03:43:01.981+0000 7fe469bdc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:43:01.984 INFO:teuthology.orchestra.run.trial104.stderr:2026-02-26T03:43:01.982+0000 7f28253a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:43:01.984 INFO:teuthology.orchestra.run.trial104.stderr:2026-02-26T03:43:01.982+0000 7fee1297e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:43:02.096 INFO:teuthology.orchestra.run.trial104.stdout:38654705674 2026-02-26T03:43:02.099 INFO:teuthology.orchestra.run.trial104.stdout:42949672969 2026-02-26T03:43:02.102 INFO:teuthology.orchestra.run.trial104.stdout:42949672969 2026-02-26T03:43:02.107 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705674 got 38654705674 for osd.0 2026-02-26T03:43:02.108 DEBUG:teuthology.parallel:result is None 2026-02-26T03:43:02.110 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672969 got 42949672969 for osd.2 2026-02-26T03:43:02.110 DEBUG:teuthology.parallel:result is None 2026-02-26T03:43:02.113 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672969 got 42949672969 for osd.1 2026-02-26T03:43:02.114 DEBUG:teuthology.parallel:result is None 2026-02-26T03:43:02.114 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-26T03:43:02.114 DEBUG:teuthology.orchestra.run.trial104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-26T03:43:02.231 INFO:teuthology.orchestra.run.trial104.stderr:2026-02-26T03:43:02.227+0000 7fa3bd95b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:43:02.232 INFO:teuthology.orchestra.run.trial104.stderr:2026-02-26T03:43:02.230+0000 7fa3bd95b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:43:02.339 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:43:02.339 INFO:teuthology.orchestra.run.trial104.stderr:dumped all 2026-02-26T03:43:02.347 INFO:teuthology.orchestra.run.trial104.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2026-02-26T03:43:02.122884+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156499968,"kb_used":2540,"kb_used_data":2540,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156497428,"statfs":{"total":160255967232,"available":160253366272,"internally_reserved":0,"allocated":2600960,"data_stored":2600960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.269813"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:43:00.884588+0000","last_change":"2026-02-26T03:43:00.887122+0000","last_active":"2026-02-26T03:43:00.884588+0000","last_peered":"2026-02-26T03:43:00.884588+0000","last_clean":"2026-02-26T03:43:00.884588+0000","last_became_active":"2026-02-26T03:42:58.908444+0000","last_became_peered":"2026-02-26T03:42:58.908444+0000","last_unstale":"2026-02-26T03:43:00.884588+0000","last_undegraded":"2026-02-26T03:43:00.884588+0000","last_fullsized":"2026-02-26T03:43:00.884588+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:42:57.862052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:42:57.862052+0000","last_clean_scrub_stamp":"2026-02-26T03:42:57.862052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:43:00.886420+0000","last_change":"2026-02-26T03:43:00.889336+0000","last_active":"2026-02-26T03:43:00.886420+0000","last_peered":"2026-02-26T03:43:00.886420+0000","last_clean":"2026-02-26T03:43:00.886420+0000","last_became_active":"2026-02-26T03:42:58.903814+0000","last_became_peered":"2026-02-26T03:42:58.903814+0000","last_unstale":"2026-02-26T03:43:00.886420+0000","last_undegraded":"2026-02-26T03:43:00.886420+0000","last_fullsized":"2026-02-26T03:43:00.886420+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:42:57.862052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:42:57.862052+0000","last_clean_scrub_stamp":"2026-02-26T03:42:57.862052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:43:00.885847+0000","last_change":"2026-02-26T03:43:00.889120+0000","last_active":"2026-02-26T03:43:00.885847+0000","last_peered":"2026-02-26T03:43:00.885847+0000","last_clean":"2026-02-26T03:43:00.885847+0000","last_became_active":"2026-02-26T03:42:58.911477+0000","last_became_peered":"2026-02-26T03:42:58.911477+0000","last_unstale":"2026-02-26T03:43:00.885847+0000","last_undegraded":"2026-02-26T03:43:00.885847+0000","last_fullsized":"2026-02-26T03:43:00.885847+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:42:57.862052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:42:57.862052+0000","last_clean_scrub_stamp":"2026-02-26T03:42:57.862052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:43:00.886583+0000","last_change":"2026-02-26T03:43:00.889109+0000","last_active":"2026-02-26T03:43:00.886583+0000","last_peered":"2026-02-26T03:43:00.886583+0000","last_clean":"2026-02-26T03:43:00.886583+0000","last_became_active":"2026-02-26T03:42:58.916964+0000","last_became_peered":"2026-02-26T03:42:58.916964+0000","last_unstale":"2026-02-26T03:43:00.886583+0000","last_undegraded":"2026-02-26T03:43:00.886583+0000","last_fullsized":"2026-02-26T03:43:00.886583+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:42:57.862052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:42:57.862052+0000","last_clean_scrub_stamp":"2026-02-26T03:42:57.862052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'1","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:43:00.885241+0000","last_change":"2026-02-26T03:43:00.885241+0000","last_active":"2026-02-26T03:43:00.885241+0000","last_peered":"2026-02-26T03:43:00.885241+0000","last_clean":"2026-02-26T03:43:00.885241+0000","last_became_active":"2026-02-26T03:42:58.900823+0000","last_became_peered":"2026-02-26T03:42:58.900823+0000","last_unstale":"2026-02-26T03:43:00.885241+0000","last_undegraded":"2026-02-26T03:43:00.885241+0000","last_fullsized":"2026-02-26T03:43:00.885241+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:42:57.862052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:42:57.862052+0000","last_clean_scrub_stamp":"2026-02-26T03:42:57.862052+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:43:00.883865+0000","last_change":"2026-02-26T03:43:00.885932+0000","last_active":"2026-02-26T03:43:00.883865+0000","last_peered":"2026-02-26T03:43:00.883865+0000","last_clean":"2026-02-26T03:43:00.883865+0000","last_became_active":"2026-02-26T03:42:58.902486+0000","last_became_peered":"2026-02-26T03:42:58.902486+0000","last_unstale":"2026-02-26T03:43:00.883865+0000","last_undegraded":"2026-02-26T03:43:00.883865+0000","last_fullsized":"2026-02-26T03:43:00.883865+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:42:57.862052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:42:57.862052+0000","last_clean_scrub_stamp":"2026-02-26T03:42:57.862052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:43:00.882401+0000","last_change":"2026-02-26T03:43:00.884903+0000","last_active":"2026-02-26T03:43:00.882401+0000","last_peered":"2026-02-26T03:43:00.882401+0000","last_clean":"2026-02-26T03:43:00.882401+0000","last_became_active":"2026-02-26T03:42:58.897287+0000","last_became_peered":"2026-02-26T03:42:58.897287+0000","last_unstale":"2026-02-26T03:43:00.882401+0000","last_undegraded":"2026-02-26T03:43:00.882401+0000","last_fullsized":"2026-02-26T03:43:00.882401+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:42:57.862052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:42:57.862052+0000","last_clean_scrub_stamp":"2026-02-26T03:42:57.862052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":16,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:43:00.886373+0000","last_change":"2026-02-26T03:43:00.888107+0000","last_active":"2026-02-26T03:43:00.886373+0000","last_peered":"2026-02-26T03:43:00.886373+0000","last_clean":"2026-02-26T03:43:00.886373+0000","last_became_active":"2026-02-26T03:42:58.896519+0000","last_became_peered":"2026-02-26T03:42:58.896519+0000","last_unstale":"2026-02-26T03:43:00.886373+0000","last_undegraded":"2026-02-26T03:43:00.886373+0000","last_fullsized":"2026-02-26T03:43:00.886373+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:42:57.862052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:42:57.862052+0000","last_clean_scrub_stamp":"2026-02-26T03:42:57.862052+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":90,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:43:00.884525+0000","last_change":"2026-02-26T03:42:55.563584+0000","last_active":"2026-02-26T03:43:00.884525+0000","last_peered":"2026-02-26T03:43:00.884525+0000","last_clean":"2026-02-26T03:43:00.884525+0000","last_became_active":"2026-02-26T03:42:55.556720+0000","last_became_peered":"2026-02-26T03:42:55.556720+0000","last_unstale":"2026-02-26T03:43:00.884525+0000","last_undegraded":"2026-02-26T03:43:00.884525+0000","last_fullsized":"2026-02-26T03:43:00.884525+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:42:54.526639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:42:54.526639+0000","last_clean_scrub_stamp":"2026-02-26T03:42:54.526639+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160255967232,"available":160253366272,"internally_reserved":0,"allocated":2600960,"data_stored":2600960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53418655744,"available":53417480192,"internally_reserved":0,"allocated":1175552,"data_stored":1175552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":316,"kb_used_data":316,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52166340,"statfs":{"total":53418655744,"available":53418332160,"internally_reserved":0,"allocated":323584,"data_stored":323584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1148,"kb_used_data":1148,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165508,"statfs":{"total":53418655744,"available":53417480192,"internally_reserved":0,"allocated":1175552,"data_stored":1175552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1076,"kb_used_data":1076,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165580,"statfs":{"total":53418655744,"available":53417553920,"internally_reserved":0,"allocated":1101824,"data_stored":1101824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53418655744,"available":53417480192,"internally_reserved":0,"allocated":1175552,"data_stored":1175552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53418655744,"available":53417553920,"internally_reserved":0,"allocated":1101824,"data_stored":1101824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53418655744,"available":53417480192,"internally_reserved":0,"allocated":1175552,"data_stored":1175552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53418655744,"available":53418332160,"internally_reserved":0,"allocated":323584,"data_stored":323584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-26T03:43:02.349 DEBUG:teuthology.orchestra.run.trial104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-26T03:43:02.426 INFO:teuthology.orchestra.run.trial104.stderr:2026-02-26T03:43:02.422+0000 7fb26710b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:43:02.427 INFO:teuthology.orchestra.run.trial104.stderr:2026-02-26T03:43:02.425+0000 7fb26710b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:43:02.534 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:43:02.534 INFO:teuthology.orchestra.run.trial104.stderr:dumped all 2026-02-26T03:43:02.543 INFO:teuthology.orchestra.run.trial104.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2026-02-26T03:43:02.122884+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156499968,"kb_used":2540,"kb_used_data":2540,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156497428,"statfs":{"total":160255967232,"available":160253366272,"internally_reserved":0,"allocated":2600960,"data_stored":2600960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.269813"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:43:00.884588+0000","last_change":"2026-02-26T03:43:00.887122+0000","last_active":"2026-02-26T03:43:00.884588+0000","last_peered":"2026-02-26T03:43:00.884588+0000","last_clean":"2026-02-26T03:43:00.884588+0000","last_became_active":"2026-02-26T03:42:58.908444+0000","last_became_peered":"2026-02-26T03:42:58.908444+0000","last_unstale":"2026-02-26T03:43:00.884588+0000","last_undegraded":"2026-02-26T03:43:00.884588+0000","last_fullsized":"2026-02-26T03:43:00.884588+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:42:57.862052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:42:57.862052+0000","last_clean_scrub_stamp":"2026-02-26T03:42:57.862052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:43:00.886420+0000","last_change":"2026-02-26T03:43:00.889336+0000","last_active":"2026-02-26T03:43:00.886420+0000","last_peered":"2026-02-26T03:43:00.886420+0000","last_clean":"2026-02-26T03:43:00.886420+0000","last_became_active":"2026-02-26T03:42:58.903814+0000","last_became_peered":"2026-02-26T03:42:58.903814+0000","last_unstale":"2026-02-26T03:43:00.886420+0000","last_undegraded":"2026-02-26T03:43:00.886420+0000","last_fullsized":"2026-02-26T03:43:00.886420+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:42:57.862052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:42:57.862052+0000","last_clean_scrub_stamp":"2026-02-26T03:42:57.862052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:43:00.885847+0000","last_change":"2026-02-26T03:43:00.889120+0000","last_active":"2026-02-26T03:43:00.885847+0000","last_peered":"2026-02-26T03:43:00.885847+0000","last_clean":"2026-02-26T03:43:00.885847+0000","last_became_active":"2026-02-26T03:42:58.911477+0000","last_became_peered":"2026-02-26T03:42:58.911477+0000","last_unstale":"2026-02-26T03:43:00.885847+0000","last_undegraded":"2026-02-26T03:43:00.885847+0000","last_fullsized":"2026-02-26T03:43:00.885847+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:42:57.862052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:42:57.862052+0000","last_clean_scrub_stamp":"2026-02-26T03:42:57.862052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:43:00.886583+0000","last_change":"2026-02-26T03:43:00.889109+0000","last_active":"2026-02-26T03:43:00.886583+0000","last_peered":"2026-02-26T03:43:00.886583+0000","last_clean":"2026-02-26T03:43:00.886583+0000","last_became_active":"2026-02-26T03:42:58.916964+0000","last_became_peered":"2026-02-26T03:42:58.916964+0000","last_unstale":"2026-02-26T03:43:00.886583+0000","last_undegraded":"2026-02-26T03:43:00.886583+0000","last_fullsized":"2026-02-26T03:43:00.886583+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:42:57.862052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:42:57.862052+0000","last_clean_scrub_stamp":"2026-02-26T03:42:57.862052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'1","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:43:00.885241+0000","last_change":"2026-02-26T03:43:00.885241+0000","last_active":"2026-02-26T03:43:00.885241+0000","last_peered":"2026-02-26T03:43:00.885241+0000","last_clean":"2026-02-26T03:43:00.885241+0000","last_became_active":"2026-02-26T03:42:58.900823+0000","last_became_peered":"2026-02-26T03:42:58.900823+0000","last_unstale":"2026-02-26T03:43:00.885241+0000","last_undegraded":"2026-02-26T03:43:00.885241+0000","last_fullsized":"2026-02-26T03:43:00.885241+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:42:57.862052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:42:57.862052+0000","last_clean_scrub_stamp":"2026-02-26T03:42:57.862052+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:43:00.883865+0000","last_change":"2026-02-26T03:43:00.885932+0000","last_active":"2026-02-26T03:43:00.883865+0000","last_peered":"2026-02-26T03:43:00.883865+0000","last_clean":"2026-02-26T03:43:00.883865+0000","last_became_active":"2026-02-26T03:42:58.902486+0000","last_became_peered":"2026-02-26T03:42:58.902486+0000","last_unstale":"2026-02-26T03:43:00.883865+0000","last_undegraded":"2026-02-26T03:43:00.883865+0000","last_fullsized":"2026-02-26T03:43:00.883865+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:42:57.862052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:42:57.862052+0000","last_clean_scrub_stamp":"2026-02-26T03:42:57.862052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:43:00.882401+0000","last_change":"2026-02-26T03:43:00.884903+0000","last_active":"2026-02-26T03:43:00.882401+0000","last_peered":"2026-02-26T03:43:00.882401+0000","last_clean":"2026-02-26T03:43:00.882401+0000","last_became_active":"2026-02-26T03:42:58.897287+0000","last_became_peered":"2026-02-26T03:42:58.897287+0000","last_unstale":"2026-02-26T03:43:00.882401+0000","last_undegraded":"2026-02-26T03:43:00.882401+0000","last_fullsized":"2026-02-26T03:43:00.882401+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:42:57.862052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:42:57.862052+0000","last_clean_scrub_stamp":"2026-02-26T03:42:57.862052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":16,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:43:00.886373+0000","last_change":"2026-02-26T03:43:00.888107+0000","last_active":"2026-02-26T03:43:00.886373+0000","last_peered":"2026-02-26T03:43:00.886373+0000","last_clean":"2026-02-26T03:43:00.886373+0000","last_became_active":"2026-02-26T03:42:58.896519+0000","last_became_peered":"2026-02-26T03:42:58.896519+0000","last_unstale":"2026-02-26T03:43:00.886373+0000","last_undegraded":"2026-02-26T03:43:00.886373+0000","last_fullsized":"2026-02-26T03:43:00.886373+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:42:57.862052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:42:57.862052+0000","last_clean_scrub_stamp":"2026-02-26T03:42:57.862052+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":90,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-26T03:43:00.884525+0000","last_change":"2026-02-26T03:42:55.563584+0000","last_active":"2026-02-26T03:43:00.884525+0000","last_peered":"2026-02-26T03:43:00.884525+0000","last_clean":"2026-02-26T03:43:00.884525+0000","last_became_active":"2026-02-26T03:42:55.556720+0000","last_became_peered":"2026-02-26T03:42:55.556720+0000","last_unstale":"2026-02-26T03:43:00.884525+0000","last_undegraded":"2026-02-26T03:43:00.884525+0000","last_fullsized":"2026-02-26T03:43:00.884525+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-26T03:42:54.526639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-26T03:42:54.526639+0000","last_clean_scrub_stamp":"2026-02-26T03:42:54.526639+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160255967232,"available":160253366272,"internally_reserved":0,"allocated":2600960,"data_stored":2600960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53418655744,"available":53417480192,"internally_reserved":0,"allocated":1175552,"data_stored":1175552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":316,"kb_used_data":316,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52166340,"statfs":{"total":53418655744,"available":53418332160,"internally_reserved":0,"allocated":323584,"data_stored":323584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1148,"kb_used_data":1148,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165508,"statfs":{"total":53418655744,"available":53417480192,"internally_reserved":0,"allocated":1175552,"data_stored":1175552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1076,"kb_used_data":1076,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165580,"statfs":{"total":53418655744,"available":53417553920,"internally_reserved":0,"allocated":1101824,"data_stored":1101824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53418655744,"available":53417480192,"internally_reserved":0,"allocated":1175552,"data_stored":1175552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53418655744,"available":53417553920,"internally_reserved":0,"allocated":1101824,"data_stored":1101824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53418655744,"available":53417480192,"internally_reserved":0,"allocated":1175552,"data_stored":1175552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53418655744,"available":53418332160,"internally_reserved":0,"allocated":323584,"data_stored":323584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-26T03:43:02.545 INFO:tasks.ceph.ceph_manager.ceph:clean! 2026-02-26T03:43:02.545 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2026-02-26T03:43:02.545 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2026-02-26T03:43:02.545 DEBUG:teuthology.orchestra.run.trial104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-26T03:43:02.621 INFO:teuthology.orchestra.run.trial104.stderr:2026-02-26T03:43:02.618+0000 7f343ab5c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:43:02.622 INFO:teuthology.orchestra.run.trial104.stderr:2026-02-26T03:43:02.621+0000 7f343ab5c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:43:02.758 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:43:02.758 INFO:teuthology.orchestra.run.trial104.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2026-02-26T03:43:02.767 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2026-02-26T03:43:02.767 INFO:teuthology.run_tasks:Running task ssh_keys... 2026-02-26T03:43:02.869 DEBUG:teuthology.orchestra.run.trial104:> echo 'StrictHostKeyChecking no 2026-02-26T03:43:02.869 DEBUG:teuthology.orchestra.run.trial104:> ' > /home/ubuntu/.ssh/config && echo 'UserKnownHostsFile ' /dev/null >> /home/ubuntu/.ssh/config && chmod 600 /home/ubuntu/.ssh/config 2026-02-26T03:43:02.885 INFO:teuthology.task.ssh_keys:generated public key AAAAB3NzaC1yc2EAAAADAQABAAABAQCpqDsT7WWVGGwF7qB8r0k5lnhg8yk+6m3HY+tqq7L2JMup2LHYF9B+LgHxB0oeW6ByoQ9NZrjyYxh6c0kADDyxazoT7flV9nS9i04NNOS6LhA4wFENqPjUQaQgDNmWck+E4BoEBLSJ1MW21nyHzVuxWxoFJ0pLbzdSqDcMJVKI64nknRpVhalVvWkjon/NcPppWKBEB4JVlm16oZAYq4matEKodUviJHtrwQPPIUiyrlYU+CDV+eZDBf7GOa7IEzJFEJi8c0LDYdCokesez9zItqtycJafnAAKS/H6J2bg/xKDRUMVKUUvlJRt7yZAOGBlyrINIw7bkYD32kUlUqbN 2026-02-26T03:43:02.885 INFO:teuthology.task.ssh_keys:pushing keys to trial104.front.sepia.ceph.com for ubuntu 2026-02-26T03:43:02.885 DEBUG:teuthology.orchestra.run.trial104:> rm -f -- /home/ubuntu/.ssh/id_rsa 2026-02-26T03:43:02.941 DEBUG:teuthology.orchestra.run.trial104:> set -ex 2026-02-26T03:43:02.941 DEBUG:teuthology.orchestra.run.trial104:> dd of=/home/ubuntu/.ssh/id_rsa 2026-02-26T03:43:02.941 DEBUG:teuthology.orchestra.run.trial104:> chmod 0500 /home/ubuntu/.ssh/id_rsa 2026-02-26T03:43:02.998 DEBUG:teuthology.orchestra.run.trial104:> rm -f -- /home/ubuntu/.ssh/id_rsa.pub 2026-02-26T03:43:03.053 DEBUG:teuthology.orchestra.run.trial104:> set -ex 2026-02-26T03:43:03.053 DEBUG:teuthology.orchestra.run.trial104:> dd of=/home/ubuntu/.ssh/id_rsa.pub 2026-02-26T03:43:03.109 DEBUG:teuthology.orchestra.run.trial104:> cp -v -a /home/ubuntu/.ssh/authorized_keys /home/ubuntu/.ssh/authorized_keys_2026-02-26_03:43:03:109617 2026-02-26T03:43:03.165 INFO:teuthology.orchestra.run.trial104.stdout:'/home/ubuntu/.ssh/authorized_keys' -> '/home/ubuntu/.ssh/authorized_keys_2026-02-26_03:43:03:109617' 2026-02-26T03:43:03.166 DEBUG:teuthology.orchestra.run.trial104:> set -ex 2026-02-26T03:43:03.166 DEBUG:teuthology.orchestra.run.trial104:> dd of=/home/ubuntu/.ssh/authorized_keys conv=notrunc oflag=append 2026-02-26T03:43:03.222 INFO:teuthology.run_tasks:Running task cbt... 2026-02-26T03:43:03.247 INFO:tasks.cbt:cbt configuration is {'cluster': {'user': 'ubuntu', 'head': 'ubuntu@trial104.front.sepia.ceph.com', 'osds': ['ubuntu@trial104.front.sepia.ceph.com'], 'mons': ['ubuntu@trial104.front.sepia.ceph.com'], 'clients': ['ubuntu@trial104.front.sepia.ceph.com'], 'rgws': {'ubuntu@trial104.front.sepia.ceph.com': None}, 'osds_per_node': 3, 'rebuild_every_test': False, 'use_existing': True, 'is_teuthology': True, 'iterations': 1, 'tmp_dir': '/tmp/cbt', 'pool_profiles': {'replicated': {'pg_size': 256, 'pgp_size': 256, 'replication': 'replicated'}}, 'pid_dir': '/var/run/ceph'}, 'benchmarks': {'radosbench': {'concurrent_ops': 4, 'concurrent_procs': 2, 'op_size': [4194304], 'pool_monitoring_list': ['collectl'], 'pool_profile': 'replicated', 'readmode': 'rand', 'run_monitoring_list': ['collectl'], 'time': 60, 'write_only': False}}, 'client_endpoints': None, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}} 2026-02-26T03:43:03.248 DEBUG:teuthology.orchestra.run.trial104:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/cbt 2026-02-26T03:43:03.283 DEBUG:teuthology.orchestra.run.trial104:> set -ex 2026-02-26T03:43:03.284 DEBUG:teuthology.orchestra.run.trial104:> dd of=/home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2026-02-26T03:43:03.336 DEBUG:teuthology.orchestra.run.trial104:> git clone --depth 1 -b master https://github.com/ceph/cbt.git /home/ubuntu/cephtest/cbt 2026-02-26T03:43:03.399 INFO:teuthology.orchestra.run.trial104.stderr:Cloning into '/home/ubuntu/cephtest/cbt'... 2026-02-26T03:43:03.625 INFO:tasks.cbt:Installing collectl 2026-02-26T03:43:03.626 DEBUG:teuthology.orchestra.run.trial104:> sudo mkdir collectl && cd collectl && sudo wget https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download -O collectl.tar.gz && sudo tar -xvf collectl.tar.gz && cd collectl-4.3.1 && sudo ./INSTALL 2026-02-26T03:43:03.700 INFO:teuthology.orchestra.run.trial104.stderr:--2026-02-26 03:43:03-- https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download 2026-02-26T03:43:03.755 INFO:teuthology.orchestra.run.trial104.stderr:Resolving sourceforge.net (sourceforge.net)... 104.18.12.149, 104.18.13.149, 2606:4700::6812:d95, ... 2026-02-26T03:43:03.758 INFO:teuthology.orchestra.run.trial104.stderr:Connecting to sourceforge.net (sourceforge.net)|104.18.12.149|:443... connected. 2026-02-26T03:43:03.999 INFO:teuthology.orchestra.run.trial104.stderr:HTTP request sent, awaiting response... 302 Found 2026-02-26T03:43:04.000 INFO:teuthology.orchestra.run.trial104.stderr:Location: https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABpn8FHn8D0AoWxjbaepehnjadWF6idp3dsxAlyqsBkJcPSQJ-tLip2u2uRtXC3Sh-tFLhzChCxX0M21SLvASXhFuJJ3Q%3D%3D&use_mirror=psychz&r= [following] 2026-02-26T03:43:04.000 INFO:teuthology.orchestra.run.trial104.stderr:--2026-02-26 03:43:03-- https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABpn8FHn8D0AoWxjbaepehnjadWF6idp3dsxAlyqsBkJcPSQJ-tLip2u2uRtXC3Sh-tFLhzChCxX0M21SLvASXhFuJJ3Q%3D%3D&use_mirror=psychz&r= 2026-02-26T03:43:04.044 INFO:teuthology.orchestra.run.trial104.stderr:Resolving downloads.sourceforge.net (downloads.sourceforge.net)... 104.18.12.149, 104.18.13.149, 2606:4700::6812:c95, ... 2026-02-26T03:43:04.047 INFO:teuthology.orchestra.run.trial104.stderr:Connecting to downloads.sourceforge.net (downloads.sourceforge.net)|104.18.12.149|:443... connected. 2026-02-26T03:43:04.284 INFO:teuthology.orchestra.run.trial104.stderr:HTTP request sent, awaiting response... 302 Found 2026-02-26T03:43:04.284 INFO:teuthology.orchestra.run.trial104.stderr:Location: https://psychz.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?viasf=1 [following] 2026-02-26T03:43:04.284 INFO:teuthology.orchestra.run.trial104.stderr:--2026-02-26 03:43:04-- https://psychz.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?viasf=1 2026-02-26T03:43:04.290 INFO:teuthology.orchestra.run.trial104.stderr:Resolving psychz.dl.sourceforge.net (psychz.dl.sourceforge.net)... 208.87.241.191 2026-02-26T03:43:04.358 INFO:teuthology.orchestra.run.trial104.stderr:Connecting to psychz.dl.sourceforge.net (psychz.dl.sourceforge.net)|208.87.241.191|:443... connected. 2026-02-26T03:43:04.675 INFO:teuthology.orchestra.run.trial104.stderr:HTTP request sent, awaiting response... 200 OK 2026-02-26T03:43:04.676 INFO:teuthology.orchestra.run.trial104.stderr:Length: 620693 (606K) [application/x-gzip] 2026-02-26T03:43:04.676 INFO:teuthology.orchestra.run.trial104.stderr:Saving to: ‘collectl.tar.gz’ 2026-02-26T03:43:04.676 INFO:teuthology.orchestra.run.trial104.stderr: 2026-02-26T03:43:04.744 INFO:teuthology.orchestra.run.trial104.stderr: 0K .......... .......... .......... .......... .......... 8% 727K 1s 2026-02-26T03:43:04.810 INFO:teuthology.orchestra.run.trial104.stderr: 50K .......... .......... .......... .......... .......... 16% 750K 1s 2026-02-26T03:43:04.812 INFO:teuthology.orchestra.run.trial104.stderr: 100K .......... .......... .......... .......... .......... 24% 29.1M 0s 2026-02-26T03:43:04.852 INFO:teuthology.orchestra.run.trial104.stderr: 150K .......... .......... .......... .......... .......... 32% 1.21M 0s 2026-02-26T03:43:04.881 INFO:teuthology.orchestra.run.trial104.stderr: 200K .......... .......... .......... .......... .......... 41% 1.82M 0s 2026-02-26T03:43:04.881 INFO:teuthology.orchestra.run.trial104.stderr: 250K .......... .......... .......... .......... .......... 49% 135M 0s 2026-02-26T03:43:04.881 INFO:teuthology.orchestra.run.trial104.stderr: 300K .......... .......... .......... .......... .......... 57% 209M 0s 2026-02-26T03:43:04.881 INFO:teuthology.orchestra.run.trial104.stderr: 350K .......... .......... .......... .......... .......... 65% 67.4M 0s 2026-02-26T03:43:04.949 INFO:teuthology.orchestra.run.trial104.stderr: 400K .......... .......... .......... .......... .......... 74% 761K 0s 2026-02-26T03:43:04.949 INFO:teuthology.orchestra.run.trial104.stderr: 450K .......... .......... .......... .......... .......... 82% 192M 0s 2026-02-26T03:43:04.949 INFO:teuthology.orchestra.run.trial104.stderr: 500K .......... .......... .......... .......... .......... 90% 139M 0s 2026-02-26T03:43:04.949 INFO:teuthology.orchestra.run.trial104.stderr: 550K .......... .......... .......... .......... .......... 98% 306M 0s 2026-02-26T03:43:04.949 INFO:teuthology.orchestra.run.trial104.stderr: 600K ...... 100% 11.4T=0.3s 2026-02-26T03:43:04.949 INFO:teuthology.orchestra.run.trial104.stderr: 2026-02-26T03:43:04.949 INFO:teuthology.orchestra.run.trial104.stderr:2026-02-26 03:43:04 (2.18 MB/s) - ‘collectl.tar.gz’ saved [620693/620693] 2026-02-26T03:43:04.950 INFO:teuthology.orchestra.run.trial104.stderr: 2026-02-26T03:43:04.971 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/ 2026-02-26T03:43:04.971 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/collectl.conf 2026-02-26T03:43:04.971 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/docs/ 2026-02-26T03:43:04.972 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/docs/SlowProc.html 2026-02-26T03:43:04.972 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/docs/BuddyInfo.html 2026-02-26T03:43:04.972 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/docs/WhySummary.html 2026-02-26T03:43:04.972 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/docs/Socket.html 2026-02-26T03:43:04.972 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/docs/colmux.html 2026-02-26T03:43:04.972 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/docs/Tutorial.html 2026-02-26T03:43:04.972 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/docs/Gexpr.html 2026-02-26T03:43:04.972 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/docs/Import.html 2026-02-26T03:43:04.972 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/docs/style.css 2026-02-26T03:43:04.972 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/docs/Ganglia3.jpg 2026-02-26T03:43:04.972 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/docs/Ganglia2.jpg 2026-02-26T03:43:04.972 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/docs/FileNaming.html 2026-02-26T03:43:04.973 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/docs/TheMath.html 2026-02-26T03:43:04.973 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/docs/Environmental.html 2026-02-26T03:43:04.973 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/docs/Process.html 2026-02-26T03:43:04.973 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/docs/OutputFiles.html 2026-02-26T03:43:04.973 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/docs/HiResTime.html 2026-02-26T03:43:04.973 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/docs/OutputFormats.html 2026-02-26T03:43:04.973 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/docs/slub.jpg 2026-02-26T03:43:04.973 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/docs/Lustre.html 2026-02-26T03:43:04.973 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/docs/Documentation.html 2026-02-26T03:43:04.973 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/docs/Inodes.html 2026-02-26T03:43:04.973 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/docs/Graphite.html 2026-02-26T03:43:04.973 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/docs/InputFiles.html 2026-02-26T03:43:04.973 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/docs/Export.html 2026-02-26T03:43:04.974 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/docs/Interrupts.html 2026-02-26T03:43:04.974 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/docs/Startup.html 2026-02-26T03:43:04.974 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/docs/Performance.html 2026-02-26T03:43:04.974 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/docs/Messages.html 2026-02-26T03:43:04.974 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/docs/Logging.html 2026-02-26T03:43:04.974 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/docs/Data-detail.html 2026-02-26T03:43:04.974 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/docs/CPUs.html 2026-02-26T03:43:04.974 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/docs/Matrix.html 2026-02-26T03:43:04.974 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/docs/Data-brief.html 2026-02-26T03:43:04.974 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/docs/SlabInfo.html 2026-02-26T03:43:04.974 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/docs/Misc.html 2026-02-26T03:43:04.974 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/docs/Examples.html 2026-02-26T03:43:04.974 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/docs/OpenStack.html 2026-02-26T03:43:04.975 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/docs/Plotfiles.html 2026-02-26T03:43:04.975 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/docs/ColmuxCPU.jpg 2026-02-26T03:43:04.975 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/docs/Tutorial-Lustre.html 2026-02-26T03:43:04.975 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/docs/Data-verbose.html 2026-02-26T03:43:04.975 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/docs/Data.html 2026-02-26T03:43:04.975 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/docs/Exceptions.html 2026-02-26T03:43:04.975 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/docs/RunningAsAService.html 2026-02-26T03:43:04.975 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/docs/FAQ-collectl.html 2026-02-26T03:43:04.975 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/docs/Ganglia.jpg 2026-02-26T03:43:04.975 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/docs/OperationalModes.html 2026-02-26T03:43:04.975 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/docs/Playback.html 2026-02-26T03:43:04.975 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/docs/Hello.html 2026-02-26T03:43:04.975 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/docs/Sockets.html 2026-02-26T03:43:04.975 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/docs/Network.html 2026-02-26T03:43:04.976 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/docs/Infiniband.html 2026-02-26T03:43:04.976 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/docs/Memory.html 2026-02-26T03:43:04.976 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/docs/ColmuxLustre.jpg 2026-02-26T03:43:04.976 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/docs/Disks.html 2026-02-26T03:43:04.976 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/docs/NfsInfo.html 2026-02-26T03:43:04.976 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/service/ 2026-02-26T03:43:04.976 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/service/collectl.service 2026-02-26T03:43:04.976 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/initd/ 2026-02-26T03:43:04.976 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/initd/collectl-suse 2026-02-26T03:43:04.976 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/initd/collectl-debian 2026-02-26T03:43:04.976 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/initd/collectl-generic 2026-02-26T03:43:04.976 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/initd/collectl 2026-02-26T03:43:04.976 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/misc.ph 2026-02-26T03:43:04.977 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/gexpr.ph 2026-02-26T03:43:04.977 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/statsd.ph 2026-02-26T03:43:04.977 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/envrules.std 2026-02-26T03:43:04.977 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/hello.ph 2026-02-26T03:43:04.977 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/vmsum.ph 2026-02-26T03:43:04.977 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/proctree.ph 2026-02-26T03:43:04.977 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/README-WINDOWS 2026-02-26T03:43:04.977 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/formatit.ph 2026-02-26T03:43:04.977 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/colmux 2026-02-26T03:43:04.977 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/INSTALL 2026-02-26T03:43:04.977 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/graphite.ph 2026-02-26T03:43:04.977 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/RELEASE-collectl 2026-02-26T03:43:04.977 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/vmstat.ph 2026-02-26T03:43:04.978 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/README 2026-02-26T03:43:04.978 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/man1/ 2026-02-26T03:43:04.978 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/man1/collectl.1 2026-02-26T03:43:04.978 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/man1/colmux.1 2026-02-26T03:43:04.978 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/lexpr.ph 2026-02-26T03:43:04.978 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/client.pl 2026-02-26T03:43:04.978 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/collectl 2026-02-26T03:43:04.978 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/COPYING 2026-02-26T03:43:04.978 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/vnet.ph 2026-02-26T03:43:04.978 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/GPL 2026-02-26T03:43:04.978 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/UNINSTALL 2026-02-26T03:43:04.978 INFO:teuthology.orchestra.run.trial104.stdout:collectl-4.3.1/ARTISTIC 2026-02-26T03:43:05.014 INFO:teuthology.orchestra.run.trial104.stderr:./INSTALL: line 90: chkconfig: command not found 2026-02-26T03:43:05.016 DEBUG:teuthology.orchestra.run.trial104:> sudo yum -y install librbd-devel pdsh pdsh-rcmd-ssh perf 2026-02-26T03:43:05.375 INFO:teuthology.orchestra.run.trial104.stdout:Last metadata expiration check: 0:00:57 ago on Thu 26 Feb 2026 03:42:08 AM UTC. 2026-02-26T03:43:05.462 INFO:teuthology.orchestra.run.trial104.stdout:Dependencies resolved. 2026-02-26T03:43:05.462 INFO:teuthology.orchestra.run.trial104.stdout:================================================================================ 2026-02-26T03:43:05.463 INFO:teuthology.orchestra.run.trial104.stdout: Package Arch Version Repository Size 2026-02-26T03:43:05.463 INFO:teuthology.orchestra.run.trial104.stdout:================================================================================ 2026-02-26T03:43:05.463 INFO:teuthology.orchestra.run.trial104.stdout:Installing: 2026-02-26T03:43:05.463 INFO:teuthology.orchestra.run.trial104.stdout: librbd-devel x86_64 2:20.3.0-5312.gabffb617.el9 ceph 24 k 2026-02-26T03:43:05.463 INFO:teuthology.orchestra.run.trial104.stdout: pdsh x86_64 2.34-7.el9 epel 118 k 2026-02-26T03:43:05.463 INFO:teuthology.orchestra.run.trial104.stdout: pdsh-rcmd-ssh x86_64 2.34-7.el9 epel 13 k 2026-02-26T03:43:05.463 INFO:teuthology.orchestra.run.trial104.stdout: perf x86_64 5.14.0-686.el9 appstream 3.3 M 2026-02-26T03:43:05.463 INFO:teuthology.orchestra.run.trial104.stdout:Installing dependencies: 2026-02-26T03:43:05.463 INFO:teuthology.orchestra.run.trial104.stdout: libradospp-devel x86_64 2:20.3.0-5312.gabffb617.el9 ceph 31 k 2026-02-26T03:43:05.463 INFO:teuthology.orchestra.run.trial104.stdout: libtraceevent x86_64 1.8.4-2.el9 baseos 290 k 2026-02-26T03:43:05.463 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:43:05.464 INFO:teuthology.orchestra.run.trial104.stdout:Transaction Summary 2026-02-26T03:43:05.464 INFO:teuthology.orchestra.run.trial104.stdout:================================================================================ 2026-02-26T03:43:05.464 INFO:teuthology.orchestra.run.trial104.stdout:Install 6 Packages 2026-02-26T03:43:05.464 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:43:05.464 INFO:teuthology.orchestra.run.trial104.stdout:Total download size: 3.8 M 2026-02-26T03:43:05.464 INFO:teuthology.orchestra.run.trial104.stdout:Installed size: 14 M 2026-02-26T03:43:05.464 INFO:teuthology.orchestra.run.trial104.stdout:Downloading Packages: 2026-02-26T03:43:05.987 INFO:teuthology.orchestra.run.trial104.stdout:(1/6): libtraceevent-1.8.4-2.el9.x86_64.rpm 5.1 MB/s | 290 kB 00:00 2026-02-26T03:43:06.287 INFO:teuthology.orchestra.run.trial104.stdout:(2/6): libradospp-devel-20.3.0-5312.gabffb617.e 88 kB/s | 31 kB 00:00 2026-02-26T03:43:06.287 INFO:teuthology.orchestra.run.trial104.stdout:(3/6): librbd-devel-20.3.0-5312.gabffb617.el9.x 67 kB/s | 24 kB 00:00 2026-02-26T03:43:06.512 INFO:teuthology.orchestra.run.trial104.stdout:(4/6): pdsh-rcmd-ssh-2.34-7.el9.x86_64.rpm 58 kB/s | 13 kB 00:00 2026-02-26T03:43:06.730 INFO:teuthology.orchestra.run.trial104.stdout:(5/6): pdsh-2.34-7.el9.x86_64.rpm 268 kB/s | 118 kB 00:00 2026-02-26T03:43:06.905 INFO:teuthology.orchestra.run.trial104.stdout:(6/6): perf-5.14.0-686.el9.x86_64.rpm 3.6 MB/s | 3.3 MB 00:00 2026-02-26T03:43:06.909 INFO:teuthology.orchestra.run.trial104.stdout:-------------------------------------------------------------------------------- 2026-02-26T03:43:06.909 INFO:teuthology.orchestra.run.trial104.stdout:Total 2.6 MB/s | 3.8 MB 00:01 2026-02-26T03:43:06.934 INFO:teuthology.orchestra.run.trial104.stdout:Running transaction check 2026-02-26T03:43:06.945 INFO:teuthology.orchestra.run.trial104.stdout:Transaction check succeeded. 2026-02-26T03:43:06.945 INFO:teuthology.orchestra.run.trial104.stdout:Running transaction test 2026-02-26T03:43:07.003 INFO:teuthology.orchestra.run.trial104.stdout:Transaction test succeeded. 2026-02-26T03:43:07.003 INFO:teuthology.orchestra.run.trial104.stdout:Running transaction 2026-02-26T03:43:07.078 INFO:teuthology.orchestra.run.trial104.stdout: Preparing : 1/1 2026-02-26T03:43:07.084 INFO:teuthology.orchestra.run.trial104.stdout: Installing : pdsh-rcmd-ssh-2.34-7.el9.x86_64 1/6 2026-02-26T03:43:07.124 INFO:teuthology.orchestra.run.trial104.stdout: Installing : pdsh-2.34-7.el9.x86_64 2/6 2026-02-26T03:43:07.128 INFO:teuthology.orchestra.run.trial104.stdout: Installing : libtraceevent-1.8.4-2.el9.x86_64 3/6 2026-02-26T03:43:07.130 INFO:teuthology.orchestra.run.trial104.stdout: Installing : libradospp-devel-2:20.3.0-5312.gabffb617.el9.x86_64 4/6 2026-02-26T03:43:07.296 INFO:teuthology.orchestra.run.trial104.stdout: Installing : librbd-devel-2:20.3.0-5312.gabffb617.el9.x86_64 5/6 2026-02-26T03:43:07.319 INFO:teuthology.orchestra.run.trial104.stdout: Installing : perf-5.14.0-686.el9.x86_64 6/6 2026-02-26T03:43:07.432 INFO:teuthology.orchestra.run.trial104.stdout: Running scriptlet: perf-5.14.0-686.el9.x86_64 6/6 2026-02-26T03:43:07.432 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : libradospp-devel-2:20.3.0-5312.gabffb617.el9.x86_64 1/6 2026-02-26T03:43:07.432 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : librbd-devel-2:20.3.0-5312.gabffb617.el9.x86_64 2/6 2026-02-26T03:43:07.432 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : libtraceevent-1.8.4-2.el9.x86_64 3/6 2026-02-26T03:43:07.432 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : perf-5.14.0-686.el9.x86_64 4/6 2026-02-26T03:43:07.432 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : pdsh-2.34-7.el9.x86_64 5/6 2026-02-26T03:43:07.498 INFO:teuthology.orchestra.run.trial104.stdout: Verifying : pdsh-rcmd-ssh-2.34-7.el9.x86_64 6/6 2026-02-26T03:43:07.498 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:43:07.498 INFO:teuthology.orchestra.run.trial104.stdout:Installed: 2026-02-26T03:43:07.498 INFO:teuthology.orchestra.run.trial104.stdout: libradospp-devel-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:43:07.498 INFO:teuthology.orchestra.run.trial104.stdout: librbd-devel-2:20.3.0-5312.gabffb617.el9.x86_64 2026-02-26T03:43:07.498 INFO:teuthology.orchestra.run.trial104.stdout: libtraceevent-1.8.4-2.el9.x86_64 2026-02-26T03:43:07.498 INFO:teuthology.orchestra.run.trial104.stdout: pdsh-2.34-7.el9.x86_64 2026-02-26T03:43:07.499 INFO:teuthology.orchestra.run.trial104.stdout: pdsh-rcmd-ssh-2.34-7.el9.x86_64 2026-02-26T03:43:07.499 INFO:teuthology.orchestra.run.trial104.stdout: perf-5.14.0-686.el9.x86_64 2026-02-26T03:43:07.499 INFO:teuthology.orchestra.run.trial104.stdout: 2026-02-26T03:43:07.499 INFO:teuthology.orchestra.run.trial104.stdout:Complete! 2026-02-26T03:43:07.529 DEBUG:teuthology.orchestra.run.trial104:> python3 -m pip install -r /home/ubuntu/cephtest/cbt/requirements.txt 2026-02-26T03:43:07.668 INFO:teuthology.orchestra.run.trial104.stdout:Defaulting to user installation because normal site-packages is not writeable 2026-02-26T03:43:07.679 INFO:teuthology.orchestra.run.trial104.stdout:Requirement already satisfied: pyyaml in /usr/lib64/python3.9/site-packages (from -r /home/ubuntu/cephtest/cbt/requirements.txt (line 1)) (5.4.1) 2026-02-26T03:43:07.679 INFO:teuthology.orchestra.run.trial104.stdout:Requirement already satisfied: lxml in /usr/lib64/python3.9/site-packages (from -r /home/ubuntu/cephtest/cbt/requirements.txt (line 2)) (4.6.5) 2026-02-26T03:43:08.013 INFO:teuthology.orchestra.run.trial104.stdout:Collecting matplotlib 2026-02-26T03:43:08.035 INFO:teuthology.orchestra.run.trial104.stdout: Downloading matplotlib-3.9.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (8.3 MB) 2026-02-26T03:43:08.440 INFO:teuthology.orchestra.run.trial104.stdout:Collecting fonttools>=4.22.0 2026-02-26T03:43:08.445 INFO:teuthology.orchestra.run.trial104.stdout: Downloading fonttools-4.60.2-cp39-cp39-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (4.8 MB) 2026-02-26T03:43:08.477 INFO:teuthology.orchestra.run.trial104.stdout:Requirement already satisfied: pyparsing>=2.3.1 in /usr/lib/python3.9/site-packages (from matplotlib->-r /home/ubuntu/cephtest/cbt/requirements.txt (line 3)) (2.4.7) 2026-02-26T03:43:08.478 INFO:teuthology.orchestra.run.trial104.stdout:Requirement already satisfied: python-dateutil>=2.7 in /usr/lib/python3.9/site-packages (from matplotlib->-r /home/ubuntu/cephtest/cbt/requirements.txt (line 3)) (2.9.0.post0) 2026-02-26T03:43:08.478 INFO:teuthology.orchestra.run.trial104.stdout:Requirement already satisfied: packaging>=20.0 in /usr/lib/python3.9/site-packages (from matplotlib->-r /home/ubuntu/cephtest/cbt/requirements.txt (line 3)) (20.9) 2026-02-26T03:43:08.647 INFO:teuthology.orchestra.run.trial104.stdout:Collecting contourpy>=1.0.1 2026-02-26T03:43:08.654 INFO:teuthology.orchestra.run.trial104.stdout: Downloading contourpy-1.3.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (321 kB) 2026-02-26T03:43:09.064 INFO:teuthology.orchestra.run.trial104.stdout:Collecting pillow>=8 2026-02-26T03:43:09.070 INFO:teuthology.orchestra.run.trial104.stdout: Downloading pillow-11.3.0-cp39-cp39-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (6.6 MB) 2026-02-26T03:43:09.126 INFO:teuthology.orchestra.run.trial104.stdout:Collecting cycler>=0.10 2026-02-26T03:43:09.130 INFO:teuthology.orchestra.run.trial104.stdout: Downloading cycler-0.12.1-py3-none-any.whl (8.3 kB) 2026-02-26T03:43:09.131 INFO:teuthology.orchestra.run.trial104.stdout:Requirement already satisfied: numpy>=1.23 in /usr/lib64/python3.9/site-packages (from matplotlib->-r /home/ubuntu/cephtest/cbt/requirements.txt (line 3)) (1.23.5) 2026-02-26T03:43:09.281 INFO:teuthology.orchestra.run.trial104.stdout:Collecting kiwisolver>=1.3.1 2026-02-26T03:43:09.289 INFO:teuthology.orchestra.run.trial104.stdout: Downloading kiwisolver-1.4.7-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.6 MB) 2026-02-26T03:43:09.331 INFO:teuthology.orchestra.run.trial104.stdout:Collecting importlib-resources>=3.2.0 2026-02-26T03:43:09.335 INFO:teuthology.orchestra.run.trial104.stdout: Downloading importlib_resources-6.5.2-py3-none-any.whl (37 kB) 2026-02-26T03:43:09.385 INFO:teuthology.orchestra.run.trial104.stdout:Collecting zipp>=3.1.0 2026-02-26T03:43:09.390 INFO:teuthology.orchestra.run.trial104.stdout: Downloading zipp-3.23.0-py3-none-any.whl (10 kB) 2026-02-26T03:43:09.401 INFO:teuthology.orchestra.run.trial104.stdout:Requirement already satisfied: six>=1.5 in /usr/lib/python3.9/site-packages (from python-dateutil>=2.7->matplotlib->-r /home/ubuntu/cephtest/cbt/requirements.txt (line 3)) (1.15.0) 2026-02-26T03:43:09.516 INFO:teuthology.orchestra.run.trial104.stdout:Installing collected packages: zipp, pillow, kiwisolver, importlib-resources, fonttools, cycler, contourpy, matplotlib 2026-02-26T03:43:10.576 INFO:teuthology.orchestra.run.trial104.stdout:Successfully installed contourpy-1.3.0 cycler-0.12.1 fonttools-4.60.2 importlib-resources-6.5.2 kiwisolver-1.4.7 matplotlib-3.9.4 pillow-11.3.0 zipp-3.23.0 2026-02-26T03:43:10.682 INFO:tasks.cbt:benchmark: radosbench 2026-02-26T03:43:10.683 DEBUG:teuthology.orchestra.run.trial104:> sudo /sbin/sysctl -q -w kernel.perf_event_paranoid=0 2026-02-26T03:43:10.709 DEBUG:teuthology.orchestra.run.trial104:> /home/ubuntu/cephtest/cbt/cbt.py -a /home/ubuntu/cephtest/archive/cbt /home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2026-02-26T03:43:10.820 INFO:teuthology.orchestra.run.trial104.stderr:03:43:10 - DEBUG - cbt - Settings.cluster: 2026-02-26T03:43:10.820 INFO:teuthology.orchestra.run.trial104.stderr: {'archive_dir': '/home/ubuntu/cephtest/archive/cbt', 2026-02-26T03:43:10.820 INFO:teuthology.orchestra.run.trial104.stderr: 'clients': ['ubuntu@trial104.front.sepia.ceph.com'], 2026-02-26T03:43:10.820 INFO:teuthology.orchestra.run.trial104.stderr: 'conf_file': '/etc/ceph/ceph.conf', 2026-02-26T03:43:10.820 INFO:teuthology.orchestra.run.trial104.stderr: 'head': 'ubuntu@trial104.front.sepia.ceph.com', 2026-02-26T03:43:10.820 INFO:teuthology.orchestra.run.trial104.stderr: 'is_teuthology': True, 2026-02-26T03:43:10.821 INFO:teuthology.orchestra.run.trial104.stderr: 'iterations': 1, 2026-02-26T03:43:10.821 INFO:teuthology.orchestra.run.trial104.stderr: 'mons': ['ubuntu@trial104.front.sepia.ceph.com'], 2026-02-26T03:43:10.821 INFO:teuthology.orchestra.run.trial104.stderr: 'osds': ['ubuntu@trial104.front.sepia.ceph.com'], 2026-02-26T03:43:10.821 INFO:teuthology.orchestra.run.trial104.stderr: 'osds_per_node': 3, 2026-02-26T03:43:10.821 INFO:teuthology.orchestra.run.trial104.stderr: 'pid_dir': '/var/run/ceph', 2026-02-26T03:43:10.821 INFO:teuthology.orchestra.run.trial104.stderr: 'pool_profiles': {'replicated': {'pg_size': 256, 2026-02-26T03:43:10.821 INFO:teuthology.orchestra.run.trial104.stderr: 'pgp_size': 256, 2026-02-26T03:43:10.821 INFO:teuthology.orchestra.run.trial104.stderr: 'replication': 'replicated'}}, 2026-02-26T03:43:10.821 INFO:teuthology.orchestra.run.trial104.stderr: 'rebuild_every_test': False, 2026-02-26T03:43:10.821 INFO:teuthology.orchestra.run.trial104.stderr: 'rgws': {'ubuntu@trial104.front.sepia.ceph.com': None}, 2026-02-26T03:43:10.821 INFO:teuthology.orchestra.run.trial104.stderr: 'tmp_dir': '/tmp/cbt', 2026-02-26T03:43:10.821 INFO:teuthology.orchestra.run.trial104.stderr: 'use_existing': True, 2026-02-26T03:43:10.821 INFO:teuthology.orchestra.run.trial104.stderr: 'user': 'ubuntu'} 2026-02-26T03:43:10.822 INFO:teuthology.orchestra.run.trial104.stderr:03:43:10 - INFO - cbt - Results dir: /home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c 2026-02-26T03:43:10.822 INFO:teuthology.orchestra.run.trial104.stderr:03:43:10 - INFO - cbt - Pausing for 60s for idle monitoring. 2026-02-26T03:43:10.822 INFO:teuthology.orchestra.run.trial104.stderr:03:43:10 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial104.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/idle_monitoring/collectl 2026-02-26T03:43:10.989 INFO:teuthology.orchestra.run.trial104.stderr:03:43:10 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial104.front.sepia.ceph.com collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/idle_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2026-02-26T03:43:10.991 INFO:teuthology.orchestra.run.trial104.stderr:03:43:10 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial104.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/idle_monitoring/perf 2026-02-26T03:43:11.153 INFO:teuthology.orchestra.run.trial104.stderr:03:43:11 - DEBUG - cbt - PerfMonitoring: remote_node 2026-02-26T03:43:11.153 INFO:teuthology.orchestra.run.trial104.stderr:03:43:11 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial104.front.sepia.ceph.com for pid in `cat /var/run/ceph/crimson-osd.*.pid`; do perf stat -p ${pid} -o /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/idle_monitoring/perf/perf_stat.${pid} & done 2026-02-26T03:44:11.215 INFO:teuthology.orchestra.run.trial104.stderr:03:44:11 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial104.front.sepia.ceph.com pkill -SIGINT -f collectl 2026-02-26T03:44:11.393 INFO:teuthology.orchestra.run.trial104.stderr:03:44:11 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial104.front.sepia.ceph.com sudo pkill -SIGINT -f perf\ 2026-02-26T03:44:11.572 INFO:teuthology.orchestra.run.trial104.stderr:03:44:11 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial104.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/idle_monitoring/perf/perf.data 2026-02-26T03:44:11.573 INFO:teuthology.orchestra.run.trial104.stderr:03:44:11 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial104.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/idle_monitoring/perf/perf_stat.* 2026-02-26T03:44:11.575 INFO:teuthology.orchestra.run.trial104.stderr:03:44:11 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial104.front.sepia.ceph.com sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/* 2026-02-26T03:44:11.753 INFO:teuthology.orchestra.run.trial104.stderr:03:44:11 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=rpdcp -f 10 -R ssh -w ubuntu@trial104.front.sepia.ceph.com -r /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/* /home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c 2026-02-26T03:44:11.917 INFO:teuthology.orchestra.run.trial104.stderr:03:44:11 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial104.front.sepia.ceph.com sudo killall -9 rados 2026-02-26T03:44:12.095 INFO:teuthology.orchestra.run.trial104.stderr:03:44:12 - INFO - cbt - Results dir: /home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c 2026-02-26T03:44:12.095 INFO:teuthology.orchestra.run.trial104.stderr:03:44:12 - INFO - cbt - Skipping existing test in /home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c. 2026-02-26T03:44:12.095 INFO:teuthology.orchestra.run.trial104.stderr:03:44:12 - INFO - cbt - Running benchmark /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004 2026-02-26T03:44:12.095 INFO:teuthology.orchestra.run.trial104.stderr:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c 2026-02-26T03:44:12.095 INFO:teuthology.orchestra.run.trial104.stderr:{'concurrent_ops': 4, 'concurrent_procs': 2, 'pool_profile': 'replicated', 'readmode': 'rand', 'time': 60, 'write_only': False, 'op_size': 4194304, 'pool_monitoring_list': 'collectl', 'run_monitoring_list': 'collectl', 'benchmark': 'radosbench', 'iteration': 0} == iteration 0 == 2026-02-26T03:44:12.095 INFO:teuthology.orchestra.run.trial104.stderr:03:44:12 - INFO - cbt - Setting OSD Read Ahead to: 0 2026-02-26T03:44:12.095 INFO:teuthology.orchestra.run.trial104.stderr:03:44:12 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial104.front.sepia.ceph.com find /dev/disk/by-partlabel/osd-device-*data -exec readlink {} \; | cut -d"/" -f 3 | sed "s/[0-9]$//" | xargs -I{} sudo sh -c "echo 0 > /sys/block/'{}'/queue/read_ahead_kb" 2026-02-26T03:44:12.096 INFO:teuthology.orchestra.run.trial104.stderr:03:44:12 - DEBUG - cbt - Cleaning existing temporary run directory: /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004 2026-02-26T03:44:12.096 INFO:teuthology.orchestra.run.trial104.stderr:03:44:12 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial104.front.sepia.ceph.com sudo rm -rf /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004 2026-02-26T03:44:12.269 INFO:teuthology.orchestra.run.trial104.stderr:03:44:12 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial104.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/pool_monitoring/collectl 2026-02-26T03:44:12.430 INFO:teuthology.orchestra.run.trial104.stderr:03:44:12 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial104.front.sepia.ceph.com collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/pool_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2026-02-26T03:44:12.432 INFO:teuthology.orchestra.run.trial104.stderr:03:44:12 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial104.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/pool_monitoring/perf 2026-02-26T03:44:12.596 INFO:teuthology.orchestra.run.trial104.stderr:03:44:12 - DEBUG - cbt - PerfMonitoring: remote_node 2026-02-26T03:44:12.596 INFO:teuthology.orchestra.run.trial104.stderr:03:44:12 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial104.front.sepia.ceph.com for pid in `cat /var/run/ceph/crimson-osd.*.pid`; do perf stat -p ${pid} -o /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/pool_monitoring/perf/perf_stat.${pid} & done 2026-02-26T03:44:12.598 INFO:teuthology.orchestra.run.trial104.stderr:03:44:12 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial104.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool delete rados-bench-cbt rados-bench-cbt --yes-i-really-really-mean-it 2026-02-26T03:44:12.943 INFO:teuthology.orchestra.run.trial104.stderr:03:44:12 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial104.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool create rados-bench-cbt 256 256 2026-02-26T03:44:14.233 INFO:teuthology.orchestra.run.trial104.stderr:03:44:14 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial104.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool application enable rados-bench-cbt radosbench 2026-02-26T03:44:15.237 INFO:teuthology.orchestra.run.trial104.stderr:03:44:15 - INFO - cbt - Checking Health after pool creation. 2026-02-26T03:44:20.240 INFO:teuthology.orchestra.run.trial104.stderr:03:44:20 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial104.front.sepia.ceph.com /usr/bin/ceph -c /etc/ceph/ceph.conf health 2026-02-26T03:44:20.603 INFO:teuthology.orchestra.run.trial104.stderr:03:44:20 - INFO - cbt - Final Pool Health Check. 2026-02-26T03:44:25.608 INFO:teuthology.orchestra.run.trial104.stderr:03:44:25 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial104.front.sepia.ceph.com /usr/bin/ceph -c /etc/ceph/ceph.conf health 2026-02-26T03:44:25.973 INFO:teuthology.orchestra.run.trial104.stderr:03:44:25 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial104.front.sepia.ceph.com pkill -SIGINT -f collectl 2026-02-26T03:44:26.140 INFO:teuthology.orchestra.run.trial104.stderr:03:44:26 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial104.front.sepia.ceph.com sudo pkill -SIGINT -f perf\ 2026-02-26T03:44:26.324 INFO:teuthology.orchestra.run.trial104.stderr:03:44:26 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial104.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/pool_monitoring/perf/perf.data 2026-02-26T03:44:26.326 INFO:teuthology.orchestra.run.trial104.stderr:03:44:26 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial104.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/pool_monitoring/perf/perf_stat.* 2026-02-26T03:44:26.328 INFO:teuthology.orchestra.run.trial104.stderr:03:44:26 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial104.front.sepia.ceph.com sync 2026-02-26T03:44:26.538 INFO:teuthology.orchestra.run.trial104.stderr:03:44:26 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial104.front.sepia.ceph.com echo 3 | sudo tee /proc/sys/vm/drop_caches 2026-02-26T03:44:27.586 INFO:teuthology.orchestra.run.trial104.stderr:03:44:27 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial104.front.sepia.ceph.com /usr/bin/rados -c /etc/ceph/ceph.conf -v 2026-02-26T03:44:27.793 INFO:teuthology.orchestra.run.trial104.stderr:03:44:27 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial104.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/write 2026-02-26T03:44:27.956 INFO:teuthology.orchestra.run.trial104.stderr:03:44:27 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial104.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf daemon osd.0 config show > /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/write/ceph_settings.out 2026-02-26T03:44:28.245 INFO:teuthology.orchestra.run.trial104.stderr:03:44:28 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial104.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/write/collectl 2026-02-26T03:44:28.407 INFO:teuthology.orchestra.run.trial104.stderr:03:44:28 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial104.front.sepia.ceph.com collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/write/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2026-02-26T03:44:28.409 INFO:teuthology.orchestra.run.trial104.stderr:03:44:28 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial104.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/write/perf 2026-02-26T03:44:28.572 INFO:teuthology.orchestra.run.trial104.stderr:03:44:28 - DEBUG - cbt - PerfMonitoring: remote_node 2026-02-26T03:44:28.572 INFO:teuthology.orchestra.run.trial104.stderr:03:44:28 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial104.front.sepia.ceph.com for pid in `cat /var/run/ceph/crimson-osd.*.pid`; do perf stat -p ${pid} -o /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/write/perf/perf_stat.${pid} & done 2026-02-26T03:44:28.573 INFO:teuthology.orchestra.run.trial104.stderr:03:44:28 - INFO - cbt - Running radosbench write test. 2026-02-26T03:44:28.574 INFO:teuthology.orchestra.run.trial104.stderr:03:44:28 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial104.front.sepia.ceph.com /usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench -b 4194304 60 write --concurrent-ios 4 --run-name `hostname -f`-0 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/write/objecter.0.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/write/output.0 2026-02-26T03:44:28.575 INFO:teuthology.orchestra.run.trial104.stderr:03:44:28 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial104.front.sepia.ceph.com /usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench -b 4194304 60 write --concurrent-ios 4 --run-name `hostname -f`-1 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/write/objecter.1.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/write/output.1 2026-02-26T03:45:29.800 INFO:teuthology.orchestra.run.trial104.stderr:03:45:29 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial104.front.sepia.ceph.com pkill -SIGINT -f collectl 2026-02-26T03:45:29.979 INFO:teuthology.orchestra.run.trial104.stderr:03:45:29 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial104.front.sepia.ceph.com sudo pkill -SIGINT -f perf\ 2026-02-26T03:45:30.156 INFO:teuthology.orchestra.run.trial104.stderr:03:45:30 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial104.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/write/perf/perf.data 2026-02-26T03:45:30.158 INFO:teuthology.orchestra.run.trial104.stderr:03:45:30 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial104.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/write/perf/perf_stat.* 2026-02-26T03:45:30.159 INFO:teuthology.orchestra.run.trial104.stderr:03:45:30 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial104.front.sepia.ceph.com find /var/run/ceph/ceph-osd*.asok -maxdepth 1 -exec sudo /usr/bin/ceph --admin-daemon {} dump_historic_ops \; > /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/write/historic_ops.out 2026-02-26T03:45:30.518 INFO:teuthology.orchestra.run.trial104.stderr:03:45:30 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial104.front.sepia.ceph.com sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/write/* 2026-02-26T03:45:30.695 INFO:teuthology.orchestra.run.trial104.stderr:03:45:30 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=rpdcp -f 10 -R ssh -w ubuntu@trial104.front.sepia.ceph.com -r /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/write/* /home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/write 2026-02-26T03:45:30.863 INFO:teuthology.orchestra.run.trial104.stderr:03:45:30 - INFO - cbt - Convert results to json format. 2026-02-26T03:45:30.864 INFO:teuthology.orchestra.run.trial104.stderr:03:45:30 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial104.front.sepia.ceph.com sync 2026-02-26T03:45:31.026 INFO:teuthology.orchestra.run.trial104.stderr:03:45:31 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial104.front.sepia.ceph.com echo 3 | sudo tee /proc/sys/vm/drop_caches 2026-02-26T03:45:31.341 INFO:teuthology.orchestra.run.trial104.stderr:03:45:31 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial104.front.sepia.ceph.com /usr/bin/rados -c /etc/ceph/ceph.conf -v 2026-02-26T03:45:31.552 INFO:teuthology.orchestra.run.trial104.stderr:03:45:31 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial104.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/rand 2026-02-26T03:45:31.713 INFO:teuthology.orchestra.run.trial104.stderr:03:45:31 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial104.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf daemon osd.0 config show > /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/rand/ceph_settings.out 2026-02-26T03:45:32.006 INFO:teuthology.orchestra.run.trial104.stderr:03:45:32 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial104.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/rand/collectl 2026-02-26T03:45:32.167 INFO:teuthology.orchestra.run.trial104.stderr:03:45:32 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial104.front.sepia.ceph.com collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/rand/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2026-02-26T03:45:32.169 INFO:teuthology.orchestra.run.trial104.stderr:03:45:32 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial104.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/rand/perf 2026-02-26T03:45:32.341 INFO:teuthology.orchestra.run.trial104.stderr:03:45:32 - DEBUG - cbt - PerfMonitoring: remote_node 2026-02-26T03:45:32.341 INFO:teuthology.orchestra.run.trial104.stderr:03:45:32 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial104.front.sepia.ceph.com for pid in `cat /var/run/ceph/crimson-osd.*.pid`; do perf stat -p ${pid} -o /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/rand/perf/perf_stat.${pid} & done 2026-02-26T03:45:32.342 INFO:teuthology.orchestra.run.trial104.stderr:03:45:32 - INFO - cbt - Running radosbench rand test. 2026-02-26T03:45:32.343 INFO:teuthology.orchestra.run.trial104.stderr:03:45:32 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial104.front.sepia.ceph.com /usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench 60 rand --concurrent-ios 4 --run-name `hostname -f`-0 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/rand/objecter.0.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/rand/output.0 2026-02-26T03:45:32.344 INFO:teuthology.orchestra.run.trial104.stderr:03:45:32 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial104.front.sepia.ceph.com /usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench 60 rand --concurrent-ios 4 --run-name `hostname -f`-1 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/rand/objecter.1.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/rand/output.1 2026-02-26T03:46:33.571 INFO:teuthology.orchestra.run.trial104.stderr:03:46:33 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial104.front.sepia.ceph.com pkill -SIGINT -f collectl 2026-02-26T03:46:33.742 INFO:teuthology.orchestra.run.trial104.stderr:03:46:33 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial104.front.sepia.ceph.com sudo pkill -SIGINT -f perf\ 2026-02-26T03:46:33.923 INFO:teuthology.orchestra.run.trial104.stderr:03:46:33 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial104.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/rand/perf/perf.data 2026-02-26T03:46:33.924 INFO:teuthology.orchestra.run.trial104.stderr:03:46:33 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial104.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/rand/perf/perf_stat.* 2026-02-26T03:46:33.932 INFO:teuthology.orchestra.run.trial104.stderr:03:46:33 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial104.front.sepia.ceph.com find /var/run/ceph/ceph-osd*.asok -maxdepth 1 -exec sudo /usr/bin/ceph --admin-daemon {} dump_historic_ops \; > /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/rand/historic_ops.out 2026-02-26T03:46:34.303 INFO:teuthology.orchestra.run.trial104.stderr:03:46:34 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial104.front.sepia.ceph.com sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/rand/* 2026-02-26T03:46:34.475 INFO:teuthology.orchestra.run.trial104.stderr:03:46:34 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=rpdcp -f 10 -R ssh -w ubuntu@trial104.front.sepia.ceph.com -r /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/rand/* /home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/rand 2026-02-26T03:46:34.639 INFO:teuthology.orchestra.run.trial104.stderr:03:46:34 - INFO - cbt - Convert results to json format. 2026-02-26T03:46:34.653 DEBUG:teuthology.run_tasks:Unwinding manager cbt 2026-02-26T03:46:34.663 DEBUG:teuthology.orchestra.run.trial104:> rm --one-file-system -rf -- /home/ubuntu/cephtest/cbt 2026-02-26T03:46:34.690 INFO:tasks.cbt_performance:Collecting CBT performance data config 2026-02-26T03:46:34.690 INFO:tasks.cbt_performance:reading cbt preferences 2026-02-26T03:46:34.691 DEBUG:teuthology.orchestra.run.trial104:> find /home/ubuntu/cephtest -name 'json_output.*' 2026-02-26T03:46:34.748 INFO:teuthology.orchestra.run.trial104.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/write/json_output.0.trial104.front.sepia.ceph.com 2026-02-26T03:46:34.748 INFO:teuthology.orchestra.run.trial104.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/write/json_output.1.trial104.front.sepia.ceph.com 2026-02-26T03:46:34.748 INFO:teuthology.orchestra.run.trial104.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/rand/json_output.0.trial104.front.sepia.ceph.com 2026-02-26T03:46:34.748 INFO:teuthology.orchestra.run.trial104.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/rand/json_output.1.trial104.front.sepia.ceph.com 2026-02-26T03:46:34.749 DEBUG:teuthology.orchestra.run.trial104:> set -ex 2026-02-26T03:46:34.749 DEBUG:teuthology.orchestra.run.trial104:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/write/json_output.0.trial104.front.sepia.ceph.com of=/dev/stdout 2026-02-26T03:46:34.806 INFO:tasks.cbt_performance:reading total cpu cycles 2026-02-26T03:46:34.807 DEBUG:teuthology.orchestra.run.trial104:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/write -name 'perf_stat.*' 2026-02-26T03:46:34.864 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2026-02-26T03:46:34.864 INFO:tasks.cbt_performance:total cpu cycles: 0 2026-02-26T03:46:34.864 DEBUG:teuthology.orchestra.run.trial104:> set -ex 2026-02-26T03:46:34.864 DEBUG:teuthology.orchestra.run.trial104:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/write/json_output.1.trial104.front.sepia.ceph.com of=/dev/stdout 2026-02-26T03:46:34.921 INFO:tasks.cbt_performance:reading total cpu cycles 2026-02-26T03:46:34.921 DEBUG:teuthology.orchestra.run.trial104:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/write -name 'perf_stat.*' 2026-02-26T03:46:34.978 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2026-02-26T03:46:34.978 INFO:tasks.cbt_performance:total cpu cycles: 0 2026-02-26T03:46:34.978 DEBUG:teuthology.orchestra.run.trial104:> set -ex 2026-02-26T03:46:34.978 DEBUG:teuthology.orchestra.run.trial104:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/rand/json_output.0.trial104.front.sepia.ceph.com of=/dev/stdout 2026-02-26T03:46:35.034 INFO:tasks.cbt_performance:reading total cpu cycles 2026-02-26T03:46:35.034 DEBUG:teuthology.orchestra.run.trial104:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/rand -name 'perf_stat.*' 2026-02-26T03:46:35.090 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2026-02-26T03:46:35.090 INFO:tasks.cbt_performance:total cpu cycles: 0 2026-02-26T03:46:35.091 DEBUG:teuthology.orchestra.run.trial104:> set -ex 2026-02-26T03:46:35.091 DEBUG:teuthology.orchestra.run.trial104:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/rand/json_output.1.trial104.front.sepia.ceph.com of=/dev/stdout 2026-02-26T03:46:35.146 INFO:tasks.cbt_performance:reading total cpu cycles 2026-02-26T03:46:35.146 DEBUG:teuthology.orchestra.run.trial104:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/rand -name 'perf_stat.*' 2026-02-26T03:46:35.202 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2026-02-26T03:46:35.202 INFO:tasks.cbt_performance:total cpu cycles: 0 2026-02-26T03:46:35.205 WARNING:tasks.cbt_performance:Could not send performance data to http://mira118.front.sepia.ceph.com:4000/cbt_performance: HTTPConnectionPool(host='mira118.front.sepia.ceph.com', port=4000): Max retries exceeded with url: /cbt_performance (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -2] Name or service not known')) 2026-02-26T03:46:35.205 WARNING:tasks.cbt_performance:Continuing without performance data collection 2026-02-26T03:46:35.205 DEBUG:teuthology.run_tasks:Unwinding manager ssh_keys 2026-02-26T03:46:35.217 INFO:teuthology.task.ssh_keys:Cleaning up SSH keys 2026-02-26T03:46:35.217 INFO:teuthology.task.ssh_keys:cleaning up keys added for testing 2026-02-26T03:46:35.217 INFO:teuthology.task.ssh_keys: cleaning up keys for user ubuntu on trial104.front.sepia.ceph.com 2026-02-26T03:46:35.217 DEBUG:teuthology.orchestra.run.trial104:> rm -- /home/ubuntu/.ssh/id_rsa 2026-02-26T03:46:35.259 DEBUG:teuthology.orchestra.run.trial104:> rm -- /home/ubuntu/.ssh/id_rsa.pub 2026-02-26T03:46:35.316 DEBUG:teuthology.orchestra.run.trial104:> stat -c '"%a"' /home/ubuntu/.ssh/authorized_keys 2026-02-26T03:46:35.373 INFO:teuthology.orchestra.run.trial104.stdout:"600" 2026-02-26T03:46:35.373 DEBUG:teuthology.orchestra.run.trial104:> mv -- /home/ubuntu/.ssh/authorized_keys_2026-02-26_03:43:03:109617 /home/ubuntu/.ssh/authorized_keys 2026-02-26T03:46:35.431 DEBUG:teuthology.orchestra.run.trial104:> chmod 600 /home/ubuntu/.ssh/authorized_keys 2026-02-26T03:46:35.487 DEBUG:teuthology.orchestra.run.trial104:> rm /home/ubuntu/.ssh/config 2026-02-26T03:46:35.543 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2026-02-26T03:46:35.560 DEBUG:teuthology.orchestra.run.trial104:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2026-02-26T03:46:35.659 INFO:teuthology.orchestra.run.trial104.stderr:2026-02-26T03:46:35.652+0000 7fe9774c2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:46:35.660 INFO:teuthology.orchestra.run.trial104.stderr:2026-02-26T03:46:35.657+0000 7fe976239640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-26T03:46:35.785 INFO:teuthology.misc:Shutting down mds daemons... 2026-02-26T03:46:35.785 INFO:teuthology.misc:Shutting down osd daemons... 2026-02-26T03:46:35.786 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2026-02-26T03:46:35.786 INFO:teuthology.orchestra.run:waiting for 300 2026-02-26T03:46:35.860 INFO:tasks.ceph.osd.0:Stopped 2026-02-26T03:46:35.860 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2026-02-26T03:46:35.860 INFO:teuthology.orchestra.run:waiting for 300 2026-02-26T03:46:35.938 INFO:tasks.ceph.osd.1:Stopped 2026-02-26T03:46:35.939 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2026-02-26T03:46:35.939 INFO:teuthology.orchestra.run:waiting for 300 2026-02-26T03:46:36.015 INFO:tasks.ceph.osd.2:Stopped 2026-02-26T03:46:36.015 INFO:teuthology.misc:Shutting down mgr daemons... 2026-02-26T03:46:36.015 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2026-02-26T03:46:36.015 INFO:teuthology.orchestra.run:waiting for 300 2026-02-26T03:46:36.055 INFO:tasks.ceph.mgr.x:Stopped 2026-02-26T03:46:36.056 INFO:teuthology.misc:Shutting down mon daemons... 2026-02-26T03:46:36.056 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2026-02-26T03:46:36.056 INFO:teuthology.orchestra.run:waiting for 300 2026-02-26T03:46:36.106 INFO:tasks.ceph.mon.a:Stopped 2026-02-26T03:46:36.106 INFO:tasks.ceph:Checking cluster log for badness... 2026-02-26T03:46:36.106 DEBUG:teuthology.orchestra.run.trial104:> sudo grep -E '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | grep -E -v '\(MDS_ALL_DOWN\)' | grep -E -v '\(MDS_UP_LESS_THAN_MAX\)' | grep -E -v '\(POOL_APP_NOT_ENABLED\)' | grep -E -v '\(PG_' | grep -E -v '\(OSD_' | grep -E -v '\(OBJECT_' | grep -E -v 'overall HEALTH' | head -n 1 2026-02-26T03:46:36.172 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@trial104.front.sepia.ceph.com 2026-02-26T03:46:36.173 DEBUG:teuthology.orchestra.run.trial104:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2026-02-26T03:46:36.243 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@trial104.front.sepia.ceph.com 2026-02-26T03:46:36.243 DEBUG:teuthology.orchestra.run.trial104:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2026-02-26T03:46:36.314 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@trial104.front.sepia.ceph.com 2026-02-26T03:46:36.314 DEBUG:teuthology.orchestra.run.trial104:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2026-02-26T03:46:36.382 INFO:tasks.ceph:Archiving mon data... 2026-02-26T03:46:36.384 DEBUG:teuthology.misc:Transferring archived files from trial104:/var/lib/ceph/mon/ceph-a to /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-26_02:22:56-crimson-rados-main-distro-debug-trial/71217/data/mon.a.tgz 2026-02-26T03:46:36.384 DEBUG:teuthology.orchestra.run.trial104:> mktemp 2026-02-26T03:46:36.438 INFO:teuthology.orchestra.run.trial104.stdout:/tmp/tmp.HF01oHRjPo 2026-02-26T03:46:36.439 DEBUG:teuthology.orchestra.run.trial104:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.HF01oHRjPo 2026-02-26T03:46:36.530 DEBUG:teuthology.orchestra.run.trial104:> sudo chmod 0666 /tmp/tmp.HF01oHRjPo 2026-02-26T03:46:36.612 DEBUG:teuthology.orchestra.remote:trial104:/tmp/tmp.HF01oHRjPo is 206KB 2026-02-26T03:46:36.675 DEBUG:teuthology.orchestra.run.trial104:> rm -fr /tmp/tmp.HF01oHRjPo 2026-02-26T03:46:36.690 INFO:tasks.ceph:Cleaning ceph cluster... 2026-02-26T03:46:36.691 DEBUG:teuthology.orchestra.run.trial104:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2026-02-26T03:46:36.811 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2026-02-26T03:46:36.811 INFO:tasks.ceph:Archiving crash dumps... 2026-02-26T03:46:36.813 DEBUG:teuthology.misc:Transferring archived files from trial104:/var/lib/ceph/crash to /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-26_02:22:56-crimson-rados-main-distro-debug-trial/71217/remote/trial104/crash 2026-02-26T03:46:36.814 DEBUG:teuthology.orchestra.run.trial104:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2026-02-26T03:46:36.875 INFO:tasks.ceph:Compressing logs... 2026-02-26T03:46:36.875 DEBUG:teuthology.orchestra.run.trial104:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2026-02-26T03:46:36.938 INFO:teuthology.orchestra.run.trial104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2026-02-26T03:46:36.939 INFO:teuthology.orchestra.run.trial104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40970.log 2026-02-26T03:46:36.939 INFO:teuthology.orchestra.run.trial104.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2026-02-26T03:46:36.939 INFO:teuthology.orchestra.run.trial104.stderr:/var/log/ceph/ceph-client.admin.40970.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2026-02-26T03:46:36.939 INFO:teuthology.orchestra.run.trial104.stderr: 66.9% -- replaced with /var/log/ceph/ceph-client.admin.40970.log.gz 2026-02-26T03:46:36.939 INFO:teuthology.orchestra.run.trial104.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40875.log 2026-02-26T03:46:36.939 INFO:teuthology.orchestra.run.trial104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44254.log 2026-02-26T03:46:36.940 INFO:teuthology.orchestra.run.trial104.stderr:/var/log/ceph/ceph-osd.0.log: /var/log/ceph/ceph-client.admin.40875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45009.log 2026-02-26T03:46:36.950 INFO:teuthology.orchestra.run.trial104.stderr: 91.2% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2026-02-26T03:46:36.951 INFO:teuthology.orchestra.run.trial104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40392.log 2026-02-26T03:46:36.951 INFO:teuthology.orchestra.run.trial104.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40875.log.gz 2026-02-26T03:46:36.951 INFO:teuthology.orchestra.run.trial104.stderr:/var/log/ceph/ceph-client.admin.44254.log: /var/log/ceph/ceph-client.admin.45009.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45007.log 2026-02-26T03:46:36.951 INFO:teuthology.orchestra.run.trial104.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44254.log.gz 2026-02-26T03:46:36.951 INFO:teuthology.orchestra.run.trial104.stderr:/var/log/ceph/ceph-client.admin.40392.log: 66.7% -- replaced with /var/log/ceph/ceph-client.admin.45009.log.gz 2026-02-26T03:46:36.951 INFO:teuthology.orchestra.run.trial104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40161.log 2026-02-26T03:46:36.951 INFO:teuthology.orchestra.run.trial104.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40392.log.gz 2026-02-26T03:46:36.951 INFO:teuthology.orchestra.run.trial104.stderr:/var/log/ceph/ceph-client.admin.45007.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44400.log 2026-02-26T03:46:36.951 INFO:teuthology.orchestra.run.trial104.stderr: 65.9% -- replaced with /var/log/ceph/ceph-client.admin.45007.log.gz 2026-02-26T03:46:36.952 INFO:teuthology.orchestra.run.trial104.stderr:/var/log/ceph/ceph-client.admin.40161.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40641.log 2026-02-26T03:46:36.952 INFO:teuthology.orchestra.run.trial104.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.40161.log.gz 2026-02-26T03:46:36.952 INFO:teuthology.orchestra.run.trial104.stderr:/var/log/ceph/ceph-client.admin.44400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41183.log 2026-02-26T03:46:36.952 INFO:teuthology.orchestra.run.trial104.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44400.log.gz 2026-02-26T03:46:36.952 INFO:teuthology.orchestra.run.trial104.stderr:/var/log/ceph/ceph-client.admin.40641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41085.log 2026-02-26T03:46:36.952 INFO:teuthology.orchestra.run.trial104.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.40641.log.gz 2026-02-26T03:46:36.952 INFO:teuthology.orchestra.run.trial104.stderr:/var/log/ceph/ceph-client.admin.41183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40802.log 2026-02-26T03:46:36.952 INFO:teuthology.orchestra.run.trial104.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41183.log.gz 2026-02-26T03:46:36.952 INFO:teuthology.orchestra.run.trial104.stderr:/var/log/ceph/ceph-client.admin.41085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40298.log 2026-02-26T03:46:36.952 INFO:teuthology.orchestra.run.trial104.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41085.log.gz 2026-02-26T03:46:36.952 INFO:teuthology.orchestra.run.trial104.stderr:/var/log/ceph/ceph-client.admin.40802.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41284.log 2026-02-26T03:46:36.952 INFO:teuthology.orchestra.run.trial104.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40802.log.gz 2026-02-26T03:46:36.952 INFO:teuthology.orchestra.run.trial104.stderr:/var/log/ceph/ceph-client.admin.40298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41526.log 2026-02-26T03:46:36.953 INFO:teuthology.orchestra.run.trial104.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40298.log.gz 2026-02-26T03:46:36.953 INFO:teuthology.orchestra.run.trial104.stderr:/var/log/ceph/ceph-client.admin.41284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41134.log 2026-02-26T03:46:36.953 INFO:teuthology.orchestra.run.trial104.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41284.log.gz 2026-02-26T03:46:36.953 INFO:teuthology.orchestra.run.trial104.stderr:/var/log/ceph/ceph-client.admin.41526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40345.log 2026-02-26T03:46:36.953 INFO:teuthology.orchestra.run.trial104.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41526.log.gz 2026-02-26T03:46:36.953 INFO:teuthology.orchestra.run.trial104.stderr:/var/log/ceph/ceph-client.admin.41134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44310.log 2026-02-26T03:46:36.953 INFO:teuthology.orchestra.run.trial104.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41134.log.gz 2026-02-26T03:46:36.953 INFO:teuthology.orchestra.run.trial104.stderr:/var/log/ceph/ceph-client.admin.40345.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40945.log 2026-02-26T03:46:36.953 INFO:teuthology.orchestra.run.trial104.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40345.log.gz 2026-02-26T03:46:36.953 INFO:teuthology.orchestra.run.trial104.stderr:/var/log/ceph/ceph-client.admin.44310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40114.log 2026-02-26T03:46:36.953 INFO:teuthology.orchestra.run.trial104.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.44310.log.gz 2026-02-26T03:46:36.953 INFO:teuthology.orchestra.run.trial104.stderr:/var/log/ceph/ceph-client.admin.40945.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2026-02-26T03:46:36.954 INFO:teuthology.orchestra.run.trial104.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.40945.log.gz 2026-02-26T03:46:36.954 INFO:teuthology.orchestra.run.trial104.stderr:/var/log/ceph/ceph-client.admin.40114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46371.log 2026-02-26T03:46:36.954 INFO:teuthology.orchestra.run.trial104.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40114.log.gz 2026-02-26T03:46:36.954 INFO:teuthology.orchestra.run.trial104.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45694.log 2026-02-26T03:46:36.954 INFO:teuthology.orchestra.run.trial104.stderr:/var/log/ceph/ceph-client.admin.46371.log: 48.4% -- replaced with /var/log/ceph/ceph-client.admin.46371.log.gz 2026-02-26T03:46:36.954 INFO:teuthology.orchestra.run.trial104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44454.log 2026-02-26T03:46:36.954 INFO:teuthology.orchestra.run.trial104.stderr: 87.7% -- replaced with /var/log/ceph/ceph.audit.log.gz 2026-02-26T03:46:36.954 INFO:teuthology.orchestra.run.trial104.stderr:/var/log/ceph/ceph-client.admin.45694.log: 65.9% -- replaced with /var/log/ceph/ceph-client.admin.45694.log.gz 2026-02-26T03:46:36.954 INFO:teuthology.orchestra.run.trial104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41432.log 2026-02-26T03:46:36.954 INFO:teuthology.orchestra.run.trial104.stderr:/var/log/ceph/ceph-client.admin.44454.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2026-02-26T03:46:36.954 INFO:teuthology.orchestra.run.trial104.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.44454.log.gz 2026-02-26T03:46:36.954 INFO:teuthology.orchestra.run.trial104.stderr:/var/log/ceph/ceph-client.admin.41432.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.38730.log 2026-02-26T03:46:36.955 INFO:teuthology.orchestra.run.trial104.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41432.log.gz 2026-02-26T03:46:36.955 INFO:teuthology.orchestra.run.trial104.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44368.log 2026-02-26T03:46:36.955 INFO:teuthology.orchestra.run.trial104.stderr:/var/log/ceph/ceph.tmp-client.admin.38730.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40019.log 2026-02-26T03:46:36.955 INFO:teuthology.orchestra.run.trial104.stderr: 68.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.38730.log.gz 2026-02-26T03:46:36.955 INFO:teuthology.orchestra.run.trial104.stderr:/var/log/ceph/ceph-client.admin.44368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41285.log 2026-02-26T03:46:36.955 INFO:teuthology.orchestra.run.trial104.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44368.log.gz 2026-02-26T03:46:36.955 INFO:teuthology.orchestra.run.trial104.stderr:/var/log/ceph/ceph-client.admin.40019.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41624.log 2026-02-26T03:46:36.955 INFO:teuthology.orchestra.run.trial104.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.40019.log.gz 2026-02-26T03:46:36.955 INFO:teuthology.orchestra.run.trial104.stderr:/var/log/ceph/ceph-client.admin.41285.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2026-02-26T03:46:36.955 INFO:teuthology.orchestra.run.trial104.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41285.log.gz 2026-02-26T03:46:36.955 INFO:teuthology.orchestra.run.trial104.stderr:/var/log/ceph/ceph-client.admin.41624.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45685.log 2026-02-26T03:46:36.955 INFO:teuthology.orchestra.run.trial104.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41624.log.gz 2026-02-26T03:46:36.955 INFO:teuthology.orchestra.run.trial104.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2026-02-26T03:46:36.956 INFO:teuthology.orchestra.run.trial104.stderr:/var/log/ceph/ceph-client.admin.45685.log: gzip 66.7% -- replaced with /var/log/ceph/ceph-client.admin.45685.log.gz 2026-02-26T03:46:36.956 INFO:teuthology.orchestra.run.trial104.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.41036.log 2026-02-26T03:46:36.956 INFO:teuthology.orchestra.run.trial104.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41431.log 2026-02-26T03:46:36.956 INFO:teuthology.orchestra.run.trial104.stderr: 85.8% -- replaced with /var/log/ceph/ceph.log.gz 2026-02-26T03:46:36.956 INFO:teuthology.orchestra.run.trial104.stderr:/var/log/ceph/ceph-client.admin.41036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41286.log 2026-02-26T03:46:36.956 INFO:teuthology.orchestra.run.trial104.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41036.log.gz 2026-02-26T03:46:36.956 INFO:teuthology.orchestra.run.trial104.stderr:/var/log/ceph/ceph-client.admin.41431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40611.log 2026-02-26T03:46:36.956 INFO:teuthology.orchestra.run.trial104.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41431.log.gz 2026-02-26T03:46:36.956 INFO:teuthology.orchestra.run.trial104.stderr:/var/log/ceph/ceph-client.admin.41286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41433.log 2026-02-26T03:46:36.956 INFO:teuthology.orchestra.run.trial104.stderr:/var/log/ceph/ceph-client.admin.40611.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41286.log.gz 2026-02-26T03:46:36.956 INFO:teuthology.orchestra.run.trial104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41575.log 2026-02-26T03:46:36.956 INFO:teuthology.orchestra.run.trial104.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.40611.log.gz 2026-02-26T03:46:36.957 INFO:teuthology.orchestra.run.trial104.stderr:/var/log/ceph/ceph-client.admin.41433.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41433.log.gz 2026-02-26T03:46:36.957 INFO:teuthology.orchestra.run.trial104.stderr:/var/log/ceph/ceph-client.admin.41575.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41575.log.gz 2026-02-26T03:46:36.958 INFO:teuthology.orchestra.run.trial104.stderr: 92.2% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2026-02-26T03:46:39.382 INFO:teuthology.orchestra.run.trial104.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2026-02-26T03:46:39.541 INFO:teuthology.orchestra.run.trial104.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2026-02-26T03:46:39.577 INFO:teuthology.orchestra.run.trial104.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2026-02-26T03:46:39.579 INFO:teuthology.orchestra.run.trial104.stderr: 2026-02-26T03:46:39.579 INFO:teuthology.orchestra.run.trial104.stderr:real 0m2.648s 2026-02-26T03:46:39.579 INFO:teuthology.orchestra.run.trial104.stderr:user 0m7.122s 2026-02-26T03:46:39.579 INFO:teuthology.orchestra.run.trial104.stderr:sys 0m0.575s 2026-02-26T03:46:39.579 INFO:tasks.ceph:Archiving logs... 2026-02-26T03:46:39.580 DEBUG:teuthology.misc:Transferring archived files from trial104:/var/log/ceph to /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-26_02:22:56-crimson-rados-main-distro-debug-trial/71217/remote/trial104/log 2026-02-26T03:46:39.581 DEBUG:teuthology.orchestra.run.trial104:> sudo tar c -f - -C /var/log/ceph -- . 2026-02-26T03:46:40.368 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-26T03:46:40.445 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-26T03:46:40.457 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2026-02-26T03:46:40.457 DEBUG:teuthology.orchestra.run.trial104:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-26T03:46:40.485 DEBUG:teuthology.run_tasks:Unwinding manager clock 2026-02-26T03:46:40.498 INFO:teuthology.task.clock:Checking final clock skew... 2026-02-26T03:46:40.498 DEBUG:teuthology.orchestra.run.trial104:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-26T03:46:40.540 INFO:teuthology.orchestra.run.trial104.stderr:bash: line 1: ntpq: command not found 2026-02-26T03:46:40.564 INFO:teuthology.orchestra.run.trial104.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-26T03:46:40.564 INFO:teuthology.orchestra.run.trial104.stdout:=============================================================================== 2026-02-26T03:46:40.564 INFO:teuthology.orchestra.run.trial104.stdout:^+ triangle.kansas.net 3 7 103 200 -3175us[-3262us] +/- 59ms 2026-02-26T03:46:40.564 INFO:teuthology.orchestra.run.trial104.stdout:^* lax1.us.ntp.li 2 7 377 72 -272us[ -362us] +/- 34ms 2026-02-26T03:46:40.564 INFO:teuthology.orchestra.run.trial104.stdout:^+ time.gslb.gravinoc.com 2 7 377 73 +1809us[+1720us] +/- 40ms 2026-02-26T03:46:40.564 INFO:teuthology.orchestra.run.trial104.stdout:^+ 23.186.168.129 2 6 377 68 +205us[ +205us] +/- 56ms 2026-02-26T03:46:40.566 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2026-02-26T03:46:40.579 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2026-02-26T03:46:40.579 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2026-02-26T03:46:40.594 DEBUG:teuthology.orchestra.run.trial104:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2026-02-26T03:46:40.666 DEBUG:teuthology.orchestra.run.trial104:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-26T03:46:40.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-26T03:46:40.692 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2026-02-26T03:46:40.704 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2026-02-26T03:46:40.723 INFO:teuthology.task.internal:Duration was 1735.884429 seconds 2026-02-26T03:46:40.723 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2026-02-26T03:46:40.735 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2026-02-26T03:46:40.735 DEBUG:teuthology.orchestra.run.trial104:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-26T03:46:40.774 INFO:teuthology.orchestra.run.trial104.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-26T03:46:41.069 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2026-02-26T03:46:41.069 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial104.front.sepia.ceph.com 2026-02-26T03:46:41.070 DEBUG:teuthology.orchestra.run.trial104:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-26T03:46:41.128 INFO:teuthology.task.internal.syslog:Gathering journactl... 2026-02-26T03:46:41.129 DEBUG:teuthology.orchestra.run.trial104:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-26T03:46:41.235 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2026-02-26T03:46:41.236 DEBUG:teuthology.orchestra.run.trial104:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-26T03:46:41.260 INFO:teuthology.orchestra.run.trial104.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-26T03:46:41.260 INFO:teuthology.orchestra.run.trial104.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-26T03:46:41.260 INFO:teuthology.orchestra.run.trial104.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-26T03:46:41.260 INFO:teuthology.orchestra.run.trial104.stderr: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-02-26T03:46:41.261 INFO:teuthology.orchestra.run.trial104.stderr:/home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-26T03:46:41.271 INFO:teuthology.orchestra.run.trial104.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: 83.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-26T03:46:41.273 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2026-02-26T03:46:41.291 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2026-02-26T03:46:41.292 DEBUG:teuthology.orchestra.run.trial104:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-26T03:46:41.338 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2026-02-26T03:46:41.357 DEBUG:teuthology.orchestra.run.trial104:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-26T03:46:41.404 INFO:teuthology.orchestra.run.trial104.stdout:kernel.core_pattern = core 2026-02-26T03:46:41.417 DEBUG:teuthology.orchestra.run.trial104:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-26T03:46:41.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-26T03:46:41.473 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2026-02-26T03:46:41.491 INFO:teuthology.task.internal:Transferring archived files... 2026-02-26T03:46:41.492 DEBUG:teuthology.misc:Transferring archived files from trial104:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-26_02:22:56-crimson-rados-main-distro-debug-trial/71217/remote/trial104 2026-02-26T03:46:41.493 DEBUG:teuthology.orchestra.run.trial104:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-26T03:46:41.566 INFO:teuthology.task.internal:Removing archive directory... 2026-02-26T03:46:41.566 DEBUG:teuthology.orchestra.run.trial104:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-26T03:46:41.598 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2026-02-26T03:46:41.613 INFO:teuthology.task.internal:Not uploading archives. 2026-02-26T03:46:41.613 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2026-02-26T03:46:41.629 INFO:teuthology.task.internal:Tidying up after the test... 2026-02-26T03:46:41.630 DEBUG:teuthology.orchestra.run.trial104:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-26T03:46:41.654 INFO:teuthology.orchestra.run.trial104.stdout: 920834 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 26 03:46 /home/ubuntu/cephtest 2026-02-26T03:46:41.655 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2026-02-26T03:46:41.673 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2026-02-26T03:46:41.709 INFO:teuthology.run:Summary data: description: crimson-rados/perf/{0-crimson_install clusters/{crimson-fixed-1} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/segmented$/{crimson_seastore_segmented} perf_ignorelist settings/optimized workloads/radosbench_4M_rand_read} duration: 1735.884429216385 flavor: debug owner: scheduled_shraddhaag@soko04.front.sepia.ceph.com success: true 2026-02-26T03:46:41.709 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-26T03:46:41.806 INFO:teuthology.run:pass