2026-02-20T12:54:02.498 INFO:root:teuthology version: 1.2.4.dev11+ge6d9099ce 2026-02-20T12:54:02.518 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-20T12:54:02.605 INFO:teuthology.run:Config: archive_path: /home/teuthworker/mnt/teuthology/jjperez-2026-02-20_10:29:19-crimson-rados-wip-perezjos-crimson-only-12-02-2026-PR66245-distro-debug-trial/60758 branch: wip-perezjos-crimson-only-12-02-2026-PR66245 description: crimson-rados/perf/{0-crimson_install clusters/{crimson-fixed-1} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/rbm$/{crimson_seastore_rbm_2q} perf_ignorelist settings/optimized workloads/sample_fio} email: null first_in_suite: false flavor: debug job_id: '60758' kernel: branch: distro kdb: 1 sha1: distro last_in_suite: false machine_type: trial name: jjperez-2026-02-20_10:29:19-crimson-rados-wip-perezjos-crimson-only-12-02-2026-PR66245-distro-debug-trial no_nested_subset: false os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-perezjos-crimson-only-12-02-2026-PR66245 ceph: conf: global: auth client required: none auth cluster required: none auth service required: none auth supported: none debug asok: 0/0 debug auth: 0/0 debug bdev: 0/0 debug bluefs: 0/0 debug bluestore: 0/0 debug buffer: 0/0 debug civetweb: 0/0 debug client: 0/0 debug compressor: 0/0 debug context: 0/0 debug crush: 0/0 debug crypto: 0/0 debug dpdk: 0/0 debug eventtrace: 0/0 debug filer: 0/0 debug filestore: 0/0 debug finisher: 0/0 debug fuse: 0/0 debug heartbeatmap: 0/0 debug javaclient: 0/0 debug journal: 0/0 debug journaler: 0/0 debug kstore: 0/0 debug leveldb: 0/0 debug lockdep: 0/0 debug mds: 0/0 debug mds balancer: 0/0 debug mds locker: 0/0 debug mds log: 0/0 debug mds log expire: 0/0 debug mds migrator: 0/0 debug memdb: 0/0 debug mgr: 0/0 debug mgrc: 0/0 debug mon: 0/0 debug monc: 0/0 debug ms: 0/0 debug objclass: 0/0 debug objectcacher: 0/0 debug objecter: 0/0 debug optracker: 0/0 debug osd: 0/0 debug paxos: 0/0 debug perfcounter: 0/0 debug rados: 0/0 debug rbd: 0/0 debug rbd mirror: 0/0 debug rbd replay: 0/0 debug refs: 0/0 debug rgw: 0/0 debug rgw lifecycle: 0/0 debug rgw notification: 0/0 debug rgw sync: 0/0 debug rocksdb: 0/0 debug striper: 0/0 debug throttle: 0/0 debug timer: 0/0 debug tp: 0/0 enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 0/0 debug ms: 0/0 debug paxos: 0/0 osd pool default crimson: true osd: crimson cpu num: 4 crimson osd obc lru size: 10 debug filestore: 0/0 debug journal: 0/0 debug ms: 0/0 debug osd: 0/0 debug seastore: 20 debug seastore backref: 20 debug seastore cache: 20 debug seastore cleaner: 20 debug seastore device: 20 debug seastore epm: 20 debug seastore fixedkv tree: 20 debug seastore journal: 20 debug seastore lba: 20 debug seastore odata: 20 debug seastore omap: 20 debug seastore onode: 20 debug seastore t: 20 debug seastore tm: 20 osd objectstore: seastore osd shutdown pgref assert: true seastore cachepin size pershard: 64M seastore cachepin type: 2Q seastore main device type: RANDOM_BLOCK_SSD seastore max concurrent transactions: 8 crimson_compat: true flavor: debug log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) - \(PG_ - \(OSD_ - \(OBJECT_ - overall HEALTH sha1: 26bc97655ae0056b78c14ab91d1307332bd6dfca ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log global: osd crush chooseleaf type: 0 osd pool default pg num: 128 osd pool default pgp num: 128 osd pool default size: 2 mon: {} install: ceph: flavor: debug sha1: 26bc97655ae0056b78c14ab91d1307332bd6dfca workunit: branch: main env: CRIMSON_COMPAT: '1' sha1: 445d5c9ba18a4040e70e25303230c36c77773acb owner: scheduled_jjperez@soko04.front.sepia.ceph.com priority: 101 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 5547 sha1: 26bc97655ae0056b78c14ab91d1307332bd6dfca sleep_before_teardown: 0 suite: crimson-rados suite_branch: main suite_path: /home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph.git suite_sha1: 445d5c9ba18a4040e70e25303230c36c77773acb targets: trial093.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ+KS8E6MM/VDOGGnZl3pISIqnht6KyA45y4MeVv3kzaeImMpRmz6kNbY5dEYtI4JfeDo+jcL3+jfWq11mWVkPE= tasks: - install: extra_packages: deb: - ceph-osd-crimson - ceph-osd-crimson-dbg rpm: - ceph-osd-crimson - ceph-osd-crimson-debuginfo - pexec: all: - sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson - install: null - ceph: conf: osd: debug monc: 20 wait-for-scrub: false - ssh_keys: null - cbt: benchmarks: librbdfio: iodepth: - 32 log_avg_msec: 100 mode: - randwrite norandommap: true op_size: - 4096 osd_ra: - 4096 pool_profile: rbd procs_per_volume: - 1 time: 60 vol_size: 4096 volumes_per_client: - 2 cluster: iterations: 1 osds_per_node: 3 pool_profiles: rbd: pg_size: 128 pgp_size: 128 replication: 3 user: ubuntu monitoring_profiles: perf: args: stat -p {pid} -o {perf_dir}/perf_stat.{pid} nodes: - osds perf_cmd: perf pid_glob: crimson-osd.*.pid teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: reimage-ssh-down teuthology_sha1: e6d9099ce03dc21193fda653159ec82465c3a94a timestamp: 2026-02-20_10:29:19 tube: trial user: jjperez verbose: false worker_log: /home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.1540699 2026-02-20T12:54:02.605 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa; will attempt to use it 2026-02-20T12:54:02.605 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks 2026-02-20T12:54:02.605 INFO:teuthology.run_tasks:Running task internal.check_packages... 2026-02-20T12:54:02.607 INFO:teuthology.task.internal:Checking packages... 2026-02-20T12:54:02.607 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'debug' and ceph hash '26bc97655ae0056b78c14ab91d1307332bd6dfca' 2026-02-20T12:54:02.607 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2026-02-20T12:54:02.607 INFO:teuthology.packaging:ref: None 2026-02-20T12:54:02.607 INFO:teuthology.packaging:tag: None 2026-02-20T12:54:02.607 INFO:teuthology.packaging:branch: wip-perezjos-crimson-only-12-02-2026-PR66245 2026-02-20T12:54:02.607 INFO:teuthology.packaging:sha1: 26bc97655ae0056b78c14ab91d1307332bd6dfca 2026-02-20T12:54:02.607 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&ref=wip-perezjos-crimson-only-12-02-2026-PR66245 2026-02-20T12:54:02.723 INFO:teuthology.task.internal:Found packages for ceph version 20.3.0-5312.g26bc9765 2026-02-20T12:54:02.724 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2026-02-20T12:54:02.736 INFO:teuthology.task.internal:no buildpackages task found 2026-02-20T12:54:02.736 INFO:teuthology.run_tasks:Running task internal.save_config... 2026-02-20T12:54:02.750 INFO:teuthology.task.internal:Saving configuration 2026-02-20T12:54:02.764 INFO:teuthology.run_tasks:Running task internal.check_lock... 2026-02-20T12:54:02.777 INFO:teuthology.task.internal.check_lock:Checking locks... 2026-02-20T12:54:02.852 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial093.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/jjperez-2026-02-20_10:29:19-crimson-rados-wip-perezjos-crimson-only-12-02-2026-PR66245-distro-debug-trial/60758', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-20 12:50:38.084311', 'locked_by': 'scheduled_jjperez@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ+KS8E6MM/VDOGGnZl3pISIqnht6KyA45y4MeVv3kzaeImMpRmz6kNbY5dEYtI4JfeDo+jcL3+jfWq11mWVkPE='} 2026-02-20T12:54:02.852 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2026-02-20T12:54:02.862 INFO:teuthology.task.internal:roles: ubuntu@trial093.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2026-02-20T12:54:02.862 INFO:teuthology.run_tasks:Running task console_log... 2026-02-20T12:54:02.948 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7ec3b9756d40>, signals=[15]) 2026-02-20T12:54:02.948 INFO:teuthology.run_tasks:Running task internal.connect... 2026-02-20T12:54:02.959 INFO:teuthology.task.internal:Opening connections... 2026-02-20T12:54:02.959 DEBUG:teuthology.task.internal:connecting to ubuntu@trial093.front.sepia.ceph.com 2026-02-20T12:54:02.960 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial093.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-20T12:54:03.026 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2026-02-20T12:54:03.038 DEBUG:teuthology.orchestra.run.trial093:> uname -m 2026-02-20T12:54:03.101 INFO:teuthology.orchestra.run.trial093.stdout:x86_64 2026-02-20T12:54:03.102 DEBUG:teuthology.orchestra.run.trial093:> cat /etc/os-release 2026-02-20T12:54:03.161 INFO:teuthology.orchestra.run.trial093.stdout:NAME="CentOS Stream" 2026-02-20T12:54:03.161 INFO:teuthology.orchestra.run.trial093.stdout:VERSION="9" 2026-02-20T12:54:03.161 INFO:teuthology.orchestra.run.trial093.stdout:ID="centos" 2026-02-20T12:54:03.161 INFO:teuthology.orchestra.run.trial093.stdout:ID_LIKE="rhel fedora" 2026-02-20T12:54:03.161 INFO:teuthology.orchestra.run.trial093.stdout:VERSION_ID="9" 2026-02-20T12:54:03.162 INFO:teuthology.orchestra.run.trial093.stdout:PLATFORM_ID="platform:el9" 2026-02-20T12:54:03.162 INFO:teuthology.orchestra.run.trial093.stdout:PRETTY_NAME="CentOS Stream 9" 2026-02-20T12:54:03.162 INFO:teuthology.orchestra.run.trial093.stdout:ANSI_COLOR="0;31" 2026-02-20T12:54:03.162 INFO:teuthology.orchestra.run.trial093.stdout:LOGO="fedora-logo-icon" 2026-02-20T12:54:03.162 INFO:teuthology.orchestra.run.trial093.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2026-02-20T12:54:03.162 INFO:teuthology.orchestra.run.trial093.stdout:HOME_URL="https://centos.org/" 2026-02-20T12:54:03.162 INFO:teuthology.orchestra.run.trial093.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2026-02-20T12:54:03.162 INFO:teuthology.orchestra.run.trial093.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2026-02-20T12:54:03.162 INFO:teuthology.orchestra.run.trial093.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2026-02-20T12:54:03.163 INFO:teuthology.lock.ops:Updating trial093.front.sepia.ceph.com on lock server 2026-02-20T12:54:03.251 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2026-02-20T12:54:03.263 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2026-02-20T12:54:03.276 INFO:teuthology.task.internal:Checking for old test directory... 2026-02-20T12:54:03.276 DEBUG:teuthology.orchestra.run.trial093:> test '!' -e /home/ubuntu/cephtest 2026-02-20T12:54:03.291 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2026-02-20T12:54:03.301 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2026-02-20T12:54:03.301 DEBUG:teuthology.orchestra.run.trial093:> test -z $(ls -A /var/lib/ceph) 2026-02-20T12:54:03.351 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2026-02-20T12:54:03.459 INFO:teuthology.run_tasks:Running task kernel... 2026-02-20T12:54:03.481 INFO:teuthology.task.kernel:normalize config orig: {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'} 2026-02-20T12:54:03.481 INFO:teuthology.task.kernel:config {'mon.a': {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'}}, timeout 300 2026-02-20T12:54:03.481 DEBUG:teuthology.orchestra.run.trial093:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-20T12:54:03.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-20T12:54:03.496 DEBUG:teuthology.orchestra.run.trial093:> uname -r 2026-02-20T12:54:03.554 INFO:teuthology.orchestra.run.trial093.stdout:5.14.0-681.el9.x86_64 2026-02-20T12:54:03.555 INFO:teuthology.task.kernel:Running kernel on trial093: 5.14.0-681.el9.x86_64 2026-02-20T12:54:03.555 DEBUG:teuthology.orchestra.run.trial093:> sudo yum install -y kernel 2026-02-20T12:54:04.804 INFO:teuthology.orchestra.run.trial093.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.9 MB 00:00 2026-02-20T12:54:07.290 INFO:teuthology.orchestra.run.trial093.stdout:CentOS Stream 9 - AppStream 14 MB/s | 27 MB 00:01 2026-02-20T12:54:10.916 INFO:teuthology.orchestra.run.trial093.stdout:CentOS Stream 9 - CRB 6.1 MB/s | 7.8 MB 00:01 2026-02-20T12:54:12.107 INFO:teuthology.orchestra.run.trial093.stdout:CentOS Stream 9 - Extras packages 43 kB/s | 20 kB 00:00 2026-02-20T12:54:13.107 INFO:teuthology.orchestra.run.trial093.stdout:Extra Packages for Enterprise Linux 22 MB/s | 20 MB 00:00 2026-02-20T12:54:16.247 INFO:teuthology.orchestra.run.trial093.stdout:lab-extras 2.5 MB/s | 50 kB 00:00 2026-02-20T12:54:17.255 INFO:teuthology.orchestra.run.trial093.stdout:Package kernel-5.14.0-645.el9.x86_64 is already installed. 2026-02-20T12:54:17.255 INFO:teuthology.orchestra.run.trial093.stdout:Package kernel-5.14.0-681.el9.x86_64 is already installed. 2026-02-20T12:54:17.274 INFO:teuthology.orchestra.run.trial093.stdout:Dependencies resolved. 2026-02-20T12:54:17.275 INFO:teuthology.orchestra.run.trial093.stdout:Nothing to do. 2026-02-20T12:54:17.275 INFO:teuthology.orchestra.run.trial093.stdout:Complete! 2026-02-20T12:54:17.298 DEBUG:teuthology.orchestra.run.trial093:> echo no | sudo yum reinstall kernel || true 2026-02-20T12:54:17.596 INFO:teuthology.orchestra.run.trial093.stdout:Last metadata expiration check: 0:00:01 ago on Fri 20 Feb 2026 12:54:16 PM UTC. 2026-02-20T12:54:17.676 INFO:teuthology.orchestra.run.trial093.stdout:Dependencies resolved. 2026-02-20T12:54:17.677 INFO:teuthology.orchestra.run.trial093.stdout:================================================================================ 2026-02-20T12:54:17.677 INFO:teuthology.orchestra.run.trial093.stdout: Package Architecture Version Repository Size 2026-02-20T12:54:17.677 INFO:teuthology.orchestra.run.trial093.stdout:================================================================================ 2026-02-20T12:54:17.677 INFO:teuthology.orchestra.run.trial093.stdout:Reinstalling: 2026-02-20T12:54:17.678 INFO:teuthology.orchestra.run.trial093.stdout: kernel x86_64 5.14.0-681.el9 baseos 803 k 2026-02-20T12:54:17.678 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T12:54:17.678 INFO:teuthology.orchestra.run.trial093.stdout:Transaction Summary 2026-02-20T12:54:17.678 INFO:teuthology.orchestra.run.trial093.stdout:================================================================================ 2026-02-20T12:54:17.678 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T12:54:17.678 INFO:teuthology.orchestra.run.trial093.stdout:Total download size: 803 k 2026-02-20T12:54:17.678 INFO:teuthology.orchestra.run.trial093.stdout:Installed size: 0 2026-02-20T12:54:17.678 INFO:teuthology.orchestra.run.trial093.stderr:Operation aborted. 2026-02-20T12:54:17.697 INFO:teuthology.orchestra.run.trial093.stdout:Is this ok [y/N]: 2026-02-20T12:54:17.697 DEBUG:teuthology.orchestra.run.trial093:> sudo yum reinstall -y kernel || true 2026-02-20T12:54:17.990 INFO:teuthology.orchestra.run.trial093.stdout:Last metadata expiration check: 0:00:01 ago on Fri 20 Feb 2026 12:54:16 PM UTC. 2026-02-20T12:54:18.070 INFO:teuthology.orchestra.run.trial093.stdout:Dependencies resolved. 2026-02-20T12:54:18.070 INFO:teuthology.orchestra.run.trial093.stdout:================================================================================ 2026-02-20T12:54:18.070 INFO:teuthology.orchestra.run.trial093.stdout: Package Architecture Version Repository Size 2026-02-20T12:54:18.070 INFO:teuthology.orchestra.run.trial093.stdout:================================================================================ 2026-02-20T12:54:18.070 INFO:teuthology.orchestra.run.trial093.stdout:Reinstalling: 2026-02-20T12:54:18.070 INFO:teuthology.orchestra.run.trial093.stdout: kernel x86_64 5.14.0-681.el9 baseos 803 k 2026-02-20T12:54:18.070 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T12:54:18.071 INFO:teuthology.orchestra.run.trial093.stdout:Transaction Summary 2026-02-20T12:54:18.071 INFO:teuthology.orchestra.run.trial093.stdout:================================================================================ 2026-02-20T12:54:18.071 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T12:54:18.071 INFO:teuthology.orchestra.run.trial093.stdout:Total download size: 803 k 2026-02-20T12:54:18.071 INFO:teuthology.orchestra.run.trial093.stdout:Installed size: 0 2026-02-20T12:54:18.071 INFO:teuthology.orchestra.run.trial093.stdout:Downloading Packages: 2026-02-20T12:54:20.993 INFO:teuthology.orchestra.run.trial093.stdout:kernel-5.14.0-681.el9.x86_64.rpm 345 kB/s | 803 kB 00:02 2026-02-20T12:54:20.993 INFO:teuthology.orchestra.run.trial093.stdout:-------------------------------------------------------------------------------- 2026-02-20T12:54:20.993 INFO:teuthology.orchestra.run.trial093.stdout:Total 275 kB/s | 803 kB 00:02 2026-02-20T12:54:21.002 INFO:teuthology.orchestra.run.trial093.stdout:Running transaction check 2026-02-20T12:54:21.037 INFO:teuthology.orchestra.run.trial093.stdout:Transaction check succeeded. 2026-02-20T12:54:21.037 INFO:teuthology.orchestra.run.trial093.stdout:Running transaction test 2026-02-20T12:54:21.041 INFO:teuthology.orchestra.run.trial093.stdout:Transaction test succeeded. 2026-02-20T12:54:21.041 INFO:teuthology.orchestra.run.trial093.stdout:Running transaction 2026-02-20T12:54:21.065 INFO:teuthology.orchestra.run.trial093.stdout: Preparing : 1/1 2026-02-20T12:54:21.069 INFO:teuthology.orchestra.run.trial093.stdout: Reinstalling : kernel-5.14.0-681.el9.x86_64 1/2 2026-02-20T12:54:21.069 INFO:teuthology.orchestra.run.trial093.stdout: Cleanup : kernel-5.14.0-681.el9.x86_64 2/2 2026-02-20T12:54:21.178 INFO:teuthology.orchestra.run.trial093.stdout: Running scriptlet: kernel-5.14.0-681.el9.x86_64 2/2 2026-02-20T12:54:21.178 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : kernel-5.14.0-681.el9.x86_64 1/2 2026-02-20T12:54:21.232 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : kernel-5.14.0-681.el9.x86_64 2/2 2026-02-20T12:54:21.232 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T12:54:21.232 INFO:teuthology.orchestra.run.trial093.stdout:Reinstalled: 2026-02-20T12:54:21.232 INFO:teuthology.orchestra.run.trial093.stdout: kernel-5.14.0-681.el9.x86_64 2026-02-20T12:54:21.232 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T12:54:21.232 INFO:teuthology.orchestra.run.trial093.stdout:Complete! 2026-02-20T12:54:21.259 DEBUG:teuthology.orchestra.run.trial093:> rpm -q kernel | sort -rV | head -n 1 2026-02-20T12:54:21.284 INFO:teuthology.orchestra.run.trial093.stdout:kernel-5.14.0-681.el9.x86_64 2026-02-20T12:54:21.285 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-681.el9.x86_64 2026-02-20T12:54:21.285 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2026-02-20T12:54:21.285 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2026-02-20T12:54:21.285 DEBUG:teuthology.orchestra.run.trial093:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-20T12:54:21.349 INFO:teuthology.orchestra.run.trial093.stdout:ttyS1 2026-02-20T12:54:21.385 DEBUG:teuthology.parallel:result is None 2026-02-20T12:54:21.385 INFO:teuthology.run_tasks:Running task internal.base... 2026-02-20T12:54:21.397 INFO:teuthology.task.internal:Creating test directory... 2026-02-20T12:54:21.397 DEBUG:teuthology.orchestra.run.trial093:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-20T12:54:21.441 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2026-02-20T12:54:21.456 INFO:teuthology.run_tasks:Running task internal.archive... 2026-02-20T12:54:21.469 INFO:teuthology.task.internal:Creating archive directory... 2026-02-20T12:54:21.469 DEBUG:teuthology.orchestra.run.trial093:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-20T12:54:21.503 INFO:teuthology.run_tasks:Running task internal.coredump... 2026-02-20T12:54:21.515 INFO:teuthology.task.internal:Enabling coredump saving... 2026-02-20T12:54:21.516 DEBUG:teuthology.orchestra.run.trial093:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-20T12:54:21.562 INFO:teuthology.orchestra.run.trial093.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T12:54:21.571 INFO:teuthology.orchestra.run.trial093.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T12:54:21.573 INFO:teuthology.run_tasks:Running task internal.sudo... 2026-02-20T12:54:21.586 INFO:teuthology.task.internal:Configuring sudo... 2026-02-20T12:54:21.586 DEBUG:teuthology.orchestra.run.trial093:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-20T12:54:21.638 INFO:teuthology.run_tasks:Running task internal.syslog... 2026-02-20T12:54:21.656 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2026-02-20T12:54:21.656 DEBUG:teuthology.orchestra.run.trial093:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-20T12:54:21.694 DEBUG:teuthology.orchestra.run.trial093:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-20T12:54:21.855 DEBUG:teuthology.orchestra.run.trial093:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-20T12:54:21.879 DEBUG:teuthology.orchestra.run.trial093:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-20T12:54:22.028 DEBUG:teuthology.orchestra.run.trial093:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-20T12:54:22.051 DEBUG:teuthology.orchestra.run.trial093:> set -ex 2026-02-20T12:54:22.051 DEBUG:teuthology.orchestra.run.trial093:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-20T12:54:22.114 DEBUG:teuthology.orchestra.run.trial093:> sudo service rsyslog restart 2026-02-20T12:54:22.183 INFO:teuthology.orchestra.run.trial093.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-20T12:54:22.544 INFO:teuthology.run_tasks:Running task internal.timer... 2026-02-20T12:54:22.556 INFO:teuthology.task.internal:Starting timer... 2026-02-20T12:54:22.556 INFO:teuthology.run_tasks:Running task pcp... 2026-02-20T12:54:22.577 INFO:teuthology.run_tasks:Running task selinux... 2026-02-20T12:54:22.596 DEBUG:teuthology.orchestra.run.trial093:> sudo service auditd rotate 2026-02-20T12:54:22.626 INFO:teuthology.orchestra.run.trial093.stdout:Rotating logs: 2026-02-20T12:54:22.627 DEBUG:teuthology.task.selinux:Getting current SELinux state 2026-02-20T12:54:22.627 DEBUG:teuthology.orchestra.run.trial093:> /usr/sbin/getenforce 2026-02-20T12:54:22.683 INFO:teuthology.orchestra.run.trial093.stdout:Permissive 2026-02-20T12:54:22.683 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@trial093.front.sepia.ceph.com': 'permissive'} 2026-02-20T12:54:22.683 DEBUG:teuthology.orchestra.run.trial093:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-20T12:54:22.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-20T12:54:22.748 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2026-02-20T12:54:22.748 DEBUG:teuthology.orchestra.run.trial093:> sudo /usr/sbin/setenforce permissive 2026-02-20T12:54:22.812 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2026-02-20T12:54:22.827 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2026-02-20T12:54:22.829 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2026-02-20T12:54:23.071 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2026-02-20T12:54:23.082 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2026-02-20T12:54:23.084 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit trial093.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2026-02-20T12:57:27.840 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@trial093.front.sepia.ceph.com')] 2026-02-20T12:57:27.841 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial093.front.sepia.ceph.com' 2026-02-20T12:57:27.842 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial093.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-20T12:57:27.954 DEBUG:teuthology.orchestra.run.trial093:> true 2026-02-20T12:57:28.030 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial093.front.sepia.ceph.com' 2026-02-20T12:57:28.030 INFO:teuthology.run_tasks:Running task clock... 2026-02-20T12:57:28.044 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2026-02-20T12:57:28.044 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-20T12:57:28.044 DEBUG:teuthology.orchestra.run.trial093:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-20T12:57:28.098 INFO:teuthology.orchestra.run.trial093.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2026-02-20T12:57:28.110 INFO:teuthology.orchestra.run.trial093.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2026-02-20T12:57:28.130 INFO:teuthology.orchestra.run.trial093.stderr:sudo: ntpd: command not found 2026-02-20T12:57:28.139 INFO:teuthology.orchestra.run.trial093.stdout:506 Cannot talk to daemon 2026-02-20T12:57:28.150 INFO:teuthology.orchestra.run.trial093.stderr:Failed to start ntp.service: Unit ntp.service not found. 2026-02-20T12:57:28.162 INFO:teuthology.orchestra.run.trial093.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2026-02-20T12:57:28.209 INFO:teuthology.orchestra.run.trial093.stderr:bash: line 1: ntpq: command not found 2026-02-20T12:57:28.210 INFO:teuthology.orchestra.run.trial093.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-20T12:57:28.211 INFO:teuthology.orchestra.run.trial093.stdout:=============================================================================== 2026-02-20T12:57:28.212 INFO:teuthology.run_tasks:Running task install... 2026-02-20T12:57:28.226 DEBUG:teuthology.task.install:project ceph 2026-02-20T12:57:28.226 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'debug', 'sha1': '26bc97655ae0056b78c14ab91d1307332bd6dfca'}} 2026-02-20T12:57:28.226 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'flavor': 'debug', 'sha1': '26bc97655ae0056b78c14ab91d1307332bd6dfca'} 2026-02-20T12:57:28.226 INFO:teuthology.task.install:Using flavor: debug 2026-02-20T12:57:28.236 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-osd', 'ceph-osd-classic', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-osd', 'ceph-osd-classic', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-20T12:57:28.236 INFO:teuthology.task.install:extra packages: {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']} 2026-02-20T12:57:28.236 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'debug', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '26bc97655ae0056b78c14ab91d1307332bd6dfca', 'tag': None, 'wait_for_package': False} 2026-02-20T12:57:28.237 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=26bc97655ae0056b78c14ab91d1307332bd6dfca 2026-02-20T12:57:28.399 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-perezjos-crimson-only-12-02-2026-PR66245/26bc97655ae0056b78c14ab91d1307332bd6dfca/centos/9/flavors/debug/ 2026-02-20T12:57:28.399 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5312.g26bc9765 2026-02-20T12:57:28.770 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-perezjos-crimson-only-12-02-2026-PR66245/26bc97655ae0056b78c14ab91d1307332bd6dfca/centos/9/flavors/debug/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-perezjos-crimson-only-12-02-2026-PR66245/26bc97655ae0056b78c14ab91d1307332bd6dfca/centos/9/flavors/debug/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-perezjos-crimson-only-12-02-2026-PR66245/26bc97655ae0056b78c14ab91d1307332bd6dfca/centos/9/flavors/debug/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-20T12:57:28.770 DEBUG:teuthology.orchestra.run.trial093:> set -ex 2026-02-20T12:57:28.771 DEBUG:teuthology.orchestra.run.trial093:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-20T12:57:28.796 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-osd-crimson, ceph-osd-crimson-debuginfo on remote rpm x86_64 2026-02-20T12:57:28.797 DEBUG:teuthology.orchestra.run.trial093:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/26bc97655ae0056b78c14ab91d1307332bd6dfca/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-20T12:57:28.862 DEBUG:teuthology.orchestra.run.trial093:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-20T12:57:28.934 DEBUG:teuthology.orchestra.run.trial093:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-20T12:57:28.996 INFO:teuthology.orchestra.run.trial093.stdout:check_obsoletes = 1 2026-02-20T12:57:28.998 DEBUG:teuthology.orchestra.run.trial093:> sudo yum clean all 2026-02-20T12:57:29.154 INFO:teuthology.orchestra.run.trial093.stdout:41 files removed 2026-02-20T12:57:29.168 DEBUG:teuthology.orchestra.run.trial093:> sudo yum -y install ceph-radosgw 2026-02-20T12:57:29.978 INFO:teuthology.orchestra.run.trial093.stdout:ceph packages for x86_64 141 kB/s | 94 kB 00:00 2026-02-20T12:57:30.614 INFO:teuthology.orchestra.run.trial093.stdout:ceph noarch packages 40 kB/s | 24 kB 00:00 2026-02-20T12:57:31.189 INFO:teuthology.orchestra.run.trial093.stdout:ceph source packages 3.7 kB/s | 2.1 kB 00:00 2026-02-20T12:57:37.905 INFO:teuthology.orchestra.run.trial093.stdout:CentOS Stream 9 - BaseOS 1.3 MB/s | 8.9 MB 00:06 2026-02-20T12:57:41.181 INFO:teuthology.orchestra.run.trial093.stdout:CentOS Stream 9 - AppStream 9.8 MB/s | 27 MB 00:02 2026-02-20T12:57:45.034 INFO:teuthology.orchestra.run.trial093.stdout:CentOS Stream 9 - CRB 5.4 MB/s | 7.8 MB 00:01 2026-02-20T12:57:46.683 INFO:teuthology.orchestra.run.trial093.stdout:CentOS Stream 9 - Extras packages 22 kB/s | 20 kB 00:00 2026-02-20T12:57:46.995 INFO:teuthology.orchestra.run.trial093.stdout:Extra Packages for Enterprise Linux 81 MB/s | 20 MB 00:00 2026-02-20T12:57:50.221 INFO:teuthology.orchestra.run.trial093.stdout:lab-extras 2.7 MB/s | 50 kB 00:00 2026-02-20T12:57:51.419 INFO:teuthology.orchestra.run.trial093.stdout:Dependencies resolved. 2026-02-20T12:57:51.420 INFO:teuthology.orchestra.run.trial093.stdout:================================================================================ 2026-02-20T12:57:51.420 INFO:teuthology.orchestra.run.trial093.stdout: Package Arch Version Repository Size 2026-02-20T12:57:51.420 INFO:teuthology.orchestra.run.trial093.stdout:================================================================================ 2026-02-20T12:57:51.420 INFO:teuthology.orchestra.run.trial093.stdout:Installing: 2026-02-20T12:57:51.421 INFO:teuthology.orchestra.run.trial093.stdout: ceph-radosgw x86_64 2:20.3.0-5312.g26bc9765.el9 ceph 11 M 2026-02-20T12:57:51.421 INFO:teuthology.orchestra.run.trial093.stdout:Upgrading: 2026-02-20T12:57:51.421 INFO:teuthology.orchestra.run.trial093.stdout: librados2 x86_64 2:20.3.0-5312.g26bc9765.el9 ceph 3.3 M 2026-02-20T12:57:51.421 INFO:teuthology.orchestra.run.trial093.stdout: librbd1 x86_64 2:20.3.0-5312.g26bc9765.el9 ceph 2.7 M 2026-02-20T12:57:51.421 INFO:teuthology.orchestra.run.trial093.stdout:Installing dependencies: 2026-02-20T12:57:51.421 INFO:teuthology.orchestra.run.trial093.stdout: boost-program-options x86_64 1.75.0-13.el9 appstream 104 k 2026-02-20T12:57:51.421 INFO:teuthology.orchestra.run.trial093.stdout: ceph-base x86_64 2:20.3.0-5312.g26bc9765.el9 ceph 5.1 M 2026-02-20T12:57:51.421 INFO:teuthology.orchestra.run.trial093.stdout: ceph-common x86_64 2:20.3.0-5312.g26bc9765.el9 ceph 21 M 2026-02-20T12:57:51.421 INFO:teuthology.orchestra.run.trial093.stdout: ceph-selinux x86_64 2:20.3.0-5312.g26bc9765.el9 ceph 25 k 2026-02-20T12:57:51.422 INFO:teuthology.orchestra.run.trial093.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2026-02-20T12:57:51.422 INFO:teuthology.orchestra.run.trial093.stdout: libarrow x86_64 9.0.0-15.el9 epel 4.4 M 2026-02-20T12:57:51.422 INFO:teuthology.orchestra.run.trial093.stdout: libarrow-doc noarch 9.0.0-15.el9 epel 25 k 2026-02-20T12:57:51.422 INFO:teuthology.orchestra.run.trial093.stdout: libcephfs2 x86_64 2:20.3.0-5312.g26bc9765.el9 ceph 860 k 2026-02-20T12:57:51.422 INFO:teuthology.orchestra.run.trial093.stdout: libnbd x86_64 1.20.3-4.el9 appstream 164 k 2026-02-20T12:57:51.422 INFO:teuthology.orchestra.run.trial093.stdout: liboath x86_64 2.6.12-1.el9 epel 49 k 2026-02-20T12:57:51.422 INFO:teuthology.orchestra.run.trial093.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2026-02-20T12:57:51.422 INFO:teuthology.orchestra.run.trial093.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2026-02-20T12:57:51.422 INFO:teuthology.orchestra.run.trial093.stdout: libradosstriper1 x86_64 2:20.3.0-5312.g26bc9765.el9 ceph 263 k 2026-02-20T12:57:51.422 INFO:teuthology.orchestra.run.trial093.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2026-02-20T12:57:51.422 INFO:teuthology.orchestra.run.trial093.stdout: librgw2 x86_64 2:20.3.0-5312.g26bc9765.el9 ceph 5.6 M 2026-02-20T12:57:51.423 INFO:teuthology.orchestra.run.trial093.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2026-02-20T12:57:51.423 INFO:teuthology.orchestra.run.trial093.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2026-02-20T12:57:51.423 INFO:teuthology.orchestra.run.trial093.stdout: mailcap noarch 2.1.49-5.el9 baseos 33 k 2026-02-20T12:57:51.423 INFO:teuthology.orchestra.run.trial093.stdout: parquet-libs x86_64 9.0.0-15.el9 epel 838 k 2026-02-20T12:57:51.423 INFO:teuthology.orchestra.run.trial093.stdout: pciutils x86_64 3.7.0-7.el9 baseos 93 k 2026-02-20T12:57:51.423 INFO:teuthology.orchestra.run.trial093.stdout: python3-ceph-argparse x86_64 2:20.3.0-5312.g26bc9765.el9 ceph 45 k 2026-02-20T12:57:51.423 INFO:teuthology.orchestra.run.trial093.stdout: python3-ceph-common x86_64 2:20.3.0-5312.g26bc9765.el9 ceph 185 k 2026-02-20T12:57:51.423 INFO:teuthology.orchestra.run.trial093.stdout: python3-cephfs x86_64 2:20.3.0-5312.g26bc9765.el9 ceph 203 k 2026-02-20T12:57:51.423 INFO:teuthology.orchestra.run.trial093.stdout: python3-rados x86_64 2:20.3.0-5312.g26bc9765.el9 ceph 324 k 2026-02-20T12:57:51.423 INFO:teuthology.orchestra.run.trial093.stdout: python3-rbd x86_64 2:20.3.0-5312.g26bc9765.el9 ceph 305 k 2026-02-20T12:57:51.423 INFO:teuthology.orchestra.run.trial093.stdout: python3-rgw x86_64 2:20.3.0-5312.g26bc9765.el9 ceph 100 k 2026-02-20T12:57:51.424 INFO:teuthology.orchestra.run.trial093.stdout: qatlib x86_64 25.08.0-2.el9 appstream 240 k 2026-02-20T12:57:51.424 INFO:teuthology.orchestra.run.trial093.stdout: qatzip-libs x86_64 1.3.1-1.el9 appstream 66 k 2026-02-20T12:57:51.424 INFO:teuthology.orchestra.run.trial093.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2026-02-20T12:57:51.424 INFO:teuthology.orchestra.run.trial093.stdout: thrift x86_64 0.15.0-4.el9 epel 1.6 M 2026-02-20T12:57:51.424 INFO:teuthology.orchestra.run.trial093.stdout:Installing weak dependencies: 2026-02-20T12:57:51.424 INFO:teuthology.orchestra.run.trial093.stdout: libcephfs-daemon x86_64 2:20.3.0-5312.g26bc9765.el9 ceph 36 k 2026-02-20T12:57:51.424 INFO:teuthology.orchestra.run.trial093.stdout: libcephfs-proxy2 x86_64 2:20.3.0-5312.g26bc9765.el9 ceph 25 k 2026-02-20T12:57:51.424 INFO:teuthology.orchestra.run.trial093.stdout: nvme-cli x86_64 2.16-1.el9 baseos 1.2 M 2026-02-20T12:57:51.424 INFO:teuthology.orchestra.run.trial093.stdout: qatlib-service x86_64 25.08.0-2.el9 appstream 37 k 2026-02-20T12:57:51.425 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T12:57:51.425 INFO:teuthology.orchestra.run.trial093.stdout:Transaction Summary 2026-02-20T12:57:51.425 INFO:teuthology.orchestra.run.trial093.stdout:================================================================================ 2026-02-20T12:57:51.425 INFO:teuthology.orchestra.run.trial093.stdout:Install 35 Packages 2026-02-20T12:57:51.425 INFO:teuthology.orchestra.run.trial093.stdout:Upgrade 2 Packages 2026-02-20T12:57:51.425 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T12:57:51.426 INFO:teuthology.orchestra.run.trial093.stdout:Total download size: 61 M 2026-02-20T12:57:51.426 INFO:teuthology.orchestra.run.trial093.stdout:Downloading Packages: 2026-02-20T12:57:54.330 INFO:teuthology.orchestra.run.trial093.stdout:(1/37): ceph-base-20.3.0-5312.g26bc9765.el9.x86 1.9 MB/s | 5.1 MB 00:02 2026-02-20T12:57:54.396 INFO:teuthology.orchestra.run.trial093.stdout:(2/37): ceph-selinux-20.3.0-5312.g26bc9765.el9. 379 kB/s | 25 kB 00:00 2026-02-20T12:57:54.463 INFO:teuthology.orchestra.run.trial093.stdout:(3/37): libcephfs-daemon-20.3.0-5312.g26bc9765. 544 kB/s | 36 kB 00:00 2026-02-20T12:57:54.529 INFO:teuthology.orchestra.run.trial093.stdout:(4/37): libcephfs-proxy2-20.3.0-5312.g26bc9765. 376 kB/s | 25 kB 00:00 2026-02-20T12:57:54.863 INFO:teuthology.orchestra.run.trial093.stdout:(5/37): libcephfs2-20.3.0-5312.g26bc9765.el9.x8 2.5 MB/s | 860 kB 00:00 2026-02-20T12:57:55.000 INFO:teuthology.orchestra.run.trial093.stdout:(6/37): libradosstriper1-20.3.0-5312.g26bc9765. 1.9 MB/s | 263 kB 00:00 2026-02-20T12:57:56.057 INFO:teuthology.orchestra.run.trial093.stdout:(7/37): ceph-radosgw-20.3.0-5312.g26bc9765.el9. 2.6 MB/s | 11 MB 00:04 2026-02-20T12:57:56.116 INFO:teuthology.orchestra.run.trial093.stdout:(8/37): python3-ceph-argparse-20.3.0-5312.g26bc 764 kB/s | 45 kB 00:00 2026-02-20T12:57:56.179 INFO:teuthology.orchestra.run.trial093.stdout:(9/37): python3-ceph-common-20.3.0-5312.g26bc97 2.9 MB/s | 185 kB 00:00 2026-02-20T12:57:56.299 INFO:teuthology.orchestra.run.trial093.stdout:(10/37): python3-cephfs-20.3.0-5312.g26bc9765.e 1.7 MB/s | 203 kB 00:00 2026-02-20T12:57:56.423 INFO:teuthology.orchestra.run.trial093.stdout:(11/37): python3-rados-20.3.0-5312.g26bc9765.el 2.6 MB/s | 324 kB 00:00 2026-02-20T12:57:56.545 INFO:teuthology.orchestra.run.trial093.stdout:(12/37): python3-rbd-20.3.0-5312.g26bc9765.el9. 2.4 MB/s | 305 kB 00:00 2026-02-20T12:57:56.612 INFO:teuthology.orchestra.run.trial093.stdout:(13/37): python3-rgw-20.3.0-5312.g26bc9765.el9. 1.5 MB/s | 100 kB 00:00 2026-02-20T12:57:56.624 INFO:teuthology.orchestra.run.trial093.stdout:(14/37): mailcap-2.1.49-5.el9.noarch.rpm 2.8 MB/s | 33 kB 00:00 2026-02-20T12:57:56.686 INFO:teuthology.orchestra.run.trial093.stdout:(15/37): nvme-cli-2.16-1.el9.x86_64.rpm 19 MB/s | 1.2 MB 00:00 2026-02-20T12:57:56.694 INFO:teuthology.orchestra.run.trial093.stdout:(16/37): pciutils-3.7.0-7.el9.x86_64.rpm 11 MB/s | 93 kB 00:00 2026-02-20T12:57:57.000 INFO:teuthology.orchestra.run.trial093.stdout:(17/37): librgw2-20.3.0-5312.g26bc9765.el9.x86_ 2.8 MB/s | 5.6 MB 00:01 2026-02-20T12:57:57.260 INFO:teuthology.orchestra.run.trial093.stdout:(18/37): boost-program-options-1.75.0-13.el9.x8 184 kB/s | 104 kB 00:00 2026-02-20T12:57:57.500 INFO:teuthology.orchestra.run.trial093.stdout:(19/37): libpmemobj-1.12.1-1.el9.x86_64.rpm 670 kB/s | 160 kB 00:00 2026-02-20T12:57:57.561 INFO:teuthology.orchestra.run.trial093.stdout:(20/37): libnbd-1.20.3-4.el9.x86_64.rpm 292 kB/s | 164 kB 00:00 2026-02-20T12:57:57.582 INFO:teuthology.orchestra.run.trial093.stdout:(21/37): librabbitmq-0.11.0-7.el9.x86_64.rpm 547 kB/s | 45 kB 00:00 2026-02-20T12:57:57.977 INFO:teuthology.orchestra.run.trial093.stdout:(22/37): lttng-ust-2.12.0-6.el9.x86_64.rpm 740 kB/s | 292 kB 00:00 2026-02-20T12:57:58.294 INFO:teuthology.orchestra.run.trial093.stdout:(23/37): qatlib-25.08.0-2.el9.x86_64.rpm 756 kB/s | 240 kB 00:00 2026-02-20T12:57:58.350 INFO:teuthology.orchestra.run.trial093.stdout:(24/37): librdkafka-1.6.1-102.el9.x86_64.rpm 840 kB/s | 662 kB 00:00 2026-02-20T12:57:58.377 INFO:teuthology.orchestra.run.trial093.stdout:(25/37): qatlib-service-25.08.0-2.el9.x86_64.rp 446 kB/s | 37 kB 00:00 2026-02-20T12:57:58.433 INFO:teuthology.orchestra.run.trial093.stdout:(26/37): qatzip-libs-1.3.1-1.el9.x86_64.rpm 799 kB/s | 66 kB 00:00 2026-02-20T12:57:59.640 INFO:teuthology.orchestra.run.trial093.stdout:(27/37): ceph-common-20.3.0-5312.g26bc9765.el9. 2.6 MB/s | 21 MB 00:07 2026-02-20T12:57:59.853 INFO:teuthology.orchestra.run.trial093.stdout:(28/37): libarrow-doc-9.0.0-15.el9.noarch.rpm 116 kB/s | 25 kB 00:00 2026-02-20T12:57:59.909 INFO:teuthology.orchestra.run.trial093.stdout:(29/37): gperftools-libs-2.9.1-3.el9.x86_64.rpm 201 kB/s | 308 kB 00:01 2026-02-20T12:57:59.917 INFO:teuthology.orchestra.run.trial093.stdout:(30/37): liboath-2.6.12-1.el9.x86_64.rpm 763 kB/s | 49 kB 00:00 2026-02-20T12:57:59.934 INFO:teuthology.orchestra.run.trial093.stdout:(31/37): libunwind-1.6.2-1.el9.x86_64.rpm 2.6 MB/s | 67 kB 00:00 2026-02-20T12:57:59.966 INFO:teuthology.orchestra.run.trial093.stdout:(32/37): re2-20211101-20.el9.x86_64.rpm 5.9 MB/s | 191 kB 00:00 2026-02-20T12:58:00.049 INFO:teuthology.orchestra.run.trial093.stdout:(33/37): thrift-0.15.0-4.el9.x86_64.rpm 19 MB/s | 1.6 MB 00:00 2026-02-20T12:58:00.057 INFO:teuthology.orchestra.run.trial093.stdout:(34/37): libarrow-9.0.0-15.el9.x86_64.rpm 2.7 MB/s | 4.4 MB 00:01 2026-02-20T12:58:00.059 INFO:teuthology.orchestra.run.trial093.stdout:(35/37): parquet-libs-9.0.0-15.el9.x86_64.rpm 5.8 MB/s | 838 kB 00:00 2026-02-20T12:58:01.198 INFO:teuthology.orchestra.run.trial093.stdout:(36/37): librbd1-20.3.0-5312.g26bc9765.el9.x86_ 2.4 MB/s | 2.7 MB 00:01 2026-02-20T12:58:02.190 INFO:teuthology.orchestra.run.trial093.stdout:(37/37): librados2-20.3.0-5312.g26bc9765.el9.x8 1.5 MB/s | 3.3 MB 00:02 2026-02-20T12:58:02.192 INFO:teuthology.orchestra.run.trial093.stdout:-------------------------------------------------------------------------------- 2026-02-20T12:58:02.192 INFO:teuthology.orchestra.run.trial093.stdout:Total 5.7 MB/s | 61 MB 00:10 2026-02-20T12:58:02.293 INFO:teuthology.orchestra.run.trial093.stdout:Running transaction check 2026-02-20T12:58:02.312 INFO:teuthology.orchestra.run.trial093.stdout:Transaction check succeeded. 2026-02-20T12:58:02.312 INFO:teuthology.orchestra.run.trial093.stdout:Running transaction test 2026-02-20T12:58:02.565 INFO:teuthology.orchestra.run.trial093.stdout:Transaction test succeeded. 2026-02-20T12:58:02.565 INFO:teuthology.orchestra.run.trial093.stdout:Running transaction 2026-02-20T12:58:02.854 INFO:teuthology.orchestra.run.trial093.stdout: Preparing : 1/1 2026-02-20T12:58:02.993 INFO:teuthology.orchestra.run.trial093.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/39 2026-02-20T12:58:02.998 INFO:teuthology.orchestra.run.trial093.stdout: Upgrading : librados2-2:20.3.0-5312.g26bc9765.el9.x86_64 2/39 2026-02-20T12:58:03.046 INFO:teuthology.orchestra.run.trial093.stdout: Running scriptlet: librados2-2:20.3.0-5312.g26bc9765.el9.x86_64 2/39 2026-02-20T12:58:03.047 INFO:teuthology.orchestra.run.trial093.stdout: Installing : libcephfs2-2:20.3.0-5312.g26bc9765.el9.x86_64 3/39 2026-02-20T12:58:03.060 INFO:teuthology.orchestra.run.trial093.stdout: Running scriptlet: libcephfs2-2:20.3.0-5312.g26bc9765.el9.x86_64 3/39 2026-02-20T12:58:03.062 INFO:teuthology.orchestra.run.trial093.stdout: Installing : libcephfs-daemon-2:20.3.0-5312.g26bc9765.el9.x86_6 4/39 2026-02-20T12:58:03.063 INFO:teuthology.orchestra.run.trial093.stdout: Installing : libcephfs-proxy2-2:20.3.0-5312.g26bc9765.el9.x86_6 5/39 2026-02-20T12:58:03.091 INFO:teuthology.orchestra.run.trial093.stdout: Running scriptlet: libcephfs-proxy2-2:20.3.0-5312.g26bc9765.el9.x86_6 5/39 2026-02-20T12:58:03.096 INFO:teuthology.orchestra.run.trial093.stdout: Installing : python3-rados-2:20.3.0-5312.g26bc9765.el9.x86_64 6/39 2026-02-20T12:58:03.104 INFO:teuthology.orchestra.run.trial093.stdout: Installing : liboath-2.6.12-1.el9.x86_64 7/39 2026-02-20T12:58:03.106 INFO:teuthology.orchestra.run.trial093.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/39 2026-02-20T12:58:03.109 INFO:teuthology.orchestra.run.trial093.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/39 2026-02-20T12:58:03.114 INFO:teuthology.orchestra.run.trial093.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/39 2026-02-20T12:58:03.125 INFO:teuthology.orchestra.run.trial093.stdout: Installing : python3-ceph-argparse-2:20.3.0-5312.g26bc9765.el9. 11/39 2026-02-20T12:58:03.138 INFO:teuthology.orchestra.run.trial093.stdout: Installing : python3-cephfs-2:20.3.0-5312.g26bc9765.el9.x86_64 12/39 2026-02-20T12:58:03.139 INFO:teuthology.orchestra.run.trial093.stdout: Installing : libradosstriper1-2:20.3.0-5312.g26bc9765.el9.x86_6 13/39 2026-02-20T12:58:03.161 INFO:teuthology.orchestra.run.trial093.stdout: Running scriptlet: libradosstriper1-2:20.3.0-5312.g26bc9765.el9.x86_6 13/39 2026-02-20T12:58:03.164 INFO:teuthology.orchestra.run.trial093.stdout: Installing : thrift-0.15.0-4.el9.x86_64 14/39 2026-02-20T12:58:03.194 INFO:teuthology.orchestra.run.trial093.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/39 2026-02-20T12:58:03.198 INFO:teuthology.orchestra.run.trial093.stdout: Installing : libarrow-9.0.0-15.el9.x86_64 16/39 2026-02-20T12:58:03.204 INFO:teuthology.orchestra.run.trial093.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/39 2026-02-20T12:58:03.207 INFO:teuthology.orchestra.run.trial093.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/39 2026-02-20T12:58:03.212 INFO:teuthology.orchestra.run.trial093.stdout: Installing : libarrow-doc-9.0.0-15.el9.noarch 19/39 2026-02-20T12:58:03.326 INFO:teuthology.orchestra.run.trial093.stdout: Installing : libnbd-1.20.3-4.el9.x86_64 20/39 2026-02-20T12:58:03.327 INFO:teuthology.orchestra.run.trial093.stdout: Upgrading : librbd1-2:20.3.0-5312.g26bc9765.el9.x86_64 21/39 2026-02-20T12:58:03.355 INFO:teuthology.orchestra.run.trial093.stdout: Running scriptlet: librbd1-2:20.3.0-5312.g26bc9765.el9.x86_64 21/39 2026-02-20T12:58:03.360 INFO:teuthology.orchestra.run.trial093.stdout: Installing : python3-rbd-2:20.3.0-5312.g26bc9765.el9.x86_64 22/39 2026-02-20T12:58:03.367 INFO:teuthology.orchestra.run.trial093.stdout: Installing : boost-program-options-1.75.0-13.el9.x86_64 23/39 2026-02-20T12:58:03.614 INFO:teuthology.orchestra.run.trial093.stdout: Installing : parquet-libs-9.0.0-15.el9.x86_64 24/39 2026-02-20T12:58:03.615 INFO:teuthology.orchestra.run.trial093.stdout: Installing : librgw2-2:20.3.0-5312.g26bc9765.el9.x86_64 25/39 2026-02-20T12:58:03.632 INFO:teuthology.orchestra.run.trial093.stdout: Running scriptlet: librgw2-2:20.3.0-5312.g26bc9765.el9.x86_64 25/39 2026-02-20T12:58:03.637 INFO:teuthology.orchestra.run.trial093.stdout: Installing : python3-rgw-2:20.3.0-5312.g26bc9765.el9.x86_64 26/39 2026-02-20T12:58:03.642 INFO:teuthology.orchestra.run.trial093.stdout: Installing : pciutils-3.7.0-7.el9.x86_64 27/39 2026-02-20T12:58:03.645 INFO:teuthology.orchestra.run.trial093.stdout: Installing : qatlib-25.08.0-2.el9.x86_64 28/39 2026-02-20T12:58:03.646 INFO:teuthology.orchestra.run.trial093.stdout: Installing : qatlib-service-25.08.0-2.el9.x86_64 29/39 2026-02-20T12:58:03.665 INFO:teuthology.orchestra.run.trial093.stdout: Running scriptlet: qatlib-service-25.08.0-2.el9.x86_64 29/39 2026-02-20T12:58:03.780 INFO:teuthology.orchestra.run.trial093.stdout: Installing : qatzip-libs-1.3.1-1.el9.x86_64 30/39 2026-02-20T12:58:03.783 INFO:teuthology.orchestra.run.trial093.stdout: Installing : nvme-cli-2.16-1.el9.x86_64 31/39 2026-02-20T12:58:04.067 INFO:teuthology.orchestra.run.trial093.stdout: Running scriptlet: nvme-cli-2.16-1.el9.x86_64 31/39 2026-02-20T12:58:04.068 INFO:teuthology.orchestra.run.trial093.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service → /usr/lib/systemd/system/nvmefc-boot-connections.service. 2026-02-20T12:58:04.068 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T12:58:04.110 INFO:teuthology.orchestra.run.trial093.stdout: Installing : mailcap-2.1.49-5.el9.noarch 32/39 2026-02-20T12:58:04.112 INFO:teuthology.orchestra.run.trial093.stdout: Installing : python3-ceph-common-2:20.3.0-5312.g26bc9765.el9.x8 33/39 2026-02-20T12:58:05.050 INFO:teuthology.orchestra.run.trial093.stdout: Running scriptlet: ceph-common-2:20.3.0-5312.g26bc9765.el9.x86_64 34/39 2026-02-20T12:58:05.053 INFO:teuthology.orchestra.run.trial093.stdout: Installing : ceph-common-2:20.3.0-5312.g26bc9765.el9.x86_64 34/39 2026-02-20T12:58:05.312 INFO:teuthology.orchestra.run.trial093.stdout: Running scriptlet: ceph-common-2:20.3.0-5312.g26bc9765.el9.x86_64 34/39 2026-02-20T12:58:05.316 INFO:teuthology.orchestra.run.trial093.stdout: Installing : ceph-base-2:20.3.0-5312.g26bc9765.el9.x86_64 35/39 2026-02-20T12:58:05.356 INFO:teuthology.orchestra.run.trial093.stdout: Running scriptlet: ceph-base-2:20.3.0-5312.g26bc9765.el9.x86_64 35/39 2026-02-20T12:58:05.356 INFO:teuthology.orchestra.run.trial093.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2026-02-20T12:58:05.356 INFO:teuthology.orchestra.run.trial093.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2026-02-20T12:58:05.356 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T12:58:05.359 INFO:teuthology.orchestra.run.trial093.stdout: Installing : ceph-selinux-2:20.3.0-5312.g26bc9765.el9.x86_64 36/39 2026-02-20T12:58:09.908 INFO:teuthology.orchestra.run.trial093.stdout: Running scriptlet: ceph-selinux-2:20.3.0-5312.g26bc9765.el9.x86_64 36/39 2026-02-20T12:58:09.908 INFO:teuthology.orchestra.run.trial093.stdout:skipping the directory /sys 2026-02-20T12:58:09.908 INFO:teuthology.orchestra.run.trial093.stdout:skipping the directory /proc 2026-02-20T12:58:09.908 INFO:teuthology.orchestra.run.trial093.stdout:skipping the directory /mnt 2026-02-20T12:58:09.908 INFO:teuthology.orchestra.run.trial093.stdout:skipping the directory /var/tmp 2026-02-20T12:58:09.908 INFO:teuthology.orchestra.run.trial093.stdout:skipping the directory /home 2026-02-20T12:58:09.909 INFO:teuthology.orchestra.run.trial093.stdout:skipping the directory /root 2026-02-20T12:58:09.909 INFO:teuthology.orchestra.run.trial093.stdout:skipping the directory /tmp 2026-02-20T12:58:09.909 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T12:58:10.401 INFO:teuthology.orchestra.run.trial093.stdout: Installing : ceph-radosgw-2:20.3.0-5312.g26bc9765.el9.x86_64 37/39 2026-02-20T12:58:10.424 INFO:teuthology.orchestra.run.trial093.stdout: Running scriptlet: ceph-radosgw-2:20.3.0-5312.g26bc9765.el9.x86_64 37/39 2026-02-20T12:58:10.424 INFO:teuthology.orchestra.run.trial093.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-20T12:58:10.425 INFO:teuthology.orchestra.run.trial093.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2026-02-20T12:58:10.425 INFO:teuthology.orchestra.run.trial093.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-20T12:58:10.425 INFO:teuthology.orchestra.run.trial093.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-20T12:58:10.425 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T12:58:10.426 INFO:teuthology.orchestra.run.trial093.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 38/39 2026-02-20T12:58:10.439 INFO:teuthology.orchestra.run.trial093.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 38/39 2026-02-20T12:58:10.439 INFO:teuthology.orchestra.run.trial093.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 39/39 2026-02-20T12:58:10.494 INFO:teuthology.orchestra.run.trial093.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 39/39 2026-02-20T12:58:10.495 INFO:teuthology.orchestra.run.trial093.stdout:Creating group 'qat' with GID 992. 2026-02-20T12:58:10.495 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T12:58:10.791 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : ceph-base-2:20.3.0-5312.g26bc9765.el9.x86_64 1/39 2026-02-20T12:58:10.791 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : ceph-common-2:20.3.0-5312.g26bc9765.el9.x86_64 2/39 2026-02-20T12:58:10.791 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : ceph-radosgw-2:20.3.0-5312.g26bc9765.el9.x86_64 3/39 2026-02-20T12:58:10.792 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : ceph-selinux-2:20.3.0-5312.g26bc9765.el9.x86_64 4/39 2026-02-20T12:58:10.792 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : libcephfs-daemon-2:20.3.0-5312.g26bc9765.el9.x86_6 5/39 2026-02-20T12:58:10.792 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : libcephfs-proxy2-2:20.3.0-5312.g26bc9765.el9.x86_6 6/39 2026-02-20T12:58:10.792 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : libcephfs2-2:20.3.0-5312.g26bc9765.el9.x86_64 7/39 2026-02-20T12:58:10.792 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : libradosstriper1-2:20.3.0-5312.g26bc9765.el9.x86_6 8/39 2026-02-20T12:58:10.792 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : librgw2-2:20.3.0-5312.g26bc9765.el9.x86_64 9/39 2026-02-20T12:58:10.792 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : python3-ceph-argparse-2:20.3.0-5312.g26bc9765.el9. 10/39 2026-02-20T12:58:10.792 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : python3-ceph-common-2:20.3.0-5312.g26bc9765.el9.x8 11/39 2026-02-20T12:58:10.792 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : python3-cephfs-2:20.3.0-5312.g26bc9765.el9.x86_64 12/39 2026-02-20T12:58:10.792 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : python3-rados-2:20.3.0-5312.g26bc9765.el9.x86_64 13/39 2026-02-20T12:58:10.792 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : python3-rbd-2:20.3.0-5312.g26bc9765.el9.x86_64 14/39 2026-02-20T12:58:10.792 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : python3-rgw-2:20.3.0-5312.g26bc9765.el9.x86_64 15/39 2026-02-20T12:58:10.793 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : mailcap-2.1.49-5.el9.noarch 16/39 2026-02-20T12:58:10.793 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : nvme-cli-2.16-1.el9.x86_64 17/39 2026-02-20T12:58:10.793 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : pciutils-3.7.0-7.el9.x86_64 18/39 2026-02-20T12:58:10.793 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : boost-program-options-1.75.0-13.el9.x86_64 19/39 2026-02-20T12:58:10.793 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : libnbd-1.20.3-4.el9.x86_64 20/39 2026-02-20T12:58:10.793 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 21/39 2026-02-20T12:58:10.793 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 22/39 2026-02-20T12:58:10.793 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 23/39 2026-02-20T12:58:10.793 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 24/39 2026-02-20T12:58:10.793 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : qatlib-25.08.0-2.el9.x86_64 25/39 2026-02-20T12:58:10.793 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : qatlib-service-25.08.0-2.el9.x86_64 26/39 2026-02-20T12:58:10.793 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : qatzip-libs-1.3.1-1.el9.x86_64 27/39 2026-02-20T12:58:10.793 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 28/39 2026-02-20T12:58:10.794 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : libarrow-9.0.0-15.el9.x86_64 29/39 2026-02-20T12:58:10.794 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : libarrow-doc-9.0.0-15.el9.noarch 30/39 2026-02-20T12:58:10.794 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : liboath-2.6.12-1.el9.x86_64 31/39 2026-02-20T12:58:10.794 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 32/39 2026-02-20T12:58:10.794 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : parquet-libs-9.0.0-15.el9.x86_64 33/39 2026-02-20T12:58:10.794 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : re2-1:20211101-20.el9.x86_64 34/39 2026-02-20T12:58:10.794 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : thrift-0.15.0-4.el9.x86_64 35/39 2026-02-20T12:58:10.794 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : librados2-2:20.3.0-5312.g26bc9765.el9.x86_64 36/39 2026-02-20T12:58:10.794 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 37/39 2026-02-20T12:58:10.794 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : librbd1-2:20.3.0-5312.g26bc9765.el9.x86_64 38/39 2026-02-20T12:58:10.854 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 39/39 2026-02-20T12:58:10.854 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T12:58:10.855 INFO:teuthology.orchestra.run.trial093.stdout:Upgraded: 2026-02-20T12:58:10.855 INFO:teuthology.orchestra.run.trial093.stdout: librados2-2:20.3.0-5312.g26bc9765.el9.x86_64 2026-02-20T12:58:10.855 INFO:teuthology.orchestra.run.trial093.stdout: librbd1-2:20.3.0-5312.g26bc9765.el9.x86_64 2026-02-20T12:58:10.855 INFO:teuthology.orchestra.run.trial093.stdout:Installed: 2026-02-20T12:58:10.855 INFO:teuthology.orchestra.run.trial093.stdout: boost-program-options-1.75.0-13.el9.x86_64 2026-02-20T12:58:10.855 INFO:teuthology.orchestra.run.trial093.stdout: ceph-base-2:20.3.0-5312.g26bc9765.el9.x86_64 2026-02-20T12:58:10.855 INFO:teuthology.orchestra.run.trial093.stdout: ceph-common-2:20.3.0-5312.g26bc9765.el9.x86_64 2026-02-20T12:58:10.855 INFO:teuthology.orchestra.run.trial093.stdout: ceph-radosgw-2:20.3.0-5312.g26bc9765.el9.x86_64 2026-02-20T12:58:10.855 INFO:teuthology.orchestra.run.trial093.stdout: ceph-selinux-2:20.3.0-5312.g26bc9765.el9.x86_64 2026-02-20T12:58:10.855 INFO:teuthology.orchestra.run.trial093.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2026-02-20T12:58:10.856 INFO:teuthology.orchestra.run.trial093.stdout: libarrow-9.0.0-15.el9.x86_64 2026-02-20T12:58:10.856 INFO:teuthology.orchestra.run.trial093.stdout: libarrow-doc-9.0.0-15.el9.noarch 2026-02-20T12:58:10.856 INFO:teuthology.orchestra.run.trial093.stdout: libcephfs-daemon-2:20.3.0-5312.g26bc9765.el9.x86_64 2026-02-20T12:58:10.856 INFO:teuthology.orchestra.run.trial093.stdout: libcephfs-proxy2-2:20.3.0-5312.g26bc9765.el9.x86_64 2026-02-20T12:58:10.856 INFO:teuthology.orchestra.run.trial093.stdout: libcephfs2-2:20.3.0-5312.g26bc9765.el9.x86_64 2026-02-20T12:58:10.856 INFO:teuthology.orchestra.run.trial093.stdout: libnbd-1.20.3-4.el9.x86_64 2026-02-20T12:58:10.856 INFO:teuthology.orchestra.run.trial093.stdout: liboath-2.6.12-1.el9.x86_64 2026-02-20T12:58:10.856 INFO:teuthology.orchestra.run.trial093.stdout: libpmemobj-1.12.1-1.el9.x86_64 2026-02-20T12:58:10.856 INFO:teuthology.orchestra.run.trial093.stdout: librabbitmq-0.11.0-7.el9.x86_64 2026-02-20T12:58:10.857 INFO:teuthology.orchestra.run.trial093.stdout: libradosstriper1-2:20.3.0-5312.g26bc9765.el9.x86_64 2026-02-20T12:58:10.857 INFO:teuthology.orchestra.run.trial093.stdout: librdkafka-1.6.1-102.el9.x86_64 2026-02-20T12:58:10.857 INFO:teuthology.orchestra.run.trial093.stdout: librgw2-2:20.3.0-5312.g26bc9765.el9.x86_64 2026-02-20T12:58:10.857 INFO:teuthology.orchestra.run.trial093.stdout: libunwind-1.6.2-1.el9.x86_64 2026-02-20T12:58:10.857 INFO:teuthology.orchestra.run.trial093.stdout: lttng-ust-2.12.0-6.el9.x86_64 2026-02-20T12:58:10.857 INFO:teuthology.orchestra.run.trial093.stdout: mailcap-2.1.49-5.el9.noarch 2026-02-20T12:58:10.857 INFO:teuthology.orchestra.run.trial093.stdout: nvme-cli-2.16-1.el9.x86_64 2026-02-20T12:58:10.857 INFO:teuthology.orchestra.run.trial093.stdout: parquet-libs-9.0.0-15.el9.x86_64 2026-02-20T12:58:10.858 INFO:teuthology.orchestra.run.trial093.stdout: pciutils-3.7.0-7.el9.x86_64 2026-02-20T12:58:10.858 INFO:teuthology.orchestra.run.trial093.stdout: python3-ceph-argparse-2:20.3.0-5312.g26bc9765.el9.x86_64 2026-02-20T12:58:10.858 INFO:teuthology.orchestra.run.trial093.stdout: python3-ceph-common-2:20.3.0-5312.g26bc9765.el9.x86_64 2026-02-20T12:58:10.858 INFO:teuthology.orchestra.run.trial093.stdout: python3-cephfs-2:20.3.0-5312.g26bc9765.el9.x86_64 2026-02-20T12:58:10.858 INFO:teuthology.orchestra.run.trial093.stdout: python3-rados-2:20.3.0-5312.g26bc9765.el9.x86_64 2026-02-20T12:58:10.858 INFO:teuthology.orchestra.run.trial093.stdout: python3-rbd-2:20.3.0-5312.g26bc9765.el9.x86_64 2026-02-20T12:58:10.858 INFO:teuthology.orchestra.run.trial093.stdout: python3-rgw-2:20.3.0-5312.g26bc9765.el9.x86_64 2026-02-20T12:58:10.858 INFO:teuthology.orchestra.run.trial093.stdout: qatlib-25.08.0-2.el9.x86_64 2026-02-20T12:58:10.858 INFO:teuthology.orchestra.run.trial093.stdout: qatlib-service-25.08.0-2.el9.x86_64 2026-02-20T12:58:10.858 INFO:teuthology.orchestra.run.trial093.stdout: qatzip-libs-1.3.1-1.el9.x86_64 2026-02-20T12:58:10.858 INFO:teuthology.orchestra.run.trial093.stdout: re2-1:20211101-20.el9.x86_64 2026-02-20T12:58:10.858 INFO:teuthology.orchestra.run.trial093.stdout: thrift-0.15.0-4.el9.x86_64 2026-02-20T12:58:10.858 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T12:58:10.859 INFO:teuthology.orchestra.run.trial093.stdout:Complete! 2026-02-20T12:58:10.897 DEBUG:teuthology.orchestra.run.trial093:> sudo yum -y install ceph-test 2026-02-20T12:58:11.208 INFO:teuthology.orchestra.run.trial093.stdout:Last metadata expiration check: 0:00:21 ago on Fri 20 Feb 2026 12:57:50 PM UTC. 2026-02-20T12:58:11.291 INFO:teuthology.orchestra.run.trial093.stdout:Dependencies resolved. 2026-02-20T12:58:11.292 INFO:teuthology.orchestra.run.trial093.stdout:================================================================================ 2026-02-20T12:58:11.292 INFO:teuthology.orchestra.run.trial093.stdout: Package Arch Version Repository Size 2026-02-20T12:58:11.292 INFO:teuthology.orchestra.run.trial093.stdout:================================================================================ 2026-02-20T12:58:11.293 INFO:teuthology.orchestra.run.trial093.stdout:Installing: 2026-02-20T12:58:11.293 INFO:teuthology.orchestra.run.trial093.stdout: ceph-test x86_64 2:20.3.0-5312.g26bc9765.el9 ceph 125 M 2026-02-20T12:58:11.293 INFO:teuthology.orchestra.run.trial093.stdout:Installing dependencies: 2026-02-20T12:58:11.293 INFO:teuthology.orchestra.run.trial093.stdout: hwloc-libs x86_64 2.4.1-6.el9 baseos 2.1 M 2026-02-20T12:58:11.293 INFO:teuthology.orchestra.run.trial093.stdout: libasan8 x86_64 15.2.1-7.el9 appstream 525 k 2026-02-20T12:58:11.293 INFO:teuthology.orchestra.run.trial093.stdout: libatomic x86_64 11.5.0-14.el9 baseos 23 k 2026-02-20T12:58:11.293 INFO:teuthology.orchestra.run.trial093.stdout: libcephsqlite x86_64 2:20.3.0-5312.g26bc9765.el9 ceph 153 k 2026-02-20T12:58:11.293 INFO:teuthology.orchestra.run.trial093.stdout: libubsan x86_64 11.5.0-14.el9 appstream 176 k 2026-02-20T12:58:11.293 INFO:teuthology.orchestra.run.trial093.stdout: libxslt x86_64 1.1.34-12.el9 appstream 233 k 2026-02-20T12:58:11.293 INFO:teuthology.orchestra.run.trial093.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2026-02-20T12:58:11.293 INFO:teuthology.orchestra.run.trial093.stdout: protobuf x86_64 3.14.0-17.el9 appstream 1.0 M 2026-02-20T12:58:11.294 INFO:teuthology.orchestra.run.trial093.stdout: socat x86_64 1.7.4.1-8.el9 appstream 303 k 2026-02-20T12:58:11.294 INFO:teuthology.orchestra.run.trial093.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2026-02-20T12:58:11.294 INFO:teuthology.orchestra.run.trial093.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2026-02-20T12:58:11.294 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T12:58:11.294 INFO:teuthology.orchestra.run.trial093.stdout:Transaction Summary 2026-02-20T12:58:11.294 INFO:teuthology.orchestra.run.trial093.stdout:================================================================================ 2026-02-20T12:58:11.294 INFO:teuthology.orchestra.run.trial093.stdout:Install 12 Packages 2026-02-20T12:58:11.294 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T12:58:11.294 INFO:teuthology.orchestra.run.trial093.stdout:Total download size: 129 M 2026-02-20T12:58:11.294 INFO:teuthology.orchestra.run.trial093.stdout:Installed size: 963 M 2026-02-20T12:58:11.294 INFO:teuthology.orchestra.run.trial093.stdout:Downloading Packages: 2026-02-20T12:58:12.271 INFO:teuthology.orchestra.run.trial093.stdout:(1/12): libcephsqlite-20.3.0-5312.g26bc9765.el9 318 kB/s | 153 kB 00:00 2026-02-20T12:58:12.751 INFO:teuthology.orchestra.run.trial093.stdout:(2/12): libatomic-11.5.0-14.el9.x86_64.rpm 48 kB/s | 23 kB 00:00 2026-02-20T12:58:12.987 INFO:teuthology.orchestra.run.trial093.stdout:(3/12): lksctp-tools-1.0.19-2.el9.x86_64.rpm 397 kB/s | 94 kB 00:00 2026-02-20T12:58:13.124 INFO:teuthology.orchestra.run.trial093.stdout:(4/12): libasan8-15.2.1-7.el9.x86_64.rpm 3.7 MB/s | 525 kB 00:00 2026-02-20T12:58:13.147 INFO:teuthology.orchestra.run.trial093.stdout:(5/12): libubsan-11.5.0-14.el9.x86_64.rpm 7.7 MB/s | 176 kB 00:00 2026-02-20T12:58:13.170 INFO:teuthology.orchestra.run.trial093.stdout:(6/12): libxslt-1.1.34-12.el9.x86_64.rpm 10 MB/s | 233 kB 00:00 2026-02-20T12:58:13.230 INFO:teuthology.orchestra.run.trial093.stdout:(7/12): protobuf-3.14.0-17.el9.x86_64.rpm 17 MB/s | 1.0 MB 00:00 2026-02-20T12:58:13.252 INFO:teuthology.orchestra.run.trial093.stdout:(8/12): socat-1.7.4.1-8.el9.x86_64.rpm 13 MB/s | 303 kB 00:00 2026-02-20T12:58:13.273 INFO:teuthology.orchestra.run.trial093.stdout:(9/12): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.9 MB/s | 64 kB 00:00 2026-02-20T12:58:13.644 INFO:teuthology.orchestra.run.trial093.stdout:(10/12): yaml-cpp-0.6.3-5.el9.x86_64.rpm 329 kB/s | 122 kB 00:00 2026-02-20T12:58:14.756 INFO:teuthology.orchestra.run.trial093.stdout:(11/12): hwloc-libs-2.4.1-6.el9.x86_64.rpm 724 kB/s | 2.1 MB 00:02 2026-02-20T12:59:22.588 INFO:teuthology.orchestra.run.trial093.stdout:(12/12): ceph-test-20.3.0-5312.g26bc9765.el9.x8 1.8 MB/s | 125 MB 01:10 2026-02-20T12:59:22.591 INFO:teuthology.orchestra.run.trial093.stdout:-------------------------------------------------------------------------------- 2026-02-20T12:59:22.591 INFO:teuthology.orchestra.run.trial093.stdout:Total 1.8 MB/s | 129 MB 01:11 2026-02-20T12:59:22.661 INFO:teuthology.orchestra.run.trial093.stdout:Running transaction check 2026-02-20T12:59:22.672 INFO:teuthology.orchestra.run.trial093.stdout:Transaction check succeeded. 2026-02-20T12:59:22.672 INFO:teuthology.orchestra.run.trial093.stdout:Running transaction test 2026-02-20T12:59:23.095 INFO:teuthology.orchestra.run.trial093.stdout:Transaction test succeeded. 2026-02-20T12:59:23.095 INFO:teuthology.orchestra.run.trial093.stdout:Running transaction 2026-02-20T12:59:23.544 INFO:teuthology.orchestra.run.trial093.stdout: Preparing : 1/1 2026-02-20T12:59:23.553 INFO:teuthology.orchestra.run.trial093.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/12 2026-02-20T12:59:23.563 INFO:teuthology.orchestra.run.trial093.stdout: Installing : socat-1.7.4.1-8.el9.x86_64 2/12 2026-02-20T12:59:23.568 INFO:teuthology.orchestra.run.trial093.stdout: Installing : protobuf-3.14.0-17.el9.x86_64 3/12 2026-02-20T12:59:23.572 INFO:teuthology.orchestra.run.trial093.stdout: Installing : libxslt-1.1.34-12.el9.x86_64 4/12 2026-02-20T12:59:23.574 INFO:teuthology.orchestra.run.trial093.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 5/12 2026-02-20T12:59:23.579 INFO:teuthology.orchestra.run.trial093.stdout: Installing : libubsan-11.5.0-14.el9.x86_64 6/12 2026-02-20T12:59:23.579 INFO:teuthology.orchestra.run.trial093.stdout: Installing : libasan8-15.2.1-7.el9.x86_64 7/12 2026-02-20T12:59:23.597 INFO:teuthology.orchestra.run.trial093.stdout: Running scriptlet: libasan8-15.2.1-7.el9.x86_64 7/12 2026-02-20T12:59:23.600 INFO:teuthology.orchestra.run.trial093.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 8/12 2026-02-20T12:59:23.610 INFO:teuthology.orchestra.run.trial093.stdout: Installing : libatomic-11.5.0-14.el9.x86_64 9/12 2026-02-20T12:59:23.618 INFO:teuthology.orchestra.run.trial093.stdout: Installing : hwloc-libs-2.4.1-6.el9.x86_64 10/12 2026-02-20T12:59:23.619 INFO:teuthology.orchestra.run.trial093.stdout: Installing : libcephsqlite-2:20.3.0-5312.g26bc9765.el9.x86_64 11/12 2026-02-20T12:59:30.513 INFO:teuthology.orchestra.run.trial093.stdout: Running scriptlet: libcephsqlite-2:20.3.0-5312.g26bc9765.el9.x86_64 11/12 2026-02-20T12:59:30.532 INFO:teuthology.orchestra.run.trial093.stdout: Installing : ceph-test-2:20.3.0-5312.g26bc9765.el9.x86_64 12/12 2026-02-20T12:59:31.285 INFO:teuthology.orchestra.run.trial093.stdout: Running scriptlet: ceph-test-2:20.3.0-5312.g26bc9765.el9.x86_64 12/12 2026-02-20T12:59:31.285 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : ceph-test-2:20.3.0-5312.g26bc9765.el9.x86_64 1/12 2026-02-20T12:59:31.285 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : libcephsqlite-2:20.3.0-5312.g26bc9765.el9.x86_64 2/12 2026-02-20T12:59:31.285 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : hwloc-libs-2.4.1-6.el9.x86_64 3/12 2026-02-20T12:59:31.285 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : libatomic-11.5.0-14.el9.x86_64 4/12 2026-02-20T12:59:31.285 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 5/12 2026-02-20T12:59:31.285 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : libasan8-15.2.1-7.el9.x86_64 6/12 2026-02-20T12:59:31.286 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : libubsan-11.5.0-14.el9.x86_64 7/12 2026-02-20T12:59:31.286 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : libxslt-1.1.34-12.el9.x86_64 8/12 2026-02-20T12:59:31.286 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : protobuf-3.14.0-17.el9.x86_64 9/12 2026-02-20T12:59:31.286 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : socat-1.7.4.1-8.el9.x86_64 10/12 2026-02-20T12:59:31.286 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 11/12 2026-02-20T12:59:31.339 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 12/12 2026-02-20T12:59:31.339 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T12:59:31.339 INFO:teuthology.orchestra.run.trial093.stdout:Installed: 2026-02-20T12:59:31.339 INFO:teuthology.orchestra.run.trial093.stdout: ceph-test-2:20.3.0-5312.g26bc9765.el9.x86_64 2026-02-20T12:59:31.339 INFO:teuthology.orchestra.run.trial093.stdout: hwloc-libs-2.4.1-6.el9.x86_64 2026-02-20T12:59:31.339 INFO:teuthology.orchestra.run.trial093.stdout: libasan8-15.2.1-7.el9.x86_64 2026-02-20T12:59:31.340 INFO:teuthology.orchestra.run.trial093.stdout: libatomic-11.5.0-14.el9.x86_64 2026-02-20T12:59:31.340 INFO:teuthology.orchestra.run.trial093.stdout: libcephsqlite-2:20.3.0-5312.g26bc9765.el9.x86_64 2026-02-20T12:59:31.340 INFO:teuthology.orchestra.run.trial093.stdout: libubsan-11.5.0-14.el9.x86_64 2026-02-20T12:59:31.340 INFO:teuthology.orchestra.run.trial093.stdout: libxslt-1.1.34-12.el9.x86_64 2026-02-20T12:59:31.340 INFO:teuthology.orchestra.run.trial093.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2026-02-20T12:59:31.340 INFO:teuthology.orchestra.run.trial093.stdout: protobuf-3.14.0-17.el9.x86_64 2026-02-20T12:59:31.340 INFO:teuthology.orchestra.run.trial093.stdout: socat-1.7.4.1-8.el9.x86_64 2026-02-20T12:59:31.340 INFO:teuthology.orchestra.run.trial093.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2026-02-20T12:59:31.340 INFO:teuthology.orchestra.run.trial093.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2026-02-20T12:59:31.340 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T12:59:31.340 INFO:teuthology.orchestra.run.trial093.stdout:Complete! 2026-02-20T12:59:31.378 DEBUG:teuthology.orchestra.run.trial093:> sudo yum -y install ceph 2026-02-20T12:59:31.687 INFO:teuthology.orchestra.run.trial093.stdout:Last metadata expiration check: 0:01:41 ago on Fri 20 Feb 2026 12:57:50 PM UTC. 2026-02-20T12:59:31.773 INFO:teuthology.orchestra.run.trial093.stdout:Dependencies resolved. 2026-02-20T12:59:31.776 INFO:teuthology.orchestra.run.trial093.stdout:======================================================================================= 2026-02-20T12:59:31.776 INFO:teuthology.orchestra.run.trial093.stdout: Package Arch Version Repository Size 2026-02-20T12:59:31.776 INFO:teuthology.orchestra.run.trial093.stdout:======================================================================================= 2026-02-20T12:59:31.776 INFO:teuthology.orchestra.run.trial093.stdout:Installing: 2026-02-20T12:59:31.776 INFO:teuthology.orchestra.run.trial093.stdout: ceph x86_64 2:20.3.0-5312.g26bc9765.el9 ceph 6.5 k 2026-02-20T12:59:31.776 INFO:teuthology.orchestra.run.trial093.stdout:Installing dependencies: 2026-02-20T12:59:31.776 INFO:teuthology.orchestra.run.trial093.stdout: abseil-cpp x86_64 20211102.0-4.el9 epel 551 k 2026-02-20T12:59:31.776 INFO:teuthology.orchestra.run.trial093.stdout: ceph-grafana-dashboards noarch 2:20.3.0-5312.g26bc9765.el9 ceph-noarch 57 k 2026-02-20T12:59:31.776 INFO:teuthology.orchestra.run.trial093.stdout: ceph-mds x86_64 2:20.3.0-5312.g26bc9765.el9 ceph 2.0 M 2026-02-20T12:59:31.777 INFO:teuthology.orchestra.run.trial093.stdout: ceph-mgr x86_64 2:20.3.0-5312.g26bc9765.el9 ceph 937 k 2026-02-20T12:59:31.777 INFO:teuthology.orchestra.run.trial093.stdout: ceph-mgr-modules-core noarch 2:20.3.0-5312.g26bc9765.el9 ceph-noarch 297 k 2026-02-20T12:59:31.777 INFO:teuthology.orchestra.run.trial093.stdout: ceph-mon x86_64 2:20.3.0-5312.g26bc9765.el9 ceph 4.9 M 2026-02-20T12:59:31.777 INFO:teuthology.orchestra.run.trial093.stdout: ceph-osd x86_64 2:20.3.0-5312.g26bc9765.el9 ceph 163 k 2026-02-20T12:59:31.777 INFO:teuthology.orchestra.run.trial093.stdout: ceph-osd-classic x86_64 2:20.3.0-5312.g26bc9765.el9 ceph 15 M 2026-02-20T12:59:31.777 INFO:teuthology.orchestra.run.trial093.stdout: ceph-osd-crimson x86_64 2:20.3.0-5312.g26bc9765.el9 ceph 85 M 2026-02-20T12:59:31.777 INFO:teuthology.orchestra.run.trial093.stdout: ceph-prometheus-alerts noarch 2:20.3.0-5312.g26bc9765.el9 ceph-noarch 17 k 2026-02-20T12:59:31.777 INFO:teuthology.orchestra.run.trial093.stdout: cephadm noarch 2:20.3.0-5312.g26bc9765.el9 ceph-noarch 1.0 M 2026-02-20T12:59:31.777 INFO:teuthology.orchestra.run.trial093.stdout: cryptsetup x86_64 2.8.1-3.el9 baseos 351 k 2026-02-20T12:59:31.777 INFO:teuthology.orchestra.run.trial093.stdout: flexiblas x86_64 3.0.4-9.el9 appstream 30 k 2026-02-20T12:59:31.777 INFO:teuthology.orchestra.run.trial093.stdout: flexiblas-netlib x86_64 3.0.4-9.el9 appstream 3.0 M 2026-02-20T12:59:31.778 INFO:teuthology.orchestra.run.trial093.stdout: flexiblas-openblas-openmp x86_64 3.0.4-9.el9 appstream 15 k 2026-02-20T12:59:31.778 INFO:teuthology.orchestra.run.trial093.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2026-02-20T12:59:31.778 INFO:teuthology.orchestra.run.trial093.stdout: ledmon-libs x86_64 1.1.0-3.el9 baseos 40 k 2026-02-20T12:59:31.778 INFO:teuthology.orchestra.run.trial093.stdout: libconfig x86_64 1.7.2-9.el9 baseos 72 k 2026-02-20T12:59:31.778 INFO:teuthology.orchestra.run.trial093.stdout: libgfortran x86_64 11.5.0-14.el9 baseos 794 k 2026-02-20T12:59:31.778 INFO:teuthology.orchestra.run.trial093.stdout: libquadmath x86_64 11.5.0-14.el9 baseos 184 k 2026-02-20T12:59:31.778 INFO:teuthology.orchestra.run.trial093.stdout: libstoragemgmt x86_64 1.10.1-1.el9 appstream 246 k 2026-02-20T12:59:31.778 INFO:teuthology.orchestra.run.trial093.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2026-02-20T12:59:31.778 INFO:teuthology.orchestra.run.trial093.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2026-02-20T12:59:31.778 INFO:teuthology.orchestra.run.trial093.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2026-02-20T12:59:31.778 INFO:teuthology.orchestra.run.trial093.stdout: openblas x86_64 0.3.29-1.el9 appstream 42 k 2026-02-20T12:59:31.778 INFO:teuthology.orchestra.run.trial093.stdout: openblas-openmp x86_64 0.3.29-1.el9 appstream 5.3 M 2026-02-20T12:59:31.779 INFO:teuthology.orchestra.run.trial093.stdout: protobuf-compiler x86_64 3.14.0-17.el9 crb 862 k 2026-02-20T12:59:31.779 INFO:teuthology.orchestra.run.trial093.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2026-02-20T12:59:31.779 INFO:teuthology.orchestra.run.trial093.stdout: python3-autocommand noarch 2.2.2-8.el9 epel 29 k 2026-02-20T12:59:31.779 INFO:teuthology.orchestra.run.trial093.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2026-02-20T12:59:31.779 INFO:teuthology.orchestra.run.trial093.stdout: python3-backports-tarfile noarch 1.2.0-1.el9 epel 60 k 2026-02-20T12:59:31.779 INFO:teuthology.orchestra.run.trial093.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2026-02-20T12:59:31.779 INFO:teuthology.orchestra.run.trial093.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2026-02-20T12:59:31.779 INFO:teuthology.orchestra.run.trial093.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2026-02-20T12:59:31.779 INFO:teuthology.orchestra.run.trial093.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2026-02-20T12:59:31.779 INFO:teuthology.orchestra.run.trial093.stdout: python3-cheroot noarch 10.0.1-4.el9 epel 173 k 2026-02-20T12:59:31.779 INFO:teuthology.orchestra.run.trial093.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2026-02-20T12:59:31.779 INFO:teuthology.orchestra.run.trial093.stdout: python3-cryptography x86_64 36.0.1-5.el9 baseos 1.2 M 2026-02-20T12:59:31.779 INFO:teuthology.orchestra.run.trial093.stdout: python3-devel x86_64 3.9.25-3.el9 appstream 244 k 2026-02-20T12:59:31.780 INFO:teuthology.orchestra.run.trial093.stdout: python3-google-auth noarch 1:2.45.0-1.el9 epel 254 k 2026-02-20T12:59:31.780 INFO:teuthology.orchestra.run.trial093.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2026-02-20T12:59:31.780 INFO:teuthology.orchestra.run.trial093.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2026-02-20T12:59:31.780 INFO:teuthology.orchestra.run.trial093.stdout: python3-isodate noarch 0.6.1-3.el9 epel 56 k 2026-02-20T12:59:31.780 INFO:teuthology.orchestra.run.trial093.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2026-02-20T12:59:31.780 INFO:teuthology.orchestra.run.trial093.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2026-02-20T12:59:31.780 INFO:teuthology.orchestra.run.trial093.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2026-02-20T12:59:31.780 INFO:teuthology.orchestra.run.trial093.stdout: python3-jaraco-context noarch 6.0.1-3.el9 epel 20 k 2026-02-20T12:59:31.780 INFO:teuthology.orchestra.run.trial093.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2026-02-20T12:59:31.780 INFO:teuthology.orchestra.run.trial093.stdout: python3-jaraco-text noarch 4.0.0-2.el9 epel 26 k 2026-02-20T12:59:31.780 INFO:teuthology.orchestra.run.trial093.stdout: python3-jinja2 noarch 2.11.3-8.el9 appstream 249 k 2026-02-20T12:59:31.780 INFO:teuthology.orchestra.run.trial093.stdout: python3-jmespath noarch 1.0.1-1.el9 appstream 48 k 2026-02-20T12:59:31.780 INFO:teuthology.orchestra.run.trial093.stdout: python3-kubernetes noarch 1:26.1.0-3.el9 epel 1.0 M 2026-02-20T12:59:31.781 INFO:teuthology.orchestra.run.trial093.stdout: python3-libstoragemgmt x86_64 1.10.1-1.el9 appstream 177 k 2026-02-20T12:59:31.781 INFO:teuthology.orchestra.run.trial093.stdout: python3-lxml x86_64 4.6.5-3.el9 appstream 1.2 M 2026-02-20T12:59:31.781 INFO:teuthology.orchestra.run.trial093.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2026-02-20T12:59:31.781 INFO:teuthology.orchestra.run.trial093.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2026-02-20T12:59:31.781 INFO:teuthology.orchestra.run.trial093.stdout: python3-msgpack x86_64 1.0.3-2.el9 epel 86 k 2026-02-20T12:59:31.781 INFO:teuthology.orchestra.run.trial093.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2026-02-20T12:59:31.781 INFO:teuthology.orchestra.run.trial093.stdout: python3-numpy x86_64 1:1.23.5-2.el9 appstream 6.1 M 2026-02-20T12:59:31.781 INFO:teuthology.orchestra.run.trial093.stdout: python3-numpy-f2py x86_64 1:1.23.5-2.el9 appstream 442 k 2026-02-20T12:59:31.781 INFO:teuthology.orchestra.run.trial093.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2026-02-20T12:59:31.781 INFO:teuthology.orchestra.run.trial093.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2026-02-20T12:59:31.781 INFO:teuthology.orchestra.run.trial093.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2026-02-20T12:59:31.781 INFO:teuthology.orchestra.run.trial093.stdout: python3-protobuf noarch 3.14.0-17.el9 appstream 267 k 2026-02-20T12:59:31.781 INFO:teuthology.orchestra.run.trial093.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2026-02-20T12:59:31.782 INFO:teuthology.orchestra.run.trial093.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2026-02-20T12:59:31.782 INFO:teuthology.orchestra.run.trial093.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2026-02-20T12:59:31.782 INFO:teuthology.orchestra.run.trial093.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2026-02-20T12:59:31.782 INFO:teuthology.orchestra.run.trial093.stdout: python3-pyparsing noarch 2.4.7-9.el9 baseos 150 k 2026-02-20T12:59:31.782 INFO:teuthology.orchestra.run.trial093.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2026-02-20T12:59:31.782 INFO:teuthology.orchestra.run.trial093.stdout: python3-requests noarch 2.25.1-10.el9 baseos 126 k 2026-02-20T12:59:31.782 INFO:teuthology.orchestra.run.trial093.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2026-02-20T12:59:31.782 INFO:teuthology.orchestra.run.trial093.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2026-02-20T12:59:31.782 INFO:teuthology.orchestra.run.trial093.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2026-02-20T12:59:31.782 INFO:teuthology.orchestra.run.trial093.stdout: python3-scipy x86_64 1.9.3-2.el9 appstream 19 M 2026-02-20T12:59:31.782 INFO:teuthology.orchestra.run.trial093.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2026-02-20T12:59:31.782 INFO:teuthology.orchestra.run.trial093.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2026-02-20T12:59:31.783 INFO:teuthology.orchestra.run.trial093.stdout: python3-typing-extensions noarch 4.15.0-1.el9 epel 86 k 2026-02-20T12:59:31.783 INFO:teuthology.orchestra.run.trial093.stdout: python3-urllib3 noarch 1.26.5-7.el9 baseos 218 k 2026-02-20T12:59:31.783 INFO:teuthology.orchestra.run.trial093.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2026-02-20T12:59:31.783 INFO:teuthology.orchestra.run.trial093.stdout: python3-xmlsec x86_64 1.3.13-1.el9 epel 48 k 2026-02-20T12:59:31.783 INFO:teuthology.orchestra.run.trial093.stdout: python3-xmltodict noarch 0.12.0-15.el9 epel 22 k 2026-02-20T12:59:31.783 INFO:teuthology.orchestra.run.trial093.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2026-02-20T12:59:31.783 INFO:teuthology.orchestra.run.trial093.stdout: unzip x86_64 6.0-59.el9 baseos 182 k 2026-02-20T12:59:31.783 INFO:teuthology.orchestra.run.trial093.stdout: xmlsec1 x86_64 1.2.29-13.el9 appstream 189 k 2026-02-20T12:59:31.783 INFO:teuthology.orchestra.run.trial093.stdout: xmlsec1-openssl x86_64 1.2.29-13.el9 appstream 90 k 2026-02-20T12:59:31.783 INFO:teuthology.orchestra.run.trial093.stdout: zip x86_64 3.0-35.el9 baseos 266 k 2026-02-20T12:59:31.783 INFO:teuthology.orchestra.run.trial093.stdout:Installing weak dependencies: 2026-02-20T12:59:31.783 INFO:teuthology.orchestra.run.trial093.stdout: ceph-mgr-cephadm noarch 2:20.3.0-5312.g26bc9765.el9 ceph-noarch 184 k 2026-02-20T12:59:31.783 INFO:teuthology.orchestra.run.trial093.stdout: ceph-mgr-dashboard noarch 2:20.3.0-5312.g26bc9765.el9 ceph-noarch 51 M 2026-02-20T12:59:31.784 INFO:teuthology.orchestra.run.trial093.stdout: ceph-mgr-diskprediction-local noarch 2:20.3.0-5312.g26bc9765.el9 ceph-noarch 7.4 M 2026-02-20T12:59:31.784 INFO:teuthology.orchestra.run.trial093.stdout: ceph-mgr-k8sevents noarch 2:20.3.0-5312.g26bc9765.el9 ceph-noarch 22 k 2026-02-20T12:59:31.784 INFO:teuthology.orchestra.run.trial093.stdout: ceph-mgr-rook noarch 2:20.3.0-5312.g26bc9765.el9 ceph-noarch 50 k 2026-02-20T12:59:31.784 INFO:teuthology.orchestra.run.trial093.stdout: ceph-volume noarch 2:20.3.0-5312.g26bc9765.el9 ceph-noarch 299 k 2026-02-20T12:59:31.784 INFO:teuthology.orchestra.run.trial093.stdout: python3-influxdb noarch 5.3.1-1.el9 epel 139 k 2026-02-20T12:59:31.784 INFO:teuthology.orchestra.run.trial093.stdout: python3-saml noarch 1.16.0-1.el9 epel 125 k 2026-02-20T12:59:31.784 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T12:59:31.784 INFO:teuthology.orchestra.run.trial093.stdout:Transaction Summary 2026-02-20T12:59:31.784 INFO:teuthology.orchestra.run.trial093.stdout:======================================================================================= 2026-02-20T12:59:31.784 INFO:teuthology.orchestra.run.trial093.stdout:Install 96 Packages 2026-02-20T12:59:31.784 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T12:59:31.785 INFO:teuthology.orchestra.run.trial093.stdout:Total download size: 223 M 2026-02-20T12:59:31.785 INFO:teuthology.orchestra.run.trial093.stdout:Installed size: 1.5 G 2026-02-20T12:59:31.785 INFO:teuthology.orchestra.run.trial093.stdout:Downloading Packages: 2026-02-20T12:59:32.569 INFO:teuthology.orchestra.run.trial093.stdout:(1/96): ceph-20.3.0-5312.g26bc9765.el9.x86_64.r 23 kB/s | 6.5 kB 00:00 2026-02-20T12:59:33.556 INFO:teuthology.orchestra.run.trial093.stdout:(2/96): ceph-mgr-20.3.0-5312.g26bc9765.el9.x86_ 741 kB/s | 937 kB 00:01 2026-02-20T12:59:33.739 INFO:teuthology.orchestra.run.trial093.stdout:(3/96): ceph-osd-20.3.0-5312.g26bc9765.el9.x86_ 895 kB/s | 163 kB 00:00 2026-02-20T12:59:33.902 INFO:teuthology.orchestra.run.trial093.stdout:(4/96): ceph-mds-20.3.0-5312.g26bc9765.el9.x86_ 1.3 MB/s | 2.0 MB 00:01 2026-02-20T12:59:36.929 INFO:teuthology.orchestra.run.trial093.stdout:(5/96): ceph-mon-20.3.0-5312.g26bc9765.el9.x86_ 1.1 MB/s | 4.9 MB 00:04 2026-02-20T12:59:36.999 INFO:teuthology.orchestra.run.trial093.stdout:(6/96): ceph-grafana-dashboards-20.3.0-5312.g26 820 kB/s | 57 kB 00:00 2026-02-20T12:59:37.140 INFO:teuthology.orchestra.run.trial093.stdout:(7/96): ceph-mgr-cephadm-20.3.0-5312.g26bc9765. 1.3 MB/s | 184 kB 00:00 2026-02-20T12:59:42.572 INFO:teuthology.orchestra.run.trial093.stdout:(8/96): ceph-osd-classic-20.3.0-5312.g26bc9765. 1.7 MB/s | 15 MB 00:08 2026-02-20T12:59:46.555 INFO:teuthology.orchestra.run.trial093.stdout:(9/96): ceph-mgr-diskprediction-local-20.3.0-53 1.9 MB/s | 7.4 MB 00:03 2026-02-20T12:59:46.616 INFO:teuthology.orchestra.run.trial093.stdout:(10/96): ceph-mgr-k8sevents-20.3.0-5312.g26bc97 367 kB/s | 22 kB 00:00 2026-02-20T12:59:46.801 INFO:teuthology.orchestra.run.trial093.stdout:(11/96): ceph-mgr-modules-core-20.3.0-5312.g26b 1.6 MB/s | 297 kB 00:00 2026-02-20T12:59:46.863 INFO:teuthology.orchestra.run.trial093.stdout:(12/96): ceph-mgr-rook-20.3.0-5312.g26bc9765.el 811 kB/s | 50 kB 00:00 2026-02-20T12:59:46.925 INFO:teuthology.orchestra.run.trial093.stdout:(13/96): ceph-prometheus-alerts-20.3.0-5312.g26 281 kB/s | 17 kB 00:00 2026-02-20T12:59:47.114 INFO:teuthology.orchestra.run.trial093.stdout:(14/96): ceph-volume-20.3.0-5312.g26bc9765.el9. 1.5 MB/s | 299 kB 00:00 2026-02-20T12:59:47.637 INFO:teuthology.orchestra.run.trial093.stdout:(15/96): cephadm-20.3.0-5312.g26bc9765.el9.noar 1.9 MB/s | 1.0 MB 00:00 2026-02-20T12:59:52.776 INFO:teuthology.orchestra.run.trial093.stdout:(16/96): cryptsetup-2.8.1-3.el9.x86_64.rpm 68 kB/s | 351 kB 00:05 2026-02-20T12:59:52.780 INFO:teuthology.orchestra.run.trial093.stdout:(17/96): ledmon-libs-1.1.0-3.el9.x86_64.rpm 9.4 MB/s | 40 kB 00:00 2026-02-20T12:59:52.785 INFO:teuthology.orchestra.run.trial093.stdout:(18/96): libconfig-1.7.2-9.el9.x86_64.rpm 16 MB/s | 72 kB 00:00 2026-02-20T12:59:52.799 INFO:teuthology.orchestra.run.trial093.stdout:(19/96): libgfortran-11.5.0-14.el9.x86_64.rpm 55 MB/s | 794 kB 00:00 2026-02-20T12:59:52.823 INFO:teuthology.orchestra.run.trial093.stdout:(20/96): libquadmath-11.5.0-14.el9.x86_64.rpm 7.8 MB/s | 184 kB 00:00 2026-02-20T12:59:52.838 INFO:teuthology.orchestra.run.trial093.stdout:(21/96): python3-cffi-1.14.5-5.el9.x86_64.rpm 16 MB/s | 253 kB 00:00 2026-02-20T12:59:52.877 INFO:teuthology.orchestra.run.trial093.stdout:(22/96): python3-cryptography-36.0.1-5.el9.x86_ 32 MB/s | 1.2 MB 00:00 2026-02-20T12:59:52.882 INFO:teuthology.orchestra.run.trial093.stdout:(23/96): python3-ply-3.11-14.el9.noarch.rpm 18 MB/s | 106 kB 00:00 2026-02-20T12:59:52.888 INFO:teuthology.orchestra.run.trial093.stdout:(24/96): python3-pycparser-2.20-6.el9.noarch.rp 22 MB/s | 135 kB 00:00 2026-02-20T12:59:52.895 INFO:teuthology.orchestra.run.trial093.stdout:(25/96): python3-pyparsing-2.4.7-9.el9.noarch.r 24 MB/s | 150 kB 00:00 2026-02-20T12:59:52.901 INFO:teuthology.orchestra.run.trial093.stdout:(26/96): python3-requests-2.25.1-10.el9.noarch. 19 MB/s | 126 kB 00:00 2026-02-20T12:59:52.908 INFO:teuthology.orchestra.run.trial093.stdout:(27/96): python3-urllib3-1.26.5-7.el9.noarch.rp 33 MB/s | 218 kB 00:00 2026-02-20T12:59:52.915 INFO:teuthology.orchestra.run.trial093.stdout:(28/96): unzip-6.0-59.el9.x86_64.rpm 26 MB/s | 182 kB 00:00 2026-02-20T12:59:52.921 INFO:teuthology.orchestra.run.trial093.stdout:(29/96): zip-3.0-35.el9.x86_64.rpm 42 MB/s | 266 kB 00:00 2026-02-20T12:59:54.842 INFO:teuthology.orchestra.run.trial093.stdout:(30/96): flexiblas-3.0.4-9.el9.x86_64.rpm 15 kB/s | 30 kB 00:01 2026-02-20T12:59:56.751 INFO:teuthology.orchestra.run.trial093.stdout:(31/96): ceph-mgr-dashboard-20.3.0-5312.g26bc97 2.6 MB/s | 51 MB 00:19 2026-02-20T12:59:57.020 INFO:teuthology.orchestra.run.trial093.stdout:(32/96): flexiblas-netlib-3.0.4-9.el9.x86_64.rp 1.4 MB/s | 3.0 MB 00:02 2026-02-20T12:59:57.069 INFO:teuthology.orchestra.run.trial093.stdout:(33/96): flexiblas-openblas-openmp-3.0.4-9.el9. 47 kB/s | 15 kB 00:00 2026-02-20T12:59:57.338 INFO:teuthology.orchestra.run.trial093.stdout:(34/96): libstoragemgmt-1.10.1-1.el9.x86_64.rpm 776 kB/s | 246 kB 00:00 2026-02-20T12:59:57.339 INFO:teuthology.orchestra.run.trial093.stdout:(35/96): lua-5.4.4-4.el9.x86_64.rpm 699 kB/s | 188 kB 00:00 2026-02-20T12:59:57.638 INFO:teuthology.orchestra.run.trial093.stdout:(36/96): openblas-0.3.29-1.el9.x86_64.rpm 141 kB/s | 42 kB 00:00 2026-02-20T12:59:58.713 INFO:teuthology.orchestra.run.trial093.stdout:(37/96): openblas-openmp-0.3.29-1.el9.x86_64.rp 3.8 MB/s | 5.3 MB 00:01 2026-02-20T12:59:59.388 INFO:teuthology.orchestra.run.trial093.stdout:(38/96): python3-devel-3.9.25-3.el9.x86_64.rpm 363 kB/s | 244 kB 00:00 2026-02-20T12:59:59.397 INFO:teuthology.orchestra.run.trial093.stdout:(39/96): python3-babel-2.9.1-2.el9.noarch.rpm 3.4 MB/s | 6.0 MB 00:01 2026-02-20T12:59:59.704 INFO:teuthology.orchestra.run.trial093.stdout:(40/96): python3-jinja2-2.11.3-8.el9.noarch.rpm 790 kB/s | 249 kB 00:00 2026-02-20T12:59:59.965 INFO:teuthology.orchestra.run.trial093.stdout:(41/96): python3-jmespath-1.0.1-1.el9.noarch.rp 84 kB/s | 48 kB 00:00 2026-02-20T13:00:00.133 INFO:teuthology.orchestra.run.trial093.stdout:(42/96): python3-libstoragemgmt-1.10.1-1.el9.x8 412 kB/s | 177 kB 00:00 2026-02-20T13:00:00.569 INFO:teuthology.orchestra.run.trial093.stdout:(43/96): python3-markupsafe-1.1.1-12.el9.x86_64 80 kB/s | 35 kB 00:00 2026-02-20T13:00:00.693 INFO:teuthology.orchestra.run.trial093.stdout:(44/96): python3-lxml-4.6.5-3.el9.x86_64.rpm 1.7 MB/s | 1.2 MB 00:00 2026-02-20T13:00:01.394 INFO:teuthology.orchestra.run.trial093.stdout:(45/96): python3-numpy-f2py-1.23.5-2.el9.x86_64 631 kB/s | 442 kB 00:00 2026-02-20T13:00:01.395 INFO:teuthology.orchestra.run.trial093.stdout:[MIRROR] python3-numpy-1.23.5-2.el9.x86_64.rpm: Curl error (18): Transferred a partial file for https://us.mirrors.cicku.me/centos/9-stream/AppStream/x86_64/os/Packages/python3-numpy-1.23.5-2.el9.x86_64.rpm [transfer closed with 3304727 bytes remaining to read] 2026-02-20T13:00:01.594 INFO:teuthology.orchestra.run.trial093.stdout:(46/96): python3-numpy-1.23.5-2.el9.x86_64.rpm 6.0 MB/s | 6.1 MB 00:01 2026-02-20T13:00:01.616 INFO:teuthology.orchestra.run.trial093.stdout:(47/96): python3-protobuf-3.14.0-17.el9.noarch. 12 MB/s | 267 kB 00:00 2026-02-20T13:00:01.638 INFO:teuthology.orchestra.run.trial093.stdout:(48/96): python3-pyasn1-0.4.8-6.el9.noarch.rpm 7.1 MB/s | 159 kB 00:00 2026-02-20T13:00:01.662 INFO:teuthology.orchestra.run.trial093.stdout:(49/96): python3-pyasn1-modules-0.4.8-6.el9.noa 12 MB/s | 279 kB 00:00 2026-02-20T13:00:01.683 INFO:teuthology.orchestra.run.trial093.stdout:(50/96): python3-requests-oauthlib-1.3.0-12.el9 2.5 MB/s | 54 kB 00:00 2026-02-20T13:00:01.705 INFO:teuthology.orchestra.run.trial093.stdout:(51/96): python3-packaging-20.9-5.el9.noarch.rp 248 kB/s | 77 kB 00:00 2026-02-20T13:00:01.773 INFO:teuthology.orchestra.run.trial093.stdout:(52/96): python3-toml-0.10.2-6.el9.noarch.rpm 613 kB/s | 42 kB 00:00 2026-02-20T13:00:01.855 INFO:teuthology.orchestra.run.trial093.stdout:(53/96): python3-scipy-1.9.3-2.el9.x86_64.rpm 112 MB/s | 19 MB 00:00 2026-02-20T13:00:01.856 INFO:teuthology.orchestra.run.trial093.stdout:(54/96): xmlsec1-1.2.29-13.el9.x86_64.rpm 2.2 MB/s | 189 kB 00:00 2026-02-20T13:00:01.863 INFO:teuthology.orchestra.run.trial093.stdout:(55/96): lua-devel-5.4.4-4.el9.x86_64.rpm 3.1 MB/s | 22 kB 00:00 2026-02-20T13:00:01.877 INFO:teuthology.orchestra.run.trial093.stdout:(56/96): xmlsec1-openssl-1.2.29-13.el9.x86_64.r 4.1 MB/s | 90 kB 00:00 2026-02-20T13:00:01.892 INFO:teuthology.orchestra.run.trial093.stdout:(57/96): protobuf-compiler-3.14.0-17.el9.x86_64 29 MB/s | 862 kB 00:00 2026-02-20T13:00:02.188 INFO:teuthology.orchestra.run.trial093.stdout:(58/96): grpc-data-1.46.7-10.el9.noarch.rpm 66 kB/s | 19 kB 00:00 2026-02-20T13:00:02.334 INFO:teuthology.orchestra.run.trial093.stdout:(59/96): luarocks-3.9.2-5.el9.noarch.rpm 1.0 MB/s | 151 kB 00:00 2026-02-20T13:00:02.372 INFO:teuthology.orchestra.run.trial093.stdout:(60/96): abseil-cpp-20211102.0-4.el9.x86_64.rpm 1.1 MB/s | 551 kB 00:00 2026-02-20T13:00:02.415 INFO:teuthology.orchestra.run.trial093.stdout:(61/96): python3-autocommand-2.2.2-8.el9.noarch 686 kB/s | 29 kB 00:00 2026-02-20T13:00:02.460 INFO:teuthology.orchestra.run.trial093.stdout:(62/96): python3-backports-tarfile-1.2.0-1.el9. 1.3 MB/s | 60 kB 00:00 2026-02-20T13:00:02.469 INFO:teuthology.orchestra.run.trial093.stdout:(63/96): python3-asyncssh-2.13.2-5.el9.noarch.r 4.0 MB/s | 548 kB 00:00 2026-02-20T13:00:02.504 INFO:teuthology.orchestra.run.trial093.stdout:(64/96): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 984 kB/s | 43 kB 00:00 2026-02-20T13:00:02.512 INFO:teuthology.orchestra.run.trial093.stdout:(65/96): python3-cachetools-4.2.4-1.el9.noarch. 749 kB/s | 32 kB 00:00 2026-02-20T13:00:02.547 INFO:teuthology.orchestra.run.trial093.stdout:(66/96): python3-certifi-2023.05.07-4.el9.noarc 331 kB/s | 14 kB 00:00 2026-02-20T13:00:02.562 INFO:teuthology.orchestra.run.trial093.stdout:(67/96): python3-cheroot-10.0.1-4.el9.noarch.rp 3.4 MB/s | 173 kB 00:00 2026-02-20T13:00:02.608 INFO:teuthology.orchestra.run.trial093.stdout:(68/96): python3-cherrypy-18.6.1-2.el9.noarch.r 5.7 MB/s | 358 kB 00:00 2026-02-20T13:00:02.615 INFO:teuthology.orchestra.run.trial093.stdout:(69/96): python3-google-auth-2.45.0-1.el9.noarc 4.6 MB/s | 254 kB 00:00 2026-02-20T13:00:02.664 INFO:teuthology.orchestra.run.trial093.stdout:(70/96): python3-grpcio-tools-1.46.7-10.el9.x86 2.9 MB/s | 144 kB 00:00 2026-02-20T13:00:02.713 INFO:teuthology.orchestra.run.trial093.stdout:(71/96): python3-influxdb-5.3.1-1.el9.noarch.rp 2.8 MB/s | 139 kB 00:00 2026-02-20T13:00:02.744 INFO:teuthology.orchestra.run.trial093.stdout:(72/96): python3-grpcio-1.46.7-10.el9.x86_64.rp 15 MB/s | 2.0 MB 00:00 2026-02-20T13:00:02.758 INFO:teuthology.orchestra.run.trial093.stdout:(73/96): python3-isodate-0.6.1-3.el9.noarch.rpm 1.2 MB/s | 56 kB 00:00 2026-02-20T13:00:02.787 INFO:teuthology.orchestra.run.trial093.stdout:(74/96): python3-jaraco-8.2.1-3.el9.noarch.rpm 250 kB/s | 11 kB 00:00 2026-02-20T13:00:02.803 INFO:teuthology.orchestra.run.trial093.stdout:(75/96): python3-jaraco-classes-3.2.1-5.el9.noa 392 kB/s | 18 kB 00:00 2026-02-20T13:00:02.830 INFO:teuthology.orchestra.run.trial093.stdout:(76/96): python3-jaraco-collections-3.0.0-8.el9 542 kB/s | 23 kB 00:00 2026-02-20T13:00:02.848 INFO:teuthology.orchestra.run.trial093.stdout:(77/96): python3-jaraco-context-6.0.1-3.el9.noa 437 kB/s | 20 kB 00:00 2026-02-20T13:00:02.873 INFO:teuthology.orchestra.run.trial093.stdout:(78/96): python3-jaraco-functools-3.5.0-2.el9.n 455 kB/s | 19 kB 00:00 2026-02-20T13:00:02.891 INFO:teuthology.orchestra.run.trial093.stdout:(79/96): python3-jaraco-text-4.0.0-2.el9.noarch 612 kB/s | 26 kB 00:00 2026-02-20T13:00:02.930 INFO:teuthology.orchestra.run.trial093.stdout:(80/96): python3-kubernetes-26.1.0-3.el9.noarch 18 MB/s | 1.0 MB 00:00 2026-02-20T13:00:02.937 INFO:teuthology.orchestra.run.trial093.stdout:(81/96): python3-more-itertools-8.12.0-2.el9.no 1.7 MB/s | 79 kB 00:00 2026-02-20T13:00:02.973 INFO:teuthology.orchestra.run.trial093.stdout:(82/96): python3-msgpack-1.0.3-2.el9.x86_64.rpm 2.0 MB/s | 86 kB 00:00 2026-02-20T13:00:02.982 INFO:teuthology.orchestra.run.trial093.stdout:(83/96): python3-natsort-7.1.1-5.el9.noarch.rpm 1.3 MB/s | 58 kB 00:00 2026-02-20T13:00:03.016 INFO:teuthology.orchestra.run.trial093.stdout:(84/96): python3-portend-3.1.0-2.el9.noarch.rpm 384 kB/s | 16 kB 00:00 2026-02-20T13:00:03.028 INFO:teuthology.orchestra.run.trial093.stdout:(85/96): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.9 MB/s | 90 kB 00:00 2026-02-20T13:00:03.059 INFO:teuthology.orchestra.run.trial093.stdout:(86/96): python3-repoze-lru-0.7-16.el9.noarch.r 716 kB/s | 31 kB 00:00 2026-02-20T13:00:03.079 INFO:teuthology.orchestra.run.trial093.stdout:(87/96): python3-routes-2.5.1-5.el9.noarch.rpm 3.6 MB/s | 188 kB 00:00 2026-02-20T13:00:03.102 INFO:teuthology.orchestra.run.trial093.stdout:(88/96): python3-rsa-4.9-2.el9.noarch.rpm 1.3 MB/s | 59 kB 00:00 2026-02-20T13:00:03.127 INFO:teuthology.orchestra.run.trial093.stdout:(89/96): python3-saml-1.16.0-1.el9.noarch.rpm 2.5 MB/s | 125 kB 00:00 2026-02-20T13:00:03.145 INFO:teuthology.orchestra.run.trial093.stdout:(90/96): python3-tempora-5.0.0-2.el9.noarch.rpm 831 kB/s | 36 kB 00:00 2026-02-20T13:00:03.173 INFO:teuthology.orchestra.run.trial093.stdout:(91/96): python3-typing-extensions-4.15.0-1.el9 1.8 MB/s | 86 kB 00:00 2026-02-20T13:00:03.188 INFO:teuthology.orchestra.run.trial093.stdout:(92/96): python3-websocket-client-1.2.3-2.el9.n 2.0 MB/s | 90 kB 00:00 2026-02-20T13:00:03.217 INFO:teuthology.orchestra.run.trial093.stdout:(93/96): python3-xmlsec-1.3.13-1.el9.x86_64.rpm 1.1 MB/s | 48 kB 00:00 2026-02-20T13:00:03.231 INFO:teuthology.orchestra.run.trial093.stdout:(94/96): python3-xmltodict-0.12.0-15.el9.noarch 517 kB/s | 22 kB 00:00 2026-02-20T13:00:03.261 INFO:teuthology.orchestra.run.trial093.stdout:(95/96): python3-zc-lockfile-2.0-10.el9.noarch. 460 kB/s | 20 kB 00:00 2026-02-20T13:00:11.652 INFO:teuthology.orchestra.run.trial093.stdout:(96/96): ceph-osd-crimson-20.3.0-5312.g26bc9765 2.3 MB/s | 85 MB 00:37 2026-02-20T13:00:11.655 INFO:teuthology.orchestra.run.trial093.stdout:-------------------------------------------------------------------------------- 2026-02-20T13:00:11.655 INFO:teuthology.orchestra.run.trial093.stdout:Total 5.6 MB/s | 223 MB 00:39 2026-02-20T13:00:12.056 INFO:teuthology.orchestra.run.trial093.stdout:Running transaction check 2026-02-20T13:00:12.095 INFO:teuthology.orchestra.run.trial093.stdout:Transaction check succeeded. 2026-02-20T13:00:12.096 INFO:teuthology.orchestra.run.trial093.stdout:Running transaction test 2026-02-20T13:00:12.887 INFO:teuthology.orchestra.run.trial093.stdout:Transaction test succeeded. 2026-02-20T13:00:12.888 INFO:teuthology.orchestra.run.trial093.stdout:Running transaction 2026-02-20T13:00:13.720 INFO:teuthology.orchestra.run.trial093.stdout: Preparing : 1/1 2026-02-20T13:00:13.722 INFO:teuthology.orchestra.run.trial093.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/96 2026-02-20T13:00:13.726 INFO:teuthology.orchestra.run.trial093.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/96 2026-02-20T13:00:13.748 INFO:teuthology.orchestra.run.trial093.stdout: Installing : xmlsec1-1.2.29-13.el9.x86_64 3/96 2026-02-20T13:00:13.781 INFO:teuthology.orchestra.run.trial093.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 4/96 2026-02-20T13:00:13.787 INFO:teuthology.orchestra.run.trial093.stdout: Installing : python3-lxml-4.6.5-3.el9.x86_64 5/96 2026-02-20T13:00:13.790 INFO:teuthology.orchestra.run.trial093.stdout: Installing : lua-5.4.4-4.el9.x86_64 6/96 2026-02-20T13:00:13.794 INFO:teuthology.orchestra.run.trial093.stdout: Installing : flexiblas-3.0.4-9.el9.x86_64 7/96 2026-02-20T13:00:13.817 INFO:teuthology.orchestra.run.trial093.stdout: Installing : unzip-6.0-59.el9.x86_64 8/96 2026-02-20T13:00:13.830 INFO:teuthology.orchestra.run.trial093.stdout: Installing : python3-urllib3-1.26.5-7.el9.noarch 9/96 2026-02-20T13:00:13.833 INFO:teuthology.orchestra.run.trial093.stdout: Installing : python3-requests-2.25.1-10.el9.noarch 10/96 2026-02-20T13:00:13.839 INFO:teuthology.orchestra.run.trial093.stdout: Installing : libquadmath-11.5.0-14.el9.x86_64 11/96 2026-02-20T13:00:13.841 INFO:teuthology.orchestra.run.trial093.stdout: Installing : libgfortran-11.5.0-14.el9.x86_64 12/96 2026-02-20T13:00:13.852 INFO:teuthology.orchestra.run.trial093.stdout: Installing : ledmon-libs-1.1.0-3.el9.x86_64 13/96 2026-02-20T13:00:13.858 INFO:teuthology.orchestra.run.trial093.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 14/96 2026-02-20T13:00:13.882 INFO:teuthology.orchestra.run.trial093.stdout: Installing : zip-3.0-35.el9.x86_64 15/96 2026-02-20T13:00:13.885 INFO:teuthology.orchestra.run.trial093.stdout: Installing : luarocks-3.9.2-5.el9.noarch 16/96 2026-02-20T13:00:13.935 INFO:teuthology.orchestra.run.trial093.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 17/96 2026-02-20T13:00:13.948 INFO:teuthology.orchestra.run.trial093.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 18/96 2026-02-20T13:00:13.950 INFO:teuthology.orchestra.run.trial093.stdout: Installing : python3-rsa-4.9-2.el9.noarch 19/96 2026-02-20T13:00:13.954 INFO:teuthology.orchestra.run.trial093.stdout: Installing : xmlsec1-openssl-1.2.29-13.el9.x86_64 20/96 2026-02-20T13:00:13.960 INFO:teuthology.orchestra.run.trial093.stdout: Installing : python3-xmlsec-1.3.13-1.el9.x86_64 21/96 2026-02-20T13:00:13.964 INFO:teuthology.orchestra.run.trial093.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 22/96 2026-02-20T13:00:13.967 INFO:teuthology.orchestra.run.trial093.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 23/96 2026-02-20T13:00:13.982 INFO:teuthology.orchestra.run.trial093.stdout: Installing : python3-xmltodict-0.12.0-15.el9.noarch 24/96 2026-02-20T13:00:13.986 INFO:teuthology.orchestra.run.trial093.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 25/96 2026-02-20T13:00:13.991 INFO:teuthology.orchestra.run.trial093.stdout: Installing : python3-typing-extensions-4.15.0-1.el9.noarch 26/96 2026-02-20T13:00:14.003 INFO:teuthology.orchestra.run.trial093.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 27/96 2026-02-20T13:00:14.012 INFO:teuthology.orchestra.run.trial093.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 28/96 2026-02-20T13:00:14.019 INFO:teuthology.orchestra.run.trial093.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 29/96 2026-02-20T13:00:14.038 INFO:teuthology.orchestra.run.trial093.stdout: Installing : python3-msgpack-1.0.3-2.el9.x86_64 30/96 2026-02-20T13:00:14.049 INFO:teuthology.orchestra.run.trial093.stdout: Installing : python3-influxdb-5.3.1-1.el9.noarch 31/96 2026-02-20T13:00:14.065 INFO:teuthology.orchestra.run.trial093.stdout: Installing : python3-isodate-0.6.1-3.el9.noarch 32/96 2026-02-20T13:00:14.069 INFO:teuthology.orchestra.run.trial093.stdout: Installing : python3-saml-1.16.0-1.el9.noarch 33/96 2026-02-20T13:00:14.077 INFO:teuthology.orchestra.run.trial093.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 34/96 2026-02-20T13:00:14.116 INFO:teuthology.orchestra.run.trial093.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 35/96 2026-02-20T13:00:14.439 INFO:teuthology.orchestra.run.trial093.stdout: Installing : python3-google-auth-1:2.45.0-1.el9.noarch 36/96 2026-02-20T13:00:14.449 INFO:teuthology.orchestra.run.trial093.stdout: Installing : python3-kubernetes-1:26.1.0-3.el9.noarch 37/96 2026-02-20T13:00:14.453 INFO:teuthology.orchestra.run.trial093.stdout: Installing : python3-backports-tarfile-1.2.0-1.el9.noarch 38/96 2026-02-20T13:00:14.459 INFO:teuthology.orchestra.run.trial093.stdout: Installing : python3-jaraco-context-6.0.1-3.el9.noarch 39/96 2026-02-20T13:00:14.460 INFO:teuthology.orchestra.run.trial093.stdout: Installing : python3-autocommand-2.2.2-8.el9.noarch 40/96 2026-02-20T13:00:14.487 INFO:teuthology.orchestra.run.trial093.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 41/96 2026-02-20T13:00:14.531 INFO:teuthology.orchestra.run.trial093.stdout: Installing : abseil-cpp-20211102.0-4.el9.x86_64 42/96 2026-02-20T13:00:14.539 INFO:teuthology.orchestra.run.trial093.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 43/96 2026-02-20T13:00:14.545 INFO:teuthology.orchestra.run.trial093.stdout: Installing : protobuf-compiler-3.14.0-17.el9.x86_64 44/96 2026-02-20T13:00:14.548 INFO:teuthology.orchestra.run.trial093.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 45/96 2026-02-20T13:00:14.554 INFO:teuthology.orchestra.run.trial093.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 46/96 2026-02-20T13:00:14.558 INFO:teuthology.orchestra.run.trial093.stdout: Installing : python3-jaraco-text-4.0.0-2.el9.noarch 47/96 2026-02-20T13:00:14.564 INFO:teuthology.orchestra.run.trial093.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 48/96 2026-02-20T13:00:14.568 INFO:teuthology.orchestra.run.trial093.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 49/96 2026-02-20T13:00:14.595 INFO:teuthology.orchestra.run.trial093.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 50/96 2026-02-20T13:00:14.605 INFO:teuthology.orchestra.run.trial093.stdout: Installing : python3-protobuf-3.14.0-17.el9.noarch 51/96 2026-02-20T13:00:14.611 INFO:teuthology.orchestra.run.trial093.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 52/96 2026-02-20T13:00:14.618 INFO:teuthology.orchestra.run.trial093.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 53/96 2026-02-20T13:00:14.652 INFO:teuthology.orchestra.run.trial093.stdout: Installing : python3-jmespath-1.0.1-1.el9.noarch 54/96 2026-02-20T13:00:14.875 INFO:teuthology.orchestra.run.trial093.stdout: Installing : python3-devel-3.9.25-3.el9.x86_64 55/96 2026-02-20T13:00:14.898 INFO:teuthology.orchestra.run.trial093.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 56/96 2026-02-20T13:00:14.902 INFO:teuthology.orchestra.run.trial093.stdout: Installing : python3-jinja2-2.11.3-8.el9.noarch 57/96 2026-02-20T13:00:14.961 INFO:teuthology.orchestra.run.trial093.stdout: Installing : openblas-0.3.29-1.el9.x86_64 58/96 2026-02-20T13:00:14.963 INFO:teuthology.orchestra.run.trial093.stdout: Installing : openblas-openmp-0.3.29-1.el9.x86_64 59/96 2026-02-20T13:00:14.985 INFO:teuthology.orchestra.run.trial093.stdout: Installing : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 60/96 2026-02-20T13:00:15.309 INFO:teuthology.orchestra.run.trial093.stdout: Installing : flexiblas-netlib-3.0.4-9.el9.x86_64 61/96 2026-02-20T13:00:15.381 INFO:teuthology.orchestra.run.trial093.stdout: Installing : python3-numpy-1:1.23.5-2.el9.x86_64 62/96 2026-02-20T13:00:16.048 INFO:teuthology.orchestra.run.trial093.stdout: Installing : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 63/96 2026-02-20T13:00:16.065 INFO:teuthology.orchestra.run.trial093.stdout: Installing : python3-scipy-1.9.3-2.el9.x86_64 64/96 2026-02-20T13:00:16.074 INFO:teuthology.orchestra.run.trial093.stdout: Installing : python3-pyparsing-2.4.7-9.el9.noarch 65/96 2026-02-20T13:00:16.081 INFO:teuthology.orchestra.run.trial093.stdout: Installing : python3-packaging-20.9-5.el9.noarch 66/96 2026-02-20T13:00:16.094 INFO:teuthology.orchestra.run.trial093.stdout: Installing : python3-ply-3.11-14.el9.noarch 67/96 2026-02-20T13:00:16.110 INFO:teuthology.orchestra.run.trial093.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 68/96 2026-02-20T13:00:16.185 INFO:teuthology.orchestra.run.trial093.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 69/96 2026-02-20T13:00:16.193 INFO:teuthology.orchestra.run.trial093.stdout: Installing : python3-cryptography-36.0.1-5.el9.x86_64 70/96 2026-02-20T13:00:16.217 INFO:teuthology.orchestra.run.trial093.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 71/96 2026-02-20T13:00:16.246 INFO:teuthology.orchestra.run.trial093.stdout: Installing : python3-cheroot-10.0.1-4.el9.noarch 72/96 2026-02-20T13:00:16.296 INFO:teuthology.orchestra.run.trial093.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 73/96 2026-02-20T13:00:16.302 INFO:teuthology.orchestra.run.trial093.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 74/96 2026-02-20T13:00:16.305 INFO:teuthology.orchestra.run.trial093.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 75/96 2026-02-20T13:00:16.307 INFO:teuthology.orchestra.run.trial093.stdout: Installing : libconfig-1.7.2-9.el9.x86_64 76/96 2026-02-20T13:00:16.324 INFO:teuthology.orchestra.run.trial093.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-20T13:00:16.324 INFO:teuthology.orchestra.run.trial093.stdout:Creating group 'libstoragemgmt' with GID 991. 2026-02-20T13:00:16.324 INFO:teuthology.orchestra.run.trial093.stdout:Creating user 'libstoragemgmt' (daemon account for libstoragemgmt) with UID 991 and GID 991. 2026-02-20T13:00:16.324 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:00:16.332 INFO:teuthology.orchestra.run.trial093.stdout: Installing : libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-20T13:00:16.356 INFO:teuthology.orchestra.run.trial093.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-20T13:00:16.356 INFO:teuthology.orchestra.run.trial093.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/libstoragemgmt.service → /usr/lib/systemd/system/libstoragemgmt.service. 2026-02-20T13:00:16.356 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:00:16.390 INFO:teuthology.orchestra.run.trial093.stdout: Installing : python3-libstoragemgmt-1.10.1-1.el9.x86_64 78/96 2026-02-20T13:00:16.975 INFO:teuthology.orchestra.run.trial093.stdout: Installing : cryptsetup-2.8.1-3.el9.x86_64 79/96 2026-02-20T13:00:22.237 INFO:teuthology.orchestra.run.trial093.stdout: Installing : ceph-osd-classic-2:20.3.0-5312.g26bc9765.el9.x86_6 80/96 2026-02-20T13:00:22.300 INFO:teuthology.orchestra.run.trial093.stdout: Installing : ceph-osd-crimson-2:20.3.0-5312.g26bc9765.el9.x86_6 81/96 2026-02-20T13:00:22.302 INFO:teuthology.orchestra.run.trial093.stdout: Installing : ceph-volume-2:20.3.0-5312.g26bc9765.el9.noarch 82/96 2026-02-20T13:00:22.316 INFO:teuthology.orchestra.run.trial093.stdout: Running scriptlet: ceph-volume-2:20.3.0-5312.g26bc9765.el9.noarch 82/96 2026-02-20T13:00:22.316 INFO:teuthology.orchestra.run.trial093.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-20T13:00:22.316 INFO:teuthology.orchestra.run.trial093.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2026-02-20T13:00:22.316 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:00:22.326 INFO:teuthology.orchestra.run.trial093.stdout: Installing : ceph-osd-2:20.3.0-5312.g26bc9765.el9.x86_64 83/96 2026-02-20T13:00:22.350 INFO:teuthology.orchestra.run.trial093.stdout: Running scriptlet: ceph-osd-2:20.3.0-5312.g26bc9765.el9.x86_64 83/96 2026-02-20T13:00:22.350 INFO:teuthology.orchestra.run.trial093.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-20T13:00:22.350 INFO:teuthology.orchestra.run.trial093.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2026-02-20T13:00:22.350 INFO:teuthology.orchestra.run.trial093.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-20T13:00:22.350 INFO:teuthology.orchestra.run.trial093.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-20T13:00:22.350 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:00:22.406 INFO:teuthology.orchestra.run.trial093.stdout: Running scriptlet: cephadm-2:20.3.0-5312.g26bc9765.el9.noarch 84/96 2026-02-20T13:00:22.408 INFO:teuthology.orchestra.run.trial093.stdout: Installing : cephadm-2:20.3.0-5312.g26bc9765.el9.noarch 84/96 2026-02-20T13:00:22.417 INFO:teuthology.orchestra.run.trial093.stdout: Installing : ceph-prometheus-alerts-2:20.3.0-5312.g26bc9765.el9 85/96 2026-02-20T13:00:22.441 INFO:teuthology.orchestra.run.trial093.stdout: Installing : ceph-grafana-dashboards-2:20.3.0-5312.g26bc9765.el 86/96 2026-02-20T13:00:22.442 INFO:teuthology.orchestra.run.trial093.stdout: Installing : ceph-mgr-cephadm-2:20.3.0-5312.g26bc9765.el9.noarc 87/96 2026-02-20T13:00:25.328 INFO:teuthology.orchestra.run.trial093.stdout: Running scriptlet: ceph-mgr-cephadm-2:20.3.0-5312.g26bc9765.el9.noarc 87/96 2026-02-20T13:00:25.335 INFO:teuthology.orchestra.run.trial093.stdout: Installing : ceph-mgr-dashboard-2:20.3.0-5312.g26bc9765.el9.noa 88/96 2026-02-20T13:00:25.791 INFO:teuthology.orchestra.run.trial093.stdout: Running scriptlet: ceph-mgr-dashboard-2:20.3.0-5312.g26bc9765.el9.noa 88/96 2026-02-20T13:00:25.792 INFO:teuthology.orchestra.run.trial093.stdout: Installing : ceph-mgr-diskprediction-local-2:20.3.0-5312.g26bc9 89/96 2026-02-20T13:00:25.805 INFO:teuthology.orchestra.run.trial093.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:20.3.0-5312.g26bc9 89/96 2026-02-20T13:00:25.806 INFO:teuthology.orchestra.run.trial093.stdout: Installing : ceph-mgr-k8sevents-2:20.3.0-5312.g26bc9765.el9.noa 90/96 2026-02-20T13:00:25.868 INFO:teuthology.orchestra.run.trial093.stdout: Running scriptlet: ceph-mgr-k8sevents-2:20.3.0-5312.g26bc9765.el9.noa 90/96 2026-02-20T13:00:25.908 INFO:teuthology.orchestra.run.trial093.stdout: Installing : ceph-mgr-modules-core-2:20.3.0-5312.g26bc9765.el9. 91/96 2026-02-20T13:00:25.909 INFO:teuthology.orchestra.run.trial093.stdout: Installing : ceph-mgr-2:20.3.0-5312.g26bc9765.el9.x86_64 92/96 2026-02-20T13:00:25.930 INFO:teuthology.orchestra.run.trial093.stdout: Running scriptlet: ceph-mgr-2:20.3.0-5312.g26bc9765.el9.x86_64 92/96 2026-02-20T13:00:25.931 INFO:teuthology.orchestra.run.trial093.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-20T13:00:25.931 INFO:teuthology.orchestra.run.trial093.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2026-02-20T13:00:25.931 INFO:teuthology.orchestra.run.trial093.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-20T13:00:25.931 INFO:teuthology.orchestra.run.trial093.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-20T13:00:25.931 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:00:25.944 INFO:teuthology.orchestra.run.trial093.stdout: Installing : ceph-mgr-rook-2:20.3.0-5312.g26bc9765.el9.noarch 93/96 2026-02-20T13:00:26.156 INFO:teuthology.orchestra.run.trial093.stdout: Running scriptlet: ceph-mgr-rook-2:20.3.0-5312.g26bc9765.el9.noarch 93/96 2026-02-20T13:00:26.158 INFO:teuthology.orchestra.run.trial093.stdout: Installing : ceph-mon-2:20.3.0-5312.g26bc9765.el9.x86_64 94/96 2026-02-20T13:00:26.179 INFO:teuthology.orchestra.run.trial093.stdout: Running scriptlet: ceph-mon-2:20.3.0-5312.g26bc9765.el9.x86_64 94/96 2026-02-20T13:00:26.179 INFO:teuthology.orchestra.run.trial093.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-20T13:00:26.179 INFO:teuthology.orchestra.run.trial093.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2026-02-20T13:00:26.179 INFO:teuthology.orchestra.run.trial093.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-20T13:00:26.179 INFO:teuthology.orchestra.run.trial093.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-20T13:00:26.179 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:00:26.261 INFO:teuthology.orchestra.run.trial093.stdout: Installing : ceph-mds-2:20.3.0-5312.g26bc9765.el9.x86_64 95/96 2026-02-20T13:00:26.282 INFO:teuthology.orchestra.run.trial093.stdout: Running scriptlet: ceph-mds-2:20.3.0-5312.g26bc9765.el9.x86_64 95/96 2026-02-20T13:00:26.283 INFO:teuthology.orchestra.run.trial093.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-20T13:00:26.283 INFO:teuthology.orchestra.run.trial093.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2026-02-20T13:00:26.283 INFO:teuthology.orchestra.run.trial093.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-20T13:00:26.283 INFO:teuthology.orchestra.run.trial093.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-20T13:00:26.283 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:00:26.284 INFO:teuthology.orchestra.run.trial093.stdout: Installing : ceph-2:20.3.0-5312.g26bc9765.el9.x86_64 96/96 2026-02-20T13:00:26.290 INFO:teuthology.orchestra.run.trial093.stdout: Running scriptlet: ceph-osd-classic-2:20.3.0-5312.g26bc9765.el9.x86_6 96/96 2026-02-20T13:00:26.315 INFO:teuthology.orchestra.run.trial093.stdout: Running scriptlet: ceph-osd-crimson-2:20.3.0-5312.g26bc9765.el9.x86_6 96/96 2026-02-20T13:00:27.709 INFO:teuthology.orchestra.run.trial093.stdout: Running scriptlet: ceph-2:20.3.0-5312.g26bc9765.el9.x86_64 96/96 2026-02-20T13:00:27.709 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : ceph-2:20.3.0-5312.g26bc9765.el9.x86_64 1/96 2026-02-20T13:00:27.709 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : ceph-mds-2:20.3.0-5312.g26bc9765.el9.x86_64 2/96 2026-02-20T13:00:27.710 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : ceph-mgr-2:20.3.0-5312.g26bc9765.el9.x86_64 3/96 2026-02-20T13:00:27.710 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : ceph-mon-2:20.3.0-5312.g26bc9765.el9.x86_64 4/96 2026-02-20T13:00:27.710 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : ceph-osd-2:20.3.0-5312.g26bc9765.el9.x86_64 5/96 2026-02-20T13:00:27.710 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : ceph-osd-classic-2:20.3.0-5312.g26bc9765.el9.x86_6 6/96 2026-02-20T13:00:27.710 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : ceph-osd-crimson-2:20.3.0-5312.g26bc9765.el9.x86_6 7/96 2026-02-20T13:00:27.710 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : ceph-grafana-dashboards-2:20.3.0-5312.g26bc9765.el 8/96 2026-02-20T13:00:27.710 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : ceph-mgr-cephadm-2:20.3.0-5312.g26bc9765.el9.noarc 9/96 2026-02-20T13:00:27.710 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : ceph-mgr-dashboard-2:20.3.0-5312.g26bc9765.el9.noa 10/96 2026-02-20T13:00:27.710 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : ceph-mgr-diskprediction-local-2:20.3.0-5312.g26bc9 11/96 2026-02-20T13:00:27.710 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : ceph-mgr-k8sevents-2:20.3.0-5312.g26bc9765.el9.noa 12/96 2026-02-20T13:00:27.710 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : ceph-mgr-modules-core-2:20.3.0-5312.g26bc9765.el9. 13/96 2026-02-20T13:00:27.710 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : ceph-mgr-rook-2:20.3.0-5312.g26bc9765.el9.noarch 14/96 2026-02-20T13:00:27.711 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : ceph-prometheus-alerts-2:20.3.0-5312.g26bc9765.el9 15/96 2026-02-20T13:00:27.711 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : ceph-volume-2:20.3.0-5312.g26bc9765.el9.noarch 16/96 2026-02-20T13:00:27.711 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : cephadm-2:20.3.0-5312.g26bc9765.el9.noarch 17/96 2026-02-20T13:00:27.711 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : cryptsetup-2.8.1-3.el9.x86_64 18/96 2026-02-20T13:00:27.711 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : ledmon-libs-1.1.0-3.el9.x86_64 19/96 2026-02-20T13:00:27.711 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : libconfig-1.7.2-9.el9.x86_64 20/96 2026-02-20T13:00:27.711 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : libgfortran-11.5.0-14.el9.x86_64 21/96 2026-02-20T13:00:27.711 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : libquadmath-11.5.0-14.el9.x86_64 22/96 2026-02-20T13:00:27.711 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 23/96 2026-02-20T13:00:27.711 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : python3-cryptography-36.0.1-5.el9.x86_64 24/96 2026-02-20T13:00:27.711 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : python3-ply-3.11-14.el9.noarch 25/96 2026-02-20T13:00:27.711 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 26/96 2026-02-20T13:00:27.711 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : python3-pyparsing-2.4.7-9.el9.noarch 27/96 2026-02-20T13:00:27.712 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : python3-requests-2.25.1-10.el9.noarch 28/96 2026-02-20T13:00:27.712 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : python3-urllib3-1.26.5-7.el9.noarch 29/96 2026-02-20T13:00:27.712 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : unzip-6.0-59.el9.x86_64 30/96 2026-02-20T13:00:27.712 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : zip-3.0-35.el9.x86_64 31/96 2026-02-20T13:00:27.712 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : flexiblas-3.0.4-9.el9.x86_64 32/96 2026-02-20T13:00:27.712 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : flexiblas-netlib-3.0.4-9.el9.x86_64 33/96 2026-02-20T13:00:27.712 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 34/96 2026-02-20T13:00:27.712 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : libstoragemgmt-1.10.1-1.el9.x86_64 35/96 2026-02-20T13:00:27.712 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : lua-5.4.4-4.el9.x86_64 36/96 2026-02-20T13:00:27.712 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : openblas-0.3.29-1.el9.x86_64 37/96 2026-02-20T13:00:27.712 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : openblas-openmp-0.3.29-1.el9.x86_64 38/96 2026-02-20T13:00:27.712 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 39/96 2026-02-20T13:00:27.712 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : python3-devel-3.9.25-3.el9.x86_64 40/96 2026-02-20T13:00:27.713 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : python3-jinja2-2.11.3-8.el9.noarch 41/96 2026-02-20T13:00:27.713 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : python3-jmespath-1.0.1-1.el9.noarch 42/96 2026-02-20T13:00:27.713 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : python3-libstoragemgmt-1.10.1-1.el9.x86_64 43/96 2026-02-20T13:00:27.713 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : python3-lxml-4.6.5-3.el9.x86_64 44/96 2026-02-20T13:00:27.713 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 45/96 2026-02-20T13:00:27.713 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : python3-numpy-1:1.23.5-2.el9.x86_64 46/96 2026-02-20T13:00:27.713 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 47/96 2026-02-20T13:00:27.713 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 48/96 2026-02-20T13:00:27.713 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : python3-protobuf-3.14.0-17.el9.noarch 49/96 2026-02-20T13:00:27.713 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 50/96 2026-02-20T13:00:27.713 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 51/96 2026-02-20T13:00:27.713 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 52/96 2026-02-20T13:00:27.713 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : python3-scipy-1.9.3-2.el9.x86_64 53/96 2026-02-20T13:00:27.714 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 54/96 2026-02-20T13:00:27.714 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : xmlsec1-1.2.29-13.el9.x86_64 55/96 2026-02-20T13:00:27.714 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : xmlsec1-openssl-1.2.29-13.el9.x86_64 56/96 2026-02-20T13:00:27.714 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 57/96 2026-02-20T13:00:27.714 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : protobuf-compiler-3.14.0-17.el9.x86_64 58/96 2026-02-20T13:00:27.714 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : abseil-cpp-20211102.0-4.el9.x86_64 59/96 2026-02-20T13:00:27.714 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 60/96 2026-02-20T13:00:27.714 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 61/96 2026-02-20T13:00:27.714 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 62/96 2026-02-20T13:00:27.714 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : python3-autocommand-2.2.2-8.el9.noarch 63/96 2026-02-20T13:00:27.714 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : python3-backports-tarfile-1.2.0-1.el9.noarch 64/96 2026-02-20T13:00:27.714 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 65/96 2026-02-20T13:00:27.715 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 66/96 2026-02-20T13:00:27.715 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 67/96 2026-02-20T13:00:27.715 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : python3-cheroot-10.0.1-4.el9.noarch 68/96 2026-02-20T13:00:27.715 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 69/96 2026-02-20T13:00:27.715 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : python3-google-auth-1:2.45.0-1.el9.noarch 70/96 2026-02-20T13:00:27.715 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 71/96 2026-02-20T13:00:27.715 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 72/96 2026-02-20T13:00:27.715 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : python3-influxdb-5.3.1-1.el9.noarch 73/96 2026-02-20T13:00:27.715 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : python3-isodate-0.6.1-3.el9.noarch 74/96 2026-02-20T13:00:27.715 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 75/96 2026-02-20T13:00:27.715 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 76/96 2026-02-20T13:00:27.715 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 77/96 2026-02-20T13:00:27.715 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : python3-jaraco-context-6.0.1-3.el9.noarch 78/96 2026-02-20T13:00:27.716 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 79/96 2026-02-20T13:00:27.716 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : python3-jaraco-text-4.0.0-2.el9.noarch 80/96 2026-02-20T13:00:27.716 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : python3-kubernetes-1:26.1.0-3.el9.noarch 81/96 2026-02-20T13:00:27.716 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 82/96 2026-02-20T13:00:27.716 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : python3-msgpack-1.0.3-2.el9.x86_64 83/96 2026-02-20T13:00:27.716 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 84/96 2026-02-20T13:00:27.716 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 85/96 2026-02-20T13:00:27.716 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/96 2026-02-20T13:00:27.716 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 87/96 2026-02-20T13:00:27.716 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 88/96 2026-02-20T13:00:27.716 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 89/96 2026-02-20T13:00:27.716 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : python3-saml-1.16.0-1.el9.noarch 90/96 2026-02-20T13:00:27.716 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 91/96 2026-02-20T13:00:27.717 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : python3-typing-extensions-4.15.0-1.el9.noarch 92/96 2026-02-20T13:00:27.717 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 93/96 2026-02-20T13:00:27.717 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : python3-xmlsec-1.3.13-1.el9.x86_64 94/96 2026-02-20T13:00:27.717 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : python3-xmltodict-0.12.0-15.el9.noarch 95/96 2026-02-20T13:00:27.772 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 96/96 2026-02-20T13:00:27.773 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:00:27.773 INFO:teuthology.orchestra.run.trial093.stdout:Installed: 2026-02-20T13:00:27.773 INFO:teuthology.orchestra.run.trial093.stdout: abseil-cpp-20211102.0-4.el9.x86_64 2026-02-20T13:00:27.773 INFO:teuthology.orchestra.run.trial093.stdout: ceph-2:20.3.0-5312.g26bc9765.el9.x86_64 2026-02-20T13:00:27.773 INFO:teuthology.orchestra.run.trial093.stdout: ceph-grafana-dashboards-2:20.3.0-5312.g26bc9765.el9.noarch 2026-02-20T13:00:27.773 INFO:teuthology.orchestra.run.trial093.stdout: ceph-mds-2:20.3.0-5312.g26bc9765.el9.x86_64 2026-02-20T13:00:27.773 INFO:teuthology.orchestra.run.trial093.stdout: ceph-mgr-2:20.3.0-5312.g26bc9765.el9.x86_64 2026-02-20T13:00:27.773 INFO:teuthology.orchestra.run.trial093.stdout: ceph-mgr-cephadm-2:20.3.0-5312.g26bc9765.el9.noarch 2026-02-20T13:00:27.773 INFO:teuthology.orchestra.run.trial093.stdout: ceph-mgr-dashboard-2:20.3.0-5312.g26bc9765.el9.noarch 2026-02-20T13:00:27.773 INFO:teuthology.orchestra.run.trial093.stdout: ceph-mgr-diskprediction-local-2:20.3.0-5312.g26bc9765.el9.noarch 2026-02-20T13:00:27.774 INFO:teuthology.orchestra.run.trial093.stdout: ceph-mgr-k8sevents-2:20.3.0-5312.g26bc9765.el9.noarch 2026-02-20T13:00:27.774 INFO:teuthology.orchestra.run.trial093.stdout: ceph-mgr-modules-core-2:20.3.0-5312.g26bc9765.el9.noarch 2026-02-20T13:00:27.774 INFO:teuthology.orchestra.run.trial093.stdout: ceph-mgr-rook-2:20.3.0-5312.g26bc9765.el9.noarch 2026-02-20T13:00:27.774 INFO:teuthology.orchestra.run.trial093.stdout: ceph-mon-2:20.3.0-5312.g26bc9765.el9.x86_64 2026-02-20T13:00:27.774 INFO:teuthology.orchestra.run.trial093.stdout: ceph-osd-2:20.3.0-5312.g26bc9765.el9.x86_64 2026-02-20T13:00:27.774 INFO:teuthology.orchestra.run.trial093.stdout: ceph-osd-classic-2:20.3.0-5312.g26bc9765.el9.x86_64 2026-02-20T13:00:27.774 INFO:teuthology.orchestra.run.trial093.stdout: ceph-osd-crimson-2:20.3.0-5312.g26bc9765.el9.x86_64 2026-02-20T13:00:27.774 INFO:teuthology.orchestra.run.trial093.stdout: ceph-prometheus-alerts-2:20.3.0-5312.g26bc9765.el9.noarch 2026-02-20T13:00:27.774 INFO:teuthology.orchestra.run.trial093.stdout: ceph-volume-2:20.3.0-5312.g26bc9765.el9.noarch 2026-02-20T13:00:27.774 INFO:teuthology.orchestra.run.trial093.stdout: cephadm-2:20.3.0-5312.g26bc9765.el9.noarch 2026-02-20T13:00:27.774 INFO:teuthology.orchestra.run.trial093.stdout: cryptsetup-2.8.1-3.el9.x86_64 2026-02-20T13:00:27.774 INFO:teuthology.orchestra.run.trial093.stdout: flexiblas-3.0.4-9.el9.x86_64 2026-02-20T13:00:27.774 INFO:teuthology.orchestra.run.trial093.stdout: flexiblas-netlib-3.0.4-9.el9.x86_64 2026-02-20T13:00:27.775 INFO:teuthology.orchestra.run.trial093.stdout: flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 2026-02-20T13:00:27.775 INFO:teuthology.orchestra.run.trial093.stdout: grpc-data-1.46.7-10.el9.noarch 2026-02-20T13:00:27.775 INFO:teuthology.orchestra.run.trial093.stdout: ledmon-libs-1.1.0-3.el9.x86_64 2026-02-20T13:00:27.775 INFO:teuthology.orchestra.run.trial093.stdout: libconfig-1.7.2-9.el9.x86_64 2026-02-20T13:00:27.775 INFO:teuthology.orchestra.run.trial093.stdout: libgfortran-11.5.0-14.el9.x86_64 2026-02-20T13:00:27.775 INFO:teuthology.orchestra.run.trial093.stdout: libquadmath-11.5.0-14.el9.x86_64 2026-02-20T13:00:27.775 INFO:teuthology.orchestra.run.trial093.stdout: libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-20T13:00:27.775 INFO:teuthology.orchestra.run.trial093.stdout: lua-5.4.4-4.el9.x86_64 2026-02-20T13:00:27.775 INFO:teuthology.orchestra.run.trial093.stdout: lua-devel-5.4.4-4.el9.x86_64 2026-02-20T13:00:27.775 INFO:teuthology.orchestra.run.trial093.stdout: luarocks-3.9.2-5.el9.noarch 2026-02-20T13:00:27.775 INFO:teuthology.orchestra.run.trial093.stdout: openblas-0.3.29-1.el9.x86_64 2026-02-20T13:00:27.775 INFO:teuthology.orchestra.run.trial093.stdout: openblas-openmp-0.3.29-1.el9.x86_64 2026-02-20T13:00:27.775 INFO:teuthology.orchestra.run.trial093.stdout: protobuf-compiler-3.14.0-17.el9.x86_64 2026-02-20T13:00:27.776 INFO:teuthology.orchestra.run.trial093.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2026-02-20T13:00:27.776 INFO:teuthology.orchestra.run.trial093.stdout: python3-autocommand-2.2.2-8.el9.noarch 2026-02-20T13:00:27.776 INFO:teuthology.orchestra.run.trial093.stdout: python3-babel-2.9.1-2.el9.noarch 2026-02-20T13:00:27.776 INFO:teuthology.orchestra.run.trial093.stdout: python3-backports-tarfile-1.2.0-1.el9.noarch 2026-02-20T13:00:27.776 INFO:teuthology.orchestra.run.trial093.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2026-02-20T13:00:27.776 INFO:teuthology.orchestra.run.trial093.stdout: python3-cachetools-4.2.4-1.el9.noarch 2026-02-20T13:00:27.776 INFO:teuthology.orchestra.run.trial093.stdout: python3-certifi-2023.05.07-4.el9.noarch 2026-02-20T13:00:27.776 INFO:teuthology.orchestra.run.trial093.stdout: python3-cffi-1.14.5-5.el9.x86_64 2026-02-20T13:00:27.776 INFO:teuthology.orchestra.run.trial093.stdout: python3-cheroot-10.0.1-4.el9.noarch 2026-02-20T13:00:27.776 INFO:teuthology.orchestra.run.trial093.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2026-02-20T13:00:27.776 INFO:teuthology.orchestra.run.trial093.stdout: python3-cryptography-36.0.1-5.el9.x86_64 2026-02-20T13:00:27.776 INFO:teuthology.orchestra.run.trial093.stdout: python3-devel-3.9.25-3.el9.x86_64 2026-02-20T13:00:27.777 INFO:teuthology.orchestra.run.trial093.stdout: python3-google-auth-1:2.45.0-1.el9.noarch 2026-02-20T13:00:27.777 INFO:teuthology.orchestra.run.trial093.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2026-02-20T13:00:27.777 INFO:teuthology.orchestra.run.trial093.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2026-02-20T13:00:27.777 INFO:teuthology.orchestra.run.trial093.stdout: python3-influxdb-5.3.1-1.el9.noarch 2026-02-20T13:00:27.777 INFO:teuthology.orchestra.run.trial093.stdout: python3-isodate-0.6.1-3.el9.noarch 2026-02-20T13:00:27.777 INFO:teuthology.orchestra.run.trial093.stdout: python3-jaraco-8.2.1-3.el9.noarch 2026-02-20T13:00:27.777 INFO:teuthology.orchestra.run.trial093.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2026-02-20T13:00:27.777 INFO:teuthology.orchestra.run.trial093.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2026-02-20T13:00:27.777 INFO:teuthology.orchestra.run.trial093.stdout: python3-jaraco-context-6.0.1-3.el9.noarch 2026-02-20T13:00:27.777 INFO:teuthology.orchestra.run.trial093.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2026-02-20T13:00:27.777 INFO:teuthology.orchestra.run.trial093.stdout: python3-jaraco-text-4.0.0-2.el9.noarch 2026-02-20T13:00:27.777 INFO:teuthology.orchestra.run.trial093.stdout: python3-jinja2-2.11.3-8.el9.noarch 2026-02-20T13:00:27.777 INFO:teuthology.orchestra.run.trial093.stdout: python3-jmespath-1.0.1-1.el9.noarch 2026-02-20T13:00:27.778 INFO:teuthology.orchestra.run.trial093.stdout: python3-kubernetes-1:26.1.0-3.el9.noarch 2026-02-20T13:00:27.778 INFO:teuthology.orchestra.run.trial093.stdout: python3-libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-20T13:00:27.778 INFO:teuthology.orchestra.run.trial093.stdout: python3-lxml-4.6.5-3.el9.x86_64 2026-02-20T13:00:27.778 INFO:teuthology.orchestra.run.trial093.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2026-02-20T13:00:27.778 INFO:teuthology.orchestra.run.trial093.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2026-02-20T13:00:27.778 INFO:teuthology.orchestra.run.trial093.stdout: python3-msgpack-1.0.3-2.el9.x86_64 2026-02-20T13:00:27.778 INFO:teuthology.orchestra.run.trial093.stdout: python3-natsort-7.1.1-5.el9.noarch 2026-02-20T13:00:27.778 INFO:teuthology.orchestra.run.trial093.stdout: python3-numpy-1:1.23.5-2.el9.x86_64 2026-02-20T13:00:27.778 INFO:teuthology.orchestra.run.trial093.stdout: python3-numpy-f2py-1:1.23.5-2.el9.x86_64 2026-02-20T13:00:27.778 INFO:teuthology.orchestra.run.trial093.stdout: python3-packaging-20.9-5.el9.noarch 2026-02-20T13:00:27.778 INFO:teuthology.orchestra.run.trial093.stdout: python3-ply-3.11-14.el9.noarch 2026-02-20T13:00:27.778 INFO:teuthology.orchestra.run.trial093.stdout: python3-portend-3.1.0-2.el9.noarch 2026-02-20T13:00:27.778 INFO:teuthology.orchestra.run.trial093.stdout: python3-protobuf-3.14.0-17.el9.noarch 2026-02-20T13:00:27.779 INFO:teuthology.orchestra.run.trial093.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2026-02-20T13:00:27.779 INFO:teuthology.orchestra.run.trial093.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2026-02-20T13:00:27.779 INFO:teuthology.orchestra.run.trial093.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2026-02-20T13:00:27.779 INFO:teuthology.orchestra.run.trial093.stdout: python3-pycparser-2.20-6.el9.noarch 2026-02-20T13:00:27.779 INFO:teuthology.orchestra.run.trial093.stdout: python3-pyparsing-2.4.7-9.el9.noarch 2026-02-20T13:00:27.779 INFO:teuthology.orchestra.run.trial093.stdout: python3-repoze-lru-0.7-16.el9.noarch 2026-02-20T13:00:27.779 INFO:teuthology.orchestra.run.trial093.stdout: python3-requests-2.25.1-10.el9.noarch 2026-02-20T13:00:27.780 INFO:teuthology.orchestra.run.trial093.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2026-02-20T13:00:27.780 INFO:teuthology.orchestra.run.trial093.stdout: python3-routes-2.5.1-5.el9.noarch 2026-02-20T13:00:27.780 INFO:teuthology.orchestra.run.trial093.stdout: python3-rsa-4.9-2.el9.noarch 2026-02-20T13:00:27.780 INFO:teuthology.orchestra.run.trial093.stdout: python3-saml-1.16.0-1.el9.noarch 2026-02-20T13:00:27.780 INFO:teuthology.orchestra.run.trial093.stdout: python3-scipy-1.9.3-2.el9.x86_64 2026-02-20T13:00:27.780 INFO:teuthology.orchestra.run.trial093.stdout: python3-tempora-5.0.0-2.el9.noarch 2026-02-20T13:00:27.780 INFO:teuthology.orchestra.run.trial093.stdout: python3-toml-0.10.2-6.el9.noarch 2026-02-20T13:00:27.780 INFO:teuthology.orchestra.run.trial093.stdout: python3-typing-extensions-4.15.0-1.el9.noarch 2026-02-20T13:00:27.780 INFO:teuthology.orchestra.run.trial093.stdout: python3-urllib3-1.26.5-7.el9.noarch 2026-02-20T13:00:27.780 INFO:teuthology.orchestra.run.trial093.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2026-02-20T13:00:27.780 INFO:teuthology.orchestra.run.trial093.stdout: python3-xmlsec-1.3.13-1.el9.x86_64 2026-02-20T13:00:27.780 INFO:teuthology.orchestra.run.trial093.stdout: python3-xmltodict-0.12.0-15.el9.noarch 2026-02-20T13:00:27.780 INFO:teuthology.orchestra.run.trial093.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2026-02-20T13:00:27.781 INFO:teuthology.orchestra.run.trial093.stdout: unzip-6.0-59.el9.x86_64 2026-02-20T13:00:27.781 INFO:teuthology.orchestra.run.trial093.stdout: xmlsec1-1.2.29-13.el9.x86_64 2026-02-20T13:00:27.781 INFO:teuthology.orchestra.run.trial093.stdout: xmlsec1-openssl-1.2.29-13.el9.x86_64 2026-02-20T13:00:27.781 INFO:teuthology.orchestra.run.trial093.stdout: zip-3.0-35.el9.x86_64 2026-02-20T13:00:27.781 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:00:27.781 INFO:teuthology.orchestra.run.trial093.stdout:Complete! 2026-02-20T13:00:27.844 DEBUG:teuthology.orchestra.run.trial093:> sudo yum -y install ceph-base 2026-02-20T13:00:28.156 INFO:teuthology.orchestra.run.trial093.stdout:Last metadata expiration check: 0:02:38 ago on Fri 20 Feb 2026 12:57:50 PM UTC. 2026-02-20T13:00:28.220 INFO:teuthology.orchestra.run.trial093.stdout:Package ceph-base-2:20.3.0-5312.g26bc9765.el9.x86_64 is already installed. 2026-02-20T13:00:28.238 INFO:teuthology.orchestra.run.trial093.stdout:Dependencies resolved. 2026-02-20T13:00:28.239 INFO:teuthology.orchestra.run.trial093.stdout:Nothing to do. 2026-02-20T13:00:28.239 INFO:teuthology.orchestra.run.trial093.stdout:Complete! 2026-02-20T13:00:28.262 DEBUG:teuthology.orchestra.run.trial093:> sudo yum -y install cephadm 2026-02-20T13:00:28.576 INFO:teuthology.orchestra.run.trial093.stdout:Last metadata expiration check: 0:02:38 ago on Fri 20 Feb 2026 12:57:50 PM UTC. 2026-02-20T13:00:28.640 INFO:teuthology.orchestra.run.trial093.stdout:Package cephadm-2:20.3.0-5312.g26bc9765.el9.noarch is already installed. 2026-02-20T13:00:28.658 INFO:teuthology.orchestra.run.trial093.stdout:Dependencies resolved. 2026-02-20T13:00:28.659 INFO:teuthology.orchestra.run.trial093.stdout:Nothing to do. 2026-02-20T13:00:28.659 INFO:teuthology.orchestra.run.trial093.stdout:Complete! 2026-02-20T13:00:28.681 DEBUG:teuthology.orchestra.run.trial093:> sudo yum -y install ceph-immutable-object-cache 2026-02-20T13:00:28.982 INFO:teuthology.orchestra.run.trial093.stdout:Last metadata expiration check: 0:02:38 ago on Fri 20 Feb 2026 12:57:50 PM UTC. 2026-02-20T13:00:29.065 INFO:teuthology.orchestra.run.trial093.stdout:Dependencies resolved. 2026-02-20T13:00:29.066 INFO:teuthology.orchestra.run.trial093.stdout:================================================================================ 2026-02-20T13:00:29.066 INFO:teuthology.orchestra.run.trial093.stdout: Package Arch Version Repo Size 2026-02-20T13:00:29.066 INFO:teuthology.orchestra.run.trial093.stdout:================================================================================ 2026-02-20T13:00:29.066 INFO:teuthology.orchestra.run.trial093.stdout:Installing: 2026-02-20T13:00:29.066 INFO:teuthology.orchestra.run.trial093.stdout: ceph-immutable-object-cache x86_64 2:20.3.0-5312.g26bc9765.el9 ceph 149 k 2026-02-20T13:00:29.066 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:00:29.066 INFO:teuthology.orchestra.run.trial093.stdout:Transaction Summary 2026-02-20T13:00:29.066 INFO:teuthology.orchestra.run.trial093.stdout:================================================================================ 2026-02-20T13:00:29.067 INFO:teuthology.orchestra.run.trial093.stdout:Install 1 Package 2026-02-20T13:00:29.067 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:00:29.067 INFO:teuthology.orchestra.run.trial093.stdout:Total download size: 149 k 2026-02-20T13:00:29.067 INFO:teuthology.orchestra.run.trial093.stdout:Installed size: 455 k 2026-02-20T13:00:29.067 INFO:teuthology.orchestra.run.trial093.stdout:Downloading Packages: 2026-02-20T13:00:29.522 INFO:teuthology.orchestra.run.trial093.stdout:ceph-immutable-object-cache-20.3.0-5312.g26bc97 327 kB/s | 149 kB 00:00 2026-02-20T13:00:29.523 INFO:teuthology.orchestra.run.trial093.stdout:-------------------------------------------------------------------------------- 2026-02-20T13:00:29.523 INFO:teuthology.orchestra.run.trial093.stdout:Total 327 kB/s | 149 kB 00:00 2026-02-20T13:00:29.523 INFO:teuthology.orchestra.run.trial093.stdout:Running transaction check 2026-02-20T13:00:29.527 INFO:teuthology.orchestra.run.trial093.stdout:Transaction check succeeded. 2026-02-20T13:00:29.527 INFO:teuthology.orchestra.run.trial093.stdout:Running transaction test 2026-02-20T13:00:29.559 INFO:teuthology.orchestra.run.trial093.stdout:Transaction test succeeded. 2026-02-20T13:00:29.559 INFO:teuthology.orchestra.run.trial093.stdout:Running transaction 2026-02-20T13:00:29.614 INFO:teuthology.orchestra.run.trial093.stdout: Preparing : 1/1 2026-02-20T13:00:29.615 INFO:teuthology.orchestra.run.trial093.stdout: Installing : ceph-immutable-object-cache-2:20.3.0-5312.g26bc9765. 1/1 2026-02-20T13:00:29.637 INFO:teuthology.orchestra.run.trial093.stdout: Running scriptlet: ceph-immutable-object-cache-2:20.3.0-5312.g26bc9765. 1/1 2026-02-20T13:00:29.637 INFO:teuthology.orchestra.run.trial093.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-20T13:00:29.637 INFO:teuthology.orchestra.run.trial093.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2026-02-20T13:00:29.638 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:00:29.947 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : ceph-immutable-object-cache-2:20.3.0-5312.g26bc9765. 1/1 2026-02-20T13:00:29.947 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:00:29.947 INFO:teuthology.orchestra.run.trial093.stdout:Installed: 2026-02-20T13:00:29.947 INFO:teuthology.orchestra.run.trial093.stdout: ceph-immutable-object-cache-2:20.3.0-5312.g26bc9765.el9.x86_64 2026-02-20T13:00:29.947 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:00:29.947 INFO:teuthology.orchestra.run.trial093.stdout:Complete! 2026-02-20T13:00:29.974 DEBUG:teuthology.orchestra.run.trial093:> sudo yum -y install ceph-mgr 2026-02-20T13:00:30.291 INFO:teuthology.orchestra.run.trial093.stdout:Last metadata expiration check: 0:02:40 ago on Fri 20 Feb 2026 12:57:50 PM UTC. 2026-02-20T13:00:30.356 INFO:teuthology.orchestra.run.trial093.stdout:Package ceph-mgr-2:20.3.0-5312.g26bc9765.el9.x86_64 is already installed. 2026-02-20T13:00:30.374 INFO:teuthology.orchestra.run.trial093.stdout:Dependencies resolved. 2026-02-20T13:00:30.375 INFO:teuthology.orchestra.run.trial093.stdout:Nothing to do. 2026-02-20T13:00:30.375 INFO:teuthology.orchestra.run.trial093.stdout:Complete! 2026-02-20T13:00:30.397 DEBUG:teuthology.orchestra.run.trial093:> sudo yum -y install ceph-mgr-dashboard 2026-02-20T13:00:30.705 INFO:teuthology.orchestra.run.trial093.stdout:Last metadata expiration check: 0:02:40 ago on Fri 20 Feb 2026 12:57:50 PM UTC. 2026-02-20T13:00:30.769 INFO:teuthology.orchestra.run.trial093.stdout:Package ceph-mgr-dashboard-2:20.3.0-5312.g26bc9765.el9.noarch is already installed. 2026-02-20T13:00:30.788 INFO:teuthology.orchestra.run.trial093.stdout:Dependencies resolved. 2026-02-20T13:00:30.788 INFO:teuthology.orchestra.run.trial093.stdout:Nothing to do. 2026-02-20T13:00:30.788 INFO:teuthology.orchestra.run.trial093.stdout:Complete! 2026-02-20T13:00:30.810 DEBUG:teuthology.orchestra.run.trial093:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-20T13:00:31.122 INFO:teuthology.orchestra.run.trial093.stdout:Last metadata expiration check: 0:02:41 ago on Fri 20 Feb 2026 12:57:50 PM UTC. 2026-02-20T13:00:31.186 INFO:teuthology.orchestra.run.trial093.stdout:Package ceph-mgr-diskprediction-local-2:20.3.0-5312.g26bc9765.el9.noarch is already installed. 2026-02-20T13:00:31.205 INFO:teuthology.orchestra.run.trial093.stdout:Dependencies resolved. 2026-02-20T13:00:31.205 INFO:teuthology.orchestra.run.trial093.stdout:Nothing to do. 2026-02-20T13:00:31.206 INFO:teuthology.orchestra.run.trial093.stdout:Complete! 2026-02-20T13:00:31.227 DEBUG:teuthology.orchestra.run.trial093:> sudo yum -y install ceph-mgr-rook 2026-02-20T13:00:31.530 INFO:teuthology.orchestra.run.trial093.stdout:Last metadata expiration check: 0:02:41 ago on Fri 20 Feb 2026 12:57:50 PM UTC. 2026-02-20T13:00:31.596 INFO:teuthology.orchestra.run.trial093.stdout:Package ceph-mgr-rook-2:20.3.0-5312.g26bc9765.el9.noarch is already installed. 2026-02-20T13:00:31.615 INFO:teuthology.orchestra.run.trial093.stdout:Dependencies resolved. 2026-02-20T13:00:31.615 INFO:teuthology.orchestra.run.trial093.stdout:Nothing to do. 2026-02-20T13:00:31.615 INFO:teuthology.orchestra.run.trial093.stdout:Complete! 2026-02-20T13:00:31.636 DEBUG:teuthology.orchestra.run.trial093:> sudo yum -y install ceph-mgr-cephadm 2026-02-20T13:00:31.941 INFO:teuthology.orchestra.run.trial093.stdout:Last metadata expiration check: 0:02:41 ago on Fri 20 Feb 2026 12:57:50 PM UTC. 2026-02-20T13:00:32.005 INFO:teuthology.orchestra.run.trial093.stdout:Package ceph-mgr-cephadm-2:20.3.0-5312.g26bc9765.el9.noarch is already installed. 2026-02-20T13:00:32.023 INFO:teuthology.orchestra.run.trial093.stdout:Dependencies resolved. 2026-02-20T13:00:32.023 INFO:teuthology.orchestra.run.trial093.stdout:Nothing to do. 2026-02-20T13:00:32.024 INFO:teuthology.orchestra.run.trial093.stdout:Complete! 2026-02-20T13:00:32.044 DEBUG:teuthology.orchestra.run.trial093:> sudo yum -y install ceph-osd 2026-02-20T13:00:32.353 INFO:teuthology.orchestra.run.trial093.stdout:Last metadata expiration check: 0:02:42 ago on Fri 20 Feb 2026 12:57:50 PM UTC. 2026-02-20T13:00:32.417 INFO:teuthology.orchestra.run.trial093.stdout:Package ceph-osd-2:20.3.0-5312.g26bc9765.el9.x86_64 is already installed. 2026-02-20T13:00:32.436 INFO:teuthology.orchestra.run.trial093.stdout:Dependencies resolved. 2026-02-20T13:00:32.436 INFO:teuthology.orchestra.run.trial093.stdout:Nothing to do. 2026-02-20T13:00:32.436 INFO:teuthology.orchestra.run.trial093.stdout:Complete! 2026-02-20T13:00:32.457 DEBUG:teuthology.orchestra.run.trial093:> sudo yum -y install ceph-osd-classic 2026-02-20T13:00:32.762 INFO:teuthology.orchestra.run.trial093.stdout:Last metadata expiration check: 0:02:42 ago on Fri 20 Feb 2026 12:57:50 PM UTC. 2026-02-20T13:00:32.825 INFO:teuthology.orchestra.run.trial093.stdout:Package ceph-osd-classic-2:20.3.0-5312.g26bc9765.el9.x86_64 is already installed. 2026-02-20T13:00:32.843 INFO:teuthology.orchestra.run.trial093.stdout:Dependencies resolved. 2026-02-20T13:00:32.844 INFO:teuthology.orchestra.run.trial093.stdout:Nothing to do. 2026-02-20T13:00:32.844 INFO:teuthology.orchestra.run.trial093.stdout:Complete! 2026-02-20T13:00:32.864 DEBUG:teuthology.orchestra.run.trial093:> sudo yum -y install ceph-fuse 2026-02-20T13:00:33.176 INFO:teuthology.orchestra.run.trial093.stdout:Last metadata expiration check: 0:02:43 ago on Fri 20 Feb 2026 12:57:50 PM UTC. 2026-02-20T13:00:33.258 INFO:teuthology.orchestra.run.trial093.stdout:Dependencies resolved. 2026-02-20T13:00:33.259 INFO:teuthology.orchestra.run.trial093.stdout:================================================================================ 2026-02-20T13:00:33.259 INFO:teuthology.orchestra.run.trial093.stdout: Package Arch Version Repo Size 2026-02-20T13:00:33.259 INFO:teuthology.orchestra.run.trial093.stdout:================================================================================ 2026-02-20T13:00:33.260 INFO:teuthology.orchestra.run.trial093.stdout:Installing: 2026-02-20T13:00:33.260 INFO:teuthology.orchestra.run.trial093.stdout: ceph-fuse x86_64 2:20.3.0-5312.g26bc9765.el9 ceph 914 k 2026-02-20T13:00:33.260 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:00:33.260 INFO:teuthology.orchestra.run.trial093.stdout:Transaction Summary 2026-02-20T13:00:33.260 INFO:teuthology.orchestra.run.trial093.stdout:================================================================================ 2026-02-20T13:00:33.260 INFO:teuthology.orchestra.run.trial093.stdout:Install 1 Package 2026-02-20T13:00:33.260 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:00:33.260 INFO:teuthology.orchestra.run.trial093.stdout:Total download size: 914 k 2026-02-20T13:00:33.260 INFO:teuthology.orchestra.run.trial093.stdout:Installed size: 2.9 M 2026-02-20T13:00:33.260 INFO:teuthology.orchestra.run.trial093.stdout:Downloading Packages: 2026-02-20T13:00:33.818 INFO:teuthology.orchestra.run.trial093.stdout:ceph-fuse-20.3.0-5312.g26bc9765.el9.x86_64.rpm 1.6 MB/s | 914 kB 00:00 2026-02-20T13:00:33.820 INFO:teuthology.orchestra.run.trial093.stdout:-------------------------------------------------------------------------------- 2026-02-20T13:00:33.820 INFO:teuthology.orchestra.run.trial093.stdout:Total 1.6 MB/s | 914 kB 00:00 2026-02-20T13:00:33.820 INFO:teuthology.orchestra.run.trial093.stdout:Running transaction check 2026-02-20T13:00:33.825 INFO:teuthology.orchestra.run.trial093.stdout:Transaction check succeeded. 2026-02-20T13:00:33.825 INFO:teuthology.orchestra.run.trial093.stdout:Running transaction test 2026-02-20T13:00:33.858 INFO:teuthology.orchestra.run.trial093.stdout:Transaction test succeeded. 2026-02-20T13:00:33.859 INFO:teuthology.orchestra.run.trial093.stdout:Running transaction 2026-02-20T13:00:33.950 INFO:teuthology.orchestra.run.trial093.stdout: Preparing : 1/1 2026-02-20T13:00:33.967 INFO:teuthology.orchestra.run.trial093.stdout: Installing : ceph-fuse-2:20.3.0-5312.g26bc9765.el9.x86_64 1/1 2026-02-20T13:00:34.222 INFO:teuthology.orchestra.run.trial093.stdout: Running scriptlet: ceph-fuse-2:20.3.0-5312.g26bc9765.el9.x86_64 1/1 2026-02-20T13:00:34.281 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : ceph-fuse-2:20.3.0-5312.g26bc9765.el9.x86_64 1/1 2026-02-20T13:00:34.281 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:00:34.281 INFO:teuthology.orchestra.run.trial093.stdout:Installed: 2026-02-20T13:00:34.281 INFO:teuthology.orchestra.run.trial093.stdout: ceph-fuse-2:20.3.0-5312.g26bc9765.el9.x86_64 2026-02-20T13:00:34.281 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:00:34.281 INFO:teuthology.orchestra.run.trial093.stdout:Complete! 2026-02-20T13:00:34.309 DEBUG:teuthology.orchestra.run.trial093:> sudo yum -y install ceph-volume 2026-02-20T13:00:34.621 INFO:teuthology.orchestra.run.trial093.stdout:Last metadata expiration check: 0:02:44 ago on Fri 20 Feb 2026 12:57:50 PM UTC. 2026-02-20T13:00:34.685 INFO:teuthology.orchestra.run.trial093.stdout:Package ceph-volume-2:20.3.0-5312.g26bc9765.el9.noarch is already installed. 2026-02-20T13:00:34.704 INFO:teuthology.orchestra.run.trial093.stdout:Dependencies resolved. 2026-02-20T13:00:34.704 INFO:teuthology.orchestra.run.trial093.stdout:Nothing to do. 2026-02-20T13:00:34.704 INFO:teuthology.orchestra.run.trial093.stdout:Complete! 2026-02-20T13:00:34.725 DEBUG:teuthology.orchestra.run.trial093:> sudo yum -y install librados-devel 2026-02-20T13:00:35.039 INFO:teuthology.orchestra.run.trial093.stdout:Last metadata expiration check: 0:02:45 ago on Fri 20 Feb 2026 12:57:50 PM UTC. 2026-02-20T13:00:35.123 INFO:teuthology.orchestra.run.trial093.stdout:Dependencies resolved. 2026-02-20T13:00:35.123 INFO:teuthology.orchestra.run.trial093.stdout:================================================================================ 2026-02-20T13:00:35.123 INFO:teuthology.orchestra.run.trial093.stdout: Package Arch Version Repo Size 2026-02-20T13:00:35.124 INFO:teuthology.orchestra.run.trial093.stdout:================================================================================ 2026-02-20T13:00:35.124 INFO:teuthology.orchestra.run.trial093.stdout:Installing: 2026-02-20T13:00:35.124 INFO:teuthology.orchestra.run.trial093.stdout: librados-devel x86_64 2:20.3.0-5312.g26bc9765.el9 ceph 124 k 2026-02-20T13:00:35.124 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:00:35.124 INFO:teuthology.orchestra.run.trial093.stdout:Transaction Summary 2026-02-20T13:00:35.125 INFO:teuthology.orchestra.run.trial093.stdout:================================================================================ 2026-02-20T13:00:35.125 INFO:teuthology.orchestra.run.trial093.stdout:Install 1 Package 2026-02-20T13:00:35.125 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:00:35.126 INFO:teuthology.orchestra.run.trial093.stdout:Total download size: 124 k 2026-02-20T13:00:35.126 INFO:teuthology.orchestra.run.trial093.stdout:Installed size: 434 k 2026-02-20T13:00:35.126 INFO:teuthology.orchestra.run.trial093.stdout:Downloading Packages: 2026-02-20T13:00:35.545 INFO:teuthology.orchestra.run.trial093.stdout:librados-devel-20.3.0-5312.g26bc9765.el9.x86_64 293 kB/s | 124 kB 00:00 2026-02-20T13:00:35.545 INFO:teuthology.orchestra.run.trial093.stdout:-------------------------------------------------------------------------------- 2026-02-20T13:00:35.546 INFO:teuthology.orchestra.run.trial093.stdout:Total 292 kB/s | 124 kB 00:00 2026-02-20T13:00:35.546 INFO:teuthology.orchestra.run.trial093.stdout:Running transaction check 2026-02-20T13:00:35.548 INFO:teuthology.orchestra.run.trial093.stdout:Transaction check succeeded. 2026-02-20T13:00:35.548 INFO:teuthology.orchestra.run.trial093.stdout:Running transaction test 2026-02-20T13:00:35.581 INFO:teuthology.orchestra.run.trial093.stdout:Transaction test succeeded. 2026-02-20T13:00:35.581 INFO:teuthology.orchestra.run.trial093.stdout:Running transaction 2026-02-20T13:00:35.635 INFO:teuthology.orchestra.run.trial093.stdout: Preparing : 1/1 2026-02-20T13:00:35.652 INFO:teuthology.orchestra.run.trial093.stdout: Installing : librados-devel-2:20.3.0-5312.g26bc9765.el9.x86_64 1/1 2026-02-20T13:00:35.734 INFO:teuthology.orchestra.run.trial093.stdout: Running scriptlet: librados-devel-2:20.3.0-5312.g26bc9765.el9.x86_64 1/1 2026-02-20T13:00:35.793 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : librados-devel-2:20.3.0-5312.g26bc9765.el9.x86_64 1/1 2026-02-20T13:00:35.794 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:00:35.794 INFO:teuthology.orchestra.run.trial093.stdout:Installed: 2026-02-20T13:00:35.794 INFO:teuthology.orchestra.run.trial093.stdout: librados-devel-2:20.3.0-5312.g26bc9765.el9.x86_64 2026-02-20T13:00:35.794 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:00:35.794 INFO:teuthology.orchestra.run.trial093.stdout:Complete! 2026-02-20T13:00:35.822 DEBUG:teuthology.orchestra.run.trial093:> sudo yum -y install libcephfs2 2026-02-20T13:00:36.131 INFO:teuthology.orchestra.run.trial093.stdout:Last metadata expiration check: 0:02:46 ago on Fri 20 Feb 2026 12:57:50 PM UTC. 2026-02-20T13:00:36.196 INFO:teuthology.orchestra.run.trial093.stdout:Package libcephfs2-2:20.3.0-5312.g26bc9765.el9.x86_64 is already installed. 2026-02-20T13:00:36.214 INFO:teuthology.orchestra.run.trial093.stdout:Dependencies resolved. 2026-02-20T13:00:36.215 INFO:teuthology.orchestra.run.trial093.stdout:Nothing to do. 2026-02-20T13:00:36.215 INFO:teuthology.orchestra.run.trial093.stdout:Complete! 2026-02-20T13:00:36.236 DEBUG:teuthology.orchestra.run.trial093:> sudo yum -y install libcephfs-devel 2026-02-20T13:00:36.549 INFO:teuthology.orchestra.run.trial093.stdout:Last metadata expiration check: 0:02:46 ago on Fri 20 Feb 2026 12:57:50 PM UTC. 2026-02-20T13:00:36.632 INFO:teuthology.orchestra.run.trial093.stdout:Dependencies resolved. 2026-02-20T13:00:36.632 INFO:teuthology.orchestra.run.trial093.stdout:================================================================================ 2026-02-20T13:00:36.632 INFO:teuthology.orchestra.run.trial093.stdout: Package Arch Version Repo Size 2026-02-20T13:00:36.633 INFO:teuthology.orchestra.run.trial093.stdout:================================================================================ 2026-02-20T13:00:36.633 INFO:teuthology.orchestra.run.trial093.stdout:Installing: 2026-02-20T13:00:36.633 INFO:teuthology.orchestra.run.trial093.stdout: libcephfs-devel x86_64 2:20.3.0-5312.g26bc9765.el9 ceph 37 k 2026-02-20T13:00:36.633 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:00:36.633 INFO:teuthology.orchestra.run.trial093.stdout:Transaction Summary 2026-02-20T13:00:36.633 INFO:teuthology.orchestra.run.trial093.stdout:================================================================================ 2026-02-20T13:00:36.633 INFO:teuthology.orchestra.run.trial093.stdout:Install 1 Package 2026-02-20T13:00:36.633 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:00:36.633 INFO:teuthology.orchestra.run.trial093.stdout:Total download size: 37 k 2026-02-20T13:00:36.633 INFO:teuthology.orchestra.run.trial093.stdout:Installed size: 171 k 2026-02-20T13:00:36.633 INFO:teuthology.orchestra.run.trial093.stdout:Downloading Packages: 2026-02-20T13:00:36.957 INFO:teuthology.orchestra.run.trial093.stdout:libcephfs-devel-20.3.0-5312.g26bc9765.el9.x86_6 115 kB/s | 37 kB 00:00 2026-02-20T13:00:36.957 INFO:teuthology.orchestra.run.trial093.stdout:-------------------------------------------------------------------------------- 2026-02-20T13:00:36.957 INFO:teuthology.orchestra.run.trial093.stdout:Total 115 kB/s | 37 kB 00:00 2026-02-20T13:00:36.957 INFO:teuthology.orchestra.run.trial093.stdout:Running transaction check 2026-02-20T13:00:36.960 INFO:teuthology.orchestra.run.trial093.stdout:Transaction check succeeded. 2026-02-20T13:00:36.960 INFO:teuthology.orchestra.run.trial093.stdout:Running transaction test 2026-02-20T13:00:36.962 INFO:teuthology.orchestra.run.trial093.stdout:Transaction test succeeded. 2026-02-20T13:00:36.962 INFO:teuthology.orchestra.run.trial093.stdout:Running transaction 2026-02-20T13:00:36.984 INFO:teuthology.orchestra.run.trial093.stdout: Preparing : 1/1 2026-02-20T13:00:37.000 INFO:teuthology.orchestra.run.trial093.stdout: Installing : libcephfs-devel-2:20.3.0-5312.g26bc9765.el9.x86_64 1/1 2026-02-20T13:00:37.062 INFO:teuthology.orchestra.run.trial093.stdout: Running scriptlet: libcephfs-devel-2:20.3.0-5312.g26bc9765.el9.x86_64 1/1 2026-02-20T13:00:37.120 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : libcephfs-devel-2:20.3.0-5312.g26bc9765.el9.x86_64 1/1 2026-02-20T13:00:37.120 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:00:37.120 INFO:teuthology.orchestra.run.trial093.stdout:Installed: 2026-02-20T13:00:37.120 INFO:teuthology.orchestra.run.trial093.stdout: libcephfs-devel-2:20.3.0-5312.g26bc9765.el9.x86_64 2026-02-20T13:00:37.121 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:00:37.121 INFO:teuthology.orchestra.run.trial093.stdout:Complete! 2026-02-20T13:00:37.147 DEBUG:teuthology.orchestra.run.trial093:> sudo yum -y install librados2 2026-02-20T13:00:37.459 INFO:teuthology.orchestra.run.trial093.stdout:Last metadata expiration check: 0:02:47 ago on Fri 20 Feb 2026 12:57:50 PM UTC. 2026-02-20T13:00:37.523 INFO:teuthology.orchestra.run.trial093.stdout:Package librados2-2:20.3.0-5312.g26bc9765.el9.x86_64 is already installed. 2026-02-20T13:00:37.542 INFO:teuthology.orchestra.run.trial093.stdout:Dependencies resolved. 2026-02-20T13:00:37.542 INFO:teuthology.orchestra.run.trial093.stdout:Nothing to do. 2026-02-20T13:00:37.543 INFO:teuthology.orchestra.run.trial093.stdout:Complete! 2026-02-20T13:00:37.564 DEBUG:teuthology.orchestra.run.trial093:> sudo yum -y install librbd1 2026-02-20T13:00:37.873 INFO:teuthology.orchestra.run.trial093.stdout:Last metadata expiration check: 0:02:47 ago on Fri 20 Feb 2026 12:57:50 PM UTC. 2026-02-20T13:00:37.939 INFO:teuthology.orchestra.run.trial093.stdout:Package librbd1-2:20.3.0-5312.g26bc9765.el9.x86_64 is already installed. 2026-02-20T13:00:37.957 INFO:teuthology.orchestra.run.trial093.stdout:Dependencies resolved. 2026-02-20T13:00:37.958 INFO:teuthology.orchestra.run.trial093.stdout:Nothing to do. 2026-02-20T13:00:37.958 INFO:teuthology.orchestra.run.trial093.stdout:Complete! 2026-02-20T13:00:37.980 DEBUG:teuthology.orchestra.run.trial093:> sudo yum -y install python3-rados 2026-02-20T13:00:38.288 INFO:teuthology.orchestra.run.trial093.stdout:Last metadata expiration check: 0:02:48 ago on Fri 20 Feb 2026 12:57:50 PM UTC. 2026-02-20T13:00:38.353 INFO:teuthology.orchestra.run.trial093.stdout:Package python3-rados-2:20.3.0-5312.g26bc9765.el9.x86_64 is already installed. 2026-02-20T13:00:38.372 INFO:teuthology.orchestra.run.trial093.stdout:Dependencies resolved. 2026-02-20T13:00:38.373 INFO:teuthology.orchestra.run.trial093.stdout:Nothing to do. 2026-02-20T13:00:38.373 INFO:teuthology.orchestra.run.trial093.stdout:Complete! 2026-02-20T13:00:38.395 DEBUG:teuthology.orchestra.run.trial093:> sudo yum -y install python3-rgw 2026-02-20T13:00:38.711 INFO:teuthology.orchestra.run.trial093.stdout:Last metadata expiration check: 0:02:48 ago on Fri 20 Feb 2026 12:57:50 PM UTC. 2026-02-20T13:00:38.775 INFO:teuthology.orchestra.run.trial093.stdout:Package python3-rgw-2:20.3.0-5312.g26bc9765.el9.x86_64 is already installed. 2026-02-20T13:00:38.794 INFO:teuthology.orchestra.run.trial093.stdout:Dependencies resolved. 2026-02-20T13:00:38.794 INFO:teuthology.orchestra.run.trial093.stdout:Nothing to do. 2026-02-20T13:00:38.794 INFO:teuthology.orchestra.run.trial093.stdout:Complete! 2026-02-20T13:00:38.816 DEBUG:teuthology.orchestra.run.trial093:> sudo yum -y install python3-cephfs 2026-02-20T13:00:39.129 INFO:teuthology.orchestra.run.trial093.stdout:Last metadata expiration check: 0:02:49 ago on Fri 20 Feb 2026 12:57:50 PM UTC. 2026-02-20T13:00:39.193 INFO:teuthology.orchestra.run.trial093.stdout:Package python3-cephfs-2:20.3.0-5312.g26bc9765.el9.x86_64 is already installed. 2026-02-20T13:00:39.211 INFO:teuthology.orchestra.run.trial093.stdout:Dependencies resolved. 2026-02-20T13:00:39.212 INFO:teuthology.orchestra.run.trial093.stdout:Nothing to do. 2026-02-20T13:00:39.212 INFO:teuthology.orchestra.run.trial093.stdout:Complete! 2026-02-20T13:00:39.234 DEBUG:teuthology.orchestra.run.trial093:> sudo yum -y install python3-rbd 2026-02-20T13:00:39.542 INFO:teuthology.orchestra.run.trial093.stdout:Last metadata expiration check: 0:02:49 ago on Fri 20 Feb 2026 12:57:50 PM UTC. 2026-02-20T13:00:39.607 INFO:teuthology.orchestra.run.trial093.stdout:Package python3-rbd-2:20.3.0-5312.g26bc9765.el9.x86_64 is already installed. 2026-02-20T13:00:39.625 INFO:teuthology.orchestra.run.trial093.stdout:Dependencies resolved. 2026-02-20T13:00:39.626 INFO:teuthology.orchestra.run.trial093.stdout:Nothing to do. 2026-02-20T13:00:39.626 INFO:teuthology.orchestra.run.trial093.stdout:Complete! 2026-02-20T13:00:39.647 DEBUG:teuthology.orchestra.run.trial093:> sudo yum -y install rbd-fuse 2026-02-20T13:00:39.959 INFO:teuthology.orchestra.run.trial093.stdout:Last metadata expiration check: 0:02:49 ago on Fri 20 Feb 2026 12:57:50 PM UTC. 2026-02-20T13:00:40.042 INFO:teuthology.orchestra.run.trial093.stdout:Dependencies resolved. 2026-02-20T13:00:40.043 INFO:teuthology.orchestra.run.trial093.stdout:================================================================================ 2026-02-20T13:00:40.043 INFO:teuthology.orchestra.run.trial093.stdout: Package Architecture Version Repository Size 2026-02-20T13:00:40.043 INFO:teuthology.orchestra.run.trial093.stdout:================================================================================ 2026-02-20T13:00:40.043 INFO:teuthology.orchestra.run.trial093.stdout:Installing: 2026-02-20T13:00:40.043 INFO:teuthology.orchestra.run.trial093.stdout: rbd-fuse x86_64 2:20.3.0-5312.g26bc9765.el9 ceph 87 k 2026-02-20T13:00:40.043 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:00:40.043 INFO:teuthology.orchestra.run.trial093.stdout:Transaction Summary 2026-02-20T13:00:40.043 INFO:teuthology.orchestra.run.trial093.stdout:================================================================================ 2026-02-20T13:00:40.043 INFO:teuthology.orchestra.run.trial093.stdout:Install 1 Package 2026-02-20T13:00:40.043 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:00:40.044 INFO:teuthology.orchestra.run.trial093.stdout:Total download size: 87 k 2026-02-20T13:00:40.044 INFO:teuthology.orchestra.run.trial093.stdout:Installed size: 236 k 2026-02-20T13:00:40.044 INFO:teuthology.orchestra.run.trial093.stdout:Downloading Packages: 2026-02-20T13:00:40.433 INFO:teuthology.orchestra.run.trial093.stdout:rbd-fuse-20.3.0-5312.g26bc9765.el9.x86_64.rpm 222 kB/s | 87 kB 00:00 2026-02-20T13:00:40.434 INFO:teuthology.orchestra.run.trial093.stdout:-------------------------------------------------------------------------------- 2026-02-20T13:00:40.434 INFO:teuthology.orchestra.run.trial093.stdout:Total 221 kB/s | 87 kB 00:00 2026-02-20T13:00:40.434 INFO:teuthology.orchestra.run.trial093.stdout:Running transaction check 2026-02-20T13:00:40.438 INFO:teuthology.orchestra.run.trial093.stdout:Transaction check succeeded. 2026-02-20T13:00:40.438 INFO:teuthology.orchestra.run.trial093.stdout:Running transaction test 2026-02-20T13:00:40.469 INFO:teuthology.orchestra.run.trial093.stdout:Transaction test succeeded. 2026-02-20T13:00:40.469 INFO:teuthology.orchestra.run.trial093.stdout:Running transaction 2026-02-20T13:00:40.521 INFO:teuthology.orchestra.run.trial093.stdout: Preparing : 1/1 2026-02-20T13:00:40.537 INFO:teuthology.orchestra.run.trial093.stdout: Installing : rbd-fuse-2:20.3.0-5312.g26bc9765.el9.x86_64 1/1 2026-02-20T13:00:40.620 INFO:teuthology.orchestra.run.trial093.stdout: Running scriptlet: rbd-fuse-2:20.3.0-5312.g26bc9765.el9.x86_64 1/1 2026-02-20T13:00:40.680 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : rbd-fuse-2:20.3.0-5312.g26bc9765.el9.x86_64 1/1 2026-02-20T13:00:40.681 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:00:40.681 INFO:teuthology.orchestra.run.trial093.stdout:Installed: 2026-02-20T13:00:40.681 INFO:teuthology.orchestra.run.trial093.stdout: rbd-fuse-2:20.3.0-5312.g26bc9765.el9.x86_64 2026-02-20T13:00:40.681 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:00:40.681 INFO:teuthology.orchestra.run.trial093.stdout:Complete! 2026-02-20T13:00:40.709 DEBUG:teuthology.orchestra.run.trial093:> sudo yum -y install rbd-mirror 2026-02-20T13:00:41.043 INFO:teuthology.orchestra.run.trial093.stdout:Last metadata expiration check: 0:02:51 ago on Fri 20 Feb 2026 12:57:50 PM UTC. 2026-02-20T13:00:41.127 INFO:teuthology.orchestra.run.trial093.stdout:Dependencies resolved. 2026-02-20T13:00:41.128 INFO:teuthology.orchestra.run.trial093.stdout:================================================================================ 2026-02-20T13:00:41.128 INFO:teuthology.orchestra.run.trial093.stdout: Package Arch Version Repo Size 2026-02-20T13:00:41.128 INFO:teuthology.orchestra.run.trial093.stdout:================================================================================ 2026-02-20T13:00:41.128 INFO:teuthology.orchestra.run.trial093.stdout:Installing: 2026-02-20T13:00:41.128 INFO:teuthology.orchestra.run.trial093.stdout: rbd-mirror x86_64 2:20.3.0-5312.g26bc9765.el9 ceph 2.6 M 2026-02-20T13:00:41.128 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:00:41.128 INFO:teuthology.orchestra.run.trial093.stdout:Transaction Summary 2026-02-20T13:00:41.129 INFO:teuthology.orchestra.run.trial093.stdout:================================================================================ 2026-02-20T13:00:41.129 INFO:teuthology.orchestra.run.trial093.stdout:Install 1 Package 2026-02-20T13:00:41.129 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:00:41.129 INFO:teuthology.orchestra.run.trial093.stdout:Total download size: 2.6 M 2026-02-20T13:00:41.129 INFO:teuthology.orchestra.run.trial093.stdout:Installed size: 11 M 2026-02-20T13:00:41.129 INFO:teuthology.orchestra.run.trial093.stdout:Downloading Packages: 2026-02-20T13:00:41.996 INFO:teuthology.orchestra.run.trial093.stdout:rbd-mirror-20.3.0-5312.g26bc9765.el9.x86_64.rpm 3.0 MB/s | 2.6 MB 00:00 2026-02-20T13:00:41.996 INFO:teuthology.orchestra.run.trial093.stdout:-------------------------------------------------------------------------------- 2026-02-20T13:00:41.997 INFO:teuthology.orchestra.run.trial093.stdout:Total 3.0 MB/s | 2.6 MB 00:00 2026-02-20T13:00:41.997 INFO:teuthology.orchestra.run.trial093.stdout:Running transaction check 2026-02-20T13:00:42.001 INFO:teuthology.orchestra.run.trial093.stdout:Transaction check succeeded. 2026-02-20T13:00:42.001 INFO:teuthology.orchestra.run.trial093.stdout:Running transaction test 2026-02-20T13:00:42.041 INFO:teuthology.orchestra.run.trial093.stdout:Transaction test succeeded. 2026-02-20T13:00:42.041 INFO:teuthology.orchestra.run.trial093.stdout:Running transaction 2026-02-20T13:00:42.203 INFO:teuthology.orchestra.run.trial093.stdout: Preparing : 1/1 2026-02-20T13:00:42.205 INFO:teuthology.orchestra.run.trial093.stdout: Installing : rbd-mirror-2:20.3.0-5312.g26bc9765.el9.x86_64 1/1 2026-02-20T13:00:42.227 INFO:teuthology.orchestra.run.trial093.stdout: Running scriptlet: rbd-mirror-2:20.3.0-5312.g26bc9765.el9.x86_64 1/1 2026-02-20T13:00:42.227 INFO:teuthology.orchestra.run.trial093.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-20T13:00:42.227 INFO:teuthology.orchestra.run.trial093.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2026-02-20T13:00:42.228 INFO:teuthology.orchestra.run.trial093.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-20T13:00:42.228 INFO:teuthology.orchestra.run.trial093.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-20T13:00:42.228 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:00:42.558 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : rbd-mirror-2:20.3.0-5312.g26bc9765.el9.x86_64 1/1 2026-02-20T13:00:42.558 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:00:42.559 INFO:teuthology.orchestra.run.trial093.stdout:Installed: 2026-02-20T13:00:42.559 INFO:teuthology.orchestra.run.trial093.stdout: rbd-mirror-2:20.3.0-5312.g26bc9765.el9.x86_64 2026-02-20T13:00:42.559 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:00:42.559 INFO:teuthology.orchestra.run.trial093.stdout:Complete! 2026-02-20T13:00:42.588 DEBUG:teuthology.orchestra.run.trial093:> sudo yum -y install rbd-nbd 2026-02-20T13:00:42.901 INFO:teuthology.orchestra.run.trial093.stdout:Last metadata expiration check: 0:02:52 ago on Fri 20 Feb 2026 12:57:50 PM UTC. 2026-02-20T13:00:42.984 INFO:teuthology.orchestra.run.trial093.stdout:Dependencies resolved. 2026-02-20T13:00:42.984 INFO:teuthology.orchestra.run.trial093.stdout:================================================================================ 2026-02-20T13:00:42.984 INFO:teuthology.orchestra.run.trial093.stdout: Package Architecture Version Repository Size 2026-02-20T13:00:42.984 INFO:teuthology.orchestra.run.trial093.stdout:================================================================================ 2026-02-20T13:00:42.984 INFO:teuthology.orchestra.run.trial093.stdout:Installing: 2026-02-20T13:00:42.984 INFO:teuthology.orchestra.run.trial093.stdout: rbd-nbd x86_64 2:20.3.0-5312.g26bc9765.el9 ceph 166 k 2026-02-20T13:00:42.984 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:00:42.984 INFO:teuthology.orchestra.run.trial093.stdout:Transaction Summary 2026-02-20T13:00:42.984 INFO:teuthology.orchestra.run.trial093.stdout:================================================================================ 2026-02-20T13:00:42.984 INFO:teuthology.orchestra.run.trial093.stdout:Install 1 Package 2026-02-20T13:00:42.984 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:00:42.984 INFO:teuthology.orchestra.run.trial093.stdout:Total download size: 166 k 2026-02-20T13:00:42.984 INFO:teuthology.orchestra.run.trial093.stdout:Installed size: 496 k 2026-02-20T13:00:42.985 INFO:teuthology.orchestra.run.trial093.stdout:Downloading Packages: 2026-02-20T13:00:43.369 INFO:teuthology.orchestra.run.trial093.stdout:rbd-nbd-20.3.0-5312.g26bc9765.el9.x86_64.rpm 433 kB/s | 166 kB 00:00 2026-02-20T13:00:43.369 INFO:teuthology.orchestra.run.trial093.stdout:-------------------------------------------------------------------------------- 2026-02-20T13:00:43.369 INFO:teuthology.orchestra.run.trial093.stdout:Total 432 kB/s | 166 kB 00:00 2026-02-20T13:00:43.369 INFO:teuthology.orchestra.run.trial093.stdout:Running transaction check 2026-02-20T13:00:43.374 INFO:teuthology.orchestra.run.trial093.stdout:Transaction check succeeded. 2026-02-20T13:00:43.374 INFO:teuthology.orchestra.run.trial093.stdout:Running transaction test 2026-02-20T13:00:43.406 INFO:teuthology.orchestra.run.trial093.stdout:Transaction test succeeded. 2026-02-20T13:00:43.406 INFO:teuthology.orchestra.run.trial093.stdout:Running transaction 2026-02-20T13:00:43.463 INFO:teuthology.orchestra.run.trial093.stdout: Preparing : 1/1 2026-02-20T13:00:43.480 INFO:teuthology.orchestra.run.trial093.stdout: Installing : rbd-nbd-2:20.3.0-5312.g26bc9765.el9.x86_64 1/1 2026-02-20T13:00:43.563 INFO:teuthology.orchestra.run.trial093.stdout: Running scriptlet: rbd-nbd-2:20.3.0-5312.g26bc9765.el9.x86_64 1/1 2026-02-20T13:00:43.622 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : rbd-nbd-2:20.3.0-5312.g26bc9765.el9.x86_64 1/1 2026-02-20T13:00:43.622 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:00:43.622 INFO:teuthology.orchestra.run.trial093.stdout:Installed: 2026-02-20T13:00:43.622 INFO:teuthology.orchestra.run.trial093.stdout: rbd-nbd-2:20.3.0-5312.g26bc9765.el9.x86_64 2026-02-20T13:00:43.622 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:00:43.622 INFO:teuthology.orchestra.run.trial093.stdout:Complete! 2026-02-20T13:00:43.651 DEBUG:teuthology.orchestra.run.trial093:> sudo yum -y install ceph-osd-crimson 2026-02-20T13:00:43.964 INFO:teuthology.orchestra.run.trial093.stdout:Last metadata expiration check: 0:02:53 ago on Fri 20 Feb 2026 12:57:50 PM UTC. 2026-02-20T13:00:44.029 INFO:teuthology.orchestra.run.trial093.stdout:Package ceph-osd-crimson-2:20.3.0-5312.g26bc9765.el9.x86_64 is already installed. 2026-02-20T13:00:44.047 INFO:teuthology.orchestra.run.trial093.stdout:Dependencies resolved. 2026-02-20T13:00:44.048 INFO:teuthology.orchestra.run.trial093.stdout:Nothing to do. 2026-02-20T13:00:44.048 INFO:teuthology.orchestra.run.trial093.stdout:Complete! 2026-02-20T13:00:44.069 DEBUG:teuthology.orchestra.run.trial093:> sudo yum -y install ceph-osd-crimson-debuginfo 2026-02-20T13:00:44.380 INFO:teuthology.orchestra.run.trial093.stdout:Last metadata expiration check: 0:02:54 ago on Fri 20 Feb 2026 12:57:50 PM UTC. 2026-02-20T13:00:44.465 INFO:teuthology.orchestra.run.trial093.stdout:Dependencies resolved. 2026-02-20T13:00:44.466 INFO:teuthology.orchestra.run.trial093.stdout:================================================================================ 2026-02-20T13:00:44.466 INFO:teuthology.orchestra.run.trial093.stdout: Package Arch Version Repo Size 2026-02-20T13:00:44.466 INFO:teuthology.orchestra.run.trial093.stdout:================================================================================ 2026-02-20T13:00:44.466 INFO:teuthology.orchestra.run.trial093.stdout:Installing: 2026-02-20T13:00:44.466 INFO:teuthology.orchestra.run.trial093.stdout: ceph-osd-crimson-debuginfo x86_64 2:20.3.0-5312.g26bc9765.el9 ceph 425 M 2026-02-20T13:00:44.466 INFO:teuthology.orchestra.run.trial093.stdout:Installing weak dependencies: 2026-02-20T13:00:44.467 INFO:teuthology.orchestra.run.trial093.stdout: ceph-debugsource x86_64 2:20.3.0-5312.g26bc9765.el9 ceph 12 M 2026-02-20T13:00:44.467 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:00:44.467 INFO:teuthology.orchestra.run.trial093.stdout:Transaction Summary 2026-02-20T13:00:44.467 INFO:teuthology.orchestra.run.trial093.stdout:================================================================================ 2026-02-20T13:00:44.467 INFO:teuthology.orchestra.run.trial093.stdout:Install 2 Packages 2026-02-20T13:00:44.467 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:00:44.467 INFO:teuthology.orchestra.run.trial093.stdout:Total download size: 437 M 2026-02-20T13:00:44.467 INFO:teuthology.orchestra.run.trial093.stdout:Installed size: 2.6 G 2026-02-20T13:00:44.467 INFO:teuthology.orchestra.run.trial093.stdout:Downloading Packages: 2026-02-20T13:00:48.654 INFO:teuthology.orchestra.run.trial093.stdout:(1/2): ceph-debugsource-20.3.0-5312.g26bc9765.e 2.9 MB/s | 12 MB 00:04 2026-02-20T13:01:30.345 INFO:teuthology.orchestra.run.trial093.stdout:(2/2): ceph-osd-crimson-debuginfo-20.3.0-5312.g 9.3 MB/s | 425 MB 00:45 2026-02-20T13:01:30.346 INFO:teuthology.orchestra.run.trial093.stdout:-------------------------------------------------------------------------------- 2026-02-20T13:01:30.346 INFO:teuthology.orchestra.run.trial093.stdout:Total 9.5 MB/s | 437 MB 00:45 2026-02-20T13:01:30.346 INFO:teuthology.orchestra.run.trial093.stdout:Running transaction check 2026-02-20T13:01:30.352 INFO:teuthology.orchestra.run.trial093.stdout:Transaction check succeeded. 2026-02-20T13:01:30.352 INFO:teuthology.orchestra.run.trial093.stdout:Running transaction test 2026-02-20T13:01:31.700 INFO:teuthology.orchestra.run.trial093.stdout:Transaction test succeeded. 2026-02-20T13:01:31.701 INFO:teuthology.orchestra.run.trial093.stdout:Running transaction 2026-02-20T13:01:35.264 INFO:teuthology.orchestra.run.trial093.stdout: Preparing : 1/1 2026-02-20T13:01:57.436 INFO:teuthology.orchestra.run.trial093.stdout: Installing : ceph-debugsource-2:20.3.0-5312.g26bc9765.el9.x86_64 1/2 2026-02-20T13:01:57.453 INFO:teuthology.orchestra.run.trial093.stdout: Installing : ceph-osd-crimson-debuginfo-2:20.3.0-5312.g26bc9765.e 2/2 2026-02-20T13:01:59.394 INFO:teuthology.orchestra.run.trial093.stdout: Running scriptlet: ceph-osd-crimson-debuginfo-2:20.3.0-5312.g26bc9765.e 2/2 2026-02-20T13:01:59.394 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : ceph-debugsource-2:20.3.0-5312.g26bc9765.el9.x86_64 1/2 2026-02-20T13:01:59.455 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : ceph-osd-crimson-debuginfo-2:20.3.0-5312.g26bc9765.e 2/2 2026-02-20T13:01:59.455 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:01:59.455 INFO:teuthology.orchestra.run.trial093.stdout:Installed: 2026-02-20T13:01:59.455 INFO:teuthology.orchestra.run.trial093.stdout: ceph-debugsource-2:20.3.0-5312.g26bc9765.el9.x86_64 2026-02-20T13:01:59.455 INFO:teuthology.orchestra.run.trial093.stdout: ceph-osd-crimson-debuginfo-2:20.3.0-5312.g26bc9765.el9.x86_64 2026-02-20T13:01:59.456 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:01:59.456 INFO:teuthology.orchestra.run.trial093.stdout:Complete! 2026-02-20T13:01:59.532 DEBUG:teuthology.parallel:result is None 2026-02-20T13:01:59.532 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=26bc97655ae0056b78c14ab91d1307332bd6dfca 2026-02-20T13:01:59.699 DEBUG:teuthology.orchestra.run.trial093:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-20T13:01:59.721 INFO:teuthology.orchestra.run.trial093.stdout:20.3.0-5312.g26bc9765.el9 2026-02-20T13:01:59.721 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5312.g26bc9765.el9 2026-02-20T13:01:59.722 INFO:teuthology.task.install:The correct ceph version 20.3.0-5312.g26bc9765 is installed. 2026-02-20T13:01:59.724 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2026-02-20T13:01:59.724 DEBUG:teuthology.orchestra.run.trial093:> set -ex 2026-02-20T13:01:59.724 DEBUG:teuthology.orchestra.run.trial093:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-20T13:01:59.788 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2026-02-20T13:01:59.788 DEBUG:teuthology.orchestra.run.trial093:> set -ex 2026-02-20T13:01:59.788 DEBUG:teuthology.orchestra.run.trial093:> sudo dd of=/usr/bin/daemon-helper 2026-02-20T13:01:59.853 DEBUG:teuthology.orchestra.run.trial093:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-20T13:01:59.916 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2026-02-20T13:01:59.917 DEBUG:teuthology.orchestra.run.trial093:> set -ex 2026-02-20T13:01:59.917 DEBUG:teuthology.orchestra.run.trial093:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-20T13:01:59.979 DEBUG:teuthology.orchestra.run.trial093:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-20T13:02:00.043 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2026-02-20T13:02:00.044 DEBUG:teuthology.orchestra.run.trial093:> set -ex 2026-02-20T13:02:00.044 DEBUG:teuthology.orchestra.run.trial093:> sudo dd of=/usr/bin/stdin-killer 2026-02-20T13:02:00.107 DEBUG:teuthology.orchestra.run.trial093:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-20T13:02:00.171 INFO:teuthology.run_tasks:Running task pexec... 2026-02-20T13:02:00.189 INFO:teuthology.task.pexec:Executing custom commands... 2026-02-20T13:02:00.190 DEBUG:teuthology.orchestra.run.trial093:> TESTDIR=/home/ubuntu/cephtest bash -s 2026-02-20T13:02:00.214 INFO:teuthology.task.pexec:Running commands on host ubuntu@trial093.front.sepia.ceph.com 2026-02-20T13:02:00.214 INFO:teuthology.task.pexec:sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson 2026-02-20T13:02:00.265 DEBUG:teuthology.parallel:result is None 2026-02-20T13:02:00.265 INFO:teuthology.run_tasks:Running task install... 2026-02-20T13:02:00.282 DEBUG:teuthology.task.install:project ceph 2026-02-20T13:02:00.283 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'debug', 'sha1': '26bc97655ae0056b78c14ab91d1307332bd6dfca'}} 2026-02-20T13:02:00.283 DEBUG:teuthology.task.install:config {'flavor': 'debug', 'sha1': '26bc97655ae0056b78c14ab91d1307332bd6dfca'} 2026-02-20T13:02:00.283 INFO:teuthology.task.install:Using flavor: debug 2026-02-20T13:02:00.292 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-osd', 'ceph-osd-classic', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-osd', 'ceph-osd-classic', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-20T13:02:00.292 INFO:teuthology.task.install:extra packages: [] 2026-02-20T13:02:00.293 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'debug', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '26bc97655ae0056b78c14ab91d1307332bd6dfca', 'tag': None, 'wait_for_package': False} 2026-02-20T13:02:00.293 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=26bc97655ae0056b78c14ab91d1307332bd6dfca 2026-02-20T13:02:00.453 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-perezjos-crimson-only-12-02-2026-PR66245/26bc97655ae0056b78c14ab91d1307332bd6dfca/centos/9/flavors/debug/ 2026-02-20T13:02:00.453 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5312.g26bc9765 2026-02-20T13:02:00.850 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-perezjos-crimson-only-12-02-2026-PR66245/26bc97655ae0056b78c14ab91d1307332bd6dfca/centos/9/flavors/debug/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-perezjos-crimson-only-12-02-2026-PR66245/26bc97655ae0056b78c14ab91d1307332bd6dfca/centos/9/flavors/debug/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-perezjos-crimson-only-12-02-2026-PR66245/26bc97655ae0056b78c14ab91d1307332bd6dfca/centos/9/flavors/debug/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-20T13:02:00.851 DEBUG:teuthology.orchestra.run.trial093:> set -ex 2026-02-20T13:02:00.851 DEBUG:teuthology.orchestra.run.trial093:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-20T13:02:00.877 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2026-02-20T13:02:00.878 DEBUG:teuthology.orchestra.run.trial093:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/26bc97655ae0056b78c14ab91d1307332bd6dfca/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-20T13:02:00.943 DEBUG:teuthology.orchestra.run.trial093:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-20T13:02:01.007 DEBUG:teuthology.orchestra.run.trial093:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-20T13:02:01.062 INFO:teuthology.orchestra.run.trial093.stdout:check_obsoletes = 1 2026-02-20T13:02:01.071 DEBUG:teuthology.orchestra.run.trial093:> sudo yum clean all 2026-02-20T13:02:01.231 INFO:teuthology.orchestra.run.trial093.stdout:56 files removed 2026-02-20T13:02:01.247 DEBUG:teuthology.orchestra.run.trial093:> sudo yum -y install ceph-radosgw 2026-02-20T13:02:02.073 INFO:teuthology.orchestra.run.trial093.stdout:ceph packages for x86_64 139 kB/s | 94 kB 00:00 2026-02-20T13:02:02.709 INFO:teuthology.orchestra.run.trial093.stdout:ceph noarch packages 40 kB/s | 24 kB 00:00 2026-02-20T13:02:03.275 INFO:teuthology.orchestra.run.trial093.stdout:ceph source packages 3.8 kB/s | 2.1 kB 00:00 2026-02-20T13:02:06.228 INFO:teuthology.orchestra.run.trial093.stdout:CentOS Stream 9 - BaseOS 3.0 MB/s | 8.9 MB 00:02 2026-02-20T13:02:25.115 INFO:teuthology.orchestra.run.trial093.stdout:CentOS Stream 9 - AppStream 1.5 MB/s | 27 MB 00:18 2026-02-20T13:02:33.084 INFO:teuthology.orchestra.run.trial093.stdout:CentOS Stream 9 - CRB 1.4 MB/s | 7.8 MB 00:05 2026-02-20T13:02:34.911 INFO:teuthology.orchestra.run.trial093.stdout:CentOS Stream 9 - Extras packages 19 kB/s | 20 kB 00:01 2026-02-20T13:02:41.376 INFO:teuthology.orchestra.run.trial093.stdout:Extra Packages for Enterprise Linux 3.2 MB/s | 20 MB 00:06 2026-02-20T13:02:44.603 INFO:teuthology.orchestra.run.trial093.stdout:lab-extras 2.4 MB/s | 50 kB 00:00 2026-02-20T13:02:45.789 INFO:teuthology.orchestra.run.trial093.stdout:Package ceph-radosgw-2:20.3.0-5312.g26bc9765.el9.x86_64 is already installed. 2026-02-20T13:02:45.808 INFO:teuthology.orchestra.run.trial093.stdout:Dependencies resolved. 2026-02-20T13:02:45.809 INFO:teuthology.orchestra.run.trial093.stdout:Nothing to do. 2026-02-20T13:02:45.809 INFO:teuthology.orchestra.run.trial093.stdout:Complete! 2026-02-20T13:02:45.832 DEBUG:teuthology.orchestra.run.trial093:> sudo yum -y install ceph-test 2026-02-20T13:02:46.146 INFO:teuthology.orchestra.run.trial093.stdout:Last metadata expiration check: 0:00:02 ago on Fri 20 Feb 2026 01:02:44 PM UTC. 2026-02-20T13:02:46.211 INFO:teuthology.orchestra.run.trial093.stdout:Package ceph-test-2:20.3.0-5312.g26bc9765.el9.x86_64 is already installed. 2026-02-20T13:02:46.229 INFO:teuthology.orchestra.run.trial093.stdout:Dependencies resolved. 2026-02-20T13:02:46.230 INFO:teuthology.orchestra.run.trial093.stdout:Nothing to do. 2026-02-20T13:02:46.230 INFO:teuthology.orchestra.run.trial093.stdout:Complete! 2026-02-20T13:02:46.252 DEBUG:teuthology.orchestra.run.trial093:> sudo yum -y install ceph 2026-02-20T13:02:46.561 INFO:teuthology.orchestra.run.trial093.stdout:Last metadata expiration check: 0:00:02 ago on Fri 20 Feb 2026 01:02:44 PM UTC. 2026-02-20T13:02:46.626 INFO:teuthology.orchestra.run.trial093.stdout:Package ceph-2:20.3.0-5312.g26bc9765.el9.x86_64 is already installed. 2026-02-20T13:02:46.644 INFO:teuthology.orchestra.run.trial093.stdout:Dependencies resolved. 2026-02-20T13:02:46.645 INFO:teuthology.orchestra.run.trial093.stdout:Nothing to do. 2026-02-20T13:02:46.645 INFO:teuthology.orchestra.run.trial093.stdout:Complete! 2026-02-20T13:02:46.666 DEBUG:teuthology.orchestra.run.trial093:> sudo yum -y install ceph-base 2026-02-20T13:02:46.973 INFO:teuthology.orchestra.run.trial093.stdout:Last metadata expiration check: 0:00:02 ago on Fri 20 Feb 2026 01:02:44 PM UTC. 2026-02-20T13:02:47.038 INFO:teuthology.orchestra.run.trial093.stdout:Package ceph-base-2:20.3.0-5312.g26bc9765.el9.x86_64 is already installed. 2026-02-20T13:02:47.057 INFO:teuthology.orchestra.run.trial093.stdout:Dependencies resolved. 2026-02-20T13:02:47.057 INFO:teuthology.orchestra.run.trial093.stdout:Nothing to do. 2026-02-20T13:02:47.057 INFO:teuthology.orchestra.run.trial093.stdout:Complete! 2026-02-20T13:02:47.079 DEBUG:teuthology.orchestra.run.trial093:> sudo yum -y install cephadm 2026-02-20T13:02:47.389 INFO:teuthology.orchestra.run.trial093.stdout:Last metadata expiration check: 0:00:03 ago on Fri 20 Feb 2026 01:02:44 PM UTC. 2026-02-20T13:02:47.454 INFO:teuthology.orchestra.run.trial093.stdout:Package cephadm-2:20.3.0-5312.g26bc9765.el9.noarch is already installed. 2026-02-20T13:02:47.473 INFO:teuthology.orchestra.run.trial093.stdout:Dependencies resolved. 2026-02-20T13:02:47.473 INFO:teuthology.orchestra.run.trial093.stdout:Nothing to do. 2026-02-20T13:02:47.473 INFO:teuthology.orchestra.run.trial093.stdout:Complete! 2026-02-20T13:02:47.495 DEBUG:teuthology.orchestra.run.trial093:> sudo yum -y install ceph-immutable-object-cache 2026-02-20T13:02:47.804 INFO:teuthology.orchestra.run.trial093.stdout:Last metadata expiration check: 0:00:03 ago on Fri 20 Feb 2026 01:02:44 PM UTC. 2026-02-20T13:02:47.868 INFO:teuthology.orchestra.run.trial093.stdout:Package ceph-immutable-object-cache-2:20.3.0-5312.g26bc9765.el9.x86_64 is already installed. 2026-02-20T13:02:47.886 INFO:teuthology.orchestra.run.trial093.stdout:Dependencies resolved. 2026-02-20T13:02:47.887 INFO:teuthology.orchestra.run.trial093.stdout:Nothing to do. 2026-02-20T13:02:47.887 INFO:teuthology.orchestra.run.trial093.stdout:Complete! 2026-02-20T13:02:47.909 DEBUG:teuthology.orchestra.run.trial093:> sudo yum -y install ceph-mgr 2026-02-20T13:02:48.213 INFO:teuthology.orchestra.run.trial093.stdout:Last metadata expiration check: 0:00:04 ago on Fri 20 Feb 2026 01:02:44 PM UTC. 2026-02-20T13:02:48.278 INFO:teuthology.orchestra.run.trial093.stdout:Package ceph-mgr-2:20.3.0-5312.g26bc9765.el9.x86_64 is already installed. 2026-02-20T13:02:48.297 INFO:teuthology.orchestra.run.trial093.stdout:Dependencies resolved. 2026-02-20T13:02:48.298 INFO:teuthology.orchestra.run.trial093.stdout:Nothing to do. 2026-02-20T13:02:48.298 INFO:teuthology.orchestra.run.trial093.stdout:Complete! 2026-02-20T13:02:48.319 DEBUG:teuthology.orchestra.run.trial093:> sudo yum -y install ceph-mgr-dashboard 2026-02-20T13:02:48.630 INFO:teuthology.orchestra.run.trial093.stdout:Last metadata expiration check: 0:00:04 ago on Fri 20 Feb 2026 01:02:44 PM UTC. 2026-02-20T13:02:48.694 INFO:teuthology.orchestra.run.trial093.stdout:Package ceph-mgr-dashboard-2:20.3.0-5312.g26bc9765.el9.noarch is already installed. 2026-02-20T13:02:48.713 INFO:teuthology.orchestra.run.trial093.stdout:Dependencies resolved. 2026-02-20T13:02:48.714 INFO:teuthology.orchestra.run.trial093.stdout:Nothing to do. 2026-02-20T13:02:48.714 INFO:teuthology.orchestra.run.trial093.stdout:Complete! 2026-02-20T13:02:48.735 DEBUG:teuthology.orchestra.run.trial093:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-20T13:02:49.050 INFO:teuthology.orchestra.run.trial093.stdout:Last metadata expiration check: 0:00:05 ago on Fri 20 Feb 2026 01:02:44 PM UTC. 2026-02-20T13:02:49.114 INFO:teuthology.orchestra.run.trial093.stdout:Package ceph-mgr-diskprediction-local-2:20.3.0-5312.g26bc9765.el9.noarch is already installed. 2026-02-20T13:02:49.132 INFO:teuthology.orchestra.run.trial093.stdout:Dependencies resolved. 2026-02-20T13:02:49.133 INFO:teuthology.orchestra.run.trial093.stdout:Nothing to do. 2026-02-20T13:02:49.133 INFO:teuthology.orchestra.run.trial093.stdout:Complete! 2026-02-20T13:02:49.155 DEBUG:teuthology.orchestra.run.trial093:> sudo yum -y install ceph-mgr-rook 2026-02-20T13:02:49.465 INFO:teuthology.orchestra.run.trial093.stdout:Last metadata expiration check: 0:00:05 ago on Fri 20 Feb 2026 01:02:44 PM UTC. 2026-02-20T13:02:49.530 INFO:teuthology.orchestra.run.trial093.stdout:Package ceph-mgr-rook-2:20.3.0-5312.g26bc9765.el9.noarch is already installed. 2026-02-20T13:02:49.549 INFO:teuthology.orchestra.run.trial093.stdout:Dependencies resolved. 2026-02-20T13:02:49.549 INFO:teuthology.orchestra.run.trial093.stdout:Nothing to do. 2026-02-20T13:02:49.550 INFO:teuthology.orchestra.run.trial093.stdout:Complete! 2026-02-20T13:02:49.572 DEBUG:teuthology.orchestra.run.trial093:> sudo yum -y install ceph-mgr-cephadm 2026-02-20T13:02:49.888 INFO:teuthology.orchestra.run.trial093.stdout:Last metadata expiration check: 0:00:05 ago on Fri 20 Feb 2026 01:02:44 PM UTC. 2026-02-20T13:02:49.953 INFO:teuthology.orchestra.run.trial093.stdout:Package ceph-mgr-cephadm-2:20.3.0-5312.g26bc9765.el9.noarch is already installed. 2026-02-20T13:02:49.971 INFO:teuthology.orchestra.run.trial093.stdout:Dependencies resolved. 2026-02-20T13:02:49.972 INFO:teuthology.orchestra.run.trial093.stdout:Nothing to do. 2026-02-20T13:02:49.972 INFO:teuthology.orchestra.run.trial093.stdout:Complete! 2026-02-20T13:02:49.993 DEBUG:teuthology.orchestra.run.trial093:> sudo yum -y install ceph-osd 2026-02-20T13:02:50.306 INFO:teuthology.orchestra.run.trial093.stdout:Last metadata expiration check: 0:00:06 ago on Fri 20 Feb 2026 01:02:44 PM UTC. 2026-02-20T13:02:50.371 INFO:teuthology.orchestra.run.trial093.stdout:Package ceph-osd-2:20.3.0-5312.g26bc9765.el9.x86_64 is already installed. 2026-02-20T13:02:50.390 INFO:teuthology.orchestra.run.trial093.stdout:Dependencies resolved. 2026-02-20T13:02:50.390 INFO:teuthology.orchestra.run.trial093.stdout:Nothing to do. 2026-02-20T13:02:50.390 INFO:teuthology.orchestra.run.trial093.stdout:Complete! 2026-02-20T13:02:50.412 DEBUG:teuthology.orchestra.run.trial093:> sudo yum -y install ceph-osd-classic 2026-02-20T13:02:50.725 INFO:teuthology.orchestra.run.trial093.stdout:Last metadata expiration check: 0:00:06 ago on Fri 20 Feb 2026 01:02:44 PM UTC. 2026-02-20T13:02:50.790 INFO:teuthology.orchestra.run.trial093.stdout:Package ceph-osd-classic-2:20.3.0-5312.g26bc9765.el9.x86_64 is already installed. 2026-02-20T13:02:50.808 INFO:teuthology.orchestra.run.trial093.stdout:Dependencies resolved. 2026-02-20T13:02:50.809 INFO:teuthology.orchestra.run.trial093.stdout:Nothing to do. 2026-02-20T13:02:50.809 INFO:teuthology.orchestra.run.trial093.stdout:Complete! 2026-02-20T13:02:50.831 DEBUG:teuthology.orchestra.run.trial093:> sudo yum -y install ceph-fuse 2026-02-20T13:02:51.144 INFO:teuthology.orchestra.run.trial093.stdout:Last metadata expiration check: 0:00:07 ago on Fri 20 Feb 2026 01:02:44 PM UTC. 2026-02-20T13:02:51.209 INFO:teuthology.orchestra.run.trial093.stdout:Package ceph-fuse-2:20.3.0-5312.g26bc9765.el9.x86_64 is already installed. 2026-02-20T13:02:51.228 INFO:teuthology.orchestra.run.trial093.stdout:Dependencies resolved. 2026-02-20T13:02:51.228 INFO:teuthology.orchestra.run.trial093.stdout:Nothing to do. 2026-02-20T13:02:51.229 INFO:teuthology.orchestra.run.trial093.stdout:Complete! 2026-02-20T13:02:51.250 DEBUG:teuthology.orchestra.run.trial093:> sudo yum -y install ceph-volume 2026-02-20T13:02:51.565 INFO:teuthology.orchestra.run.trial093.stdout:Last metadata expiration check: 0:00:07 ago on Fri 20 Feb 2026 01:02:44 PM UTC. 2026-02-20T13:02:51.630 INFO:teuthology.orchestra.run.trial093.stdout:Package ceph-volume-2:20.3.0-5312.g26bc9765.el9.noarch is already installed. 2026-02-20T13:02:51.649 INFO:teuthology.orchestra.run.trial093.stdout:Dependencies resolved. 2026-02-20T13:02:51.650 INFO:teuthology.orchestra.run.trial093.stdout:Nothing to do. 2026-02-20T13:02:51.650 INFO:teuthology.orchestra.run.trial093.stdout:Complete! 2026-02-20T13:02:51.671 DEBUG:teuthology.orchestra.run.trial093:> sudo yum -y install librados-devel 2026-02-20T13:02:51.982 INFO:teuthology.orchestra.run.trial093.stdout:Last metadata expiration check: 0:00:07 ago on Fri 20 Feb 2026 01:02:44 PM UTC. 2026-02-20T13:02:52.047 INFO:teuthology.orchestra.run.trial093.stdout:Package librados-devel-2:20.3.0-5312.g26bc9765.el9.x86_64 is already installed. 2026-02-20T13:02:52.066 INFO:teuthology.orchestra.run.trial093.stdout:Dependencies resolved. 2026-02-20T13:02:52.066 INFO:teuthology.orchestra.run.trial093.stdout:Nothing to do. 2026-02-20T13:02:52.066 INFO:teuthology.orchestra.run.trial093.stdout:Complete! 2026-02-20T13:02:52.088 DEBUG:teuthology.orchestra.run.trial093:> sudo yum -y install libcephfs2 2026-02-20T13:02:52.398 INFO:teuthology.orchestra.run.trial093.stdout:Last metadata expiration check: 0:00:08 ago on Fri 20 Feb 2026 01:02:44 PM UTC. 2026-02-20T13:02:52.462 INFO:teuthology.orchestra.run.trial093.stdout:Package libcephfs2-2:20.3.0-5312.g26bc9765.el9.x86_64 is already installed. 2026-02-20T13:02:52.481 INFO:teuthology.orchestra.run.trial093.stdout:Dependencies resolved. 2026-02-20T13:02:52.482 INFO:teuthology.orchestra.run.trial093.stdout:Nothing to do. 2026-02-20T13:02:52.482 INFO:teuthology.orchestra.run.trial093.stdout:Complete! 2026-02-20T13:02:52.503 DEBUG:teuthology.orchestra.run.trial093:> sudo yum -y install libcephfs-devel 2026-02-20T13:02:52.815 INFO:teuthology.orchestra.run.trial093.stdout:Last metadata expiration check: 0:00:08 ago on Fri 20 Feb 2026 01:02:44 PM UTC. 2026-02-20T13:02:52.879 INFO:teuthology.orchestra.run.trial093.stdout:Package libcephfs-devel-2:20.3.0-5312.g26bc9765.el9.x86_64 is already installed. 2026-02-20T13:02:52.898 INFO:teuthology.orchestra.run.trial093.stdout:Dependencies resolved. 2026-02-20T13:02:52.899 INFO:teuthology.orchestra.run.trial093.stdout:Nothing to do. 2026-02-20T13:02:52.899 INFO:teuthology.orchestra.run.trial093.stdout:Complete! 2026-02-20T13:02:52.919 DEBUG:teuthology.orchestra.run.trial093:> sudo yum -y install librados2 2026-02-20T13:02:53.229 INFO:teuthology.orchestra.run.trial093.stdout:Last metadata expiration check: 0:00:09 ago on Fri 20 Feb 2026 01:02:44 PM UTC. 2026-02-20T13:02:53.295 INFO:teuthology.orchestra.run.trial093.stdout:Package librados2-2:20.3.0-5312.g26bc9765.el9.x86_64 is already installed. 2026-02-20T13:02:53.314 INFO:teuthology.orchestra.run.trial093.stdout:Dependencies resolved. 2026-02-20T13:02:53.314 INFO:teuthology.orchestra.run.trial093.stdout:Nothing to do. 2026-02-20T13:02:53.314 INFO:teuthology.orchestra.run.trial093.stdout:Complete! 2026-02-20T13:02:53.337 DEBUG:teuthology.orchestra.run.trial093:> sudo yum -y install librbd1 2026-02-20T13:02:53.648 INFO:teuthology.orchestra.run.trial093.stdout:Last metadata expiration check: 0:00:09 ago on Fri 20 Feb 2026 01:02:44 PM UTC. 2026-02-20T13:02:53.713 INFO:teuthology.orchestra.run.trial093.stdout:Package librbd1-2:20.3.0-5312.g26bc9765.el9.x86_64 is already installed. 2026-02-20T13:02:53.732 INFO:teuthology.orchestra.run.trial093.stdout:Dependencies resolved. 2026-02-20T13:02:53.733 INFO:teuthology.orchestra.run.trial093.stdout:Nothing to do. 2026-02-20T13:02:53.733 INFO:teuthology.orchestra.run.trial093.stdout:Complete! 2026-02-20T13:02:53.754 DEBUG:teuthology.orchestra.run.trial093:> sudo yum -y install python3-rados 2026-02-20T13:02:54.067 INFO:teuthology.orchestra.run.trial093.stdout:Last metadata expiration check: 0:00:10 ago on Fri 20 Feb 2026 01:02:44 PM UTC. 2026-02-20T13:02:54.132 INFO:teuthology.orchestra.run.trial093.stdout:Package python3-rados-2:20.3.0-5312.g26bc9765.el9.x86_64 is already installed. 2026-02-20T13:02:54.151 INFO:teuthology.orchestra.run.trial093.stdout:Dependencies resolved. 2026-02-20T13:02:54.152 INFO:teuthology.orchestra.run.trial093.stdout:Nothing to do. 2026-02-20T13:02:54.152 INFO:teuthology.orchestra.run.trial093.stdout:Complete! 2026-02-20T13:02:54.173 DEBUG:teuthology.orchestra.run.trial093:> sudo yum -y install python3-rgw 2026-02-20T13:02:54.490 INFO:teuthology.orchestra.run.trial093.stdout:Last metadata expiration check: 0:00:10 ago on Fri 20 Feb 2026 01:02:44 PM UTC. 2026-02-20T13:02:54.554 INFO:teuthology.orchestra.run.trial093.stdout:Package python3-rgw-2:20.3.0-5312.g26bc9765.el9.x86_64 is already installed. 2026-02-20T13:02:54.573 INFO:teuthology.orchestra.run.trial093.stdout:Dependencies resolved. 2026-02-20T13:02:54.573 INFO:teuthology.orchestra.run.trial093.stdout:Nothing to do. 2026-02-20T13:02:54.573 INFO:teuthology.orchestra.run.trial093.stdout:Complete! 2026-02-20T13:02:54.595 DEBUG:teuthology.orchestra.run.trial093:> sudo yum -y install python3-cephfs 2026-02-20T13:02:54.908 INFO:teuthology.orchestra.run.trial093.stdout:Last metadata expiration check: 0:00:10 ago on Fri 20 Feb 2026 01:02:44 PM UTC. 2026-02-20T13:02:54.973 INFO:teuthology.orchestra.run.trial093.stdout:Package python3-cephfs-2:20.3.0-5312.g26bc9765.el9.x86_64 is already installed. 2026-02-20T13:02:54.992 INFO:teuthology.orchestra.run.trial093.stdout:Dependencies resolved. 2026-02-20T13:02:54.992 INFO:teuthology.orchestra.run.trial093.stdout:Nothing to do. 2026-02-20T13:02:54.992 INFO:teuthology.orchestra.run.trial093.stdout:Complete! 2026-02-20T13:02:55.014 DEBUG:teuthology.orchestra.run.trial093:> sudo yum -y install python3-rbd 2026-02-20T13:02:55.324 INFO:teuthology.orchestra.run.trial093.stdout:Last metadata expiration check: 0:00:11 ago on Fri 20 Feb 2026 01:02:44 PM UTC. 2026-02-20T13:02:55.389 INFO:teuthology.orchestra.run.trial093.stdout:Package python3-rbd-2:20.3.0-5312.g26bc9765.el9.x86_64 is already installed. 2026-02-20T13:02:55.408 INFO:teuthology.orchestra.run.trial093.stdout:Dependencies resolved. 2026-02-20T13:02:55.409 INFO:teuthology.orchestra.run.trial093.stdout:Nothing to do. 2026-02-20T13:02:55.409 INFO:teuthology.orchestra.run.trial093.stdout:Complete! 2026-02-20T13:02:55.430 DEBUG:teuthology.orchestra.run.trial093:> sudo yum -y install rbd-fuse 2026-02-20T13:02:55.748 INFO:teuthology.orchestra.run.trial093.stdout:Last metadata expiration check: 0:00:11 ago on Fri 20 Feb 2026 01:02:44 PM UTC. 2026-02-20T13:02:55.812 INFO:teuthology.orchestra.run.trial093.stdout:Package rbd-fuse-2:20.3.0-5312.g26bc9765.el9.x86_64 is already installed. 2026-02-20T13:02:55.831 INFO:teuthology.orchestra.run.trial093.stdout:Dependencies resolved. 2026-02-20T13:02:55.831 INFO:teuthology.orchestra.run.trial093.stdout:Nothing to do. 2026-02-20T13:02:55.831 INFO:teuthology.orchestra.run.trial093.stdout:Complete! 2026-02-20T13:02:55.854 DEBUG:teuthology.orchestra.run.trial093:> sudo yum -y install rbd-mirror 2026-02-20T13:02:56.168 INFO:teuthology.orchestra.run.trial093.stdout:Last metadata expiration check: 0:00:12 ago on Fri 20 Feb 2026 01:02:44 PM UTC. 2026-02-20T13:02:56.234 INFO:teuthology.orchestra.run.trial093.stdout:Package rbd-mirror-2:20.3.0-5312.g26bc9765.el9.x86_64 is already installed. 2026-02-20T13:02:56.252 INFO:teuthology.orchestra.run.trial093.stdout:Dependencies resolved. 2026-02-20T13:02:56.253 INFO:teuthology.orchestra.run.trial093.stdout:Nothing to do. 2026-02-20T13:02:56.253 INFO:teuthology.orchestra.run.trial093.stdout:Complete! 2026-02-20T13:02:56.274 DEBUG:teuthology.orchestra.run.trial093:> sudo yum -y install rbd-nbd 2026-02-20T13:02:56.580 INFO:teuthology.orchestra.run.trial093.stdout:Last metadata expiration check: 0:00:12 ago on Fri 20 Feb 2026 01:02:44 PM UTC. 2026-02-20T13:02:56.645 INFO:teuthology.orchestra.run.trial093.stdout:Package rbd-nbd-2:20.3.0-5312.g26bc9765.el9.x86_64 is already installed. 2026-02-20T13:02:56.664 INFO:teuthology.orchestra.run.trial093.stdout:Dependencies resolved. 2026-02-20T13:02:56.665 INFO:teuthology.orchestra.run.trial093.stdout:Nothing to do. 2026-02-20T13:02:56.665 INFO:teuthology.orchestra.run.trial093.stdout:Complete! 2026-02-20T13:02:56.690 DEBUG:teuthology.parallel:result is None 2026-02-20T13:02:56.690 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=26bc97655ae0056b78c14ab91d1307332bd6dfca 2026-02-20T13:02:56.856 DEBUG:teuthology.orchestra.run.trial093:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-20T13:02:56.876 INFO:teuthology.orchestra.run.trial093.stdout:20.3.0-5312.g26bc9765.el9 2026-02-20T13:02:56.877 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5312.g26bc9765.el9 2026-02-20T13:02:56.877 INFO:teuthology.task.install:The correct ceph version 20.3.0-5312.g26bc9765 is installed. 2026-02-20T13:02:56.878 INFO:teuthology.task.install.util:Utilities already shipped, skip it... 2026-02-20T13:02:56.878 INFO:teuthology.run_tasks:Running task ceph... 2026-02-20T13:02:56.948 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2026-02-20T13:02:56.948 DEBUG:teuthology.orchestra.run.trial093:> sudo chmod 777 /var/log/ceph 2026-02-20T13:02:56.973 INFO:tasks.ceph:Disabling ceph logrotate... 2026-02-20T13:02:56.974 DEBUG:teuthology.orchestra.run.trial093:> sudo rm -f -- /etc/logrotate.d/ceph 2026-02-20T13:02:57.038 INFO:tasks.ceph:Creating extra log directories... 2026-02-20T13:02:57.038 DEBUG:teuthology.orchestra.run.trial093:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2026-02-20T13:02:57.116 INFO:tasks.ceph:Creating ceph cluster ceph... 2026-02-20T13:02:57.116 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore cachepin type': '2Q', 'seastore main device type': 'RANDOM_BLOCK_SSD', 'seastore max concurrent transactions': 8}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2026-02-20T13:02:57.117 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/mnt/teuthology/jjperez-2026-02-20_10:29:19-crimson-rados-wip-perezjos-crimson-only-12-02-2026-PR66245-distro-debug-trial/60758', 'branch': 'wip-perezjos-crimson-only-12-02-2026-PR66245', 'description': 'crimson-rados/perf/{0-crimson_install clusters/{crimson-fixed-1} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/rbm$/{crimson_seastore_rbm_2q} perf_ignorelist settings/optimized workloads/sample_fio}', 'email': None, 'first_in_suite': False, 'flavor': 'debug', 'job_id': '60758', 'kernel': {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'trial', 'name': 'jjperez-2026-02-20_10:29:19-crimson-rados-wip-perezjos-crimson-only-12-02-2026-PR66245-distro-debug-trial', 'no_nested_subset': False, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-perezjos-crimson-only-12-02-2026-PR66245'}, 'ceph': {'conf': {'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd': {'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore cachepin type': '2Q', 'seastore main device type': 'RANDOM_BLOCK_SSD', 'seastore max concurrent transactions': 8}}, 'crimson_compat': True, 'flavor': 'debug', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH'], 'sha1': '26bc97655ae0056b78c14ab91d1307332bd6dfca'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'global': {'osd crush chooseleaf type': 0, 'osd pool default pg num': 128, 'osd pool default pgp num': 128, 'osd pool default size': 2}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'debug', 'sha1': '26bc97655ae0056b78c14ab91d1307332bd6dfca'}}, 'workunit': {'branch': 'main', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '445d5c9ba18a4040e70e25303230c36c77773acb'}}, 'owner': 'scheduled_jjperez@soko04.front.sepia.ceph.com', 'priority': 101, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 5547, 'sha1': '26bc97655ae0056b78c14ab91d1307332bd6dfca', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph.git', 'suite_sha1': '445d5c9ba18a4040e70e25303230c36c77773acb', 'targets': {'trial093.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ+KS8E6MM/VDOGGnZl3pISIqnht6KyA45y4MeVv3kzaeImMpRmz6kNbY5dEYtI4JfeDo+jcL3+jfWq11mWVkPE='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'flavor': 'debug', 'sha1': '26bc97655ae0056b78c14ab91d1307332bd6dfca'}}, {'pexec': {'all': ['sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson']}}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore cachepin type': '2Q', 'seastore main device type': 'RANDOM_BLOCK_SSD', 'seastore max concurrent transactions': 8}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}}, 'wait-for-scrub': False, 'crimson_compat': True, 'flavor': 'debug', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH'], 'sha1': '26bc97655ae0056b78c14ab91d1307332bd6dfca', 'cluster': 'ceph'}}, {'ssh_keys': None}, {'cbt': {'benchmarks': {'librbdfio': {'iodepth': [32], 'log_avg_msec': 100, 'mode': ['randwrite'], 'norandommap': True, 'op_size': [4096], 'osd_ra': [4096], 'pool_profile': 'rbd', 'procs_per_volume': [1], 'time': 60, 'vol_size': 4096, 'volumes_per_client': [2]}}, 'cluster': {'iterations': 1, 'osds_per_node': 3, 'pool_profiles': {'rbd': {'pg_size': 128, 'pgp_size': 128, 'replication': 3}}, 'user': 'ubuntu'}, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'reimage-ssh-down', 'teuthology_sha1': 'e6d9099ce03dc21193fda653159ec82465c3a94a', 'timestamp': '2026-02-20_10:29:19', 'tube': 'trial', 'user': 'jjperez', 'verbose': False, 'worker_log': '/home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.1540699'} 2026-02-20T13:02:57.118 DEBUG:teuthology.orchestra.run.trial093:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2026-02-20T13:02:57.161 DEBUG:teuthology.orchestra.run.trial093:> sudo install -d -m0777 -- /var/run/ceph 2026-02-20T13:02:57.225 DEBUG:teuthology.orchestra.run.trial093:> set -ex 2026-02-20T13:02:57.226 DEBUG:teuthology.orchestra.run.trial093:> dd if=/scratch_devs of=/dev/stdout 2026-02-20T13:02:57.283 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2026-02-20T13:02:57.283 DEBUG:teuthology.orchestra.run.trial093:> stat /dev/vg_nvme/lv_1 2026-02-20T13:02:57.339 INFO:teuthology.orchestra.run.trial093.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2026-02-20T13:02:57.339 INFO:teuthology.orchestra.run.trial093.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-20T13:02:57.340 INFO:teuthology.orchestra.run.trial093.stdout:Device: 6h/6d Inode: 1287 Links: 1 2026-02-20T13:02:57.340 INFO:teuthology.orchestra.run.trial093.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-20T13:02:57.340 INFO:teuthology.orchestra.run.trial093.stdout:Context: system_u:object_r:device_t:s0 2026-02-20T13:02:57.340 INFO:teuthology.orchestra.run.trial093.stdout:Access: 2026-02-20 13:00:42.370664173 +0000 2026-02-20T13:02:57.340 INFO:teuthology.orchestra.run.trial093.stdout:Modify: 2026-02-20 12:58:04.092735651 +0000 2026-02-20T13:02:57.340 INFO:teuthology.orchestra.run.trial093.stdout:Change: 2026-02-20 12:58:04.092735651 +0000 2026-02-20T13:02:57.340 INFO:teuthology.orchestra.run.trial093.stdout: Birth: 2026-02-20 12:58:04.092735651 +0000 2026-02-20T13:02:57.341 DEBUG:teuthology.orchestra.run.trial093:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2026-02-20T13:02:57.402 INFO:teuthology.orchestra.run.trial093.stderr:1+0 records in 2026-02-20T13:02:57.403 INFO:teuthology.orchestra.run.trial093.stderr:1+0 records out 2026-02-20T13:02:57.403 INFO:teuthology.orchestra.run.trial093.stderr:512 bytes copied, 0.000154727 s, 3.3 MB/s 2026-02-20T13:02:57.404 DEBUG:teuthology.orchestra.run.trial093:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2026-02-20T13:02:57.461 DEBUG:teuthology.orchestra.run.trial093:> stat /dev/vg_nvme/lv_2 2026-02-20T13:02:57.518 INFO:teuthology.orchestra.run.trial093.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2026-02-20T13:02:57.518 INFO:teuthology.orchestra.run.trial093.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-20T13:02:57.518 INFO:teuthology.orchestra.run.trial093.stdout:Device: 6h/6d Inode: 1309 Links: 1 2026-02-20T13:02:57.518 INFO:teuthology.orchestra.run.trial093.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-20T13:02:57.518 INFO:teuthology.orchestra.run.trial093.stdout:Context: system_u:object_r:device_t:s0 2026-02-20T13:02:57.519 INFO:teuthology.orchestra.run.trial093.stdout:Access: 2026-02-20 13:00:42.370664173 +0000 2026-02-20T13:02:57.519 INFO:teuthology.orchestra.run.trial093.stdout:Modify: 2026-02-20 12:58:04.093735624 +0000 2026-02-20T13:02:57.519 INFO:teuthology.orchestra.run.trial093.stdout:Change: 2026-02-20 12:58:04.093735624 +0000 2026-02-20T13:02:57.519 INFO:teuthology.orchestra.run.trial093.stdout: Birth: 2026-02-20 12:58:04.093735624 +0000 2026-02-20T13:02:57.519 DEBUG:teuthology.orchestra.run.trial093:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2026-02-20T13:02:57.582 INFO:teuthology.orchestra.run.trial093.stderr:1+0 records in 2026-02-20T13:02:57.583 INFO:teuthology.orchestra.run.trial093.stderr:1+0 records out 2026-02-20T13:02:57.583 INFO:teuthology.orchestra.run.trial093.stderr:512 bytes copied, 0.0001563 s, 3.3 MB/s 2026-02-20T13:02:57.584 DEBUG:teuthology.orchestra.run.trial093:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2026-02-20T13:02:57.641 DEBUG:teuthology.orchestra.run.trial093:> stat /dev/vg_nvme/lv_3 2026-02-20T13:02:57.699 INFO:teuthology.orchestra.run.trial093.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2026-02-20T13:02:57.699 INFO:teuthology.orchestra.run.trial093.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-20T13:02:57.699 INFO:teuthology.orchestra.run.trial093.stdout:Device: 6h/6d Inode: 1294 Links: 1 2026-02-20T13:02:57.699 INFO:teuthology.orchestra.run.trial093.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-20T13:02:57.699 INFO:teuthology.orchestra.run.trial093.stdout:Context: system_u:object_r:device_t:s0 2026-02-20T13:02:57.700 INFO:teuthology.orchestra.run.trial093.stdout:Access: 2026-02-20 13:00:42.370664173 +0000 2026-02-20T13:02:57.700 INFO:teuthology.orchestra.run.trial093.stdout:Modify: 2026-02-20 12:58:04.092735651 +0000 2026-02-20T13:02:57.700 INFO:teuthology.orchestra.run.trial093.stdout:Change: 2026-02-20 12:58:04.092735651 +0000 2026-02-20T13:02:57.700 INFO:teuthology.orchestra.run.trial093.stdout: Birth: 2026-02-20 12:58:04.092735651 +0000 2026-02-20T13:02:57.700 DEBUG:teuthology.orchestra.run.trial093:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2026-02-20T13:02:57.763 INFO:teuthology.orchestra.run.trial093.stderr:1+0 records in 2026-02-20T13:02:57.763 INFO:teuthology.orchestra.run.trial093.stderr:1+0 records out 2026-02-20T13:02:57.763 INFO:teuthology.orchestra.run.trial093.stderr:512 bytes copied, 0.00015603 s, 3.3 MB/s 2026-02-20T13:02:57.764 DEBUG:teuthology.orchestra.run.trial093:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2026-02-20T13:02:57.822 DEBUG:teuthology.orchestra.run.trial093:> stat /dev/vg_nvme/lv_4 2026-02-20T13:02:57.879 INFO:teuthology.orchestra.run.trial093.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2026-02-20T13:02:57.880 INFO:teuthology.orchestra.run.trial093.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-20T13:02:57.880 INFO:teuthology.orchestra.run.trial093.stdout:Device: 6h/6d Inode: 1301 Links: 1 2026-02-20T13:02:57.880 INFO:teuthology.orchestra.run.trial093.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-20T13:02:57.880 INFO:teuthology.orchestra.run.trial093.stdout:Context: system_u:object_r:device_t:s0 2026-02-20T13:02:57.880 INFO:teuthology.orchestra.run.trial093.stdout:Access: 2026-02-20 13:00:42.371664149 +0000 2026-02-20T13:02:57.880 INFO:teuthology.orchestra.run.trial093.stdout:Modify: 2026-02-20 12:58:04.092735651 +0000 2026-02-20T13:02:57.880 INFO:teuthology.orchestra.run.trial093.stdout:Change: 2026-02-20 12:58:04.092735651 +0000 2026-02-20T13:02:57.880 INFO:teuthology.orchestra.run.trial093.stdout: Birth: 2026-02-20 12:58:04.092735651 +0000 2026-02-20T13:02:57.881 DEBUG:teuthology.orchestra.run.trial093:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2026-02-20T13:02:57.941 INFO:teuthology.orchestra.run.trial093.stderr:1+0 records in 2026-02-20T13:02:57.942 INFO:teuthology.orchestra.run.trial093.stderr:1+0 records out 2026-02-20T13:02:57.942 INFO:teuthology.orchestra.run.trial093.stderr:512 bytes copied, 0.000141973 s, 3.6 MB/s 2026-02-20T13:02:57.943 DEBUG:teuthology.orchestra.run.trial093:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2026-02-20T13:02:58.000 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-20T13:02:58.000 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@trial093.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2026-02-20T13:02:58.000 INFO:tasks.ceph:Generating config... 2026-02-20T13:02:58.003 INFO:tasks.ceph:[osd] debug monc = 20 2026-02-20T13:02:58.003 INFO:tasks.ceph:[osd] crimson cpu num = 4 2026-02-20T13:02:58.003 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2026-02-20T13:02:58.003 INFO:tasks.ceph:[osd] debug filestore = 0/0 2026-02-20T13:02:58.003 INFO:tasks.ceph:[osd] debug journal = 0/0 2026-02-20T13:02:58.003 INFO:tasks.ceph:[osd] debug ms = 0/0 2026-02-20T13:02:58.003 INFO:tasks.ceph:[osd] debug osd = 0/0 2026-02-20T13:02:58.004 INFO:tasks.ceph:[osd] debug seastore = 20 2026-02-20T13:02:58.004 INFO:tasks.ceph:[osd] debug seastore backref = 20 2026-02-20T13:02:58.004 INFO:tasks.ceph:[osd] debug seastore cache = 20 2026-02-20T13:02:58.004 INFO:tasks.ceph:[osd] debug seastore cleaner = 20 2026-02-20T13:02:58.004 INFO:tasks.ceph:[osd] debug seastore device = 20 2026-02-20T13:02:58.004 INFO:tasks.ceph:[osd] debug seastore epm = 20 2026-02-20T13:02:58.004 INFO:tasks.ceph:[osd] debug seastore fixedkv tree = 20 2026-02-20T13:02:58.004 INFO:tasks.ceph:[osd] debug seastore journal = 20 2026-02-20T13:02:58.004 INFO:tasks.ceph:[osd] debug seastore lba = 20 2026-02-20T13:02:58.004 INFO:tasks.ceph:[osd] debug seastore odata = 20 2026-02-20T13:02:58.004 INFO:tasks.ceph:[osd] debug seastore omap = 20 2026-02-20T13:02:58.005 INFO:tasks.ceph:[osd] debug seastore onode = 20 2026-02-20T13:02:58.005 INFO:tasks.ceph:[osd] debug seastore t = 20 2026-02-20T13:02:58.005 INFO:tasks.ceph:[osd] debug seastore tm = 20 2026-02-20T13:02:58.005 INFO:tasks.ceph:[osd] osd objectstore = seastore 2026-02-20T13:02:58.005 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2026-02-20T13:02:58.005 INFO:tasks.ceph:[osd] seastore cachepin size pershard = 64M 2026-02-20T13:02:58.005 INFO:tasks.ceph:[osd] seastore cachepin type = 2Q 2026-02-20T13:02:58.005 INFO:tasks.ceph:[osd] seastore main device type = RANDOM_BLOCK_SSD 2026-02-20T13:02:58.005 INFO:tasks.ceph:[osd] seastore max concurrent transactions = 8 2026-02-20T13:02:58.005 INFO:tasks.ceph:[global] auth client required = none 2026-02-20T13:02:58.005 INFO:tasks.ceph:[global] auth cluster required = none 2026-02-20T13:02:58.006 INFO:tasks.ceph:[global] auth service required = none 2026-02-20T13:02:58.006 INFO:tasks.ceph:[global] auth supported = none 2026-02-20T13:02:58.006 INFO:tasks.ceph:[global] debug asok = 0/0 2026-02-20T13:02:58.006 INFO:tasks.ceph:[global] debug auth = 0/0 2026-02-20T13:02:58.006 INFO:tasks.ceph:[global] debug bdev = 0/0 2026-02-20T13:02:58.006 INFO:tasks.ceph:[global] debug bluefs = 0/0 2026-02-20T13:02:58.006 INFO:tasks.ceph:[global] debug bluestore = 0/0 2026-02-20T13:02:58.006 INFO:tasks.ceph:[global] debug buffer = 0/0 2026-02-20T13:02:58.006 INFO:tasks.ceph:[global] debug civetweb = 0/0 2026-02-20T13:02:58.006 INFO:tasks.ceph:[global] debug client = 0/0 2026-02-20T13:02:58.006 INFO:tasks.ceph:[global] debug compressor = 0/0 2026-02-20T13:02:58.007 INFO:tasks.ceph:[global] debug context = 0/0 2026-02-20T13:02:58.007 INFO:tasks.ceph:[global] debug crush = 0/0 2026-02-20T13:02:58.007 INFO:tasks.ceph:[global] debug crypto = 0/0 2026-02-20T13:02:58.007 INFO:tasks.ceph:[global] debug dpdk = 0/0 2026-02-20T13:02:58.007 INFO:tasks.ceph:[global] debug eventtrace = 0/0 2026-02-20T13:02:58.007 INFO:tasks.ceph:[global] debug filer = 0/0 2026-02-20T13:02:58.007 INFO:tasks.ceph:[global] debug filestore = 0/0 2026-02-20T13:02:58.007 INFO:tasks.ceph:[global] debug finisher = 0/0 2026-02-20T13:02:58.007 INFO:tasks.ceph:[global] debug fuse = 0/0 2026-02-20T13:02:58.007 INFO:tasks.ceph:[global] debug heartbeatmap = 0/0 2026-02-20T13:02:58.007 INFO:tasks.ceph:[global] debug javaclient = 0/0 2026-02-20T13:02:58.008 INFO:tasks.ceph:[global] debug journal = 0/0 2026-02-20T13:02:58.008 INFO:tasks.ceph:[global] debug journaler = 0/0 2026-02-20T13:02:58.008 INFO:tasks.ceph:[global] debug kstore = 0/0 2026-02-20T13:02:58.008 INFO:tasks.ceph:[global] debug leveldb = 0/0 2026-02-20T13:02:58.008 INFO:tasks.ceph:[global] debug lockdep = 0/0 2026-02-20T13:02:58.008 INFO:tasks.ceph:[global] debug mds = 0/0 2026-02-20T13:02:58.008 INFO:tasks.ceph:[global] debug mds balancer = 0/0 2026-02-20T13:02:58.008 INFO:tasks.ceph:[global] debug mds locker = 0/0 2026-02-20T13:02:58.008 INFO:tasks.ceph:[global] debug mds log = 0/0 2026-02-20T13:02:58.008 INFO:tasks.ceph:[global] debug mds log expire = 0/0 2026-02-20T13:02:58.008 INFO:tasks.ceph:[global] debug mds migrator = 0/0 2026-02-20T13:02:58.008 INFO:tasks.ceph:[global] debug memdb = 0/0 2026-02-20T13:02:58.009 INFO:tasks.ceph:[global] debug mgr = 0/0 2026-02-20T13:02:58.009 INFO:tasks.ceph:[global] debug mgrc = 0/0 2026-02-20T13:02:58.009 INFO:tasks.ceph:[global] debug mon = 0/0 2026-02-20T13:02:58.009 INFO:tasks.ceph:[global] debug monc = 0/0 2026-02-20T13:02:58.009 INFO:tasks.ceph:[global] debug ms = 0/0 2026-02-20T13:02:58.009 INFO:tasks.ceph:[global] debug objclass = 0/0 2026-02-20T13:02:58.009 INFO:tasks.ceph:[global] debug objectcacher = 0/0 2026-02-20T13:02:58.009 INFO:tasks.ceph:[global] debug objecter = 0/0 2026-02-20T13:02:58.009 INFO:tasks.ceph:[global] debug optracker = 0/0 2026-02-20T13:02:58.009 INFO:tasks.ceph:[global] debug osd = 0/0 2026-02-20T13:02:58.009 INFO:tasks.ceph:[global] debug paxos = 0/0 2026-02-20T13:02:58.009 INFO:tasks.ceph:[global] debug perfcounter = 0/0 2026-02-20T13:02:58.010 INFO:tasks.ceph:[global] debug rados = 0/0 2026-02-20T13:02:58.010 INFO:tasks.ceph:[global] debug rbd = 0/0 2026-02-20T13:02:58.010 INFO:tasks.ceph:[global] debug rbd mirror = 0/0 2026-02-20T13:02:58.010 INFO:tasks.ceph:[global] debug rbd replay = 0/0 2026-02-20T13:02:58.010 INFO:tasks.ceph:[global] debug refs = 0/0 2026-02-20T13:02:58.010 INFO:tasks.ceph:[global] debug rgw = 0/0 2026-02-20T13:02:58.010 INFO:tasks.ceph:[global] debug rgw lifecycle = 0/0 2026-02-20T13:02:58.010 INFO:tasks.ceph:[global] debug rgw notification = 0/0 2026-02-20T13:02:58.010 INFO:tasks.ceph:[global] debug rgw sync = 0/0 2026-02-20T13:02:58.011 INFO:tasks.ceph:[global] debug rocksdb = 0/0 2026-02-20T13:02:58.011 INFO:tasks.ceph:[global] debug striper = 0/0 2026-02-20T13:02:58.011 INFO:tasks.ceph:[global] debug throttle = 0/0 2026-02-20T13:02:58.011 INFO:tasks.ceph:[global] debug timer = 0/0 2026-02-20T13:02:58.011 INFO:tasks.ceph:[global] debug tp = 0/0 2026-02-20T13:02:58.011 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2026-02-20T13:02:58.011 INFO:tasks.ceph:[global] ms client mode = crc 2026-02-20T13:02:58.011 INFO:tasks.ceph:[global] ms cluster mode = crc 2026-02-20T13:02:58.011 INFO:tasks.ceph:[global] ms mon client mode = crc 2026-02-20T13:02:58.011 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2026-02-20T13:02:58.011 INFO:tasks.ceph:[global] ms mon service mode = crc 2026-02-20T13:02:58.011 INFO:tasks.ceph:[global] ms service mode = crc 2026-02-20T13:02:58.012 INFO:tasks.ceph:[mgr] debug mgr = 20 2026-02-20T13:02:58.012 INFO:tasks.ceph:[mgr] debug ms = 1 2026-02-20T13:02:58.012 INFO:tasks.ceph:[mon] debug mon = 0/0 2026-02-20T13:02:58.012 INFO:tasks.ceph:[mon] debug ms = 0/0 2026-02-20T13:02:58.012 INFO:tasks.ceph:[mon] debug paxos = 0/0 2026-02-20T13:02:58.012 INFO:tasks.ceph:[mon] osd pool default crimson = True 2026-02-20T13:02:58.012 INFO:tasks.ceph:Setting up mon.a... 2026-02-20T13:02:58.012 DEBUG:teuthology.orchestra.run.trial093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2026-02-20T13:02:58.081 INFO:teuthology.orchestra.run.trial093.stdout:creating /etc/ceph/ceph.keyring 2026-02-20T13:02:58.084 DEBUG:teuthology.orchestra.run.trial093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2026-02-20T13:02:58.168 DEBUG:teuthology.orchestra.run.trial093:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-20T13:02:58.232 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '10.20.193.93')] 2026-02-20T13:02:58.233 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'none', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '10.20.193.93', 'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': 'true', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore cachepin type': '2Q', 'seastore main device type': 'RANDOM_BLOCK_SSD', 'seastore max concurrent transactions': 8}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': '0/0', 'debug mon': '0/0', 'debug paxos': '0/0', 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}} 2026-02-20T13:02:58.235 DEBUG:teuthology.orchestra.run.trial093:> set -ex 2026-02-20T13:02:58.235 DEBUG:teuthology.orchestra.run.trial093:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2026-02-20T13:02:58.291 DEBUG:teuthology.orchestra.run.trial093:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 10.20.193.93 --print /home/ubuntu/cephtest/ceph.monmap 2026-02-20T13:02:58.372 INFO:teuthology.orchestra.run.trial093.stderr:2026-02-20T13:02:58.370+0000 7f350302ff40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T13:02:58.373 INFO:teuthology.orchestra.run.trial093.stderr:2026-02-20T13:02:58.370+0000 7f350302ff40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T13:02:58.373 INFO:teuthology.orchestra.run.trial093.stderr:2026-02-20T13:02:58.370+0000 7f350302ff40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T13:02:58.374 INFO:teuthology.orchestra.run.trial093.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2026-02-20T13:02:58.374 INFO:teuthology.orchestra.run.trial093.stdout:monmaptool: generated fsid 062d8521-4eb6-4504-8170-0bfd8fbdcb1a 2026-02-20T13:02:58.374 INFO:teuthology.orchestra.run.trial093.stdout:setting min_mon_release = tentacle 2026-02-20T13:02:58.374 INFO:teuthology.orchestra.run.trial093.stdout:epoch 0 2026-02-20T13:02:58.375 INFO:teuthology.orchestra.run.trial093.stdout:fsid 062d8521-4eb6-4504-8170-0bfd8fbdcb1a 2026-02-20T13:02:58.375 INFO:teuthology.orchestra.run.trial093.stdout:last_changed 2026-02-20T13:02:58.371654+0000 2026-02-20T13:02:58.375 INFO:teuthology.orchestra.run.trial093.stdout:created 2026-02-20T13:02:58.371654+0000 2026-02-20T13:02:58.375 INFO:teuthology.orchestra.run.trial093.stdout:min_mon_release 20 (tentacle) 2026-02-20T13:02:58.375 INFO:teuthology.orchestra.run.trial093.stdout:election_strategy: 1 2026-02-20T13:02:58.375 INFO:teuthology.orchestra.run.trial093.stdout:0: [v2:10.20.193.93:3300/0,v1:10.20.193.93:6789/0] mon.a 2026-02-20T13:02:58.375 INFO:teuthology.orchestra.run.trial093.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2026-02-20T13:02:58.376 DEBUG:teuthology.orchestra.run.trial093:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2026-02-20T13:02:58.430 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 062d8521-4eb6-4504-8170-0bfd8fbdcb1a... 2026-02-20T13:02:58.432 DEBUG:teuthology.orchestra.run.trial093:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2026-02-20T13:02:58.512 INFO:teuthology.orchestra.run.trial093.stdout:[global] 2026-02-20T13:02:58.512 INFO:teuthology.orchestra.run.trial093.stdout: chdir = "" 2026-02-20T13:02:58.513 INFO:teuthology.orchestra.run.trial093.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2026-02-20T13:02:58.513 INFO:teuthology.orchestra.run.trial093.stdout: auth supported = none 2026-02-20T13:02:58.513 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:02:58.513 INFO:teuthology.orchestra.run.trial093.stdout: filestore xattr use omap = true 2026-02-20T13:02:58.513 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:02:58.513 INFO:teuthology.orchestra.run.trial093.stdout: mon clock drift allowed = 1.000 2026-02-20T13:02:58.513 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:02:58.513 INFO:teuthology.orchestra.run.trial093.stdout: osd crush chooseleaf type = 0 2026-02-20T13:02:58.513 INFO:teuthology.orchestra.run.trial093.stdout: auth debug = true 2026-02-20T13:02:58.513 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:02:58.513 INFO:teuthology.orchestra.run.trial093.stdout: ms die on old message = true 2026-02-20T13:02:58.514 INFO:teuthology.orchestra.run.trial093.stdout: ms die on bug = true 2026-02-20T13:02:58.514 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:02:58.514 INFO:teuthology.orchestra.run.trial093.stdout: mon max pg per osd = 10000 # >= luminous 2026-02-20T13:02:58.514 INFO:teuthology.orchestra.run.trial093.stdout: mon pg warn max object skew = 0 2026-02-20T13:02:58.514 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:02:58.514 INFO:teuthology.orchestra.run.trial093.stdout: # disable pg_autoscaler by default for new pools 2026-02-20T13:02:58.514 INFO:teuthology.orchestra.run.trial093.stdout: osd_pool_default_pg_autoscale_mode = off 2026-02-20T13:02:58.514 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:02:58.514 INFO:teuthology.orchestra.run.trial093.stdout: osd pool default size = 2 2026-02-20T13:02:58.514 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:02:58.514 INFO:teuthology.orchestra.run.trial093.stdout: mon osd allow primary affinity = true 2026-02-20T13:02:58.514 INFO:teuthology.orchestra.run.trial093.stdout: mon osd allow pg remap = true 2026-02-20T13:02:58.515 INFO:teuthology.orchestra.run.trial093.stdout: mon warn on legacy crush tunables = false 2026-02-20T13:02:58.515 INFO:teuthology.orchestra.run.trial093.stdout: mon warn on crush straw calc version zero = false 2026-02-20T13:02:58.515 INFO:teuthology.orchestra.run.trial093.stdout: mon warn on no sortbitwise = false 2026-02-20T13:02:58.515 INFO:teuthology.orchestra.run.trial093.stdout: mon warn on osd down out interval zero = false 2026-02-20T13:02:58.515 INFO:teuthology.orchestra.run.trial093.stdout: mon warn on too few osds = false 2026-02-20T13:02:58.515 INFO:teuthology.orchestra.run.trial093.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2026-02-20T13:02:58.515 INFO:teuthology.orchestra.run.trial093.stdout: mon_warn_on_pool_no_redundancy = false 2026-02-20T13:02:58.515 INFO:teuthology.orchestra.run.trial093.stdout: mon_allow_pool_size_one = true 2026-02-20T13:02:58.515 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:02:58.515 INFO:teuthology.orchestra.run.trial093.stdout: osd pool default erasure code profile = plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2026-02-20T13:02:58.515 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:02:58.515 INFO:teuthology.orchestra.run.trial093.stdout: osd default data pool replay window = 5 2026-02-20T13:02:58.516 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:02:58.516 INFO:teuthology.orchestra.run.trial093.stdout: mon allow pool delete = true 2026-02-20T13:02:58.516 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:02:58.516 INFO:teuthology.orchestra.run.trial093.stdout: mon cluster log file level = debug 2026-02-20T13:02:58.516 INFO:teuthology.orchestra.run.trial093.stdout: debug asserts on shutdown = true 2026-02-20T13:02:58.516 INFO:teuthology.orchestra.run.trial093.stdout: mon health detail to clog = false 2026-02-20T13:02:58.516 INFO:teuthology.orchestra.run.trial093.stdout: mon host = 10.20.193.93 2026-02-20T13:02:58.516 INFO:teuthology.orchestra.run.trial093.stdout: auth client required = none 2026-02-20T13:02:58.516 INFO:teuthology.orchestra.run.trial093.stdout: auth cluster required = none 2026-02-20T13:02:58.517 INFO:teuthology.orchestra.run.trial093.stdout: auth service required = none 2026-02-20T13:02:58.517 INFO:teuthology.orchestra.run.trial093.stdout: debug asok = 0/0 2026-02-20T13:02:58.517 INFO:teuthology.orchestra.run.trial093.stdout: debug auth = 0/0 2026-02-20T13:02:58.517 INFO:teuthology.orchestra.run.trial093.stdout: debug bdev = 0/0 2026-02-20T13:02:58.517 INFO:teuthology.orchestra.run.trial093.stdout: debug bluefs = 0/0 2026-02-20T13:02:58.517 INFO:teuthology.orchestra.run.trial093.stdout: debug bluestore = 0/0 2026-02-20T13:02:58.517 INFO:teuthology.orchestra.run.trial093.stdout: debug buffer = 0/0 2026-02-20T13:02:58.517 INFO:teuthology.orchestra.run.trial093.stdout: debug civetweb = 0/0 2026-02-20T13:02:58.517 INFO:teuthology.orchestra.run.trial093.stdout: debug client = 0/0 2026-02-20T13:02:58.517 INFO:teuthology.orchestra.run.trial093.stdout: debug compressor = 0/0 2026-02-20T13:02:58.517 INFO:teuthology.orchestra.run.trial093.stdout: debug context = 0/0 2026-02-20T13:02:58.517 INFO:teuthology.orchestra.run.trial093.stdout: debug crush = 0/0 2026-02-20T13:02:58.517 INFO:teuthology.orchestra.run.trial093.stdout: debug crypto = 0/0 2026-02-20T13:02:58.518 INFO:teuthology.orchestra.run.trial093.stdout: debug dpdk = 0/0 2026-02-20T13:02:58.518 INFO:teuthology.orchestra.run.trial093.stdout: debug eventtrace = 0/0 2026-02-20T13:02:58.518 INFO:teuthology.orchestra.run.trial093.stdout: debug filer = 0/0 2026-02-20T13:02:58.518 INFO:teuthology.orchestra.run.trial093.stdout: debug filestore = 0/0 2026-02-20T13:02:58.518 INFO:teuthology.orchestra.run.trial093.stdout: debug finisher = 0/0 2026-02-20T13:02:58.518 INFO:teuthology.orchestra.run.trial093.stdout: debug fuse = 0/0 2026-02-20T13:02:58.518 INFO:teuthology.orchestra.run.trial093.stdout: debug heartbeatmap = 0/0 2026-02-20T13:02:58.518 INFO:teuthology.orchestra.run.trial093.stdout: debug javaclient = 0/0 2026-02-20T13:02:58.518 INFO:teuthology.orchestra.run.trial093.stdout: debug journal = 0/0 2026-02-20T13:02:58.518 INFO:teuthology.orchestra.run.trial093.stdout: debug journaler = 0/0 2026-02-20T13:02:58.518 INFO:teuthology.orchestra.run.trial093.stdout: debug kstore = 0/0 2026-02-20T13:02:58.518 INFO:teuthology.orchestra.run.trial093.stdout: debug leveldb = 0/0 2026-02-20T13:02:58.518 INFO:teuthology.orchestra.run.trial093.stdout: debug lockdep = 0/0 2026-02-20T13:02:58.519 INFO:teuthology.orchestra.run.trial093.stdout: debug mds = 0/0 2026-02-20T13:02:58.519 INFO:teuthology.orchestra.run.trial093.stdout: debug mds balancer = 0/0 2026-02-20T13:02:58.519 INFO:teuthology.orchestra.run.trial093.stdout: debug mds locker = 0/0 2026-02-20T13:02:58.519 INFO:teuthology.orchestra.run.trial093.stdout: debug mds log = 0/0 2026-02-20T13:02:58.519 INFO:teuthology.orchestra.run.trial093.stdout: debug mds log expire = 0/0 2026-02-20T13:02:58.519 INFO:teuthology.orchestra.run.trial093.stdout: debug mds migrator = 0/0 2026-02-20T13:02:58.519 INFO:teuthology.orchestra.run.trial093.stdout: debug memdb = 0/0 2026-02-20T13:02:58.519 INFO:teuthology.orchestra.run.trial093.stdout: debug mgr = 0/0 2026-02-20T13:02:58.519 INFO:teuthology.orchestra.run.trial093.stdout: debug mgrc = 0/0 2026-02-20T13:02:58.519 INFO:teuthology.orchestra.run.trial093.stdout: debug mon = 0/0 2026-02-20T13:02:58.519 INFO:teuthology.orchestra.run.trial093.stdout: debug monc = 0/0 2026-02-20T13:02:58.519 INFO:teuthology.orchestra.run.trial093.stdout: debug ms = 0/0 2026-02-20T13:02:58.519 INFO:teuthology.orchestra.run.trial093.stdout: debug objclass = 0/0 2026-02-20T13:02:58.519 INFO:teuthology.orchestra.run.trial093.stdout: debug objectcacher = 0/0 2026-02-20T13:02:58.520 INFO:teuthology.orchestra.run.trial093.stdout: debug objecter = 0/0 2026-02-20T13:02:58.520 INFO:teuthology.orchestra.run.trial093.stdout: debug optracker = 0/0 2026-02-20T13:02:58.520 INFO:teuthology.orchestra.run.trial093.stdout: debug osd = 0/0 2026-02-20T13:02:58.520 INFO:teuthology.orchestra.run.trial093.stdout: debug paxos = 0/0 2026-02-20T13:02:58.520 INFO:teuthology.orchestra.run.trial093.stdout: debug perfcounter = 0/0 2026-02-20T13:02:58.520 INFO:teuthology.orchestra.run.trial093.stdout: debug rados = 0/0 2026-02-20T13:02:58.520 INFO:teuthology.orchestra.run.trial093.stdout: debug rbd = 0/0 2026-02-20T13:02:58.520 INFO:teuthology.orchestra.run.trial093.stdout: debug rbd mirror = 0/0 2026-02-20T13:02:58.520 INFO:teuthology.orchestra.run.trial093.stdout: debug rbd replay = 0/0 2026-02-20T13:02:58.520 INFO:teuthology.orchestra.run.trial093.stdout: debug refs = 0/0 2026-02-20T13:02:58.520 INFO:teuthology.orchestra.run.trial093.stdout: debug rgw = 0/0 2026-02-20T13:02:58.520 INFO:teuthology.orchestra.run.trial093.stdout: debug rgw lifecycle = 0/0 2026-02-20T13:02:58.520 INFO:teuthology.orchestra.run.trial093.stdout: debug rgw notification = 0/0 2026-02-20T13:02:58.521 INFO:teuthology.orchestra.run.trial093.stdout: debug rgw sync = 0/0 2026-02-20T13:02:58.521 INFO:teuthology.orchestra.run.trial093.stdout: debug rocksdb = 0/0 2026-02-20T13:02:58.521 INFO:teuthology.orchestra.run.trial093.stdout: debug striper = 0/0 2026-02-20T13:02:58.521 INFO:teuthology.orchestra.run.trial093.stdout: debug throttle = 0/0 2026-02-20T13:02:58.521 INFO:teuthology.orchestra.run.trial093.stdout: debug timer = 0/0 2026-02-20T13:02:58.521 INFO:teuthology.orchestra.run.trial093.stdout: debug tp = 0/0 2026-02-20T13:02:58.521 INFO:teuthology.orchestra.run.trial093.stdout: enable experimental unrecoverable data corrupting features = crimson 2026-02-20T13:02:58.521 INFO:teuthology.orchestra.run.trial093.stdout: ms client mode = crc 2026-02-20T13:02:58.521 INFO:teuthology.orchestra.run.trial093.stdout: ms cluster mode = crc 2026-02-20T13:02:58.521 INFO:teuthology.orchestra.run.trial093.stdout: ms mon client mode = crc 2026-02-20T13:02:58.521 INFO:teuthology.orchestra.run.trial093.stdout: ms mon cluster mode = crc 2026-02-20T13:02:58.521 INFO:teuthology.orchestra.run.trial093.stdout: ms mon service mode = crc 2026-02-20T13:02:58.521 INFO:teuthology.orchestra.run.trial093.stdout: ms service mode = crc 2026-02-20T13:02:58.522 INFO:teuthology.orchestra.run.trial093.stdout: fsid = 062d8521-4eb6-4504-8170-0bfd8fbdcb1a 2026-02-20T13:02:58.522 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:02:58.522 INFO:teuthology.orchestra.run.trial093.stdout:[osd] 2026-02-20T13:02:58.522 INFO:teuthology.orchestra.run.trial093.stdout: osd journal size = 100 2026-02-20T13:02:58.522 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:02:58.522 INFO:teuthology.orchestra.run.trial093.stdout: osd scrub load threshold = 5.0 2026-02-20T13:02:58.522 INFO:teuthology.orchestra.run.trial093.stdout: osd scrub max interval = 600 2026-02-20T13:02:58.522 INFO:teuthology.orchestra.run.trial093.stdout: osd mclock profile = high_recovery_ops 2026-02-20T13:02:58.522 INFO:teuthology.orchestra.run.trial093.stdout: osd mclock skip benchmark = true 2026-02-20T13:02:58.522 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:02:58.522 INFO:teuthology.orchestra.run.trial093.stdout: osd recover clone overlap = true 2026-02-20T13:02:58.522 INFO:teuthology.orchestra.run.trial093.stdout: osd recovery max chunk = 1048576 2026-02-20T13:02:58.523 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:02:58.523 INFO:teuthology.orchestra.run.trial093.stdout: osd debug shutdown = true 2026-02-20T13:02:58.523 INFO:teuthology.orchestra.run.trial093.stdout: osd debug op order = true 2026-02-20T13:02:58.523 INFO:teuthology.orchestra.run.trial093.stdout: osd debug verify stray on activate = true 2026-02-20T13:02:58.523 INFO:teuthology.orchestra.run.trial093.stdout: osd debug trim objects = true 2026-02-20T13:02:58.523 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:02:58.523 INFO:teuthology.orchestra.run.trial093.stdout: osd open classes on start = true 2026-02-20T13:02:58.523 INFO:teuthology.orchestra.run.trial093.stdout: osd debug pg log writeout = true 2026-02-20T13:02:58.523 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:02:58.523 INFO:teuthology.orchestra.run.trial093.stdout: osd deep scrub update digest min age = 30 2026-02-20T13:02:58.523 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:02:58.524 INFO:teuthology.orchestra.run.trial093.stdout: osd map max advance = 10 2026-02-20T13:02:58.524 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:02:58.524 INFO:teuthology.orchestra.run.trial093.stdout: journal zero on create = true 2026-02-20T13:02:58.524 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:02:58.524 INFO:teuthology.orchestra.run.trial093.stdout: filestore ondisk finisher threads = 3 2026-02-20T13:02:58.524 INFO:teuthology.orchestra.run.trial093.stdout: filestore apply finisher threads = 3 2026-02-20T13:02:58.524 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:02:58.524 INFO:teuthology.orchestra.run.trial093.stdout: bdev debug aio = true 2026-02-20T13:02:58.524 INFO:teuthology.orchestra.run.trial093.stdout: osd debug misdirected ops = true 2026-02-20T13:02:58.524 INFO:teuthology.orchestra.run.trial093.stdout: debug monc = 20 2026-02-20T13:02:58.524 INFO:teuthology.orchestra.run.trial093.stdout: crimson cpu num = 4 2026-02-20T13:02:58.524 INFO:teuthology.orchestra.run.trial093.stdout: crimson osd obc lru size = 10 2026-02-20T13:02:58.524 INFO:teuthology.orchestra.run.trial093.stdout: debug filestore = 0/0 2026-02-20T13:02:58.525 INFO:teuthology.orchestra.run.trial093.stdout: debug journal = 0/0 2026-02-20T13:02:58.525 INFO:teuthology.orchestra.run.trial093.stdout: debug ms = 0/0 2026-02-20T13:02:58.525 INFO:teuthology.orchestra.run.trial093.stdout: debug osd = 0/0 2026-02-20T13:02:58.525 INFO:teuthology.orchestra.run.trial093.stdout: debug seastore = 20 2026-02-20T13:02:58.525 INFO:teuthology.orchestra.run.trial093.stdout: debug seastore backref = 20 2026-02-20T13:02:58.525 INFO:teuthology.orchestra.run.trial093.stdout: debug seastore cache = 20 2026-02-20T13:02:58.525 INFO:teuthology.orchestra.run.trial093.stdout: debug seastore cleaner = 20 2026-02-20T13:02:58.525 INFO:teuthology.orchestra.run.trial093.stdout: debug seastore device = 20 2026-02-20T13:02:58.525 INFO:teuthology.orchestra.run.trial093.stdout: debug seastore epm = 20 2026-02-20T13:02:58.525 INFO:teuthology.orchestra.run.trial093.stdout: debug seastore fixedkv tree = 20 2026-02-20T13:02:58.525 INFO:teuthology.orchestra.run.trial093.stdout: debug seastore journal = 20 2026-02-20T13:02:58.525 INFO:teuthology.orchestra.run.trial093.stdout: debug seastore lba = 20 2026-02-20T13:02:58.525 INFO:teuthology.orchestra.run.trial093.stdout: debug seastore odata = 20 2026-02-20T13:02:58.526 INFO:teuthology.orchestra.run.trial093.stdout: debug seastore omap = 20 2026-02-20T13:02:58.526 INFO:teuthology.orchestra.run.trial093.stdout: debug seastore onode = 20 2026-02-20T13:02:58.526 INFO:teuthology.orchestra.run.trial093.stdout: debug seastore t = 20 2026-02-20T13:02:58.526 INFO:teuthology.orchestra.run.trial093.stdout: debug seastore tm = 20 2026-02-20T13:02:58.526 INFO:teuthology.orchestra.run.trial093.stdout: osd objectstore = seastore 2026-02-20T13:02:58.526 INFO:teuthology.orchestra.run.trial093.stdout: osd shutdown pgref assert = True 2026-02-20T13:02:58.526 INFO:teuthology.orchestra.run.trial093.stdout: seastore cachepin size pershard = 64M 2026-02-20T13:02:58.526 INFO:teuthology.orchestra.run.trial093.stdout: seastore cachepin type = 2Q 2026-02-20T13:02:58.526 INFO:teuthology.orchestra.run.trial093.stdout: seastore main device type = RANDOM_BLOCK_SSD 2026-02-20T13:02:58.526 INFO:teuthology.orchestra.run.trial093.stdout: seastore max concurrent transactions = 8 2026-02-20T13:02:58.527 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:02:58.527 INFO:teuthology.orchestra.run.trial093.stdout:[mgr] 2026-02-20T13:02:58.527 INFO:teuthology.orchestra.run.trial093.stdout: debug ms = 1 2026-02-20T13:02:58.527 INFO:teuthology.orchestra.run.trial093.stdout: debug mgr = 20 2026-02-20T13:02:58.527 INFO:teuthology.orchestra.run.trial093.stdout: debug mon = 20 2026-02-20T13:02:58.527 INFO:teuthology.orchestra.run.trial093.stdout: debug auth = 20 2026-02-20T13:02:58.527 INFO:teuthology.orchestra.run.trial093.stdout: mon reweight min pgs per osd = 4 2026-02-20T13:02:58.527 INFO:teuthology.orchestra.run.trial093.stdout: mon reweight min bytes per osd = 10 2026-02-20T13:02:58.527 INFO:teuthology.orchestra.run.trial093.stdout: mgr/telemetry/nag = false 2026-02-20T13:02:58.528 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:02:58.528 INFO:teuthology.orchestra.run.trial093.stdout:[mon] 2026-02-20T13:02:58.528 INFO:teuthology.orchestra.run.trial093.stdout: debug ms = 0/0 2026-02-20T13:02:58.528 INFO:teuthology.orchestra.run.trial093.stdout: debug mon = 0/0 2026-02-20T13:02:58.528 INFO:teuthology.orchestra.run.trial093.stdout: debug paxos = 0/0 2026-02-20T13:02:58.528 INFO:teuthology.orchestra.run.trial093.stdout: debug auth = 20 2026-02-20T13:02:58.528 INFO:teuthology.orchestra.run.trial093.stdout: mon data avail warn = 5 2026-02-20T13:02:58.528 INFO:teuthology.orchestra.run.trial093.stdout: mon mgr mkfs grace = 240 2026-02-20T13:02:58.528 INFO:teuthology.orchestra.run.trial093.stdout: mon reweight min pgs per osd = 4 2026-02-20T13:02:58.528 INFO:teuthology.orchestra.run.trial093.stdout: mon osd reporter subtree level = osd 2026-02-20T13:02:58.528 INFO:teuthology.orchestra.run.trial093.stdout: mon osd prime pg temp = true 2026-02-20T13:02:58.528 INFO:teuthology.orchestra.run.trial093.stdout: mon reweight min bytes per osd = 10 2026-02-20T13:02:58.529 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:02:58.529 INFO:teuthology.orchestra.run.trial093.stdout: # rotate auth tickets quickly to exercise renewal paths 2026-02-20T13:02:58.529 INFO:teuthology.orchestra.run.trial093.stdout: auth mon ticket ttl = 660 # 11m 2026-02-20T13:02:58.529 INFO:teuthology.orchestra.run.trial093.stdout: auth service ticket ttl = 240 # 4m 2026-02-20T13:02:58.529 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:02:58.529 INFO:teuthology.orchestra.run.trial093.stdout: # don't complain about insecure global_id in the test suite 2026-02-20T13:02:58.529 INFO:teuthology.orchestra.run.trial093.stdout: mon_warn_on_insecure_global_id_reclaim = false 2026-02-20T13:02:58.529 INFO:teuthology.orchestra.run.trial093.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2026-02-20T13:02:58.529 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:02:58.529 INFO:teuthology.orchestra.run.trial093.stdout: # 1m isn't quite enough 2026-02-20T13:02:58.529 INFO:teuthology.orchestra.run.trial093.stdout: mon_down_mkfs_grace = 2m 2026-02-20T13:02:58.530 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:02:58.530 INFO:teuthology.orchestra.run.trial093.stdout: mon_warn_on_filestore_osds = false 2026-02-20T13:02:58.530 INFO:teuthology.orchestra.run.trial093.stdout: osd pool default crimson = True 2026-02-20T13:02:58.530 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:02:58.530 INFO:teuthology.orchestra.run.trial093.stdout:[client] 2026-02-20T13:02:58.530 INFO:teuthology.orchestra.run.trial093.stdout: rgw cache enabled = true 2026-02-20T13:02:58.530 INFO:teuthology.orchestra.run.trial093.stdout: rgw enable ops log = true 2026-02-20T13:02:58.530 INFO:teuthology.orchestra.run.trial093.stdout: rgw enable usage log = true 2026-02-20T13:02:58.530 INFO:teuthology.orchestra.run.trial093.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2026-02-20T13:02:58.530 INFO:teuthology.orchestra.run.trial093.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2026-02-20T13:02:58.530 INFO:teuthology.orchestra.run.trial093.stdout:[mon.a] 2026-02-20T13:02:58.532 INFO:tasks.ceph:Creating admin key on mon.a... 2026-02-20T13:02:58.532 DEBUG:teuthology.orchestra.run.trial093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2026-02-20T13:02:58.623 INFO:tasks.ceph:Copying monmap to all nodes... 2026-02-20T13:02:58.623 DEBUG:teuthology.orchestra.run.trial093:> set -ex 2026-02-20T13:02:58.623 DEBUG:teuthology.orchestra.run.trial093:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2026-02-20T13:02:58.640 DEBUG:teuthology.orchestra.run.trial093:> set -ex 2026-02-20T13:02:58.640 DEBUG:teuthology.orchestra.run.trial093:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2026-02-20T13:02:58.697 INFO:tasks.ceph:Sending monmap to node ubuntu@trial093.front.sepia.ceph.com 2026-02-20T13:02:58.698 DEBUG:teuthology.orchestra.run.trial093:> set -ex 2026-02-20T13:02:58.698 DEBUG:teuthology.orchestra.run.trial093:> sudo dd of=/etc/ceph/ceph.keyring 2026-02-20T13:02:58.698 DEBUG:teuthology.orchestra.run.trial093:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-20T13:02:58.773 DEBUG:teuthology.orchestra.run.trial093:> set -ex 2026-02-20T13:02:58.773 DEBUG:teuthology.orchestra.run.trial093:> dd of=/home/ubuntu/cephtest/ceph.monmap 2026-02-20T13:02:58.829 INFO:tasks.ceph:Setting up mon nodes... 2026-02-20T13:02:58.829 INFO:tasks.ceph:Setting up mgr nodes... 2026-02-20T13:02:58.829 DEBUG:teuthology.orchestra.run.trial093:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2026-02-20T13:02:58.920 INFO:teuthology.orchestra.run.trial093.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2026-02-20T13:02:58.924 INFO:tasks.ceph:Setting up mds nodes... 2026-02-20T13:02:58.924 INFO:tasks.ceph_client:Setting up client nodes... 2026-02-20T13:02:58.925 DEBUG:teuthology.orchestra.run.trial093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2026-02-20T13:02:58.966 INFO:teuthology.orchestra.run.trial093.stdout:creating /etc/ceph/ceph.client.0.keyring 2026-02-20T13:02:58.978 INFO:tasks.ceph:Running mkfs on osd nodes... 2026-02-20T13:02:58.979 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@trial093.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2026-02-20T13:02:58.979 DEBUG:teuthology.orchestra.run.trial093:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2026-02-20T13:02:59.042 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-20T13:02:59.043 INFO:tasks.ceph:role: osd.0 2026-02-20T13:02:59.043 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@trial093.front.sepia.ceph.com 2026-02-20T13:02:59.043 DEBUG:teuthology.orchestra.run.trial093:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2026-02-20T13:02:59.106 INFO:teuthology.orchestra.run.trial093.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=45875200 blks 2026-02-20T13:02:59.106 INFO:teuthology.orchestra.run.trial093.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-20T13:02:59.106 INFO:teuthology.orchestra.run.trial093.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-20T13:02:59.106 INFO:teuthology.orchestra.run.trial093.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-20T13:02:59.106 INFO:teuthology.orchestra.run.trial093.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-20T13:02:59.106 INFO:teuthology.orchestra.run.trial093.stdout: = sunit=0 swidth=0 blks 2026-02-20T13:02:59.106 INFO:teuthology.orchestra.run.trial093.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-20T13:02:59.106 INFO:teuthology.orchestra.run.trial093.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-20T13:02:59.106 INFO:teuthology.orchestra.run.trial093.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-20T13:02:59.106 INFO:teuthology.orchestra.run.trial093.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-20T13:02:59.680 INFO:teuthology.orchestra.run.trial093.stdout:Discarding blocks...Done. 2026-02-20T13:02:59.682 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@trial093.front.sepia.ceph.com -o noatime 2026-02-20T13:02:59.682 DEBUG:teuthology.orchestra.run.trial093:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2026-02-20T13:02:59.753 DEBUG:teuthology.orchestra.run.trial093:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2026-02-20T13:02:59.820 DEBUG:teuthology.orchestra.run.trial093:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2026-02-20T13:02:59.883 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-20T13:02:59.884 INFO:tasks.ceph:role: osd.1 2026-02-20T13:02:59.884 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@trial093.front.sepia.ceph.com 2026-02-20T13:02:59.884 DEBUG:teuthology.orchestra.run.trial093:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2026-02-20T13:02:59.950 INFO:teuthology.orchestra.run.trial093.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=45875200 blks 2026-02-20T13:02:59.950 INFO:teuthology.orchestra.run.trial093.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-20T13:02:59.950 INFO:teuthology.orchestra.run.trial093.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-20T13:02:59.950 INFO:teuthology.orchestra.run.trial093.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-20T13:02:59.950 INFO:teuthology.orchestra.run.trial093.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-20T13:02:59.950 INFO:teuthology.orchestra.run.trial093.stdout: = sunit=0 swidth=0 blks 2026-02-20T13:02:59.950 INFO:teuthology.orchestra.run.trial093.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-20T13:02:59.950 INFO:teuthology.orchestra.run.trial093.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-20T13:02:59.950 INFO:teuthology.orchestra.run.trial093.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-20T13:02:59.951 INFO:teuthology.orchestra.run.trial093.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-20T13:03:00.515 INFO:teuthology.orchestra.run.trial093.stdout:Discarding blocks...Done. 2026-02-20T13:03:00.517 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@trial093.front.sepia.ceph.com -o noatime 2026-02-20T13:03:00.517 DEBUG:teuthology.orchestra.run.trial093:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2026-02-20T13:03:00.584 DEBUG:teuthology.orchestra.run.trial093:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2026-02-20T13:03:00.651 DEBUG:teuthology.orchestra.run.trial093:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2026-02-20T13:03:00.714 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-20T13:03:00.714 INFO:tasks.ceph:role: osd.2 2026-02-20T13:03:00.714 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@trial093.front.sepia.ceph.com 2026-02-20T13:03:00.714 DEBUG:teuthology.orchestra.run.trial093:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2026-02-20T13:03:00.778 INFO:teuthology.orchestra.run.trial093.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=45875200 blks 2026-02-20T13:03:00.779 INFO:teuthology.orchestra.run.trial093.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-20T13:03:00.779 INFO:teuthology.orchestra.run.trial093.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-20T13:03:00.779 INFO:teuthology.orchestra.run.trial093.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-20T13:03:00.779 INFO:teuthology.orchestra.run.trial093.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-20T13:03:00.779 INFO:teuthology.orchestra.run.trial093.stdout: = sunit=0 swidth=0 blks 2026-02-20T13:03:00.779 INFO:teuthology.orchestra.run.trial093.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-20T13:03:00.779 INFO:teuthology.orchestra.run.trial093.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-20T13:03:00.779 INFO:teuthology.orchestra.run.trial093.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-20T13:03:00.779 INFO:teuthology.orchestra.run.trial093.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-20T13:03:01.333 INFO:teuthology.orchestra.run.trial093.stdout:Discarding blocks...Done. 2026-02-20T13:03:01.335 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@trial093.front.sepia.ceph.com -o noatime 2026-02-20T13:03:01.335 DEBUG:teuthology.orchestra.run.trial093:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2026-02-20T13:03:01.402 DEBUG:teuthology.orchestra.run.trial093:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2026-02-20T13:03:01.471 DEBUG:teuthology.orchestra.run.trial093:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-20T13:03:01.619 INFO:teuthology.orchestra.run.trial093.stderr:INFO  osd - OSD::main: parsing early config 2026-02-20T13:03:01.674 INFO:teuthology.orchestra.run.trial093.stderr:INFO 2026-02-20 13:03:01,673 seastar - Reactor backend: linux-aio 2026-02-20T13:03:01.688 INFO:teuthology.orchestra.run.trial093.stderr:INFO 2026-02-20 13:03:01,687 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-20T13:03:01.688 INFO:teuthology.orchestra.run.trial093.stderr:INFO 2026-02-20 13:03:01,687 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-20T13:03:01.690 INFO:teuthology.orchestra.run.trial093.stderr:==39329==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-20T13:03:01.750 INFO:teuthology.orchestra.run.trial093.stderr:DEBUG 2026-02-20 13:03:01,749 [shard 0:main] seastar - reactor::drain 2026-02-20T13:03:03.778 INFO:teuthology.orchestra.run.trial093.stderr:----------------------------------------------------- 2026-02-20T13:03:03.778 INFO:teuthology.orchestra.run.trial093.stderr:Suppressions used: 2026-02-20T13:03:03.778 INFO:teuthology.orchestra.run.trial093.stderr: count bytes template 2026-02-20T13:03:03.778 INFO:teuthology.orchestra.run.trial093.stderr: 1 8 MallocExtension::Register 2026-02-20T13:03:03.778 INFO:teuthology.orchestra.run.trial093.stderr:----------------------------------------------------- 2026-02-20T13:03:03.778 INFO:teuthology.orchestra.run.trial093.stderr: 2026-02-20T13:03:03.797 INFO:teuthology.orchestra.run.trial093.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-20T13:03:03.797 INFO:teuthology.orchestra.run.trial093.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-20T13:03:03.840 INFO:teuthology.orchestra.run.trial093.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-20T13:03:03.840 INFO:teuthology.orchestra.run.trial093.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-20T13:03:03.847 INFO:teuthology.orchestra.run.trial093.stderr:INFO 2026-02-20 13:03:03,846 seastar - Reactor backend: linux-aio 2026-02-20T13:03:03.857 INFO:teuthology.orchestra.run.trial093.stderr:INFO 2026-02-20 13:03:03,856 [shard 2:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-20T13:03:03.857 INFO:teuthology.orchestra.run.trial093.stderr:INFO 2026-02-20 13:03:03,856 [shard 2:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-20T13:03:03.861 INFO:teuthology.orchestra.run.trial093.stderr:==39327==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-20T13:03:03.861 INFO:teuthology.orchestra.run.trial093.stderr:INFO 2026-02-20 13:03:03,860 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-20T13:03:04.000 INFO:teuthology.orchestra.run.trial093.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9c8bb26 0x9abe76d 0x9af96d3 0x9afcc8c 0x9afcefa 0x9afd042 0x9afd501 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc597 /lib64/libasan.so.8+0x91c1 /lib64/libasan.so.8+0xee7dc 0x9808b5b 0x98beab2 0x98bebc8 0x98beca4 0x98bef20 0x98bf4c7 0x98bf8cb 0x98bfb88 0x9a5bebd 0x99c20b5 0x99c10b5 0x7b60901 0x7b6a470 0x7a0ec75 0x7a27c05 0x70e904b 0x6f6f022 0x6f72e23 0x6f789cd 0x73e30d9 0x7412614 0x7256527 0x725661f 0x732cdbe 0x732ce77 0x732d02e 0x7412695 0x725669c 0x7256721 0x728124b 0x73ebced 0x73ebd72 0x73ebe50 0x740ced4 0x7387fb8 0x7552c41 0x73f3ad4 0x73f3b4f 0x74a8152 0x759b55c 0x9bc0b9d 0x9ac6d0c 0x9b7f759 0x9bcfcf7 0x9d147e2 0x9d14e5e 0x265f87f 0xa34f610 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T13:03:04.000 INFO:teuthology.orchestra.run.trial093.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff985c9616 0xffffffff985d3aeb 0xffffffff985d3f4b 0xffffffff982910d2 0xffffffff98f4d451 0xffffffff99000ba2 2026-02-20T13:03:04.000 INFO:teuthology.orchestra.run.trial093.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9c8bb26 0x9abe76d 0x9af96d3 0x9afcc8c 0x9afcefa 0x9afd042 0x9afd501 /lib64/libc.so.6+0x3fc2f 0x98bef5a 0x98bf4c7 0x98bf8cb 0x98bfb88 0x9a3b3d7 0x9a3b5b5 0x9a3b811 0x9a3b902 0x9a48c46 0x9a4924a 0x9a495dd 0x9a5c268 0x99c20b5 0x99c10b5 0x7b60901 0x7b6a470 0x7a0ec75 0x7a27c05 0x70e904b 0x6f6f022 0x6f72e23 0x6f789cd 0x73e30d9 0x7412614 0x7256527 0x725661f 0x732cdbe 0x732ce77 0x732d02e 0x7412695 0x725669c 0x7256721 0x728124b 0x73ebced 0x73ebd72 0x73ebe50 0x740ced4 0x7387fb8 0x7552c41 0x73f3ad4 0x73f3b4f 0x74a8152 0x759b55c 0x9bc0b9d 0x9ac6d0c 0x9b7f759 0x9bcfcf7 0x9d147e2 0x9d14e5e 0x265f87f 0xa34f610 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T13:03:04.000 INFO:teuthology.orchestra.run.trial093.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9c8bb26 0x9abe76d 0x9af96d3 0x9afcc8c 0x9afcefa 0x9afd042 0x9afd501 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x151f5 0xb4659b 0xb89c50 0xd7a634 0x18a66f6 0x9a414ce 0x9a41874 0x9a59007 0x9a5dcef 0x99c20b5 0x99c10b5 0x7b60901 0x7b6a470 0x7a0ec75 0x7a27c05 0x70e904b 0x6f6f022 0x6f72e23 0x6f789cd 0x73e30d9 0x7412614 0x7256527 0x725661f 0x732cdbe 0x732ce77 0x732d02e 0x7412695 0x725669c 0x7256721 0x728124b 0x73ebced 0x73ebd72 0x73ebe50 0x740ced4 0x7387fb8 0x7552c41 0x73f3ad4 0x73f3b4f 0x74a8152 0x759b55c 0x9bc0b9d 0x9ac6d0c 0x9b7f759 0x9bcfcf7 0x9d147e2 0x9d14e5e 0x265f87f 0xa34f610 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T13:03:04.146 INFO:teuthology.orchestra.run.trial093.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9c8bb26 0x9abe76d 0x9af96d3 0x9afcc8c 0x9afcefa 0x9afd042 0x9afd501 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc65bc /lib64/libasan.so.8+0x15197 0x10dd764 0x10dd7ff 0x993a244 0x993a541 0x993a7b4 0x993c4db 0x993ca28 0x993cc27 0x9a4ee89 0x9a4f5e4 0x9a59876 0x9a5dcef 0x99c20b5 0x99c10b5 0x7b60901 0x7b6a470 0x7a0ec75 0x7a27c05 0x70e904b 0x6f0b27c 0x6f0d5e4 0x73e2d78 0x740ccd4 0x7254c6e 0x7254d66 0x732c65e 0x732c717 0x732c8ce 0x740cd55 0x7254de3 0x7254e68 0x727c2ab 0x73ea73c 0x73ea7c1 0x73ea89f 0x740ced4 0x7387fb8 0x7552c41 0x73f3ad4 0x73f3b4f 0x74a8152 0x759b55c 0x9bc0b9d 0x9ac6d0c 0x9b7f759 0x9bcfcf7 0x9d147e2 0x9d14e5e 0x265f87f 0xa34f610 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T13:03:04.146 INFO:teuthology.orchestra.run.trial093.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9c8bb26 0x9abe76d 0x9af96d3 0x9afcc8c 0x9afcefa 0x9afd042 0x9afd501 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc6592 /lib64/libasan.so.8+0x15197 0x993a0e0 0x993a541 0x993a7b4 0x993aa6a 0x993b02c 0x9940151 0x9940831 0x9a59644 0x9a5dcef 0x99c20b5 0x99c10b5 0x7b60901 0x7b6a470 0x7a0ec75 0x7a27c05 0x70e904b 0x6f0b27c 0x6f0d5e4 0x73e2d78 0x740ccd4 0x7254c6e 0x7254d66 0x732c65e 0x732c717 0x732c8ce 0x740cd55 0x7254de3 0x7254e68 0x727c2ab 0x73ea73c 0x73ea7c1 0x73ea89f 0x740ced4 0x7387fb8 0x7552c41 0x73f3ad4 0x73f3b4f 0x74a8152 0x759b55c 0x9bc0b9d 0x9ac6d0c 0x9b7f759 0x9bcfcf7 0x9d147e2 0x9d14e5e 0x265f87f 0xa34f610 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T13:03:04.146 INFO:teuthology.orchestra.run.trial093.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9c8bb26 0x9abe76d 0x9af96d3 0x9afcc8c 0x9afcefa 0x9afd042 0x9afd501 /lib64/libc.so.6+0x3fc2f 0x9a499c2 0x9a4a3ae 0x9a4a731 0x9a5a88c 0x9a5dcef 0x99c20b5 0x99c10b5 0x7b60901 0x7b6a470 0x7a0ec75 0x7a27c05 0x70e904b 0x6f0b27c 0x6f0d5e4 0x73e2d78 0x740ccd4 0x7254c6e 0x7254d66 0x732c65e 0x732c717 0x732c8ce 0x740cd55 0x7254de3 0x7254e68 0x727c2ab 0x73ea73c 0x73ea7c1 0x73ea89f 0x740ced4 0x7387fb8 0x7552c41 0x73f3ad4 0x73f3b4f 0x74a8152 0x759b55c 0x9bc0b9d 0x9ac6d0c 0x9b7f759 0x9bcfcf7 0x9d147e2 0x9d14e5e 0x265f87f 0xa34f610 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T13:03:06.226 INFO:teuthology.orchestra.run.trial093.stderr:----------------------------------------------------- 2026-02-20T13:03:06.226 INFO:teuthology.orchestra.run.trial093.stderr:Suppressions used: 2026-02-20T13:03:06.227 INFO:teuthology.orchestra.run.trial093.stderr: count bytes template 2026-02-20T13:03:06.227 INFO:teuthology.orchestra.run.trial093.stderr: 1 8 MallocExtension::Register 2026-02-20T13:03:06.227 INFO:teuthology.orchestra.run.trial093.stderr:----------------------------------------------------- 2026-02-20T13:03:06.227 INFO:teuthology.orchestra.run.trial093.stderr: 2026-02-20T13:03:06.255 DEBUG:teuthology.orchestra.run.trial093:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2026-02-20T13:03:06.281 DEBUG:teuthology.orchestra.run.trial093:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-20T13:03:06.429 INFO:teuthology.orchestra.run.trial093.stderr:INFO  osd - OSD::main: parsing early config 2026-02-20T13:03:06.486 INFO:teuthology.orchestra.run.trial093.stderr:INFO 2026-02-20 13:03:06,484 seastar - Reactor backend: linux-aio 2026-02-20T13:03:06.495 INFO:teuthology.orchestra.run.trial093.stderr:INFO 2026-02-20 13:03:06,494 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-20T13:03:06.495 INFO:teuthology.orchestra.run.trial093.stderr:INFO 2026-02-20 13:03:06,494 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-20T13:03:06.497 INFO:teuthology.orchestra.run.trial093.stderr:==39394==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-20T13:03:06.556 INFO:teuthology.orchestra.run.trial093.stderr:DEBUG 2026-02-20 13:03:06,555 [shard 0:main] seastar - reactor::drain 2026-02-20T13:03:08.601 INFO:teuthology.orchestra.run.trial093.stderr:----------------------------------------------------- 2026-02-20T13:03:08.601 INFO:teuthology.orchestra.run.trial093.stderr:Suppressions used: 2026-02-20T13:03:08.601 INFO:teuthology.orchestra.run.trial093.stderr: count bytes template 2026-02-20T13:03:08.601 INFO:teuthology.orchestra.run.trial093.stderr: 1 8 MallocExtension::Register 2026-02-20T13:03:08.601 INFO:teuthology.orchestra.run.trial093.stderr:----------------------------------------------------- 2026-02-20T13:03:08.601 INFO:teuthology.orchestra.run.trial093.stderr: 2026-02-20T13:03:08.620 INFO:teuthology.orchestra.run.trial093.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-20T13:03:08.620 INFO:teuthology.orchestra.run.trial093.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-20T13:03:08.667 INFO:teuthology.orchestra.run.trial093.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-20T13:03:08.667 INFO:teuthology.orchestra.run.trial093.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-20T13:03:08.674 INFO:teuthology.orchestra.run.trial093.stderr:INFO 2026-02-20 13:03:08,673 seastar - Reactor backend: linux-aio 2026-02-20T13:03:08.683 INFO:teuthology.orchestra.run.trial093.stderr:INFO 2026-02-20 13:03:08,683 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-20T13:03:08.683 INFO:teuthology.orchestra.run.trial093.stderr:INFO 2026-02-20 13:03:08,683 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-20T13:03:08.688 INFO:teuthology.orchestra.run.trial093.stderr:==39392==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-20T13:03:08.688 INFO:teuthology.orchestra.run.trial093.stderr:INFO 2026-02-20 13:03:08,688 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-20T13:03:08.828 INFO:teuthology.orchestra.run.trial093.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9c8bb26 0x9abe76d 0x9af96d3 0x9afcc8c 0x9afcefa 0x9afd042 0x9afd501 /lib64/libc.so.6+0x3fc2f 0x993aa20 0x993b02c 0x9940151 0x9940831 0x9a59644 0x9a5dcef 0x99c20b5 0x99c10b5 0x7b60901 0x7b6a470 0x7a0ec75 0x7a27c05 0x70e904b 0x6f6f022 0x6f72e23 0x6f789cd 0x73e30d9 0x7412614 0x7256527 0x725661f 0x732cdbe 0x732ce77 0x732d02e 0x7412695 0x725669c 0x7256721 0x728124b 0x73ebced 0x73ebd72 0x73ebe50 0x740ced4 0x7387fb8 0x7552c41 0x73f3ad4 0x73f3b4f 0x74a8152 0x759b55c 0x9bc0b9d 0x9ac6d0c 0x9b7f759 0x9bcfcf7 0x9d147e2 0x9d14e5e 0x265f87f 0xa34f610 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T13:03:08.829 INFO:teuthology.orchestra.run.trial093.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9c8bb26 0x9abe76d 0x9af96d3 0x9afcc8c 0x9afcefa 0x9afd042 0x9afd501 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1526c 0x10dd764 0x10dd7ff 0x993a077 0x993a541 0x993a7b4 0x993c4db 0x9a4cdf2 0x9a4ea44 0x9a4f09e 0x9a4f5e4 0x9a59876 0x9a5dcef 0x99c20b5 0x99c10b5 0x7b60901 0x7b6a470 0x7a0ec75 0x7a27c05 0x70e904b 0x6f6f022 0x6f72e23 0x6f789cd 0x73e30d9 0x7412614 0x7256527 0x725661f 0x732cdbe 0x732ce77 0x732d02e 0x7412695 0x725669c 0x7256721 0x728124b 0x73ebced 0x73ebd72 0x73ebe50 0x740ced4 0x7387fb8 0x7552c41 0x73f3ad4 0x73f3b4f 0x74a8152 0x759b55c 0x9bc0b9d 0x9ac6d0c 0x9b7f759 0x9bcfcf7 0x9d147e2 0x9d14e5e 0x265f87f 0xa34f610 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T13:03:08.829 INFO:teuthology.orchestra.run.trial093.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9c8bb26 0x9abe76d 0x9af96d3 0x9afcc8c 0x9afcefa 0x9afd042 0x9afd501 /lib64/libc.so.6+0x3fc2f 0xc80bcf 0x10dd6de 0x10dd7ff 0x993a244 0x993a541 0x993a7b4 0x993c4db 0x993ca28 0x993cc27 0x9a4ee89 0x9a4f5e4 0x9a59876 0x9a5dcef 0x99c20b5 0x99c10b5 0x7b60901 0x7b6a470 0x7a0ec75 0x7a27c05 0x70e904b 0x6f6f022 0x6f72e23 0x6f789cd 0x73e30d9 0x7412614 0x7256527 0x725661f 0x732cdbe 0x732ce77 0x732d02e 0x7412695 0x725669c 0x7256721 0x728124b 0x73ebced 0x73ebd72 0x73ebe50 0x740ced4 0x7387fb8 0x7552c41 0x73f3ad4 0x73f3b4f 0x74a8152 0x759b55c 0x9bc0b9d 0x9ac6d0c 0x9b7f759 0x9bcfcf7 0x9d147e2 0x9d14e5e 0x265f87f 0xa34f610 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T13:03:08.973 INFO:teuthology.orchestra.run.trial093.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9c8bb26 0x9abe76d 0x9af96d3 0x9afcc8c 0x9afcefa 0x9afd042 0x9afd501 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc250 /lib64/libasan.so.8+0xc553 /lib64/libasan.so.8+0x8eea /lib64/libasan.so.8+0xedc45 0xd91811 0x786f139 0x7b60233 0x7b6a470 0x7a0ec75 0x7a27c05 0x70e904b 0x6f0b27c 0x6f0d5e4 0x73e2d78 0x740ccd4 0x7254c6e 0x7254d66 0x732c65e 0x732c717 0x732c8ce 0x740cd55 0x7254de3 0x7254e68 0x727c2ab 0x73ea73c 0x73ea7c1 0x73ea89f 0x740ced4 0x7387fb8 0x7552c41 0x73f3ad4 0x73f3b4f 0x74a8152 0x759b55c 0x9bc0b9d 0x9ac6d0c 0x9b7f759 0x9bcfcf7 0x9d147e2 0x9d14e5e 0x265f87f 0xa34f610 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T13:03:08.973 INFO:teuthology.orchestra.run.trial093.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9c8bb26 0x9abe76d 0x9af96d3 0x9afcc8c 0x9afcefa 0x9afd042 0x9afd501 /lib64/libc.so.6+0x3fc2f 0x9944463 0x9a36285 0x9a3644d 0x9a4137f 0x9a41874 0x9a59007 0x9a5dcef 0x99c20b5 0x99c10b5 0x7b60901 0x7b6a470 0x7a0ec75 0x7a27c05 0x70e904b 0x6f0b27c 0x6f0d5e4 0x73e2d78 0x740ccd4 0x7254c6e 0x7254d66 0x732c65e 0x732c717 0x732c8ce 0x740cd55 0x7254de3 0x7254e68 0x727c2ab 0x73ea73c 0x73ea7c1 0x73ea89f 0x740ced4 0x7387fb8 0x7552c41 0x73f3ad4 0x73f3b4f 0x74a8152 0x759b55c 0x9bc0b9d 0x9ac6d0c 0x9b7f759 0x9bcfcf7 0x9d147e2 0x9d14e5e 0x265f87f 0xa34f610 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T13:03:08.973 INFO:teuthology.orchestra.run.trial093.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9c8bb26 0x9abe76d 0x9af96d3 0x9afcc8c 0x9afcefa 0x9afd042 0x9afd501 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x151f5 0x10dd764 0x10dd7ff 0x993a077 0x993a541 0x993a7b4 0x993aa6a 0x993b519 0x993ba17 0x993c047 0x993c1b1 0x9a49461 0x9a5c268 0x99c20b5 0x99c10b5 0x7b60901 0x7b6a470 0x7a0ec75 0x7a27c05 0x70e904b 0x6f0b27c 0x6f0d5e4 0x73e2d78 0x740ccd4 0x7254c6e 0x7254d66 0x732c65e 0x732c717 0x732c8ce 0x740cd55 0x7254de3 0x7254e68 0x727c2ab 0x73ea73c 0x73ea7c1 0x73ea89f 0x740ced4 0x7387fb8 0x7552c41 0x73f3ad4 0x73f3b4f 0x74a8152 0x759b55c 0x9bc0b9d 0x9ac6d0c 0x9b7f759 0x9bcfcf7 0x9d147e2 0x9d14e5e 0x265f87f 0xa34f610 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T13:03:11.100 INFO:teuthology.orchestra.run.trial093.stderr:----------------------------------------------------- 2026-02-20T13:03:11.101 INFO:teuthology.orchestra.run.trial093.stderr:Suppressions used: 2026-02-20T13:03:11.101 INFO:teuthology.orchestra.run.trial093.stderr: count bytes template 2026-02-20T13:03:11.101 INFO:teuthology.orchestra.run.trial093.stderr: 1 8 MallocExtension::Register 2026-02-20T13:03:11.101 INFO:teuthology.orchestra.run.trial093.stderr:----------------------------------------------------- 2026-02-20T13:03:11.101 INFO:teuthology.orchestra.run.trial093.stderr: 2026-02-20T13:03:11.131 DEBUG:teuthology.orchestra.run.trial093:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2026-02-20T13:03:11.156 DEBUG:teuthology.orchestra.run.trial093:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-20T13:03:11.308 INFO:teuthology.orchestra.run.trial093.stderr:INFO  osd - OSD::main: parsing early config 2026-02-20T13:03:11.360 INFO:teuthology.orchestra.run.trial093.stderr:INFO 2026-02-20 13:03:11,358 seastar - Reactor backend: linux-aio 2026-02-20T13:03:11.369 INFO:teuthology.orchestra.run.trial093.stderr:INFO 2026-02-20 13:03:11,368 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-20T13:03:11.369 INFO:teuthology.orchestra.run.trial093.stderr:INFO 2026-02-20 13:03:11,368 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-20T13:03:11.372 INFO:teuthology.orchestra.run.trial093.stderr:==39458==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-20T13:03:11.431 INFO:teuthology.orchestra.run.trial093.stderr:DEBUG 2026-02-20 13:03:11,430 [shard 0:main] seastar - reactor::drain 2026-02-20T13:03:13.495 INFO:teuthology.orchestra.run.trial093.stderr:----------------------------------------------------- 2026-02-20T13:03:13.496 INFO:teuthology.orchestra.run.trial093.stderr:Suppressions used: 2026-02-20T13:03:13.496 INFO:teuthology.orchestra.run.trial093.stderr: count bytes template 2026-02-20T13:03:13.496 INFO:teuthology.orchestra.run.trial093.stderr: 1 8 MallocExtension::Register 2026-02-20T13:03:13.496 INFO:teuthology.orchestra.run.trial093.stderr:----------------------------------------------------- 2026-02-20T13:03:13.496 INFO:teuthology.orchestra.run.trial093.stderr: 2026-02-20T13:03:13.515 INFO:teuthology.orchestra.run.trial093.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-20T13:03:13.515 INFO:teuthology.orchestra.run.trial093.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-20T13:03:13.561 INFO:teuthology.orchestra.run.trial093.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-20T13:03:13.561 INFO:teuthology.orchestra.run.trial093.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-20T13:03:13.568 INFO:teuthology.orchestra.run.trial093.stderr:INFO 2026-02-20 13:03:13,567 seastar - Reactor backend: linux-aio 2026-02-20T13:03:13.582 INFO:teuthology.orchestra.run.trial093.stderr:INFO 2026-02-20 13:03:13,581 [shard 2:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-20T13:03:13.582 INFO:teuthology.orchestra.run.trial093.stderr:INFO 2026-02-20 13:03:13,581 [shard 2:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-20T13:03:13.586 INFO:teuthology.orchestra.run.trial093.stderr:==39456==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-20T13:03:13.586 INFO:teuthology.orchestra.run.trial093.stderr:INFO 2026-02-20 13:03:13,585 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-20T13:03:13.726 INFO:teuthology.orchestra.run.trial093.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9c8bb26 0x9abe76d 0x9af96d3 0x9afcc8c 0x9afcefa 0x9afd042 0x9afd501 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xf4e9a /lib64/libasan.so.8+0x10069c /lib64/libasan.so.8+0xfd19a /lib64/libasan.so.8+0xee859 0x9a3e7ca 0x9a3e896 0x9a58b45 0x9a5dcef 0x99c20b5 0x99c10b5 0x7b60901 0x7b6a470 0x7a0ec75 0x7a27c05 0x70e904b 0x6f6f022 0x6f72e23 0x6f789cd 0x73e30d9 0x7412614 0x7256527 0x725661f 0x732cdbe 0x732ce77 0x732d02e 0x7412695 0x725669c 0x7256721 0x728124b 0x73ebced 0x73ebd72 0x73ebe50 0x740ced4 0x7387fb8 0x7552c41 0x73f3ad4 0x73f3b4f 0x74a8152 0x759b55c 0x9bc0b9d 0x9ac6d0c 0x9b7f759 0x9bcfcf7 0x9d147e2 0x9d14e5e 0x265f87f 0xa34f610 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T13:03:13.727 INFO:teuthology.orchestra.run.trial093.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9c8bb26 0x9abe76d 0x9af96d3 0x9afcc8c 0x9afcefa 0x9afd042 0x9afd501 /lib64/libc.so.6+0x3fc2f 0x313124b 0x993a522 0x993a7b4 0x993aa6a 0x993b519 0x993ba17 0x993c047 0x993c1b1 0x9a49461 0x9a5c268 0x99c20b5 0x99c10b5 0x7b60901 0x7b6a470 0x7a0ec75 0x7a27c05 0x70e904b 0x6f6f022 0x6f72e23 0x6f789cd 0x73e30d9 0x7412614 0x7256527 0x725661f 0x732cdbe 0x732ce77 0x732d02e 0x7412695 0x725669c 0x7256721 0x728124b 0x73ebced 0x73ebd72 0x73ebe50 0x740ced4 0x7387fb8 0x7552c41 0x73f3ad4 0x73f3b4f 0x74a8152 0x759b55c 0x9bc0b9d 0x9ac6d0c 0x9b7f759 0x9bcfcf7 0x9d147e2 0x9d14e5e 0x265f87f 0xa34f610 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T13:03:13.727 INFO:teuthology.orchestra.run.trial093.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9c8bb26 0x9abe76d 0x9af96d3 0x9afcc8c 0x9afcefa 0x9afd042 0x9afd501 /lib64/libc.so.6+0x3fc2f 0xd915a8 0x98be5da 0x98be794 0x98be879 0x98bead5 0x98bebc8 0x98beca4 0x98bef20 0x98befd9 0x98bf4c7 0x98bf8cb 0x98bfb88 0x9a5bebd 0x99c20b5 0x99c10b5 0x7b60901 0x7b6a470 0x7a0ec75 0x7a27c05 0x70e904b 0x6f6f022 0x6f72e23 0x6f789cd 0x73e30d9 0x7412614 0x7256527 0x725661f 0x732cdbe 0x732ce77 0x732d02e 0x7412695 0x725669c 0x7256721 0x728124b 0x73ebced 0x73ebd72 0x73ebe50 0x740ced4 0x7387fb8 0x7552c41 0x73f3ad4 0x73f3b4f 0x74a8152 0x759b55c 0x9bc0b9d 0x9ac6d0c 0x9b7f759 0x9bcfcf7 0x9d147e2 0x9d14e5e 0x265f87f 0xa34f610 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T13:03:13.870 INFO:teuthology.orchestra.run.trial093.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9c8bb26 0x9abe76d 0x9af96d3 0x9afcc8c 0x9afcefa 0x9afd042 0x9afd501 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x104e3a /lib64/libasan.so.8+0xea9a /lib64/libasan.so.8+0xef905 0x3135646 0x3135820 0x3135ad7 0x3135a9c 0x3135bcc 0x6f3edc2 0x9a4341a 0x9a43632 0x9a43904 0x7b6093f 0x7b6a470 0x7a0ec75 0x7a27c05 0x70e904b 0x6f0b27c 0x6f0d5e4 0x73e2d78 0x740ccd4 0x7254c6e 0x7254d66 0x732c65e 0x732c717 0x732c8ce 0x740cd55 0x7254de3 0x7254e68 0x727c2ab 0x73ea73c 0x73ea7c1 0x73ea89f 0x740ced4 0x7387fb8 0x7552c41 0x73f3ad4 0x73f3b4f 0x74a8152 0x759b55c 0x9bc0b9d 0x9ac6d0c 0x9b7f759 0x9bcfcf7 0x9d147e2 0x9d14e5e 0x265f87f 0xa34f610 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T13:03:13.870 INFO:teuthology.orchestra.run.trial093.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9c8bb26 0x9abe76d 0x9af96d3 0x9afcc8c 0x9afcefa 0x9afd042 0x9afd501 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc597 /lib64/libasan.so.8+0x8eea /lib64/libasan.so.8+0xedc45 0xd91811 0xd91b68 0x9a5fcf1 0x786f2eb 0x7b60233 0x7b6a470 0x7a0ec75 0x7a27c05 0x70e904b 0x6f0b27c 0x6f0d5e4 0x73e2d78 0x740ccd4 0x7254c6e 0x7254d66 0x732c65e 0x732c717 0x732c8ce 0x740cd55 0x7254de3 0x7254e68 0x727c2ab 0x73ea73c 0x73ea7c1 0x73ea89f 0x740ced4 0x7387fb8 0x7552c41 0x73f3ad4 0x73f3b4f 0x74a8152 0x759b55c 0x9bc0b9d 0x9ac6d0c 0x9b7f759 0x9bcfcf7 0x9d147e2 0x9d14e5e 0x265f87f 0xa34f610 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T13:03:13.871 INFO:teuthology.orchestra.run.trial093.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff98927ef7 0xffffffff985f7f16 0xffffffff985fb8d0 0xffffffff985fcc31 0xffffffff98624893 0xffffffff98624a55 0xffffffff985c97c8 0xffffffff985d3aeb 0xffffffff985d3f4b 0xffffffff982910d2 0xffffffff98f4d451 0xffffffff99000ba2 2026-02-20T13:03:13.871 INFO:teuthology.orchestra.run.trial093.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9c8bb26 0x9abe76d 0x9af96d3 0x9afcc8c 0x9afcefa 0x9afd042 0x9afd501 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xea7eb 0xcaed44 0x35d531e 0x35d5627 0x9a41437 0x9a41874 0x9a59007 0x9a5dcef 0x99c20b5 0x99c10b5 0x7b60901 0x7b6a470 0x7a0ec75 0x7a27c05 0x70e904b 0x6f0b27c 0x6f0d5e4 0x73e2d78 0x740ccd4 0x7254c6e 0x7254d66 0x732c65e 0x732c717 0x732c8ce 0x740cd55 0x7254de3 0x7254e68 0x727c2ab 0x73ea73c 0x73ea7c1 0x73ea89f 0x740ced4 0x7387fb8 0x7552c41 0x73f3ad4 0x73f3b4f 0x74a8152 0x759b55c 0x9bc0b9d 0x9ac6d0c 0x9b7f759 0x9bcfcf7 0x9d147e2 0x9d14e5e 0x265f87f 0xa34f610 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T13:03:15.954 INFO:teuthology.orchestra.run.trial093.stderr:----------------------------------------------------- 2026-02-20T13:03:15.955 INFO:teuthology.orchestra.run.trial093.stderr:Suppressions used: 2026-02-20T13:03:15.955 INFO:teuthology.orchestra.run.trial093.stderr: count bytes template 2026-02-20T13:03:15.955 INFO:teuthology.orchestra.run.trial093.stderr: 1 8 MallocExtension::Register 2026-02-20T13:03:15.955 INFO:teuthology.orchestra.run.trial093.stderr:----------------------------------------------------- 2026-02-20T13:03:15.955 INFO:teuthology.orchestra.run.trial093.stderr: 2026-02-20T13:03:15.984 DEBUG:teuthology.orchestra.run.trial093:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2026-02-20T13:03:16.010 INFO:tasks.ceph:Reading keys from all nodes... 2026-02-20T13:03:16.010 DEBUG:teuthology.orchestra.run.trial093:> set -ex 2026-02-20T13:03:16.010 DEBUG:teuthology.orchestra.run.trial093:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2026-02-20T13:03:16.073 DEBUG:teuthology.orchestra.run.trial093:> set -ex 2026-02-20T13:03:16.074 DEBUG:teuthology.orchestra.run.trial093:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2026-02-20T13:03:16.138 DEBUG:teuthology.orchestra.run.trial093:> set -ex 2026-02-20T13:03:16.139 DEBUG:teuthology.orchestra.run.trial093:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2026-02-20T13:03:16.202 DEBUG:teuthology.orchestra.run.trial093:> set -ex 2026-02-20T13:03:16.202 DEBUG:teuthology.orchestra.run.trial093:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2026-02-20T13:03:16.266 DEBUG:teuthology.orchestra.run.trial093:> set -ex 2026-02-20T13:03:16.266 DEBUG:teuthology.orchestra.run.trial093:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2026-02-20T13:03:16.321 INFO:tasks.ceph:Adding keys to all mons... 2026-02-20T13:03:16.322 DEBUG:teuthology.orchestra.run.trial093:> sudo tee -a /etc/ceph/ceph.keyring 2026-02-20T13:03:16.383 INFO:teuthology.orchestra.run.trial093.stdout:[mgr.x] 2026-02-20T13:03:16.384 INFO:teuthology.orchestra.run.trial093.stdout: key = AQCCW5hpNjjPNhAA6UKEgpemqeI72Yy226X1oQ== 2026-02-20T13:03:16.384 INFO:teuthology.orchestra.run.trial093.stdout:[osd.0] 2026-02-20T13:03:16.384 INFO:teuthology.orchestra.run.trial093.stdout: key = AQCHW5hpKrolNxAAnb7B+fqewREp9hgJxQ5OjA== 2026-02-20T13:03:16.384 INFO:teuthology.orchestra.run.trial093.stdout:[osd.1] 2026-02-20T13:03:16.384 INFO:teuthology.orchestra.run.trial093.stdout: key = AQCMW5hpIWfeLBAAvTzDfWjCI3zyKx4dC+sZXw== 2026-02-20T13:03:16.384 INFO:teuthology.orchestra.run.trial093.stdout:[osd.2] 2026-02-20T13:03:16.384 INFO:teuthology.orchestra.run.trial093.stdout: key = AQCRW5hpv/vaJhAAdc8hyb0EVHmkFB4Hq3Ff7Q== 2026-02-20T13:03:16.384 INFO:teuthology.orchestra.run.trial093.stdout:[client.0] 2026-02-20T13:03:16.384 INFO:teuthology.orchestra.run.trial093.stdout: key = AQCCW5hpT56QORAAwggJ1OA9kgrXHd1EYbKRUw== 2026-02-20T13:03:16.386 DEBUG:teuthology.orchestra.run.trial093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2026-02-20T13:03:16.473 DEBUG:teuthology.orchestra.run.trial093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-20T13:03:16.522 DEBUG:teuthology.orchestra.run.trial093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-20T13:03:16.571 DEBUG:teuthology.orchestra.run.trial093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-20T13:03:16.620 DEBUG:teuthology.orchestra.run.trial093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-20T13:03:16.669 INFO:tasks.ceph:Running mkfs on mon nodes... 2026-02-20T13:03:16.670 DEBUG:teuthology.orchestra.run.trial093:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2026-02-20T13:03:16.695 DEBUG:teuthology.orchestra.run.trial093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2026-02-20T13:03:16.781 INFO:teuthology.orchestra.run.trial093.stderr:2026-02-20T13:03:16.778+0000 7f06caa85c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T13:03:16.781 INFO:teuthology.orchestra.run.trial093.stderr:2026-02-20T13:03:16.778+0000 7f06caa85c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T13:03:16.788 DEBUG:teuthology.orchestra.run.trial093:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2026-02-20T13:03:16.813 DEBUG:teuthology.orchestra.run.trial093:> rm -- /home/ubuntu/cephtest/ceph.monmap 2026-02-20T13:03:16.869 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2026-02-20T13:03:16.869 INFO:tasks.ceph.mon.a:Restarting daemon 2026-02-20T13:03:16.869 DEBUG:teuthology.orchestra.run.trial093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2026-02-20T13:03:16.911 INFO:tasks.ceph.mon.a:Started 2026-02-20T13:03:16.911 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2026-02-20T13:03:16.912 INFO:tasks.ceph.mgr.x:Restarting daemon 2026-02-20T13:03:16.912 DEBUG:teuthology.orchestra.run.trial093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2026-02-20T13:03:16.915 INFO:tasks.ceph.mgr.x:Started 2026-02-20T13:03:16.915 DEBUG:tasks.ceph:set 0 configs 2026-02-20T13:03:16.915 DEBUG:teuthology.orchestra.run.trial093:> sudo ceph --cluster ceph config dump 2026-02-20T13:03:16.975 INFO:tasks.ceph.mon.a.trial093.stderr:2026-02-20T13:03:16.972+0000 7f7af949ec00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T13:03:16.975 INFO:tasks.ceph.mon.a.trial093.stderr:2026-02-20T13:03:16.972+0000 7f7af949ec00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T13:03:16.975 INFO:tasks.ceph.mon.a.trial093.stderr:2026-02-20T13:03:16.972+0000 7f7af949ec00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T13:03:16.977 INFO:tasks.ceph.mgr.x.trial093.stderr:2026-02-20T13:03:16.974+0000 7fee863c6fc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T13:03:16.989 INFO:teuthology.orchestra.run.trial093.stderr:2026-02-20T13:03:16.985+0000 7fc266dfd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T13:03:16.991 INFO:teuthology.orchestra.run.trial093.stderr:2026-02-20T13:03:16.988+0000 7fc266dfd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T13:03:17.021 INFO:teuthology.orchestra.run.trial093.stdout:WHO MASK LEVEL OPTION VALUE RO 2026-02-20T13:03:17.029 INFO:tasks.ceph:Setting crush tunables to default 2026-02-20T13:03:17.029 DEBUG:teuthology.orchestra.run.trial093:> sudo ceph --cluster ceph osd crush tunables default 2026-02-20T13:03:17.102 INFO:teuthology.orchestra.run.trial093.stderr:2026-02-20T13:03:17.097+0000 7ff91b1a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T13:03:17.103 INFO:teuthology.orchestra.run.trial093.stderr:2026-02-20T13:03:17.101+0000 7ff91b1a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T13:03:17.137 INFO:teuthology.orchestra.run.trial093.stderr:adjusted tunables profile to default 2026-02-20T13:03:17.145 INFO:tasks.ceph:check_enable_crimson: True 2026-02-20T13:03:17.146 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2026-02-20T13:03:17.146 DEBUG:teuthology.orchestra.run.trial093:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2026-02-20T13:03:17.180 INFO:tasks.ceph.mgr.x.trial093.stderr:2026-02-20T13:03:17.177+0000 7fee863c6fc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T13:03:17.180 INFO:tasks.ceph.mgr.x.trial093.stderr:2026-02-20T13:03:17.178+0000 7fee863c6fc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T13:03:17.219 INFO:teuthology.orchestra.run.trial093.stderr:2026-02-20T13:03:17.214+0000 7fa0067b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T13:03:17.220 INFO:teuthology.orchestra.run.trial093.stderr:2026-02-20T13:03:17.218+0000 7fa0067b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T13:03:17.259 INFO:tasks.ceph.mon.a.trial093.stderr:2026-02-20T13:03:17.257+0000 7f7af0bf1640 -1 WARNING: experimental feature 'crimson' is enabled 2026-02-20T13:03:17.259 INFO:tasks.ceph.mon.a.trial093.stderr:Please be aware that this feature is experimental, untested, 2026-02-20T13:03:17.259 INFO:tasks.ceph.mon.a.trial093.stderr:unsupported, and may result in data corruption, data loss, 2026-02-20T13:03:17.259 INFO:tasks.ceph.mon.a.trial093.stderr:and/or irreparable damage to your cluster. Do not use 2026-02-20T13:03:17.260 INFO:tasks.ceph.mon.a.trial093.stderr:feature with important data. 2026-02-20T13:03:17.260 INFO:tasks.ceph.mon.a.trial093.stderr: 2026-02-20T13:03:18.019 INFO:tasks.ceph.mgr.x.trial093.stderr:/usr/lib64/python3.9/site-packages/scipy/__init__.py:73: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2026-02-20T13:03:18.019 INFO:tasks.ceph.mgr.x.trial093.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2026-02-20T13:03:18.019 INFO:tasks.ceph.mgr.x.trial093.stderr: from numpy import show_config as show_numpy_config 2026-02-20T13:03:18.147 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2026-02-20T13:03:18.148 DEBUG:teuthology.orchestra.run.trial093:> set -ex 2026-02-20T13:03:18.148 DEBUG:teuthology.orchestra.run.trial093:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2026-02-20T13:03:18.213 DEBUG:teuthology.orchestra.run.trial093:> set -ex 2026-02-20T13:03:18.213 DEBUG:teuthology.orchestra.run.trial093:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2026-02-20T13:03:18.278 DEBUG:teuthology.orchestra.run.trial093:> set -ex 2026-02-20T13:03:18.278 DEBUG:teuthology.orchestra.run.trial093:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2026-02-20T13:03:18.342 DEBUG:teuthology.orchestra.run.trial093:> sudo ceph --cluster ceph osd new 37351dce-f6fb-4ecc-9f47-8d7763e34108 0 2026-02-20T13:03:18.457 INFO:teuthology.orchestra.run.trial093.stderr:2026-02-20T13:03:18.451+0000 7fcd62c5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T13:03:18.458 INFO:teuthology.orchestra.run.trial093.stderr:2026-02-20T13:03:18.455+0000 7fcd62c5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T13:03:18.496 INFO:teuthology.orchestra.run.trial093.stdout:0 2026-02-20T13:03:18.505 DEBUG:teuthology.orchestra.run.trial093:> sudo ceph --cluster ceph osd new 78c4b821-1cda-4461-aa92-9f13af27c598 1 2026-02-20T13:03:18.578 INFO:teuthology.orchestra.run.trial093.stderr:2026-02-20T13:03:18.573+0000 7f74f755c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T13:03:18.579 INFO:teuthology.orchestra.run.trial093.stderr:2026-02-20T13:03:18.577+0000 7f74f755c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T13:03:18.615 INFO:teuthology.orchestra.run.trial093.stdout:1 2026-02-20T13:03:18.624 DEBUG:teuthology.orchestra.run.trial093:> sudo ceph --cluster ceph osd new dae0b123-eba1-43c5-b0dd-428379e2afe2 2 2026-02-20T13:03:18.698 INFO:teuthology.orchestra.run.trial093.stderr:2026-02-20T13:03:18.692+0000 7f192161a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T13:03:18.699 INFO:teuthology.orchestra.run.trial093.stderr:2026-02-20T13:03:18.697+0000 7f192161a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T13:03:18.735 INFO:teuthology.orchestra.run.trial093.stdout:2 2026-02-20T13:03:18.743 DEBUG:teuthology.orchestra.run.trial093:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2026-02-20T13:03:18.771 INFO:tasks.ceph.osd.0:Restarting daemon 2026-02-20T13:03:18.771 DEBUG:teuthology.orchestra.run.trial093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2026-02-20T13:03:18.813 INFO:tasks.ceph.osd.0:Started 2026-02-20T13:03:18.813 DEBUG:teuthology.orchestra.run.trial093:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2026-02-20T13:03:18.842 INFO:tasks.ceph.osd.1:Restarting daemon 2026-02-20T13:03:18.842 DEBUG:teuthology.orchestra.run.trial093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2026-02-20T13:03:18.885 INFO:tasks.ceph.osd.1:Started 2026-02-20T13:03:18.885 DEBUG:teuthology.orchestra.run.trial093:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2026-02-20T13:03:18.912 INFO:tasks.ceph.osd.2:Restarting daemon 2026-02-20T13:03:18.912 DEBUG:teuthology.orchestra.run.trial093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2026-02-20T13:03:18.954 INFO:tasks.ceph.osd.2:Started 2026-02-20T13:03:18.954 DEBUG:teuthology.orchestra.run.trial093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-20T13:03:19.031 INFO:teuthology.orchestra.run.trial093.stderr:2026-02-20T13:03:19.026+0000 7fd5c81cc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T13:03:19.032 INFO:teuthology.orchestra.run.trial093.stderr:2026-02-20T13:03:19.029+0000 7fd5c81cc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T13:03:19.063 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:03:19.064 INFO:teuthology.orchestra.run.trial093.stdout:{"epoch":6,"fsid":"062d8521-4eb6-4504-8170-0bfd8fbdcb1a","created":"2026-02-20T13:03:16.984298+0000","modified":"2026-02-20T13:03:18.733635+0000","last_up_change":"0.000000","last_in_change":"2026-02-20T13:03:18.733635+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"37351dce-f6fb-4ecc-9f47-8d7763e34108","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78c4b821-1cda-4461-aa92-9f13af27c598","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dae0b123-eba1-43c5-b0dd-428379e2afe2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-20T13:03:19.072 INFO:tasks.ceph.ceph_manager.ceph:[] 2026-02-20T13:03:19.072 INFO:tasks.ceph:Waiting for OSDs to come up 2026-02-20T13:03:19.374 DEBUG:teuthology.orchestra.run.trial093:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-20T13:03:19.442 INFO:teuthology.misc.health.trial093.stderr:2026-02-20T13:03:19.437+0000 7f165cdd3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T13:03:19.443 INFO:teuthology.misc.health.trial093.stderr:2026-02-20T13:03:19.440+0000 7f165cdd3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T13:03:19.475 INFO:teuthology.misc.health.trial093.stdout: 2026-02-20T13:03:19.475 INFO:teuthology.misc.health.trial093.stdout:{"epoch":6,"fsid":"062d8521-4eb6-4504-8170-0bfd8fbdcb1a","created":"2026-02-20T13:03:16.984298+0000","modified":"2026-02-20T13:03:18.733635+0000","last_up_change":"0.000000","last_in_change":"2026-02-20T13:03:18.733635+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"37351dce-f6fb-4ecc-9f47-8d7763e34108","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78c4b821-1cda-4461-aa92-9f13af27c598","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dae0b123-eba1-43c5-b0dd-428379e2afe2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-20T13:03:19.482 DEBUG:teuthology.misc:0 of 3 OSDs are up 2026-02-20T13:03:20.406 INFO:tasks.ceph.mgr.x.trial093.stderr:2026-02-20T13:03:20.403+0000 7feda1287640 -1 mgr.server handle_report got status from non-daemon mon.a 2026-02-20T13:03:25.785 DEBUG:teuthology.orchestra.run.trial093:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-20T13:03:25.852 INFO:teuthology.misc.health.trial093.stderr:2026-02-20T13:03:25.849+0000 7fa9f67fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T13:03:25.853 INFO:teuthology.misc.health.trial093.stderr:2026-02-20T13:03:25.852+0000 7fa9f67fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T13:03:25.963 INFO:teuthology.misc.health.trial093.stdout: 2026-02-20T13:03:25.964 INFO:teuthology.misc.health.trial093.stdout:{"epoch":8,"fsid":"062d8521-4eb6-4504-8170-0bfd8fbdcb1a","created":"2026-02-20T13:03:16.984298+0000","modified":"2026-02-20T13:03:25.780000+0000","last_up_change":"0.000000","last_in_change":"2026-02-20T13:03:18.733635+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"37351dce-f6fb-4ecc-9f47-8d7763e34108","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78c4b821-1cda-4461-aa92-9f13af27c598","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dae0b123-eba1-43c5-b0dd-428379e2afe2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-20T13:03:25.970 DEBUG:teuthology.misc:0 of 3 OSDs are up 2026-02-20T13:03:32.272 DEBUG:teuthology.orchestra.run.trial093:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-20T13:03:32.341 INFO:teuthology.misc.health.trial093.stderr:2026-02-20T13:03:32.337+0000 7f5d818cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T13:03:32.342 INFO:teuthology.misc.health.trial093.stderr:2026-02-20T13:03:32.340+0000 7f5d818cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T13:03:32.451 INFO:teuthology.misc.health.trial093.stdout: 2026-02-20T13:03:32.452 INFO:teuthology.misc.health.trial093.stdout:{"epoch":13,"fsid":"062d8521-4eb6-4504-8170-0bfd8fbdcb1a","created":"2026-02-20T13:03:16.984298+0000","modified":"2026-02-20T13:03:31.804012+0000","last_up_change":"2026-02-20T13:03:27.791695+0000","last_in_change":"2026-02-20T13:03:18.733635+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-20T13:03:29.413108+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"37351dce-f6fb-4ecc-9f47-8d7763e34108","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.93:6804","nonce":4139173107}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.93:6805","nonce":4139173107}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.93:6810","nonce":4139173107}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.93:6811","nonce":4139173107}]},"public_addr":"10.20.193.93:6804/4139173107","cluster_addr":"10.20.193.93:6805/4139173107","heartbeat_back_addr":"10.20.193.93:6810/4139173107","heartbeat_front_addr":"10.20.193.93:6811/4139173107","state":["exists","up"]},{"osd":1,"uuid":"78c4b821-1cda-4461-aa92-9f13af27c598","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.93:6803","nonce":630708317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.93:6802","nonce":630708317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.93:6808","nonce":630708317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.93:6809","nonce":630708317}]},"public_addr":"10.20.193.93:6803/630708317","cluster_addr":"10.20.193.93:6802/630708317","heartbeat_back_addr":"10.20.193.93:6808/630708317","heartbeat_front_addr":"10.20.193.93:6809/630708317","state":["exists","up"]},{"osd":2,"uuid":"dae0b123-eba1-43c5-b0dd-428379e2afe2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.93:6806","nonce":1103946719}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.93:6807","nonce":1103946719}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.93:6812","nonce":1103946719}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.93:6813","nonce":1103946719}]},"public_addr":"10.20.193.93:6806/1103946719","cluster_addr":"10.20.193.93:6807/1103946719","heartbeat_back_addr":"10.20.193.93:6812/1103946719","heartbeat_front_addr":"10.20.193.93:6813/1103946719","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-20T13:03:32.460 DEBUG:teuthology.misc:3 of 3 OSDs are up 2026-02-20T13:03:32.460 INFO:tasks.ceph:Creating RBD pool 2026-02-20T13:03:32.460 DEBUG:teuthology.orchestra.run.trial093:> sudo ceph --cluster ceph osd pool create rbd 8 2026-02-20T13:03:32.535 INFO:teuthology.orchestra.run.trial093.stderr:2026-02-20T13:03:32.530+0000 7fab311cc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T13:03:32.536 INFO:teuthology.orchestra.run.trial093.stderr:2026-02-20T13:03:32.535+0000 7fab311cc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T13:03:33.133 INFO:teuthology.orchestra.run.trial093.stderr:pool 'rbd' created 2026-02-20T13:03:33.148 DEBUG:teuthology.orchestra.run.trial093:> rbd --cluster ceph pool init rbd 2026-02-20T13:03:33.190 INFO:teuthology.orchestra.run.trial093.stderr:2026-02-20T13:03:33.188+0000 7efc45c19240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T13:03:33.190 INFO:teuthology.orchestra.run.trial093.stderr:2026-02-20T13:03:33.188+0000 7efc45c19240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T13:03:33.191 INFO:teuthology.orchestra.run.trial093.stderr:2026-02-20T13:03:33.190+0000 7efc45c19240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T13:03:36.168 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2026-02-20T13:03:36.168 DEBUG:teuthology.orchestra.run.trial093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2026-02-20T13:03:36.169 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2026-02-20T13:03:36.286 INFO:teuthology.orchestra.run.trial093.stderr:2026-02-20T13:03:36.282+0000 7f8e4afad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T13:03:36.287 INFO:teuthology.orchestra.run.trial093.stderr:2026-02-20T13:03:36.286+0000 7f8e4afad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T13:03:36.396 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:03:36.405 INFO:teuthology.orchestra.run.trial093.stdout:[{"version":1,"timestamp":"0.000000","name":"","changes":[]}] 2026-02-20T13:03:36.405 INFO:tasks.ceph_manager:config epoch is 1 2026-02-20T13:03:36.406 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2026-02-20T13:03:36.406 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2026-02-20T13:03:36.406 DEBUG:teuthology.orchestra.run.trial093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2026-02-20T13:03:36.485 INFO:teuthology.orchestra.run.trial093.stderr:2026-02-20T13:03:36.481+0000 7f36a895b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T13:03:36.486 INFO:teuthology.orchestra.run.trial093.stderr:2026-02-20T13:03:36.485+0000 7f36a895b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T13:03:36.627 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:03:36.634 INFO:teuthology.orchestra.run.trial093.stdout:{"epoch":5,"flags":0,"active_gid":4104,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.93:6800","nonce":3573797502},{"type":"v1","addr":"10.20.193.93:6801","nonce":3573797502}]},"active_addr":"10.20.193.93:6801/3573797502","active_change":"2026-02-20T13:03:19.399628+0000","active_mgr_features":4544132024016699391,"available":true,"standbys":[],"modules":["iostat","nfs"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_pg_upmap_activity":{"name":"update_pg_upmap_activity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Updates pg_upmap activity stats to be used in `balancer status detail`","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often the agent on each host will gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"The first TCP port the agent will try to bind to. Up to 1000 subsequent ports will be attempted if this port cannot be bound","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this option can facilitate debugging daemons that encounter runtime problems.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"How frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"Fraction of total system memory to divide among autotuned daemons. Converged systems hosting both Ceph and compute should set a lower value; systems with more than 10 GiB of memory per OSD may benefit from a slightly larger value so that the abundance of RAM can be fully utilized","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command persistent logging","long_desc":"","tags":[],"see_also":[]},"certificate_automated_rotation_enabled":{"name":"certificate_automated_rotation_enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"This flag controls whether cephadm automatically rotates certificates upon expiration.","long_desc":"","tags":[],"see_also":[]},"certificate_check_debug_mode":{"name":"certificate_check_debug_mode","type":"bool","level":"dev","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"FOR TESTING ONLY: This flag forces the certificate check instead of waiting for certificate_check_period.","long_desc":"","tags":[],"see_also":[]},"certificate_check_period":{"name":"certificate_check_period","type":"int","level":"advanced","flags":0,"default_value":"1","min":"0","max":"30","enum_allowed":[],"desc":"Specifies how often (in days) the certificate should be checked for validity.","long_desc":"","tags":[],"see_also":[]},"certificate_duration_days":{"name":"certificate_duration_days","type":"int","level":"advanced","flags":0,"default_value":"1095","min":"90","max":"3650","enum_allowed":[],"desc":"Specifies the duration of self-signed certificates generated and signed by the cephadm root CA","long_desc":"","tags":[],"see_also":[]},"certificate_renewal_threshold_days":{"name":"certificate_renewal_threshold_days","type":"int","level":"advanced","flags":0,"default_value":"30","min":"10","max":"90","enum_allowed":[],"desc":"Specifies the lead time in days to initiate certificate renewal before expiration.","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently Podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.28.1","min":"","max":"","enum_allowed":[],"desc":"Alertmanager container image","long_desc":"","tags":[],"see_also":[]},"container_image_alloy":{"name":"container_image_alloy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/alloy:latest","min":"","max":"","enum_allowed":[],"desc":"Alloy container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"Elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:12.3.1","min":"","max":"","enum_allowed":[],"desc":"Grafana container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"Haproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.9.1","min":"","max":"","enum_allowed":[],"desc":"Node exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.6","min":"","max":"","enum_allowed":[],"desc":"Nvmeof container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"Oauth2 proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v3.6.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba metrics container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"Snmp gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"quay.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"Location of dashboards to include in Grafana","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"How frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether the hardware monitoring daemon be deployed on every host?","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at the debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"Max number of daemons per service per host: guard against mistakes and runaways","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"Max number of OSDs that will be drained simultaneously when OSDs are removed","long_desc":"","tags":[],"see_also":[]},"max_parallel_osd_upgrades":{"name":"max_parallel_osd_upgrades","type":"int","level":"advanced","flags":0,"default_value":"16","min":"","max":"","enum_allowed":[],"desc":"Maximum number of OSD daemons upgraded in parallel.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"Mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default IP address for RedFish API (OOB management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"Location of alerts to include in Prometheus Alertmanager","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry URL for login purposes. This is not the default registry.","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"Cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Customized SSH config file to distribute to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many sequential SSH connection liveness check failures before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often SSH connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"stray_daemon_check_interval":{"name":"stray_daemon_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"How frequently cephadm should check for the presence of stray daemons","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use the cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest to ensure that all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROM_ALERT_CREDENTIAL_CACHE_TTL":{"name":"PROM_ALERT_CREDENTIAL_CACHE_TTL","type":"int","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_HOSTNAME_PER_DAEMON":{"name":"RGW_HOSTNAME_PER_DAEMON","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crypto_caller":{"name":"crypto_caller","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":true,"error_string":"","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"healthcheck_history_max_entries":{"name":"healthcheck_history_max_entries","type":"int","level":"advanced","flags":1,"default_value":"1000","min":"","max":"","enum_allowed":[],"desc":"Maximum number of health check history entries to keep","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"prometheus_tls_secret_name":{"name":"prometheus_tls_secret_name","type":"str","level":"advanced","flags":0,"default_value":"rook-ceph-prometheus-server-tls","min":"","max":"","enum_allowed":[],"desc":"name of tls secret in k8s for prometheus","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_cloning":{"name":"pause_cloning","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_purging":{"name":"pause_purging","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous subvolume purge threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"tentacle":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"force_disabled_modules":{},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"10.20.193.93:0","nonce":1221874361}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"10.20.193.93:0","nonce":1569232424}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"10.20.193.93:0","nonce":3997790789}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"10.20.193.93:0","nonce":689783378}]}]} 2026-02-20T13:03:36.638 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2026-02-20T13:03:36.638 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2026-02-20T13:03:36.638 DEBUG:teuthology.orchestra.run.trial093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-20T13:03:36.716 INFO:teuthology.orchestra.run.trial093.stderr:2026-02-20T13:03:36.712+0000 7fa2d8d0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T13:03:36.717 INFO:teuthology.orchestra.run.trial093.stderr:2026-02-20T13:03:36.715+0000 7fa2d8d0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T13:03:36.826 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:03:36.826 INFO:teuthology.orchestra.run.trial093.stdout:{"epoch":17,"fsid":"062d8521-4eb6-4504-8170-0bfd8fbdcb1a","created":"2026-02-20T13:03:16.984298+0000","modified":"2026-02-20T13:03:36.141098+0000","last_up_change":"2026-02-20T13:03:27.791695+0000","last_in_change":"2026-02-20T13:03:18.733635+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-20T13:03:29.413108+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-20T13:03:32.647159+0000","flags":270337,"flags_names":"hashpspool,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"37351dce-f6fb-4ecc-9f47-8d7763e34108","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.93:6804","nonce":4139173107}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.93:6805","nonce":4139173107}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.93:6810","nonce":4139173107}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.93:6811","nonce":4139173107}]},"public_addr":"10.20.193.93:6804/4139173107","cluster_addr":"10.20.193.93:6805/4139173107","heartbeat_back_addr":"10.20.193.93:6810/4139173107","heartbeat_front_addr":"10.20.193.93:6811/4139173107","state":["exists","up"]},{"osd":1,"uuid":"78c4b821-1cda-4461-aa92-9f13af27c598","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.93:6803","nonce":630708317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.93:6802","nonce":630708317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.93:6808","nonce":630708317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.93:6809","nonce":630708317}]},"public_addr":"10.20.193.93:6803/630708317","cluster_addr":"10.20.193.93:6802/630708317","heartbeat_back_addr":"10.20.193.93:6808/630708317","heartbeat_front_addr":"10.20.193.93:6809/630708317","state":["exists","up"]},{"osd":2,"uuid":"dae0b123-eba1-43c5-b0dd-428379e2afe2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.93:6806","nonce":1103946719}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.93:6807","nonce":1103946719}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.93:6812","nonce":1103946719}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.93:6813","nonce":1103946719}]},"public_addr":"10.20.193.93:6806/1103946719","cluster_addr":"10.20.193.93:6807/1103946719","heartbeat_back_addr":"10.20.193.93:6812/1103946719","heartbeat_front_addr":"10.20.193.93:6813/1103946719","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-20T13:03:36.837 INFO:tasks.ceph.ceph_manager.ceph:all up! 2026-02-20T13:03:36.838 DEBUG:teuthology.orchestra.run.trial093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-20T13:03:36.916 INFO:teuthology.orchestra.run.trial093.stderr:2026-02-20T13:03:36.912+0000 7f51623a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T13:03:36.917 INFO:teuthology.orchestra.run.trial093.stderr:2026-02-20T13:03:36.915+0000 7f51623a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T13:03:37.026 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:03:37.027 INFO:teuthology.orchestra.run.trial093.stdout:{"epoch":17,"fsid":"062d8521-4eb6-4504-8170-0bfd8fbdcb1a","created":"2026-02-20T13:03:16.984298+0000","modified":"2026-02-20T13:03:36.141098+0000","last_up_change":"2026-02-20T13:03:27.791695+0000","last_in_change":"2026-02-20T13:03:18.733635+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-20T13:03:29.413108+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-20T13:03:32.647159+0000","flags":270337,"flags_names":"hashpspool,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"37351dce-f6fb-4ecc-9f47-8d7763e34108","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.93:6804","nonce":4139173107}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.93:6805","nonce":4139173107}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.93:6810","nonce":4139173107}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.93:6811","nonce":4139173107}]},"public_addr":"10.20.193.93:6804/4139173107","cluster_addr":"10.20.193.93:6805/4139173107","heartbeat_back_addr":"10.20.193.93:6810/4139173107","heartbeat_front_addr":"10.20.193.93:6811/4139173107","state":["exists","up"]},{"osd":1,"uuid":"78c4b821-1cda-4461-aa92-9f13af27c598","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.93:6803","nonce":630708317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.93:6802","nonce":630708317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.93:6808","nonce":630708317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.93:6809","nonce":630708317}]},"public_addr":"10.20.193.93:6803/630708317","cluster_addr":"10.20.193.93:6802/630708317","heartbeat_back_addr":"10.20.193.93:6808/630708317","heartbeat_front_addr":"10.20.193.93:6809/630708317","state":["exists","up"]},{"osd":2,"uuid":"dae0b123-eba1-43c5-b0dd-428379e2afe2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.93:6806","nonce":1103946719}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.93:6807","nonce":1103946719}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.93:6812","nonce":1103946719}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.93:6813","nonce":1103946719}]},"public_addr":"10.20.193.93:6806/1103946719","cluster_addr":"10.20.193.93:6807/1103946719","heartbeat_back_addr":"10.20.193.93:6812/1103946719","heartbeat_front_addr":"10.20.193.93:6813/1103946719","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-20T13:03:37.036 DEBUG:teuthology.orchestra.run.trial093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-20T13:03:37.036 DEBUG:teuthology.orchestra.run.trial093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-20T13:03:37.037 DEBUG:teuthology.orchestra.run.trial093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-20T13:03:37.116 INFO:teuthology.orchestra.run.trial093.stderr:2026-02-20T13:03:37.112+0000 7fe64917e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T13:03:37.117 INFO:teuthology.orchestra.run.trial093.stderr:2026-02-20T13:03:37.112+0000 7f5c76b7b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T13:03:37.118 INFO:teuthology.orchestra.run.trial093.stderr:2026-02-20T13:03:37.112+0000 7f68a4251640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T13:03:37.118 INFO:teuthology.orchestra.run.trial093.stderr:2026-02-20T13:03:37.116+0000 7fe64917e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T13:03:37.119 INFO:teuthology.orchestra.run.trial093.stderr:2026-02-20T13:03:37.117+0000 7f68a4251640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T13:03:37.119 INFO:teuthology.orchestra.run.trial093.stderr:2026-02-20T13:03:37.117+0000 7f5c76b7b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T13:03:37.133 INFO:teuthology.orchestra.run.trial093.stdout:38654705674 2026-02-20T13:03:37.133 DEBUG:teuthology.orchestra.run.trial093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-20T13:03:37.134 INFO:teuthology.orchestra.run.trial093.stdout:42949672969 2026-02-20T13:03:37.135 DEBUG:teuthology.orchestra.run.trial093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-20T13:03:37.136 INFO:teuthology.orchestra.run.trial093.stdout:42949672969 2026-02-20T13:03:37.136 DEBUG:teuthology.orchestra.run.trial093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-20T13:03:37.252 INFO:teuthology.orchestra.run.trial093.stderr:2026-02-20T13:03:37.248+0000 7f89dc1ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T13:03:37.252 INFO:teuthology.orchestra.run.trial093.stderr:2026-02-20T13:03:37.248+0000 7fb2942cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T13:03:37.253 INFO:teuthology.orchestra.run.trial093.stderr:2026-02-20T13:03:37.251+0000 7f89dc1ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T13:03:37.253 INFO:teuthology.orchestra.run.trial093.stderr:2026-02-20T13:03:37.249+0000 7f1fb9d0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T13:03:37.254 INFO:teuthology.orchestra.run.trial093.stderr:2026-02-20T13:03:37.252+0000 7fb2942cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T13:03:37.258 INFO:teuthology.orchestra.run.trial093.stderr:2026-02-20T13:03:37.257+0000 7f1fb9d0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T13:03:37.365 INFO:teuthology.orchestra.run.trial093.stdout:38654705674 2026-02-20T13:03:37.369 INFO:teuthology.orchestra.run.trial093.stdout:42949672969 2026-02-20T13:03:37.371 INFO:teuthology.orchestra.run.trial093.stdout:42949672969 2026-02-20T13:03:37.376 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705674 got 38654705674 for osd.1 2026-02-20T13:03:37.376 DEBUG:teuthology.parallel:result is None 2026-02-20T13:03:37.380 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672969 got 42949672969 for osd.0 2026-02-20T13:03:37.381 DEBUG:teuthology.parallel:result is None 2026-02-20T13:03:37.383 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672969 got 42949672969 for osd.2 2026-02-20T13:03:37.383 DEBUG:teuthology.parallel:result is None 2026-02-20T13:03:37.383 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-20T13:03:37.383 DEBUG:teuthology.orchestra.run.trial093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-20T13:03:37.498 INFO:teuthology.orchestra.run.trial093.stderr:2026-02-20T13:03:37.495+0000 7f369b4b7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T13:03:37.499 INFO:teuthology.orchestra.run.trial093.stderr:2026-02-20T13:03:37.498+0000 7f369b4b7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T13:03:37.607 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:03:37.607 INFO:teuthology.orchestra.run.trial093.stderr:dumped all 2026-02-20T13:03:37.615 INFO:teuthology.orchestra.run.trial093.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2026-02-20T13:03:37.403285+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":157286352,"kb_used":62917052,"kb_used_data":62917052,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":94369300,"statfs":{"total":161061224448,"available":96634163200,"internally_reserved":0,"allocated":64427061248,"data_stored":64427061248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.596908"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-20T13:03:36.154906+0000","last_change":"2026-02-20T13:03:36.157834+0000","last_active":"2026-02-20T13:03:36.154906+0000","last_peered":"2026-02-20T13:03:36.154906+0000","last_clean":"2026-02-20T13:03:36.154906+0000","last_became_active":"2026-02-20T13:03:34.169262+0000","last_became_peered":"2026-02-20T13:03:34.169262+0000","last_unstale":"2026-02-20T13:03:36.154906+0000","last_undegraded":"2026-02-20T13:03:36.154906+0000","last_fullsized":"2026-02-20T13:03:36.154906+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T13:03:33.131410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T13:03:33.131410+0000","last_clean_scrub_stamp":"2026-02-20T13:03:33.131410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-20T13:03:36.156086+0000","last_change":"2026-02-20T13:03:36.158061+0000","last_active":"2026-02-20T13:03:36.156086+0000","last_peered":"2026-02-20T13:03:36.156086+0000","last_clean":"2026-02-20T13:03:36.156086+0000","last_became_active":"2026-02-20T13:03:34.166599+0000","last_became_peered":"2026-02-20T13:03:34.166599+0000","last_unstale":"2026-02-20T13:03:36.156086+0000","last_undegraded":"2026-02-20T13:03:36.156086+0000","last_fullsized":"2026-02-20T13:03:36.156086+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T13:03:33.131410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T13:03:33.131410+0000","last_clean_scrub_stamp":"2026-02-20T13:03:33.131410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-20T13:03:36.156583+0000","last_change":"2026-02-20T13:03:36.158972+0000","last_active":"2026-02-20T13:03:36.156583+0000","last_peered":"2026-02-20T13:03:36.156583+0000","last_clean":"2026-02-20T13:03:36.156583+0000","last_became_active":"2026-02-20T13:03:34.172645+0000","last_became_peered":"2026-02-20T13:03:34.172645+0000","last_unstale":"2026-02-20T13:03:36.156583+0000","last_undegraded":"2026-02-20T13:03:36.156583+0000","last_fullsized":"2026-02-20T13:03:36.156583+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T13:03:33.131410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T13:03:33.131410+0000","last_clean_scrub_stamp":"2026-02-20T13:03:33.131410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-20T13:03:36.156072+0000","last_change":"2026-02-20T13:03:36.157620+0000","last_active":"2026-02-20T13:03:36.156072+0000","last_peered":"2026-02-20T13:03:36.156072+0000","last_clean":"2026-02-20T13:03:36.156072+0000","last_became_active":"2026-02-20T13:03:34.169637+0000","last_became_peered":"2026-02-20T13:03:34.169637+0000","last_unstale":"2026-02-20T13:03:36.156072+0000","last_undegraded":"2026-02-20T13:03:36.156072+0000","last_fullsized":"2026-02-20T13:03:36.156072+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T13:03:33.131410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T13:03:33.131410+0000","last_clean_scrub_stamp":"2026-02-20T13:03:33.131410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"17'2","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-20T13:03:36.155508+0000","last_change":"2026-02-20T13:03:36.155922+0000","last_active":"2026-02-20T13:03:36.155508+0000","last_peered":"2026-02-20T13:03:36.155508+0000","last_clean":"2026-02-20T13:03:36.155508+0000","last_became_active":"2026-02-20T13:03:34.178652+0000","last_became_peered":"2026-02-20T13:03:34.178652+0000","last_unstale":"2026-02-20T13:03:36.155508+0000","last_undegraded":"2026-02-20T13:03:36.155508+0000","last_fullsized":"2026-02-20T13:03:36.155508+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T13:03:33.131410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T13:03:33.131410+0000","last_clean_scrub_stamp":"2026-02-20T13:03:33.131410+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-20T13:03:36.153752+0000","last_change":"2026-02-20T13:03:36.155731+0000","last_active":"2026-02-20T13:03:36.153752+0000","last_peered":"2026-02-20T13:03:36.153752+0000","last_clean":"2026-02-20T13:03:36.153752+0000","last_became_active":"2026-02-20T13:03:34.176581+0000","last_became_peered":"2026-02-20T13:03:34.176581+0000","last_unstale":"2026-02-20T13:03:36.153752+0000","last_undegraded":"2026-02-20T13:03:36.153752+0000","last_fullsized":"2026-02-20T13:03:36.153752+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T13:03:33.131410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T13:03:33.131410+0000","last_clean_scrub_stamp":"2026-02-20T13:03:33.131410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-20T13:03:36.152554+0000","last_change":"2026-02-20T13:03:36.155018+0000","last_active":"2026-02-20T13:03:36.152554+0000","last_peered":"2026-02-20T13:03:36.152554+0000","last_clean":"2026-02-20T13:03:36.152554+0000","last_became_active":"2026-02-20T13:03:34.176780+0000","last_became_peered":"2026-02-20T13:03:34.176780+0000","last_unstale":"2026-02-20T13:03:36.152554+0000","last_undegraded":"2026-02-20T13:03:36.152554+0000","last_fullsized":"2026-02-20T13:03:36.152554+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T13:03:33.131410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T13:03:33.131410+0000","last_clean_scrub_stamp":"2026-02-20T13:03:33.131410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":16,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-20T13:03:36.155853+0000","last_change":"2026-02-20T13:03:36.158942+0000","last_active":"2026-02-20T13:03:36.155853+0000","last_peered":"2026-02-20T13:03:36.155853+0000","last_clean":"2026-02-20T13:03:36.155853+0000","last_became_active":"2026-02-20T13:03:34.166167+0000","last_became_peered":"2026-02-20T13:03:34.166167+0000","last_unstale":"2026-02-20T13:03:36.155853+0000","last_undegraded":"2026-02-20T13:03:36.155853+0000","last_fullsized":"2026-02-20T13:03:36.155853+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T13:03:33.131410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T13:03:33.131410+0000","last_clean_scrub_stamp":"2026-02-20T13:03:33.131410+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":90,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-20T13:03:36.154650+0000","last_change":"2026-02-20T13:03:30.833889+0000","last_active":"2026-02-20T13:03:36.154650+0000","last_peered":"2026-02-20T13:03:36.154650+0000","last_clean":"2026-02-20T13:03:36.154650+0000","last_became_active":"2026-02-20T13:03:30.827292+0000","last_became_peered":"2026-02-20T13:03:30.827292+0000","last_unstale":"2026-02-20T13:03:36.154650+0000","last_undegraded":"2026-02-20T13:03:36.154650+0000","last_fullsized":"2026-02-20T13:03:36.154650+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T13:03:29.796415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T13:03:29.796415+0000","last_clean_scrub_stamp":"2026-02-20T13:03:29.796415+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":161061224448,"available":96634163200,"internally_reserved":0,"allocated":64427061248,"data_stored":64427061248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53687074816,"available":32211079168,"internally_reserved":0,"allocated":21475995648,"data_stored":21475995648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20971820,"kb_used_data":20971820,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456964,"statfs":{"total":53687074816,"available":32211931136,"internally_reserved":0,"allocated":21475143680,"data_stored":21475143680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972580,"kb_used_data":20972580,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456204,"statfs":{"total":53687074816,"available":32211152896,"internally_reserved":0,"allocated":21475921920,"data_stored":21475921920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972652,"kb_used_data":20972652,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456132,"statfs":{"total":53687074816,"available":32211079168,"internally_reserved":0,"allocated":21475995648,"data_stored":21475995648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53687074816,"available":32211079168,"internally_reserved":0,"allocated":21475995648,"data_stored":21475995648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53687074816,"available":32211152896,"internally_reserved":0,"allocated":21475921920,"data_stored":21475921920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53687074816,"available":32211079168,"internally_reserved":0,"allocated":21475995648,"data_stored":21475995648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53687074816,"available":32211931136,"internally_reserved":0,"allocated":21475143680,"data_stored":21475143680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-20T13:03:37.616 DEBUG:teuthology.orchestra.run.trial093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-20T13:03:37.691 INFO:teuthology.orchestra.run.trial093.stderr:2026-02-20T13:03:37.688+0000 7f6b44c8b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T13:03:37.692 INFO:teuthology.orchestra.run.trial093.stderr:2026-02-20T13:03:37.691+0000 7f6b44c8b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T13:03:37.801 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:03:37.801 INFO:teuthology.orchestra.run.trial093.stderr:dumped all 2026-02-20T13:03:37.810 INFO:teuthology.orchestra.run.trial093.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2026-02-20T13:03:37.403285+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":157286352,"kb_used":62917052,"kb_used_data":62917052,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":94369300,"statfs":{"total":161061224448,"available":96634163200,"internally_reserved":0,"allocated":64427061248,"data_stored":64427061248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.596908"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-20T13:03:36.154906+0000","last_change":"2026-02-20T13:03:36.157834+0000","last_active":"2026-02-20T13:03:36.154906+0000","last_peered":"2026-02-20T13:03:36.154906+0000","last_clean":"2026-02-20T13:03:36.154906+0000","last_became_active":"2026-02-20T13:03:34.169262+0000","last_became_peered":"2026-02-20T13:03:34.169262+0000","last_unstale":"2026-02-20T13:03:36.154906+0000","last_undegraded":"2026-02-20T13:03:36.154906+0000","last_fullsized":"2026-02-20T13:03:36.154906+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T13:03:33.131410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T13:03:33.131410+0000","last_clean_scrub_stamp":"2026-02-20T13:03:33.131410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-20T13:03:36.156086+0000","last_change":"2026-02-20T13:03:36.158061+0000","last_active":"2026-02-20T13:03:36.156086+0000","last_peered":"2026-02-20T13:03:36.156086+0000","last_clean":"2026-02-20T13:03:36.156086+0000","last_became_active":"2026-02-20T13:03:34.166599+0000","last_became_peered":"2026-02-20T13:03:34.166599+0000","last_unstale":"2026-02-20T13:03:36.156086+0000","last_undegraded":"2026-02-20T13:03:36.156086+0000","last_fullsized":"2026-02-20T13:03:36.156086+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T13:03:33.131410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T13:03:33.131410+0000","last_clean_scrub_stamp":"2026-02-20T13:03:33.131410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-20T13:03:36.156583+0000","last_change":"2026-02-20T13:03:36.158972+0000","last_active":"2026-02-20T13:03:36.156583+0000","last_peered":"2026-02-20T13:03:36.156583+0000","last_clean":"2026-02-20T13:03:36.156583+0000","last_became_active":"2026-02-20T13:03:34.172645+0000","last_became_peered":"2026-02-20T13:03:34.172645+0000","last_unstale":"2026-02-20T13:03:36.156583+0000","last_undegraded":"2026-02-20T13:03:36.156583+0000","last_fullsized":"2026-02-20T13:03:36.156583+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T13:03:33.131410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T13:03:33.131410+0000","last_clean_scrub_stamp":"2026-02-20T13:03:33.131410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-20T13:03:36.156072+0000","last_change":"2026-02-20T13:03:36.157620+0000","last_active":"2026-02-20T13:03:36.156072+0000","last_peered":"2026-02-20T13:03:36.156072+0000","last_clean":"2026-02-20T13:03:36.156072+0000","last_became_active":"2026-02-20T13:03:34.169637+0000","last_became_peered":"2026-02-20T13:03:34.169637+0000","last_unstale":"2026-02-20T13:03:36.156072+0000","last_undegraded":"2026-02-20T13:03:36.156072+0000","last_fullsized":"2026-02-20T13:03:36.156072+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T13:03:33.131410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T13:03:33.131410+0000","last_clean_scrub_stamp":"2026-02-20T13:03:33.131410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"17'2","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-20T13:03:36.155508+0000","last_change":"2026-02-20T13:03:36.155922+0000","last_active":"2026-02-20T13:03:36.155508+0000","last_peered":"2026-02-20T13:03:36.155508+0000","last_clean":"2026-02-20T13:03:36.155508+0000","last_became_active":"2026-02-20T13:03:34.178652+0000","last_became_peered":"2026-02-20T13:03:34.178652+0000","last_unstale":"2026-02-20T13:03:36.155508+0000","last_undegraded":"2026-02-20T13:03:36.155508+0000","last_fullsized":"2026-02-20T13:03:36.155508+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T13:03:33.131410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T13:03:33.131410+0000","last_clean_scrub_stamp":"2026-02-20T13:03:33.131410+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-20T13:03:36.153752+0000","last_change":"2026-02-20T13:03:36.155731+0000","last_active":"2026-02-20T13:03:36.153752+0000","last_peered":"2026-02-20T13:03:36.153752+0000","last_clean":"2026-02-20T13:03:36.153752+0000","last_became_active":"2026-02-20T13:03:34.176581+0000","last_became_peered":"2026-02-20T13:03:34.176581+0000","last_unstale":"2026-02-20T13:03:36.153752+0000","last_undegraded":"2026-02-20T13:03:36.153752+0000","last_fullsized":"2026-02-20T13:03:36.153752+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T13:03:33.131410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T13:03:33.131410+0000","last_clean_scrub_stamp":"2026-02-20T13:03:33.131410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-20T13:03:36.152554+0000","last_change":"2026-02-20T13:03:36.155018+0000","last_active":"2026-02-20T13:03:36.152554+0000","last_peered":"2026-02-20T13:03:36.152554+0000","last_clean":"2026-02-20T13:03:36.152554+0000","last_became_active":"2026-02-20T13:03:34.176780+0000","last_became_peered":"2026-02-20T13:03:34.176780+0000","last_unstale":"2026-02-20T13:03:36.152554+0000","last_undegraded":"2026-02-20T13:03:36.152554+0000","last_fullsized":"2026-02-20T13:03:36.152554+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T13:03:33.131410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T13:03:33.131410+0000","last_clean_scrub_stamp":"2026-02-20T13:03:33.131410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":16,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-20T13:03:36.155853+0000","last_change":"2026-02-20T13:03:36.158942+0000","last_active":"2026-02-20T13:03:36.155853+0000","last_peered":"2026-02-20T13:03:36.155853+0000","last_clean":"2026-02-20T13:03:36.155853+0000","last_became_active":"2026-02-20T13:03:34.166167+0000","last_became_peered":"2026-02-20T13:03:34.166167+0000","last_unstale":"2026-02-20T13:03:36.155853+0000","last_undegraded":"2026-02-20T13:03:36.155853+0000","last_fullsized":"2026-02-20T13:03:36.155853+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T13:03:33.131410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T13:03:33.131410+0000","last_clean_scrub_stamp":"2026-02-20T13:03:33.131410+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":90,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-20T13:03:36.154650+0000","last_change":"2026-02-20T13:03:30.833889+0000","last_active":"2026-02-20T13:03:36.154650+0000","last_peered":"2026-02-20T13:03:36.154650+0000","last_clean":"2026-02-20T13:03:36.154650+0000","last_became_active":"2026-02-20T13:03:30.827292+0000","last_became_peered":"2026-02-20T13:03:30.827292+0000","last_unstale":"2026-02-20T13:03:36.154650+0000","last_undegraded":"2026-02-20T13:03:36.154650+0000","last_fullsized":"2026-02-20T13:03:36.154650+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T13:03:29.796415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T13:03:29.796415+0000","last_clean_scrub_stamp":"2026-02-20T13:03:29.796415+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":161061224448,"available":96634163200,"internally_reserved":0,"allocated":64427061248,"data_stored":64427061248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53687074816,"available":32211079168,"internally_reserved":0,"allocated":21475995648,"data_stored":21475995648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20971820,"kb_used_data":20971820,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456964,"statfs":{"total":53687074816,"available":32211931136,"internally_reserved":0,"allocated":21475143680,"data_stored":21475143680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972580,"kb_used_data":20972580,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456204,"statfs":{"total":53687074816,"available":32211152896,"internally_reserved":0,"allocated":21475921920,"data_stored":21475921920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972652,"kb_used_data":20972652,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456132,"statfs":{"total":53687074816,"available":32211079168,"internally_reserved":0,"allocated":21475995648,"data_stored":21475995648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53687074816,"available":32211079168,"internally_reserved":0,"allocated":21475995648,"data_stored":21475995648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53687074816,"available":32211152896,"internally_reserved":0,"allocated":21475921920,"data_stored":21475921920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53687074816,"available":32211079168,"internally_reserved":0,"allocated":21475995648,"data_stored":21475995648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53687074816,"available":32211931136,"internally_reserved":0,"allocated":21475143680,"data_stored":21475143680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-20T13:03:37.811 INFO:tasks.ceph.ceph_manager.ceph:clean! 2026-02-20T13:03:37.811 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2026-02-20T13:03:37.811 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2026-02-20T13:03:37.812 DEBUG:teuthology.orchestra.run.trial093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-20T13:03:37.887 INFO:teuthology.orchestra.run.trial093.stderr:2026-02-20T13:03:37.884+0000 7efeb305a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T13:03:37.888 INFO:teuthology.orchestra.run.trial093.stderr:2026-02-20T13:03:37.887+0000 7efeb305a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T13:03:38.023 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:03:38.023 INFO:teuthology.orchestra.run.trial093.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2026-02-20T13:03:38.033 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2026-02-20T13:03:38.033 INFO:teuthology.run_tasks:Running task ssh_keys... 2026-02-20T13:03:38.087 DEBUG:teuthology.orchestra.run.trial093:> echo 'StrictHostKeyChecking no 2026-02-20T13:03:38.087 DEBUG:teuthology.orchestra.run.trial093:> ' > /home/ubuntu/.ssh/config && echo 'UserKnownHostsFile ' /dev/null >> /home/ubuntu/.ssh/config && chmod 600 /home/ubuntu/.ssh/config 2026-02-20T13:03:38.103 INFO:teuthology.task.ssh_keys:generated public key AAAAB3NzaC1yc2EAAAADAQABAAABAQCuXVjjj378lzZkp47r4opGvEFWJGoxHZk6RTy/ygXyTqDpP9ZUFke89L2+TWs2owVNXrk81J7qm8sG0uC2EjTriEVQfEW+cy5Enzej40EXaZW64YGy7E17ZAp4/JtHvIynLC7e6P1Tms/p3hQpYgyOOC9Gam+FF1kOg6usfDzecsqr9BmTvrHkB0NURZzwe+Ts91rn6YUhGdtJzwUPSYErAvAWesAXUvnTF1ANfd6jgHA0ss0pN3uckbuIsqQ9K96xgVTD6q6lUzvbvZSELs8nXzkZLSGDsKrabpCxPshtaStkDv9P579jMkXUl7zILnwfgOI8SG5EXhSU615Fs6lz 2026-02-20T13:03:38.103 INFO:teuthology.task.ssh_keys:pushing keys to trial093.front.sepia.ceph.com for ubuntu 2026-02-20T13:03:38.103 DEBUG:teuthology.orchestra.run.trial093:> rm -f -- /home/ubuntu/.ssh/id_rsa 2026-02-20T13:03:38.158 DEBUG:teuthology.orchestra.run.trial093:> set -ex 2026-02-20T13:03:38.158 DEBUG:teuthology.orchestra.run.trial093:> dd of=/home/ubuntu/.ssh/id_rsa 2026-02-20T13:03:38.158 DEBUG:teuthology.orchestra.run.trial093:> chmod 0500 /home/ubuntu/.ssh/id_rsa 2026-02-20T13:03:38.215 DEBUG:teuthology.orchestra.run.trial093:> rm -f -- /home/ubuntu/.ssh/id_rsa.pub 2026-02-20T13:03:38.273 DEBUG:teuthology.orchestra.run.trial093:> set -ex 2026-02-20T13:03:38.273 DEBUG:teuthology.orchestra.run.trial093:> dd of=/home/ubuntu/.ssh/id_rsa.pub 2026-02-20T13:03:38.329 DEBUG:teuthology.orchestra.run.trial093:> cp -v -a /home/ubuntu/.ssh/authorized_keys /home/ubuntu/.ssh/authorized_keys_2026-02-20_13:03:38:329396 2026-02-20T13:03:38.386 INFO:teuthology.orchestra.run.trial093.stdout:'/home/ubuntu/.ssh/authorized_keys' -> '/home/ubuntu/.ssh/authorized_keys_2026-02-20_13:03:38:329396' 2026-02-20T13:03:38.387 DEBUG:teuthology.orchestra.run.trial093:> set -ex 2026-02-20T13:03:38.387 DEBUG:teuthology.orchestra.run.trial093:> dd of=/home/ubuntu/.ssh/authorized_keys conv=notrunc oflag=append 2026-02-20T13:03:38.443 INFO:teuthology.run_tasks:Running task cbt... 2026-02-20T13:03:38.468 INFO:tasks.cbt:cbt configuration is {'cluster': {'user': 'ubuntu', 'head': 'ubuntu@trial093.front.sepia.ceph.com', 'osds': ['ubuntu@trial093.front.sepia.ceph.com'], 'mons': ['ubuntu@trial093.front.sepia.ceph.com'], 'clients': ['ubuntu@trial093.front.sepia.ceph.com'], 'rgws': {'ubuntu@trial093.front.sepia.ceph.com': None}, 'osds_per_node': 3, 'rebuild_every_test': False, 'use_existing': True, 'is_teuthology': True, 'iterations': 1, 'tmp_dir': '/tmp/cbt', 'pool_profiles': {'rbd': {'pg_size': 128, 'pgp_size': 128, 'replication': 3}}, 'pid_dir': '/var/run/ceph'}, 'benchmarks': {'librbdfio': {'iodepth': [32], 'log_avg_msec': 100, 'mode': ['randwrite'], 'norandommap': True, 'op_size': [4096], 'osd_ra': [4096], 'pool_profile': 'rbd', 'procs_per_volume': [1], 'time': 60, 'vol_size': 4096, 'volumes_per_client': [2], 'cmd_path': '/home/ubuntu/cephtest/fio/fio'}}, 'client_endpoints': None, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}} 2026-02-20T13:03:38.468 DEBUG:teuthology.orchestra.run.trial093:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/cbt 2026-02-20T13:03:38.503 DEBUG:teuthology.orchestra.run.trial093:> set -ex 2026-02-20T13:03:38.503 DEBUG:teuthology.orchestra.run.trial093:> dd of=/home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2026-02-20T13:03:38.555 DEBUG:teuthology.orchestra.run.trial093:> git clone --depth 1 -b master https://github.com/ceph/cbt.git /home/ubuntu/cephtest/cbt 2026-02-20T13:03:38.617 INFO:teuthology.orchestra.run.trial093.stderr:Cloning into '/home/ubuntu/cephtest/cbt'... 2026-02-20T13:03:39.834 INFO:tasks.cbt:Installing collectl 2026-02-20T13:03:39.834 DEBUG:teuthology.orchestra.run.trial093:> sudo mkdir collectl && cd collectl && sudo wget https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download -O collectl.tar.gz && sudo tar -xvf collectl.tar.gz && cd collectl-4.3.1 && sudo ./INSTALL 2026-02-20T13:03:39.909 INFO:teuthology.orchestra.run.trial093.stderr:--2026-02-20 13:03:39-- https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download 2026-02-20T13:03:45.209 INFO:teuthology.orchestra.run.trial093.stderr:Resolving sourceforge.net (sourceforge.net)... 104.18.13.149, 104.18.12.149, 2606:4700::6812:d95, ... 2026-02-20T13:03:45.335 INFO:teuthology.orchestra.run.trial093.stderr:Connecting to sourceforge.net (sourceforge.net)|104.18.13.149|:443... connected. 2026-02-20T13:03:47.382 INFO:teuthology.orchestra.run.trial093.stderr:HTTP request sent, awaiting response... 302 Found 2026-02-20T13:03:47.382 INFO:teuthology.orchestra.run.trial093.stderr:Location: https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABpmFuxgHz6Il_yqe56IkOPxdZR3T-nqkc4iuDsZbzUYTIHjknwxt7NtrQokIUGNPstxDfPFyaV_7t_dzjbmKmD2dBBTg%3D%3D&use_mirror=cytranet-dal&r= [following] 2026-02-20T13:03:47.382 INFO:teuthology.orchestra.run.trial093.stderr:--2026-02-20 13:03:47-- https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABpmFuxgHz6Il_yqe56IkOPxdZR3T-nqkc4iuDsZbzUYTIHjknwxt7NtrQokIUGNPstxDfPFyaV_7t_dzjbmKmD2dBBTg%3D%3D&use_mirror=cytranet-dal&r= 2026-02-20T13:03:52.710 INFO:teuthology.orchestra.run.trial093.stderr:Resolving downloads.sourceforge.net (downloads.sourceforge.net)... 104.18.13.149, 104.18.12.149, 2606:4700::6812:d95, ... 2026-02-20T13:03:52.822 INFO:teuthology.orchestra.run.trial093.stderr:Connecting to downloads.sourceforge.net (downloads.sourceforge.net)|104.18.13.149|:443... connected. 2026-02-20T13:03:53.224 INFO:teuthology.orchestra.run.trial093.stderr:HTTP request sent, awaiting response... 302 Found 2026-02-20T13:03:53.225 INFO:teuthology.orchestra.run.trial093.stderr:Location: https://cytranet-dal.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?viasf=1 [following] 2026-02-20T13:03:53.225 INFO:teuthology.orchestra.run.trial093.stderr:--2026-02-20 13:03:53-- https://cytranet-dal.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?viasf=1 2026-02-20T13:03:58.353 INFO:teuthology.orchestra.run.trial093.stderr:Resolving cytranet-dal.dl.sourceforge.net (cytranet-dal.dl.sourceforge.net)... 162.226.127.129 2026-02-20T13:03:58.393 INFO:teuthology.orchestra.run.trial093.stderr:Connecting to cytranet-dal.dl.sourceforge.net (cytranet-dal.dl.sourceforge.net)|162.226.127.129|:443... connected. 2026-02-20T13:03:58.918 INFO:teuthology.orchestra.run.trial093.stderr:HTTP request sent, awaiting response... 200 OK 2026-02-20T13:03:58.919 INFO:teuthology.orchestra.run.trial093.stderr:Length: 620693 (606K) [application/x-gzip] 2026-02-20T13:03:58.919 INFO:teuthology.orchestra.run.trial093.stderr:Saving to: ‘collectl.tar.gz’ 2026-02-20T13:03:58.919 INFO:teuthology.orchestra.run.trial093.stderr: 2026-02-20T13:03:58.960 INFO:teuthology.orchestra.run.trial093.stderr: 0K .......... .......... .......... .......... .......... 8% 1.16M 0s 2026-02-20T13:03:59.000 INFO:teuthology.orchestra.run.trial093.stderr: 50K .......... .......... .......... .......... .......... 16% 1.22M 0s 2026-02-20T13:03:59.039 INFO:teuthology.orchestra.run.trial093.stderr: 100K .......... .......... .......... .......... .......... 24% 1.27M 0s 2026-02-20T13:03:59.079 INFO:teuthology.orchestra.run.trial093.stderr: 150K .......... .......... .......... .......... .......... 32% 1.28M 0s 2026-02-20T13:03:59.080 INFO:teuthology.orchestra.run.trial093.stderr: 200K .......... .......... .......... .......... .......... 41% 31.7M 0s 2026-02-20T13:03:59.080 INFO:teuthology.orchestra.run.trial093.stderr: 250K .......... .......... .......... .......... .......... 49% 43.7M 0s 2026-02-20T13:03:59.081 INFO:teuthology.orchestra.run.trial093.stderr: 300K .......... .......... .......... .......... .......... 57% 43.8M 0s 2026-02-20T13:03:59.116 INFO:teuthology.orchestra.run.trial093.stderr: 350K .......... .......... .......... .......... .......... 65% 1.36M 0s 2026-02-20T13:03:59.117 INFO:teuthology.orchestra.run.trial093.stderr: 400K .......... .......... .......... .......... .......... 74% 67.7M 0s 2026-02-20T13:03:59.117 INFO:teuthology.orchestra.run.trial093.stderr: 450K .......... .......... .......... .......... .......... 82% 89.4M 0s 2026-02-20T13:03:59.122 INFO:teuthology.orchestra.run.trial093.stderr: 500K .......... .......... .......... .......... .......... 90% 13.2M 0s 2026-02-20T13:03:59.122 INFO:teuthology.orchestra.run.trial093.stderr: 550K .......... .......... .......... .......... .......... 98% 309M 0s 2026-02-20T13:03:59.122 INFO:teuthology.orchestra.run.trial093.stderr: 600K ...... 100% 11.4T=0.2s 2026-02-20T13:03:59.122 INFO:teuthology.orchestra.run.trial093.stderr: 2026-02-20T13:03:59.122 INFO:teuthology.orchestra.run.trial093.stderr:2026-02-20 13:03:59 (2.91 MB/s) - ‘collectl.tar.gz’ saved [620693/620693] 2026-02-20T13:03:59.122 INFO:teuthology.orchestra.run.trial093.stderr: 2026-02-20T13:03:59.140 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/ 2026-02-20T13:03:59.140 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/collectl.conf 2026-02-20T13:03:59.140 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/docs/ 2026-02-20T13:03:59.141 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/docs/SlowProc.html 2026-02-20T13:03:59.141 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/docs/BuddyInfo.html 2026-02-20T13:03:59.141 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/docs/WhySummary.html 2026-02-20T13:03:59.141 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/docs/Socket.html 2026-02-20T13:03:59.141 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/docs/colmux.html 2026-02-20T13:03:59.141 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/docs/Tutorial.html 2026-02-20T13:03:59.141 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/docs/Gexpr.html 2026-02-20T13:03:59.141 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/docs/Import.html 2026-02-20T13:03:59.141 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/docs/style.css 2026-02-20T13:03:59.141 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/docs/Ganglia3.jpg 2026-02-20T13:03:59.141 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/docs/Ganglia2.jpg 2026-02-20T13:03:59.141 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/docs/FileNaming.html 2026-02-20T13:03:59.141 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/docs/TheMath.html 2026-02-20T13:03:59.141 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/docs/Environmental.html 2026-02-20T13:03:59.141 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/docs/Process.html 2026-02-20T13:03:59.141 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/docs/OutputFiles.html 2026-02-20T13:03:59.141 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/docs/HiResTime.html 2026-02-20T13:03:59.141 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/docs/OutputFormats.html 2026-02-20T13:03:59.141 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/docs/slub.jpg 2026-02-20T13:03:59.141 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/docs/Lustre.html 2026-02-20T13:03:59.141 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/docs/Documentation.html 2026-02-20T13:03:59.141 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/docs/Inodes.html 2026-02-20T13:03:59.141 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/docs/Graphite.html 2026-02-20T13:03:59.141 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/docs/InputFiles.html 2026-02-20T13:03:59.141 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/docs/Export.html 2026-02-20T13:03:59.141 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/docs/Interrupts.html 2026-02-20T13:03:59.141 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/docs/Startup.html 2026-02-20T13:03:59.141 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/docs/Performance.html 2026-02-20T13:03:59.141 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/docs/Messages.html 2026-02-20T13:03:59.141 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/docs/Logging.html 2026-02-20T13:03:59.141 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/docs/Data-detail.html 2026-02-20T13:03:59.142 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/docs/CPUs.html 2026-02-20T13:03:59.142 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/docs/Matrix.html 2026-02-20T13:03:59.142 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/docs/Data-brief.html 2026-02-20T13:03:59.142 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/docs/SlabInfo.html 2026-02-20T13:03:59.142 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/docs/Misc.html 2026-02-20T13:03:59.142 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/docs/Examples.html 2026-02-20T13:03:59.142 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/docs/OpenStack.html 2026-02-20T13:03:59.142 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/docs/Plotfiles.html 2026-02-20T13:03:59.142 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/docs/ColmuxCPU.jpg 2026-02-20T13:03:59.143 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/docs/Tutorial-Lustre.html 2026-02-20T13:03:59.143 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/docs/Data-verbose.html 2026-02-20T13:03:59.143 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/docs/Data.html 2026-02-20T13:03:59.143 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/docs/Exceptions.html 2026-02-20T13:03:59.143 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/docs/RunningAsAService.html 2026-02-20T13:03:59.143 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/docs/FAQ-collectl.html 2026-02-20T13:03:59.143 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/docs/Ganglia.jpg 2026-02-20T13:03:59.143 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/docs/OperationalModes.html 2026-02-20T13:03:59.143 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/docs/Playback.html 2026-02-20T13:03:59.143 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/docs/Hello.html 2026-02-20T13:03:59.143 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/docs/Sockets.html 2026-02-20T13:03:59.143 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/docs/Network.html 2026-02-20T13:03:59.143 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/docs/Infiniband.html 2026-02-20T13:03:59.144 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/docs/Memory.html 2026-02-20T13:03:59.144 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/docs/ColmuxLustre.jpg 2026-02-20T13:03:59.144 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/docs/Disks.html 2026-02-20T13:03:59.144 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/docs/NfsInfo.html 2026-02-20T13:03:59.144 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/service/ 2026-02-20T13:03:59.144 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/service/collectl.service 2026-02-20T13:03:59.144 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/initd/ 2026-02-20T13:03:59.144 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/initd/collectl-suse 2026-02-20T13:03:59.144 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/initd/collectl-debian 2026-02-20T13:03:59.144 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/initd/collectl-generic 2026-02-20T13:03:59.144 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/initd/collectl 2026-02-20T13:03:59.144 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/misc.ph 2026-02-20T13:03:59.144 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/gexpr.ph 2026-02-20T13:03:59.144 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/statsd.ph 2026-02-20T13:03:59.144 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/envrules.std 2026-02-20T13:03:59.144 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/hello.ph 2026-02-20T13:03:59.144 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/vmsum.ph 2026-02-20T13:03:59.144 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/proctree.ph 2026-02-20T13:03:59.144 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/README-WINDOWS 2026-02-20T13:03:59.144 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/formatit.ph 2026-02-20T13:03:59.145 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/colmux 2026-02-20T13:03:59.145 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/INSTALL 2026-02-20T13:03:59.145 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/graphite.ph 2026-02-20T13:03:59.145 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/RELEASE-collectl 2026-02-20T13:03:59.145 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/vmstat.ph 2026-02-20T13:03:59.145 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/README 2026-02-20T13:03:59.145 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/man1/ 2026-02-20T13:03:59.145 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/man1/collectl.1 2026-02-20T13:03:59.145 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/man1/colmux.1 2026-02-20T13:03:59.145 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/lexpr.ph 2026-02-20T13:03:59.145 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/client.pl 2026-02-20T13:03:59.145 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/collectl 2026-02-20T13:03:59.146 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/COPYING 2026-02-20T13:03:59.146 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/vnet.ph 2026-02-20T13:03:59.146 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/GPL 2026-02-20T13:03:59.146 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/UNINSTALL 2026-02-20T13:03:59.146 INFO:teuthology.orchestra.run.trial093.stdout:collectl-4.3.1/ARTISTIC 2026-02-20T13:03:59.190 INFO:teuthology.orchestra.run.trial093.stderr:./INSTALL: line 90: chkconfig: command not found 2026-02-20T13:03:59.192 DEBUG:teuthology.orchestra.run.trial093:> sudo yum -y install librbd-devel pdsh pdsh-rcmd-ssh perf 2026-02-20T13:03:59.542 INFO:teuthology.orchestra.run.trial093.stdout:Last metadata expiration check: 0:01:15 ago on Fri 20 Feb 2026 01:02:44 PM UTC. 2026-02-20T13:03:59.627 INFO:teuthology.orchestra.run.trial093.stdout:Dependencies resolved. 2026-02-20T13:03:59.628 INFO:teuthology.orchestra.run.trial093.stdout:================================================================================ 2026-02-20T13:03:59.628 INFO:teuthology.orchestra.run.trial093.stdout: Package Arch Version Repository Size 2026-02-20T13:03:59.628 INFO:teuthology.orchestra.run.trial093.stdout:================================================================================ 2026-02-20T13:03:59.628 INFO:teuthology.orchestra.run.trial093.stdout:Installing: 2026-02-20T13:03:59.628 INFO:teuthology.orchestra.run.trial093.stdout: librbd-devel x86_64 2:20.3.0-5312.g26bc9765.el9 ceph 24 k 2026-02-20T13:03:59.628 INFO:teuthology.orchestra.run.trial093.stdout: pdsh x86_64 2.34-7.el9 epel 118 k 2026-02-20T13:03:59.628 INFO:teuthology.orchestra.run.trial093.stdout: pdsh-rcmd-ssh x86_64 2.34-7.el9 epel 13 k 2026-02-20T13:03:59.628 INFO:teuthology.orchestra.run.trial093.stdout: perf x86_64 5.14.0-681.el9 appstream 3.2 M 2026-02-20T13:03:59.628 INFO:teuthology.orchestra.run.trial093.stdout:Installing dependencies: 2026-02-20T13:03:59.628 INFO:teuthology.orchestra.run.trial093.stdout: libradospp-devel x86_64 2:20.3.0-5312.g26bc9765.el9 ceph 31 k 2026-02-20T13:03:59.628 INFO:teuthology.orchestra.run.trial093.stdout: libtraceevent x86_64 1.8.4-2.el9 baseos 290 k 2026-02-20T13:03:59.628 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:03:59.628 INFO:teuthology.orchestra.run.trial093.stdout:Transaction Summary 2026-02-20T13:03:59.628 INFO:teuthology.orchestra.run.trial093.stdout:================================================================================ 2026-02-20T13:03:59.628 INFO:teuthology.orchestra.run.trial093.stdout:Install 6 Packages 2026-02-20T13:03:59.628 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:03:59.629 INFO:teuthology.orchestra.run.trial093.stdout:Total download size: 3.7 M 2026-02-20T13:03:59.629 INFO:teuthology.orchestra.run.trial093.stdout:Installed size: 14 M 2026-02-20T13:03:59.629 INFO:teuthology.orchestra.run.trial093.stdout:Downloading Packages: 2026-02-20T13:04:00.270 INFO:teuthology.orchestra.run.trial093.stdout:(1/6): libradospp-devel-20.3.0-5312.g26bc9765.e 159 kB/s | 31 kB 00:00 2026-02-20T13:04:00.272 INFO:teuthology.orchestra.run.trial093.stdout:(2/6): librbd-devel-20.3.0-5312.g26bc9765.el9.x 120 kB/s | 24 kB 00:00 2026-02-20T13:04:00.533 INFO:teuthology.orchestra.run.trial093.stdout:(3/6): pdsh-2.34-7.el9.x86_64.rpm 453 kB/s | 118 kB 00:00 2026-02-20T13:04:00.554 INFO:teuthology.orchestra.run.trial093.stdout:(4/6): pdsh-rcmd-ssh-2.34-7.el9.x86_64.rpm 633 kB/s | 13 kB 00:00 2026-02-20T13:04:00.615 INFO:teuthology.orchestra.run.trial093.stdout:(5/6): perf-5.14.0-681.el9.x86_64.rpm 9.3 MB/s | 3.2 MB 00:00 2026-02-20T13:04:00.799 INFO:teuthology.orchestra.run.trial093.stdout:(6/6): libtraceevent-1.8.4-2.el9.x86_64.rpm 400 kB/s | 290 kB 00:00 2026-02-20T13:04:00.805 INFO:teuthology.orchestra.run.trial093.stdout:-------------------------------------------------------------------------------- 2026-02-20T13:04:00.805 INFO:teuthology.orchestra.run.trial093.stdout:Total 3.1 MB/s | 3.7 MB 00:01 2026-02-20T13:04:00.829 INFO:teuthology.orchestra.run.trial093.stdout:Running transaction check 2026-02-20T13:04:00.839 INFO:teuthology.orchestra.run.trial093.stdout:Transaction check succeeded. 2026-02-20T13:04:00.839 INFO:teuthology.orchestra.run.trial093.stdout:Running transaction test 2026-02-20T13:04:00.896 INFO:teuthology.orchestra.run.trial093.stdout:Transaction test succeeded. 2026-02-20T13:04:00.897 INFO:teuthology.orchestra.run.trial093.stdout:Running transaction 2026-02-20T13:04:00.971 INFO:teuthology.orchestra.run.trial093.stdout: Preparing : 1/1 2026-02-20T13:04:00.976 INFO:teuthology.orchestra.run.trial093.stdout: Installing : pdsh-rcmd-ssh-2.34-7.el9.x86_64 1/6 2026-02-20T13:04:01.017 INFO:teuthology.orchestra.run.trial093.stdout: Installing : pdsh-2.34-7.el9.x86_64 2/6 2026-02-20T13:04:01.021 INFO:teuthology.orchestra.run.trial093.stdout: Installing : libtraceevent-1.8.4-2.el9.x86_64 3/6 2026-02-20T13:04:01.024 INFO:teuthology.orchestra.run.trial093.stdout: Installing : libradospp-devel-2:20.3.0-5312.g26bc9765.el9.x86_64 4/6 2026-02-20T13:04:01.185 INFO:teuthology.orchestra.run.trial093.stdout: Installing : librbd-devel-2:20.3.0-5312.g26bc9765.el9.x86_64 5/6 2026-02-20T13:04:01.206 INFO:teuthology.orchestra.run.trial093.stdout: Installing : perf-5.14.0-681.el9.x86_64 6/6 2026-02-20T13:04:01.320 INFO:teuthology.orchestra.run.trial093.stdout: Running scriptlet: perf-5.14.0-681.el9.x86_64 6/6 2026-02-20T13:04:01.320 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : libradospp-devel-2:20.3.0-5312.g26bc9765.el9.x86_64 1/6 2026-02-20T13:04:01.320 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : librbd-devel-2:20.3.0-5312.g26bc9765.el9.x86_64 2/6 2026-02-20T13:04:01.320 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : libtraceevent-1.8.4-2.el9.x86_64 3/6 2026-02-20T13:04:01.320 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : perf-5.14.0-681.el9.x86_64 4/6 2026-02-20T13:04:01.320 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : pdsh-2.34-7.el9.x86_64 5/6 2026-02-20T13:04:01.381 INFO:teuthology.orchestra.run.trial093.stdout: Verifying : pdsh-rcmd-ssh-2.34-7.el9.x86_64 6/6 2026-02-20T13:04:01.381 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:04:01.381 INFO:teuthology.orchestra.run.trial093.stdout:Installed: 2026-02-20T13:04:01.381 INFO:teuthology.orchestra.run.trial093.stdout: libradospp-devel-2:20.3.0-5312.g26bc9765.el9.x86_64 2026-02-20T13:04:01.381 INFO:teuthology.orchestra.run.trial093.stdout: librbd-devel-2:20.3.0-5312.g26bc9765.el9.x86_64 2026-02-20T13:04:01.381 INFO:teuthology.orchestra.run.trial093.stdout: libtraceevent-1.8.4-2.el9.x86_64 2026-02-20T13:04:01.381 INFO:teuthology.orchestra.run.trial093.stdout: pdsh-2.34-7.el9.x86_64 2026-02-20T13:04:01.381 INFO:teuthology.orchestra.run.trial093.stdout: pdsh-rcmd-ssh-2.34-7.el9.x86_64 2026-02-20T13:04:01.381 INFO:teuthology.orchestra.run.trial093.stdout: perf-5.14.0-681.el9.x86_64 2026-02-20T13:04:01.381 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:04:01.381 INFO:teuthology.orchestra.run.trial093.stdout:Complete! 2026-02-20T13:04:01.411 DEBUG:teuthology.orchestra.run.trial093:> python3 -m pip install -r /home/ubuntu/cephtest/cbt/requirements.txt 2026-02-20T13:04:01.549 INFO:teuthology.orchestra.run.trial093.stdout:Defaulting to user installation because normal site-packages is not writeable 2026-02-20T13:04:01.559 INFO:teuthology.orchestra.run.trial093.stdout:Requirement already satisfied: pyyaml in /usr/lib64/python3.9/site-packages (from -r /home/ubuntu/cephtest/cbt/requirements.txt (line 1)) (5.4.1) 2026-02-20T13:04:01.559 INFO:teuthology.orchestra.run.trial093.stdout:Requirement already satisfied: lxml in /usr/lib64/python3.9/site-packages (from -r /home/ubuntu/cephtest/cbt/requirements.txt (line 2)) (4.6.5) 2026-02-20T13:04:03.704 INFO:teuthology.orchestra.run.trial093.stdout:Collecting matplotlib 2026-02-20T13:04:04.825 INFO:teuthology.orchestra.run.trial093.stdout: Downloading matplotlib-3.9.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (8.3 MB) 2026-02-20T13:11:39.323 INFO:teuthology.orchestra.run.trial093.stdout:Requirement already satisfied: packaging>=20.0 in /usr/lib/python3.9/site-packages (from matplotlib->-r /home/ubuntu/cephtest/cbt/requirements.txt (line 3)) (20.9) 2026-02-20T13:11:40.011 INFO:teuthology.orchestra.run.trial093.stdout:Collecting importlib-resources>=3.2.0 2026-02-20T13:11:40.182 INFO:teuthology.orchestra.run.trial093.stdout: Downloading importlib_resources-6.5.2-py3-none-any.whl (37 kB) 2026-02-20T13:11:42.925 INFO:teuthology.orchestra.run.trial093.stdout:Collecting cycler>=0.10 2026-02-20T13:11:43.054 INFO:teuthology.orchestra.run.trial093.stdout: Downloading cycler-0.12.1-py3-none-any.whl (8.3 kB) 2026-02-20T13:11:58.179 INFO:teuthology.orchestra.run.trial093.stdout:Collecting fonttools>=4.22.0 2026-02-20T13:11:58.309 INFO:teuthology.orchestra.run.trial093.stdout: Downloading fonttools-4.60.2-cp39-cp39-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (4.8 MB) 2026-02-20T13:16:08.591 INFO:teuthology.orchestra.run.trial093.stdout:Collecting contourpy>=1.0.1 2026-02-20T13:16:09.281 INFO:teuthology.orchestra.run.trial093.stdout: Downloading contourpy-1.3.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (321 kB) 2026-02-20T13:16:25.303 INFO:teuthology.orchestra.run.trial093.stdout:Requirement already satisfied: pyparsing>=2.3.1 in /usr/lib/python3.9/site-packages (from matplotlib->-r /home/ubuntu/cephtest/cbt/requirements.txt (line 3)) (2.4.7) 2026-02-20T13:16:42.927 INFO:teuthology.orchestra.run.trial093.stdout:Collecting pillow>=8 2026-02-20T13:16:43.064 INFO:teuthology.orchestra.run.trial093.stdout: Downloading pillow-11.3.0-cp39-cp39-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (6.6 MB) 2026-02-20T13:22:35.534 INFO:teuthology.orchestra.run.trial093.stdout:Requirement already satisfied: numpy>=1.23 in /usr/lib64/python3.9/site-packages (from matplotlib->-r /home/ubuntu/cephtest/cbt/requirements.txt (line 3)) (1.23.5) 2026-02-20T13:22:35.546 INFO:teuthology.orchestra.run.trial093.stdout:Requirement already satisfied: python-dateutil>=2.7 in /usr/lib/python3.9/site-packages (from matplotlib->-r /home/ubuntu/cephtest/cbt/requirements.txt (line 3)) (2.9.0.post0) 2026-02-20T13:22:39.084 INFO:teuthology.orchestra.run.trial093.stdout:Collecting kiwisolver>=1.3.1 2026-02-20T13:22:39.223 INFO:teuthology.orchestra.run.trial093.stdout: Downloading kiwisolver-1.4.7-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.6 MB) 2026-02-20T13:24:11.093 INFO:teuthology.orchestra.run.trial093.stdout:Collecting zipp>=3.1.0 2026-02-20T13:24:11.718 INFO:teuthology.orchestra.run.trial093.stdout: Downloading zipp-3.23.0-py3-none-any.whl (10 kB) 2026-02-20T13:24:12.686 INFO:teuthology.orchestra.run.trial093.stdout:Requirement already satisfied: six>=1.5 in /usr/lib/python3.9/site-packages (from python-dateutil>=2.7->matplotlib->-r /home/ubuntu/cephtest/cbt/requirements.txt (line 3)) (1.15.0) 2026-02-20T13:24:12.811 INFO:teuthology.orchestra.run.trial093.stdout:Installing collected packages: zipp, pillow, kiwisolver, importlib-resources, fonttools, cycler, contourpy, matplotlib 2026-02-20T13:24:13.853 INFO:teuthology.orchestra.run.trial093.stdout:Successfully installed contourpy-1.3.0 cycler-0.12.1 fonttools-4.60.2 importlib-resources-6.5.2 kiwisolver-1.4.7 matplotlib-3.9.4 pillow-11.3.0 zipp-3.23.0 2026-02-20T13:24:17.526 INFO:tasks.cbt:benchmark: librbdfio 2026-02-20T13:24:17.526 DEBUG:teuthology.orchestra.run.trial093:> git clone -b master https://github.com/axboe/fio.git /home/ubuntu/cephtest/fio 2026-02-20T13:24:17.542 INFO:teuthology.orchestra.run.trial093.stderr:Cloning into '/home/ubuntu/cephtest/fio'... 2026-02-20T13:24:18.441 DEBUG:teuthology.orchestra.run.trial093:> cd /home/ubuntu/cephtest/fio && ./configure && make 2026-02-20T13:24:18.609 INFO:teuthology.orchestra.run.trial093.stdout:Operating system Linux 2026-02-20T13:24:18.609 INFO:teuthology.orchestra.run.trial093.stdout:CPU x86_64 2026-02-20T13:24:18.609 INFO:teuthology.orchestra.run.trial093.stdout:Big endian no 2026-02-20T13:24:18.609 INFO:teuthology.orchestra.run.trial093.stdout:Compiler gcc 2026-02-20T13:24:18.609 INFO:teuthology.orchestra.run.trial093.stdout:Cross compile no 2026-02-20T13:24:18.609 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:24:18.609 INFO:teuthology.orchestra.run.trial093.stdout:Static build no 2026-02-20T13:24:18.657 INFO:teuthology.orchestra.run.trial093.stdout:Wordsize 64 2026-02-20T13:24:18.663 INFO:teuthology.orchestra.run.trial093.stdout:zlib no 2026-02-20T13:24:18.673 INFO:teuthology.orchestra.run.trial093.stdout:fcntl(F_FULLFSYNC) no 2026-02-20T13:24:18.683 INFO:teuthology.orchestra.run.trial093.stdout:Linux AIO support no 2026-02-20T13:24:18.683 INFO:teuthology.orchestra.run.trial093.stdout:Linux AIO support rw flags no 2026-02-20T13:24:18.705 INFO:teuthology.orchestra.run.trial093.stdout:POSIX AIO support yes 2026-02-20T13:24:18.705 INFO:teuthology.orchestra.run.trial093.stdout:POSIX AIO support needs -lrt no 2026-02-20T13:24:18.725 INFO:teuthology.orchestra.run.trial093.stdout:POSIX AIO fsync yes 2026-02-20T13:24:18.746 INFO:teuthology.orchestra.run.trial093.stdout:POSIX pshared support yes 2026-02-20T13:24:18.771 INFO:teuthology.orchestra.run.trial093.stdout:pthread_condattr_setclock() yes 2026-02-20T13:24:18.795 INFO:teuthology.orchestra.run.trial093.stdout:pthread_sigmask() yes 2026-02-20T13:24:18.820 INFO:teuthology.orchestra.run.trial093.stdout:pthread_getaffinity_np() yes 2026-02-20T13:24:18.826 INFO:teuthology.orchestra.run.trial093.stdout:Solaris AIO support no 2026-02-20T13:24:18.846 INFO:teuthology.orchestra.run.trial093.stdout:__sync_fetch_and_add yes 2026-02-20T13:24:18.868 INFO:teuthology.orchestra.run.trial093.stdout:__sync_synchronize yes 2026-02-20T13:24:18.888 INFO:teuthology.orchestra.run.trial093.stdout:__sync_val_compare_and_swap yes 2026-02-20T13:24:18.894 INFO:teuthology.orchestra.run.trial093.stdout:libverbs no 2026-02-20T13:24:18.903 INFO:teuthology.orchestra.run.trial093.stdout:rdmacm no 2026-02-20T13:24:18.923 INFO:teuthology.orchestra.run.trial093.stdout:asprintf() yes 2026-02-20T13:24:18.943 INFO:teuthology.orchestra.run.trial093.stdout:vasprintf() yes 2026-02-20T13:24:18.966 INFO:teuthology.orchestra.run.trial093.stdout:Linux fallocate yes 2026-02-20T13:24:18.988 INFO:teuthology.orchestra.run.trial093.stdout:POSIX fadvise yes 2026-02-20T13:24:19.008 INFO:teuthology.orchestra.run.trial093.stdout:POSIX fallocate yes 2026-02-20T13:24:19.030 INFO:teuthology.orchestra.run.trial093.stdout:sched_setaffinity(3 arg) yes 2026-02-20T13:24:19.031 INFO:teuthology.orchestra.run.trial093.stdout:sched_setaffinity(2 arg) no 2026-02-20T13:24:19.051 INFO:teuthology.orchestra.run.trial093.stdout:clock_gettime yes 2026-02-20T13:24:19.072 INFO:teuthology.orchestra.run.trial093.stdout:CLOCK_MONOTONIC yes 2026-02-20T13:24:19.093 INFO:teuthology.orchestra.run.trial093.stdout:gettimeofday yes 2026-02-20T13:24:19.114 INFO:teuthology.orchestra.run.trial093.stdout:fdatasync yes 2026-02-20T13:24:19.135 INFO:teuthology.orchestra.run.trial093.stdout:pipe() yes 2026-02-20T13:24:19.156 INFO:teuthology.orchestra.run.trial093.stdout:pipe2() yes 2026-02-20T13:24:19.177 INFO:teuthology.orchestra.run.trial093.stdout:pread() yes 2026-02-20T13:24:19.201 INFO:teuthology.orchestra.run.trial093.stdout:sync_file_range yes 2026-02-20T13:24:19.206 INFO:teuthology.orchestra.run.trial093.stdout:ASharedMemory_create no 2026-02-20T13:24:19.213 INFO:teuthology.orchestra.run.trial093.stdout:EXT4 move extent yes 2026-02-20T13:24:19.232 INFO:teuthology.orchestra.run.trial093.stdout:Linux splice(2) yes 2026-02-20T13:24:19.238 INFO:teuthology.orchestra.run.trial093.stdout:libnuma no 2026-02-20T13:24:19.259 INFO:teuthology.orchestra.run.trial093.stdout:strsep yes 2026-02-20T13:24:19.279 INFO:teuthology.orchestra.run.trial093.stdout:strcasestr yes 2026-02-20T13:24:19.288 INFO:teuthology.orchestra.run.trial093.stdout:strlcat no 2026-02-20T13:24:19.310 INFO:teuthology.orchestra.run.trial093.stdout:getopt_long_only() yes 2026-02-20T13:24:19.334 INFO:teuthology.orchestra.run.trial093.stdout:inet_aton yes 2026-02-20T13:24:19.355 INFO:teuthology.orchestra.run.trial093.stdout:socklen_t yes 2026-02-20T13:24:19.376 INFO:teuthology.orchestra.run.trial093.stdout:__thread yes 2026-02-20T13:24:19.397 INFO:teuthology.orchestra.run.trial093.stdout:RUSAGE_THREAD yes 2026-02-20T13:24:19.417 INFO:teuthology.orchestra.run.trial093.stdout:SCHED_IDLE yes 2026-02-20T13:24:19.437 INFO:teuthology.orchestra.run.trial093.stdout:TCP_NODELAY yes 2026-02-20T13:24:19.459 INFO:teuthology.orchestra.run.trial093.stdout:vsock yes 2026-02-20T13:24:19.481 INFO:teuthology.orchestra.run.trial093.stdout:Net engine window_size yes 2026-02-20T13:24:19.504 INFO:teuthology.orchestra.run.trial093.stdout:TCP_MAXSEG yes 2026-02-20T13:24:19.524 INFO:teuthology.orchestra.run.trial093.stdout:RLIMIT_MEMLOCK yes 2026-02-20T13:24:19.546 INFO:teuthology.orchestra.run.trial093.stdout:pwritev/preadv yes 2026-02-20T13:24:19.568 INFO:teuthology.orchestra.run.trial093.stdout:pwritev2/preadv2 yes 2026-02-20T13:24:19.592 INFO:teuthology.orchestra.run.trial093.stdout:IPv6 helpers yes 2026-02-20T13:24:19.604 INFO:teuthology.orchestra.run.trial093.stdout:http engine no 2026-02-20T13:24:19.655 INFO:teuthology.orchestra.run.trial093.stdout:Rados engine yes 2026-02-20T13:24:19.721 INFO:teuthology.orchestra.run.trial093.stdout:Rados Block Device engine yes 2026-02-20T13:24:19.786 INFO:teuthology.orchestra.run.trial093.stdout:rbd_poll yes 2026-02-20T13:24:19.853 INFO:teuthology.orchestra.run.trial093.stdout:rbd_invalidate_cache yes 2026-02-20T13:24:19.917 INFO:teuthology.orchestra.run.trial093.stdout:rbd_encryption_load yes 2026-02-20T13:24:19.938 INFO:teuthology.orchestra.run.trial093.stdout:setvbuf yes 2026-02-20T13:24:19.944 INFO:teuthology.orchestra.run.trial093.stdout:Gluster API engine no 2026-02-20T13:24:19.955 INFO:teuthology.orchestra.run.trial093.stdout:s390_z196_facilities no 2026-02-20T13:24:19.955 INFO:teuthology.orchestra.run.trial093.stdout:HDFS engine no 2026-02-20T13:24:19.975 INFO:teuthology.orchestra.run.trial093.stdout:MTD yes 2026-02-20T13:24:19.981 INFO:teuthology.orchestra.run.trial093.stdout:libpmem no 2026-02-20T13:24:19.982 INFO:teuthology.orchestra.run.trial093.stdout:libpmem1_5 no 2026-02-20T13:24:19.987 INFO:teuthology.orchestra.run.trial093.stdout:libpmem2 no 2026-02-20T13:24:19.987 INFO:teuthology.orchestra.run.trial093.stdout:PMDK dev-dax engine no 2026-02-20T13:24:19.987 INFO:teuthology.orchestra.run.trial093.stdout:PMDK libpmem engine no 2026-02-20T13:24:19.992 INFO:teuthology.orchestra.run.trial093.stdout:DDN's Infinite Memory Engine no 2026-02-20T13:24:19.992 INFO:teuthology.orchestra.run.trial093.stdout:iscsi engine no 2026-02-20T13:24:19.992 INFO:teuthology.orchestra.run.trial093.stdout:NBD engine no 2026-02-20T13:24:19.998 INFO:teuthology.orchestra.run.trial093.stdout:DAOS File System (dfs) Engine no 2026-02-20T13:24:20.005 INFO:teuthology.orchestra.run.trial093.stdout:NFS engine no 2026-02-20T13:24:20.005 INFO:teuthology.orchestra.run.trial093.stdout:lex/yacc for arithmetic no 2026-02-20T13:24:20.025 INFO:teuthology.orchestra.run.trial093.stdout:getmntent yes 2026-02-20T13:24:20.037 INFO:teuthology.orchestra.run.trial093.stdout:getmntinfo no 2026-02-20T13:24:20.067 INFO:teuthology.orchestra.run.trial093.stdout:Static Assert yes 2026-02-20T13:24:20.086 INFO:teuthology.orchestra.run.trial093.stdout:bool yes 2026-02-20T13:24:20.108 INFO:teuthology.orchestra.run.trial093.stdout:strndup yes 2026-02-20T13:24:20.113 INFO:teuthology.orchestra.run.trial093.stdout:Valgrind headers no 2026-02-20T13:24:20.132 INFO:teuthology.orchestra.run.trial093.stdout:Zoned block device support yes 2026-02-20T13:24:20.152 INFO:teuthology.orchestra.run.trial093.stdout:Zoned block device capacity yes 2026-02-20T13:24:20.157 INFO:teuthology.orchestra.run.trial093.stdout:libzbc engine no 2026-02-20T13:24:20.177 INFO:teuthology.orchestra.run.trial093.stdout:NVMe uring command support yes 2026-02-20T13:24:20.188 INFO:teuthology.orchestra.run.trial093.stdout:xnvme engine no 2026-02-20T13:24:20.194 INFO:teuthology.orchestra.run.trial093.stdout:isal no 2026-02-20T13:24:20.195 INFO:teuthology.orchestra.run.trial093.stdout:isal CRC64 no 2026-02-20T13:24:20.206 INFO:teuthology.orchestra.run.trial093.stdout:libblkio engine no 2026-02-20T13:24:20.206 INFO:teuthology.orchestra.run.trial093.stdout:march_armv8_a_crc_crypto no 2026-02-20T13:24:20.206 INFO:teuthology.orchestra.run.trial093.stdout:cuda no 2026-02-20T13:24:20.206 INFO:teuthology.orchestra.run.trial093.stdout:libcufile no 2026-02-20T13:24:20.227 INFO:teuthology.orchestra.run.trial093.stdout:Build march=native yes 2026-02-20T13:24:20.232 INFO:teuthology.orchestra.run.trial093.stdout:CUnit no 2026-02-20T13:24:20.252 INFO:teuthology.orchestra.run.trial093.stdout:__kernel_rwf_t yes 2026-02-20T13:24:20.272 INFO:teuthology.orchestra.run.trial093.stdout:-Wimplicit-fallthrough=2 yes 2026-02-20T13:24:20.292 INFO:teuthology.orchestra.run.trial093.stdout:-Wno-stringop-truncation yes 2026-02-20T13:24:20.312 INFO:teuthology.orchestra.run.trial093.stdout:MADV_HUGEPAGE yes 2026-02-20T13:24:20.333 INFO:teuthology.orchestra.run.trial093.stdout:gettid yes 2026-02-20T13:24:20.356 INFO:teuthology.orchestra.run.trial093.stdout:statx(2)/libc yes 2026-02-20T13:24:20.380 INFO:teuthology.orchestra.run.trial093.stdout:statx(2)/syscall yes 2026-02-20T13:24:20.382 INFO:teuthology.orchestra.run.trial093.stdout:Windows PDB generation no 2026-02-20T13:24:20.402 INFO:teuthology.orchestra.run.trial093.stdout:timerfd_create yes 2026-02-20T13:24:20.403 INFO:teuthology.orchestra.run.trial093.stdout:Consider installing zlib1g-dev (zlib-devel) as some fio features depend on it. 2026-02-20T13:24:20.403 INFO:teuthology.orchestra.run.trial093.stdout:Lib-based ioengines dynamic no 2026-02-20T13:24:20.445 INFO:teuthology.orchestra.run.trial093.stdout:TCMalloc support yes 2026-02-20T13:24:20.446 INFO:teuthology.orchestra.run.trial093.stdout:seed_buckets 4 2026-02-20T13:24:20.491 INFO:teuthology.orchestra.run.trial093.stderr:FIO_VERSION = fio-3.41-110-g1a04e5 2026-02-20T13:24:20.515 INFO:teuthology.orchestra.run.trial093.stdout: CC crc/crc16.o 2026-02-20T13:24:20.550 INFO:teuthology.orchestra.run.trial093.stdout: CC crc/crc32.o 2026-02-20T13:24:20.577 INFO:teuthology.orchestra.run.trial093.stdout: CC crc/crc32c-arm64.o 2026-02-20T13:24:20.621 INFO:teuthology.orchestra.run.trial093.stdout: CC crc/crc32c-intel.o 2026-02-20T13:24:20.652 INFO:teuthology.orchestra.run.trial093.stdout: CC crc/crc32c.o 2026-02-20T13:24:20.680 INFO:teuthology.orchestra.run.trial093.stdout: CC crc/crc64.o 2026-02-20T13:24:20.709 INFO:teuthology.orchestra.run.trial093.stdout: CC crc/crc7.o 2026-02-20T13:24:20.733 INFO:teuthology.orchestra.run.trial093.stdout: CC crc/crct10dif_common.o 2026-02-20T13:24:20.758 INFO:teuthology.orchestra.run.trial093.stdout: CC crc/fnv.o 2026-02-20T13:24:20.787 INFO:teuthology.orchestra.run.trial093.stdout: CC crc/md5.o 2026-02-20T13:24:20.843 INFO:teuthology.orchestra.run.trial093.stdout: CC crc/murmur3.o 2026-02-20T13:24:20.871 INFO:teuthology.orchestra.run.trial093.stdout: CC crc/sha1.o 2026-02-20T13:24:20.967 INFO:teuthology.orchestra.run.trial093.stdout: CC crc/sha256.o 2026-02-20T13:24:21.105 INFO:teuthology.orchestra.run.trial093.stdout: CC crc/sha3.o 2026-02-20T13:24:21.185 INFO:teuthology.orchestra.run.trial093.stdout: CC crc/sha512.o 2026-02-20T13:24:21.252 INFO:teuthology.orchestra.run.trial093.stdout: CC crc/test.o 2026-02-20T13:24:21.344 INFO:teuthology.orchestra.run.trial093.stdout: CC crc/xxhash.o 2026-02-20T13:24:21.403 INFO:teuthology.orchestra.run.trial093.stdout: CC lib/axmap.o 2026-02-20T13:24:21.475 INFO:teuthology.orchestra.run.trial093.stdout: CC lib/bloom.o 2026-02-20T13:24:21.526 INFO:teuthology.orchestra.run.trial093.stdout: CC lib/flist_sort.o 2026-02-20T13:24:21.571 INFO:teuthology.orchestra.run.trial093.stdout: CC lib/gauss.o 2026-02-20T13:24:21.622 INFO:teuthology.orchestra.run.trial093.stdout: CC lib/getrusage.o 2026-02-20T13:24:21.648 INFO:teuthology.orchestra.run.trial093.stdout: CC lib/hweight.o 2026-02-20T13:24:21.676 INFO:teuthology.orchestra.run.trial093.stdout: CC lib/ieee754.o 2026-02-20T13:24:21.709 INFO:teuthology.orchestra.run.trial093.stdout: CC lib/lfsr.o 2026-02-20T13:24:21.768 INFO:teuthology.orchestra.run.trial093.stdout: CC lib/memalign.o 2026-02-20T13:24:21.800 INFO:teuthology.orchestra.run.trial093.stdout: CC lib/memcpy.o 2026-02-20T13:24:21.883 INFO:teuthology.orchestra.run.trial093.stdout: CC lib/mountcheck.o 2026-02-20T13:24:21.913 INFO:teuthology.orchestra.run.trial093.stdout: CC lib/num2str.o 2026-02-20T13:24:21.957 INFO:teuthology.orchestra.run.trial093.stdout: CC lib/output_buffer.o 2026-02-20T13:24:21.990 INFO:teuthology.orchestra.run.trial093.stdout: CC lib/pattern.o 2026-02-20T13:24:22.068 INFO:teuthology.orchestra.run.trial093.stdout: CC lib/prio_tree.o 2026-02-20T13:24:22.137 INFO:teuthology.orchestra.run.trial093.stdout: CC lib/rand.o 2026-02-20T13:24:22.214 INFO:teuthology.orchestra.run.trial093.stdout: CC lib/rbtree.o 2026-02-20T13:24:22.285 INFO:teuthology.orchestra.run.trial093.stdout: CC lib/strntol.o 2026-02-20T13:24:22.319 INFO:teuthology.orchestra.run.trial093.stdout: CC lib/zipf.o 2026-02-20T13:24:22.383 INFO:teuthology.orchestra.run.trial093.stdout: CC gettime.o 2026-02-20T13:24:22.537 INFO:teuthology.orchestra.run.trial093.stdout: CC ioengines.o 2026-02-20T13:24:22.676 INFO:teuthology.orchestra.run.trial093.stdout: CC init.o 2026-02-20T13:24:23.053 INFO:teuthology.orchestra.run.trial093.stdout: CC stat.o 2026-02-20T13:24:23.755 INFO:teuthology.orchestra.run.trial093.stdout: CC log.o 2026-02-20T13:24:23.827 INFO:teuthology.orchestra.run.trial093.stdout: CC time.o 2026-02-20T13:24:23.917 INFO:teuthology.orchestra.run.trial093.stdout: CC filesetup.o 2026-02-20T13:24:24.255 INFO:teuthology.orchestra.run.trial093.stdout: CC eta.o 2026-02-20T13:24:24.417 INFO:teuthology.orchestra.run.trial093.stdout: CC verify.o 2026-02-20T13:24:24.754 INFO:teuthology.orchestra.run.trial093.stdout: CC memory.o 2026-02-20T13:24:24.845 INFO:teuthology.orchestra.run.trial093.stdout: CC io_u.o 2026-02-20T13:24:25.228 INFO:teuthology.orchestra.run.trial093.stdout: CC parse.o 2026-02-20T13:24:25.450 INFO:teuthology.orchestra.run.trial093.stdout: CC fio_sem.o 2026-02-20T13:24:25.513 INFO:teuthology.orchestra.run.trial093.stdout: CC rwlock.o 2026-02-20T13:24:25.565 INFO:teuthology.orchestra.run.trial093.stdout: CC pshared.o 2026-02-20T13:24:25.608 INFO:teuthology.orchestra.run.trial093.stdout: CC options.o 2026-02-20T13:24:26.014 INFO:teuthology.orchestra.run.trial093.stdout: CC fio_shared_sem.o 2026-02-20T13:24:26.047 INFO:teuthology.orchestra.run.trial093.stdout: CC smalloc.o 2026-02-20T13:24:26.153 INFO:teuthology.orchestra.run.trial093.stdout: CC filehash.o 2026-02-20T13:24:26.240 INFO:teuthology.orchestra.run.trial093.stdout: CC profile.o 2026-02-20T13:24:26.311 INFO:teuthology.orchestra.run.trial093.stdout: CC debug.o 2026-02-20T13:24:26.341 INFO:teuthology.orchestra.run.trial093.stdout: CC engines/cpu.o 2026-02-20T13:24:26.417 INFO:teuthology.orchestra.run.trial093.stdout: CC engines/mmap.o 2026-02-20T13:24:26.509 INFO:teuthology.orchestra.run.trial093.stdout: CC engines/sync.o 2026-02-20T13:24:26.629 INFO:teuthology.orchestra.run.trial093.stdout: CC engines/null.o 2026-02-20T13:24:26.698 INFO:teuthology.orchestra.run.trial093.stdout: CC engines/net.o 2026-02-20T13:24:26.908 INFO:teuthology.orchestra.run.trial093.stdout: CC engines/ftruncate.o 2026-02-20T13:24:26.968 INFO:teuthology.orchestra.run.trial093.stdout: CC engines/fileoperations.o 2026-02-20T13:24:27.054 INFO:teuthology.orchestra.run.trial093.stdout: CC engines/exec.o 2026-02-20T13:24:27.137 INFO:teuthology.orchestra.run.trial093.stdout: CC server.o 2026-02-20T13:24:27.495 INFO:teuthology.orchestra.run.trial093.stdout: CC client.o 2026-02-20T13:24:27.778 INFO:teuthology.orchestra.run.trial093.stdout: CC iolog.o 2026-02-20T13:24:28.007 INFO:teuthology.orchestra.run.trial093.stdout: CC backend.o 2026-02-20T13:24:28.422 INFO:teuthology.orchestra.run.trial093.stdout: CC libfio.o 2026-02-20T13:24:28.521 INFO:teuthology.orchestra.run.trial093.stdout: CC flow.o 2026-02-20T13:24:28.591 INFO:teuthology.orchestra.run.trial093.stdout: CC cconv.o 2026-02-20T13:24:28.809 INFO:teuthology.orchestra.run.trial093.stdout: CC gettime-thread.o 2026-02-20T13:24:28.872 INFO:teuthology.orchestra.run.trial093.stdout: CC helpers.o 2026-02-20T13:24:28.911 INFO:teuthology.orchestra.run.trial093.stdout: CC json.o 2026-02-20T13:24:29.009 INFO:teuthology.orchestra.run.trial093.stdout: CC idletime.o 2026-02-20T13:24:29.126 INFO:teuthology.orchestra.run.trial093.stdout: CC td_error.o 2026-02-20T13:24:29.188 INFO:teuthology.orchestra.run.trial093.stdout: CC profiles/tiobench.o 2026-02-20T13:24:29.247 INFO:teuthology.orchestra.run.trial093.stdout: CC profiles/act.o 2026-02-20T13:24:29.336 INFO:teuthology.orchestra.run.trial093.stdout: CC io_u_queue.o 2026-02-20T13:24:29.370 INFO:teuthology.orchestra.run.trial093.stdout: CC filelock.o 2026-02-20T13:24:29.443 INFO:teuthology.orchestra.run.trial093.stdout: CC workqueue.o 2026-02-20T13:24:29.541 INFO:teuthology.orchestra.run.trial093.stdout: CC rate-submit.o 2026-02-20T13:24:29.625 INFO:teuthology.orchestra.run.trial093.stdout: CC optgroup.o 2026-02-20T13:24:29.656 INFO:teuthology.orchestra.run.trial093.stdout: CC helper_thread.o 2026-02-20T13:24:29.755 INFO:teuthology.orchestra.run.trial093.stdout: CC steadystate.o 2026-02-20T13:24:29.900 INFO:teuthology.orchestra.run.trial093.stdout: CC zone-dist.o 2026-02-20T13:24:29.978 INFO:teuthology.orchestra.run.trial093.stdout: CC zbd.o 2026-02-20T13:24:30.346 INFO:teuthology.orchestra.run.trial093.stdout: CC dedupe.o 2026-02-20T13:24:30.426 INFO:teuthology.orchestra.run.trial093.stdout: CC dataplacement.o 2026-02-20T13:24:30.520 INFO:teuthology.orchestra.run.trial093.stdout: CC sprandom.o 2026-02-20T13:24:30.680 INFO:teuthology.orchestra.run.trial093.stdout: CC engines/posixaio.o 2026-02-20T13:24:30.755 INFO:teuthology.orchestra.run.trial093.stdout: CC engines/falloc.o 2026-02-20T13:24:30.823 INFO:teuthology.orchestra.run.trial093.stdout: CC engines/e4defrag.o 2026-02-20T13:24:30.900 INFO:teuthology.orchestra.run.trial093.stdout: CC engines/splice.o 2026-02-20T13:24:30.998 INFO:teuthology.orchestra.run.trial093.stdout: CC oslib/asprintf.o 2026-02-20T13:24:31.025 INFO:teuthology.orchestra.run.trial093.stdout: CC oslib/strlcat.o 2026-02-20T13:24:31.056 INFO:teuthology.orchestra.run.trial093.stdout: CC engines/mtd.o 2026-02-20T13:24:31.147 INFO:teuthology.orchestra.run.trial093.stdout: CC oslib/libmtd.o 2026-02-20T13:24:31.337 INFO:teuthology.orchestra.run.trial093.stdout: CC oslib/libmtd_legacy.o 2026-02-20T13:24:31.409 INFO:teuthology.orchestra.run.trial093.stdout: CC diskutil.o 2026-02-20T13:24:31.522 INFO:teuthology.orchestra.run.trial093.stdout: CC fifo.o 2026-02-20T13:24:31.559 INFO:teuthology.orchestra.run.trial093.stdout: CC blktrace.o 2026-02-20T13:24:31.724 INFO:teuthology.orchestra.run.trial093.stdout: CC cgroup.o 2026-02-20T13:24:31.807 INFO:teuthology.orchestra.run.trial093.stdout: CC trim.o 2026-02-20T13:24:31.877 INFO:teuthology.orchestra.run.trial093.stdout: CC engines/sg.o 2026-02-20T13:24:32.068 INFO:teuthology.orchestra.run.trial093.stdout: CC oslib/linux-dev-lookup.o 2026-02-20T13:24:32.105 INFO:teuthology.orchestra.run.trial093.stdout: CC engines/io_uring.o 2026-02-20T13:24:32.340 INFO:teuthology.orchestra.run.trial093.stdout: CC engines/nvme.o 2026-02-20T13:24:32.500 INFO:teuthology.orchestra.run.trial093.stdout: CC oslib/linux-blkzoned.o 2026-02-20T13:24:32.589 INFO:teuthology.orchestra.run.trial093.stdout: CC engines/cmdprio.o 2026-02-20T13:24:32.723 INFO:teuthology.orchestra.run.trial093.stdout: CC engines/rados.o 2026-02-20T13:24:32.827 INFO:teuthology.orchestra.run.trial093.stdout: CC engines/rbd.o 2026-02-20T13:24:32.932 INFO:teuthology.orchestra.run.trial093.stdout: CC fio.o 2026-02-20T13:24:32.988 INFO:teuthology.orchestra.run.trial093.stdout: LINK fio 2026-02-20T13:24:33.108 INFO:teuthology.orchestra.run.trial093.stdout: CC t/log.o 2026-02-20T13:24:33.138 INFO:teuthology.orchestra.run.trial093.stdout: CC t/genzipf.o 2026-02-20T13:24:33.214 INFO:teuthology.orchestra.run.trial093.stdout: CC oslib/strcasestr.o 2026-02-20T13:24:33.235 INFO:teuthology.orchestra.run.trial093.stdout: CC oslib/strndup.o 2026-02-20T13:24:33.256 INFO:teuthology.orchestra.run.trial093.stdout: LINK t/fio-genzipf 2026-02-20T13:24:33.310 INFO:teuthology.orchestra.run.trial093.stdout: CC t/btrace2fio.o 2026-02-20T13:24:33.510 INFO:teuthology.orchestra.run.trial093.stdout: LINK t/fio-btrace2fio 2026-02-20T13:24:33.564 INFO:teuthology.orchestra.run.trial093.stdout: CC t/verify-state.o 2026-02-20T13:24:33.624 INFO:teuthology.orchestra.run.trial093.stdout: CC t/debug.o 2026-02-20T13:24:33.650 INFO:teuthology.orchestra.run.trial093.stdout: LINK t/fio-verify-state 2026-02-20T13:24:33.703 INFO:teuthology.orchestra.run.trial093.stdout: CC t/stest.o 2026-02-20T13:24:33.751 INFO:teuthology.orchestra.run.trial093.stdout: CC t/arch.o 2026-02-20T13:24:33.773 INFO:teuthology.orchestra.run.trial093.stdout: LINK t/stest 2026-02-20T13:24:33.828 INFO:teuthology.orchestra.run.trial093.stdout: CC t/ieee754.o 2026-02-20T13:24:33.857 INFO:teuthology.orchestra.run.trial093.stdout: LINK t/ieee754 2026-02-20T13:24:33.910 INFO:teuthology.orchestra.run.trial093.stdout: CC t/axmap.o 2026-02-20T13:24:33.969 INFO:teuthology.orchestra.run.trial093.stdout: LINK t/axmap 2026-02-20T13:24:34.023 INFO:teuthology.orchestra.run.trial093.stdout: CC t/lfsr-test.o 2026-02-20T13:24:34.078 INFO:teuthology.orchestra.run.trial093.stdout: LINK t/lfsr-test 2026-02-20T13:24:34.131 INFO:teuthology.orchestra.run.trial093.stdout: CC t/gen-rand.o 2026-02-20T13:24:34.187 INFO:teuthology.orchestra.run.trial093.stdout: LINK t/gen-rand 2026-02-20T13:24:34.242 INFO:teuthology.orchestra.run.trial093.stdout: CC t/memlock.o 2026-02-20T13:24:34.283 INFO:teuthology.orchestra.run.trial093.stdout: LINK t/memlock 2026-02-20T13:24:34.337 INFO:teuthology.orchestra.run.trial093.stdout: CC t/read-to-pipe-async.o 2026-02-20T13:24:34.439 INFO:teuthology.orchestra.run.trial093.stdout: LINK t/read-to-pipe-async 2026-02-20T13:24:34.492 INFO:teuthology.orchestra.run.trial093.stdout: CC t/io_uring.o 2026-02-20T13:24:34.763 INFO:teuthology.orchestra.run.trial093.stdout: LINK t/io_uring 2026-02-20T13:24:34.819 DEBUG:teuthology.orchestra.run.trial093:> sudo /sbin/sysctl -q -w kernel.perf_event_paranoid=0 2026-02-20T13:24:34.847 DEBUG:teuthology.orchestra.run.trial093:> /home/ubuntu/cephtest/cbt/cbt.py -a /home/ubuntu/cephtest/archive/cbt /home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2026-02-20T13:24:34.958 INFO:teuthology.orchestra.run.trial093.stderr:13:24:34 - DEBUG - cbt - Settings.cluster: 2026-02-20T13:24:34.959 INFO:teuthology.orchestra.run.trial093.stderr: {'archive_dir': '/home/ubuntu/cephtest/archive/cbt', 2026-02-20T13:24:34.959 INFO:teuthology.orchestra.run.trial093.stderr: 'clients': ['ubuntu@trial093.front.sepia.ceph.com'], 2026-02-20T13:24:34.959 INFO:teuthology.orchestra.run.trial093.stderr: 'conf_file': '/etc/ceph/ceph.conf', 2026-02-20T13:24:34.959 INFO:teuthology.orchestra.run.trial093.stderr: 'head': 'ubuntu@trial093.front.sepia.ceph.com', 2026-02-20T13:24:34.959 INFO:teuthology.orchestra.run.trial093.stderr: 'is_teuthology': True, 2026-02-20T13:24:34.959 INFO:teuthology.orchestra.run.trial093.stderr: 'iterations': 1, 2026-02-20T13:24:34.959 INFO:teuthology.orchestra.run.trial093.stderr: 'mons': ['ubuntu@trial093.front.sepia.ceph.com'], 2026-02-20T13:24:34.959 INFO:teuthology.orchestra.run.trial093.stderr: 'osds': ['ubuntu@trial093.front.sepia.ceph.com'], 2026-02-20T13:24:34.959 INFO:teuthology.orchestra.run.trial093.stderr: 'osds_per_node': 3, 2026-02-20T13:24:34.959 INFO:teuthology.orchestra.run.trial093.stderr: 'pid_dir': '/var/run/ceph', 2026-02-20T13:24:34.959 INFO:teuthology.orchestra.run.trial093.stderr: 'pool_profiles': {'rbd': {'pg_size': 128, 'pgp_size': 128, 'replication': 3}}, 2026-02-20T13:24:34.959 INFO:teuthology.orchestra.run.trial093.stderr: 'rebuild_every_test': False, 2026-02-20T13:24:34.959 INFO:teuthology.orchestra.run.trial093.stderr: 'rgws': {'ubuntu@trial093.front.sepia.ceph.com': None}, 2026-02-20T13:24:34.959 INFO:teuthology.orchestra.run.trial093.stderr: 'tmp_dir': '/tmp/cbt', 2026-02-20T13:24:34.959 INFO:teuthology.orchestra.run.trial093.stderr: 'use_existing': True, 2026-02-20T13:24:34.959 INFO:teuthology.orchestra.run.trial093.stderr: 'user': 'ubuntu'} 2026-02-20T13:24:34.960 INFO:teuthology.orchestra.run.trial093.stderr:13:24:34 - INFO - cbt - Results dir: /home/ubuntu/cephtest/archive/cbt/results/00000000/id-7da5bbcd 2026-02-20T13:24:34.960 INFO:teuthology.orchestra.run.trial093.stderr:13:24:34 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial093.front.sepia.ceph.com if [ -d "/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite" ]; then rm -rf /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite; fi 2026-02-20T13:24:35.132 INFO:teuthology.orchestra.run.trial093.stderr:13:24:35 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial093.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite 2026-02-20T13:24:35.298 INFO:teuthology.orchestra.run.trial093.stderr:13:24:35 - INFO - cbt - Pausing for 60s for idle monitoring. 2026-02-20T13:24:35.299 INFO:teuthology.orchestra.run.trial093.stderr:13:24:35 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial093.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite/idle_monitoring/collectl 2026-02-20T13:24:35.461 INFO:teuthology.orchestra.run.trial093.stderr:13:24:35 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial093.front.sepia.ceph.com collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite/idle_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2026-02-20T13:24:35.463 INFO:teuthology.orchestra.run.trial093.stderr:13:24:35 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial093.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite/idle_monitoring/perf 2026-02-20T13:24:35.628 INFO:teuthology.orchestra.run.trial093.stderr:13:24:35 - DEBUG - cbt - PerfMonitoring: remote_node 2026-02-20T13:24:35.628 INFO:teuthology.orchestra.run.trial093.stderr:13:24:35 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial093.front.sepia.ceph.com for pid in `cat /var/run/ceph/crimson-osd.*.pid`; do perf stat -p ${pid} -o /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite/idle_monitoring/perf/perf_stat.${pid} & done 2026-02-20T13:25:35.692 INFO:teuthology.orchestra.run.trial093.stderr:13:25:35 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial093.front.sepia.ceph.com pkill -SIGINT -f collectl 2026-02-20T13:25:35.869 INFO:teuthology.orchestra.run.trial093.stderr:13:25:35 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial093.front.sepia.ceph.com sudo pkill -SIGINT -f perf\ 2026-02-20T13:25:36.061 INFO:teuthology.orchestra.run.trial093.stderr:13:25:36 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial093.front.sepia.ceph.com sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite/* 2026-02-20T13:25:36.232 INFO:teuthology.orchestra.run.trial093.stderr:13:25:36 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=rpdcp -f 10 -R ssh -w ubuntu@trial093.front.sepia.ceph.com -r /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite/* /home/ubuntu/cephtest/archive/cbt/results/00000000/id-7da5bbcd 2026-02-20T13:25:36.398 INFO:teuthology.orchestra.run.trial093.stderr:13:25:36 - INFO - cbt - Creating fio images... 2026-02-20T13:25:36.399 INFO:teuthology.orchestra.run.trial093.stderr:13:25:36 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial093.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite/pool_monitoring/collectl 2026-02-20T13:25:36.562 INFO:teuthology.orchestra.run.trial093.stderr:13:25:36 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial093.front.sepia.ceph.com collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite/pool_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2026-02-20T13:25:36.564 INFO:teuthology.orchestra.run.trial093.stderr:13:25:36 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial093.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite/pool_monitoring/perf 2026-02-20T13:25:36.732 INFO:teuthology.orchestra.run.trial093.stderr:13:25:36 - DEBUG - cbt - PerfMonitoring: remote_node 2026-02-20T13:25:36.733 INFO:teuthology.orchestra.run.trial093.stderr:13:25:36 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial093.front.sepia.ceph.com for pid in `cat /var/run/ceph/crimson-osd.*.pid`; do perf stat -p ${pid} -o /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite/pool_monitoring/perf/perf_stat.${pid} & done 2026-02-20T13:25:36.734 INFO:teuthology.orchestra.run.trial093.stderr:13:25:36 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial093.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool delete cbt-librbdfio cbt-librbdfio --yes-i-really-really-mean-it 2026-02-20T13:25:37.086 INFO:teuthology.orchestra.run.trial093.stderr:13:25:37 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial093.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool create cbt-librbdfio 128 128 2026-02-20T13:25:38.132 INFO:teuthology.orchestra.run.trial093.stderr:13:25:38 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial093.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool application enable cbt-librbdfio rbd 2026-02-20T13:25:39.137 INFO:teuthology.orchestra.run.trial093.stderr:13:25:39 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial093.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool set cbt-librbdfio size 3 2026-02-20T13:25:40.140 INFO:teuthology.orchestra.run.trial093.stderr:13:25:40 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial093.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool set cbt-librbdfio min_size 2 2026-02-20T13:25:41.147 INFO:teuthology.orchestra.run.trial093.stderr:13:25:41 - INFO - cbt - Checking Health after pool creation. 2026-02-20T13:25:46.153 INFO:teuthology.orchestra.run.trial093.stderr:13:25:46 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial093.front.sepia.ceph.com /usr/bin/ceph -c /etc/ceph/ceph.conf health 2026-02-20T13:25:46.516 INFO:teuthology.orchestra.run.trial093.stderr:13:25:46 - INFO - cbt - Final Pool Health Check. 2026-02-20T13:25:51.521 INFO:teuthology.orchestra.run.trial093.stderr:13:25:51 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial093.front.sepia.ceph.com /usr/bin/ceph -c /etc/ceph/ceph.conf health 2026-02-20T13:25:51.887 INFO:teuthology.orchestra.run.trial093.stderr:13:25:51 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial093.front.sepia.ceph.com hostname -f 2026-02-20T13:25:52.055 INFO:teuthology.orchestra.run.trial093.stderr:13:25:52 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial093.front.sepia.ceph.com /usr/bin/rbd -c /etc/ceph/ceph.conf create cbt-librbdfio-trial093-0 --size 4096 --pool cbt-librbdfio --order 22 2026-02-20T13:25:54.272 INFO:teuthology.orchestra.run.trial093.stderr:13:25:54 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial093.front.sepia.ceph.com /usr/bin/rbd -c /etc/ceph/ceph.conf create cbt-librbdfio-trial093-1 --size 4096 --pool cbt-librbdfio --order 22 2026-02-20T13:25:54.523 INFO:teuthology.orchestra.run.trial093.stderr:13:25:54 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial093.front.sepia.ceph.com pkill -SIGINT -f collectl 2026-02-20T13:25:54.696 INFO:teuthology.orchestra.run.trial093.stderr:13:25:54 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial093.front.sepia.ceph.com sudo pkill -SIGINT -f perf\ 2026-02-20T13:25:54.878 INFO:teuthology.orchestra.run.trial093.stderr:13:25:54 - INFO - cbt - Attempting to prefill fio images... 2026-02-20T13:25:54.878 INFO:teuthology.orchestra.run.trial093.stderr:13:25:54 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial093.front.sepia.ceph.com sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-0 --invalidate=0 --rw=write --numjobs=1 --bs=4M --size 4096M --name=cbt-librbdfio-`hostname -f`-file-0 --output-format=json,normal > /dev/null 2026-02-20T13:25:54.879 INFO:teuthology.orchestra.run.trial093.stderr:13:25:54 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial093.front.sepia.ceph.com sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-1 --invalidate=0 --rw=write --numjobs=1 --bs=4M --size 4096M --name=cbt-librbdfio-`hostname -f`-file-0 --output-format=json,normal > /dev/null 2026-02-20T13:26:41.928 INFO:teuthology.orchestra.run.trial093.stderr:13:26:41 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial093.front.sepia.ceph.com sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-0 --invalidate=0 --rw=write --numjobs=1 --bs=4M --size 4096M --name=cbt-librbdfio-`hostname -f`-file-0 --output-format=json,normal > /dev/null 2026-02-20T13:26:41.929 INFO:teuthology.orchestra.run.trial093.stderr:13:26:41 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial093.front.sepia.ceph.com sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-1 --invalidate=0 --rw=write --numjobs=1 --bs=4M --size 4096M --name=cbt-librbdfio-`hostname -f`-file-0 --output-format=json,normal > /dev/null 2026-02-20T13:27:04.981 INFO:teuthology.orchestra.run.trial093.stderr:13:27:04 - INFO - cbt - Results dir: /home/ubuntu/cephtest/archive/cbt/results/00000000/id-7da5bbcd 2026-02-20T13:27:04.982 INFO:teuthology.orchestra.run.trial093.stderr:13:27:04 - INFO - cbt - Skipping existing test in /home/ubuntu/cephtest/archive/cbt/results/00000000/id-7da5bbcd. 2026-02-20T13:27:04.982 INFO:teuthology.orchestra.run.trial093.stderr:13:27:04 - INFO - cbt - Running benchmark /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite 2026-02-20T13:27:04.982 INFO:teuthology.orchestra.run.trial093.stderr:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-7da5bbcd 2026-02-20T13:27:04.982 INFO:teuthology.orchestra.run.trial093.stderr:{'cmd_path': '/home/ubuntu/cephtest/fio/fio', 'log_avg_msec': 100, 'norandommap': True, 'pool_profile': 'rbd', 'time': 60, 'vol_size': 4096, 'iodepth': 32, 'mode': 'randwrite', 'op_size': 4096, 'osd_ra': 4096, 'procs_per_volume': 1, 'volumes_per_client': 2, 'benchmark': 'librbdfio', 'iteration': 0} == iteration 0 == 2026-02-20T13:27:04.982 INFO:teuthology.orchestra.run.trial093.stderr:13:27:04 - INFO - cbt - Setting OSD Read Ahead to: 4096 2026-02-20T13:27:04.982 INFO:teuthology.orchestra.run.trial093.stderr:13:27:04 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial093.front.sepia.ceph.com find /dev/disk/by-partlabel/osd-device-*data -exec readlink {} \; | cut -d"/" -f 3 | sed "s/[0-9]$//" | xargs -I{} sudo sh -c "echo 4096 > /sys/block/'{}'/queue/read_ahead_kb" 2026-02-20T13:27:04.983 INFO:teuthology.orchestra.run.trial093.stderr:13:27:04 - DEBUG - cbt - Cleaning existing temporary run directory: /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite 2026-02-20T13:27:04.983 INFO:teuthology.orchestra.run.trial093.stderr:13:27:04 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial093.front.sepia.ceph.com sudo rm -rf /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite 2026-02-20T13:27:05.166 INFO:teuthology.orchestra.run.trial093.stderr:13:27:05 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial093.front.sepia.ceph.com sync 2026-02-20T13:27:05.474 INFO:teuthology.orchestra.run.trial093.stderr:13:27:05 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial093.front.sepia.ceph.com echo 3 | sudo tee /proc/sys/vm/drop_caches 2026-02-20T13:27:06.665 INFO:teuthology.orchestra.run.trial093.stderr:13:27:06 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial093.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite 2026-02-20T13:27:06.854 INFO:teuthology.orchestra.run.trial093.stderr:13:27:06 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial093.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf daemon osd.0 config show > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite/ceph_settings.out 2026-02-20T13:27:32.178 INFO:teuthology.orchestra.run.trial093.stderr:13:27:32 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial093.front.sepia.ceph.com echo "[`date`] ceph progress" >> /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite/pgautoscaler.log;/usr/bin/ceph -c /etc/ceph/ceph.conf progress | tee -a /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite/pgautoscaler.log 2026-02-20T13:27:32.557 INFO:teuthology.orchestra.run.trial093.stderr:13:27:32 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial093.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite/collectl 2026-02-20T13:27:32.721 INFO:teuthology.orchestra.run.trial093.stderr:13:27:32 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial093.front.sepia.ceph.com collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2026-02-20T13:27:32.723 INFO:teuthology.orchestra.run.trial093.stderr:13:27:32 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial093.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite/perf 2026-02-20T13:27:32.889 INFO:teuthology.orchestra.run.trial093.stderr:13:27:32 - DEBUG - cbt - PerfMonitoring: remote_node 2026-02-20T13:27:32.889 INFO:teuthology.orchestra.run.trial093.stderr:13:27:32 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial093.front.sepia.ceph.com for pid in `cat /var/run/ceph/crimson-osd.*.pid`; do perf stat -p ${pid} -o /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite/perf/perf_stat.${pid} & done 2026-02-20T13:27:32.891 INFO:teuthology.orchestra.run.trial093.stderr:13:27:32 - INFO - cbt - Running rbd fio randwrite test. 2026-02-20T13:27:32.891 INFO:teuthology.orchestra.run.trial093.stderr:13:27:32 - DEBUG - cbt - Using rbdname cbt-librbdfio-`hostname -f`-0 2026-02-20T13:27:32.891 INFO:teuthology.orchestra.run.trial093.stderr:13:27:32 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial093.front.sepia.ceph.com sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-0 --invalidate=0 --rw=randwrite --output-format=json,normal --runtime=60 --numjobs=1 --direct=1 --bs=4096B --iodepth=32 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite/output.0 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite/output.0 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite/output.0 --log_avg_msec=100 --name=cbt-librbdfio-`hostname -f`-file-0 > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite/output.0 2026-02-20T13:27:32.892 INFO:teuthology.orchestra.run.trial093.stderr:13:27:32 - DEBUG - cbt - Using rbdname cbt-librbdfio-`hostname -f`-1 2026-02-20T13:27:32.893 INFO:teuthology.orchestra.run.trial093.stderr:13:27:32 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial093.front.sepia.ceph.com sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-1 --invalidate=0 --rw=randwrite --output-format=json,normal --runtime=60 --numjobs=1 --direct=1 --bs=4096B --iodepth=32 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite/output.1 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite/output.1 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite/output.1 --log_avg_msec=100 --name=cbt-librbdfio-`hostname -f`-file-0 > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite/output.1 2026-02-20T13:28:33.633 INFO:teuthology.orchestra.run.trial093.stderr:13:28:33 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial093.front.sepia.ceph.com pkill -SIGINT -f collectl 2026-02-20T13:28:33.821 INFO:teuthology.orchestra.run.trial093.stderr:13:28:33 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial093.front.sepia.ceph.com sudo pkill -SIGINT -f perf\ 2026-02-20T13:28:34.006 INFO:teuthology.orchestra.run.trial093.stderr:13:28:34 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial093.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite/perf/perf.data 2026-02-20T13:28:34.007 INFO:teuthology.orchestra.run.trial093.stderr:13:28:34 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial093.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite/perf/perf_stat.* 2026-02-20T13:28:34.009 INFO:teuthology.orchestra.run.trial093.stderr:13:28:34 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial093.front.sepia.ceph.com find /var/run/ceph/ceph-osd*.asok -maxdepth 1 -exec sudo /usr/bin/ceph --admin-daemon {} dump_historic_ops \; > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite/historic_ops.out 2026-02-20T13:28:34.362 INFO:teuthology.orchestra.run.trial093.stderr:13:28:34 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial093.front.sepia.ceph.com sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite/* 2026-02-20T13:28:34.537 INFO:teuthology.orchestra.run.trial093.stderr:13:28:34 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=rpdcp -f 10 -R ssh -w ubuntu@trial093.front.sepia.ceph.com -r /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite/* /home/ubuntu/cephtest/archive/cbt/results/00000000/id-7da5bbcd 2026-02-20T13:28:34.710 INFO:teuthology.orchestra.run.trial093.stderr:13:28:34 - INFO - cbt - Convert results to json format. 2026-02-20T13:28:34.710 INFO:teuthology.orchestra.run.trial093.stdout:cleaning remote dir /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite 2026-02-20T13:28:34.711 INFO:teuthology.orchestra.run.trial093.stdout:trial093: trial093 2026-02-20T13:28:34.711 INFO:teuthology.orchestra.run.trial093.stdout: 2026-02-20T13:28:34.711 INFO:teuthology.orchestra.run.trial093.stdout:['trial093'] 2026-02-20T13:28:34.721 DEBUG:teuthology.run_tasks:Unwinding manager cbt 2026-02-20T13:28:34.735 DEBUG:teuthology.orchestra.run.trial093:> rm --one-file-system -rf -- /home/ubuntu/cephtest/cbt 2026-02-20T13:28:34.761 DEBUG:teuthology.orchestra.run.trial093:> rm --one-file-system -rf -- /home/ubuntu/cephtest/fio 2026-02-20T13:28:34.840 INFO:tasks.cbt_performance:Collecting CBT performance data config 2026-02-20T13:28:34.840 INFO:tasks.cbt_performance:reading cbt preferences 2026-02-20T13:28:34.841 DEBUG:teuthology.orchestra.run.trial093:> find /home/ubuntu/cephtest -name 'json_output.*' 2026-02-20T13:28:34.899 INFO:teuthology.orchestra.run.trial093.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-7da5bbcd/json_output.1.trial093.front.sepia.ceph.com 2026-02-20T13:28:34.899 INFO:teuthology.orchestra.run.trial093.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-7da5bbcd/json_output.0.trial093.front.sepia.ceph.com 2026-02-20T13:28:34.900 DEBUG:teuthology.orchestra.run.trial093:> set -ex 2026-02-20T13:28:34.900 DEBUG:teuthology.orchestra.run.trial093:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-7da5bbcd/json_output.1.trial093.front.sepia.ceph.com of=/dev/stdout 2026-02-20T13:28:34.959 INFO:tasks.cbt_performance:reading total cpu cycles 2026-02-20T13:28:34.959 DEBUG:teuthology.orchestra.run.trial093:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-7da5bbcd -name 'perf_stat.*' 2026-02-20T13:28:35.014 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2026-02-20T13:28:35.014 INFO:tasks.cbt_performance:total cpu cycles: 0 2026-02-20T13:28:35.014 DEBUG:teuthology.orchestra.run.trial093:> set -ex 2026-02-20T13:28:35.014 DEBUG:teuthology.orchestra.run.trial093:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-7da5bbcd/json_output.0.trial093.front.sepia.ceph.com of=/dev/stdout 2026-02-20T13:28:35.073 INFO:tasks.cbt_performance:reading total cpu cycles 2026-02-20T13:28:35.074 DEBUG:teuthology.orchestra.run.trial093:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-7da5bbcd -name 'perf_stat.*' 2026-02-20T13:28:35.130 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2026-02-20T13:28:35.130 INFO:tasks.cbt_performance:total cpu cycles: 0 2026-02-20T13:28:35.136 WARNING:tasks.cbt_performance:Could not send performance data to http://mira118.front.sepia.ceph.com:4000/cbt_performance: HTTPConnectionPool(host='mira118.front.sepia.ceph.com', port=4000): Max retries exceeded with url: /cbt_performance (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -2] Name or service not known')) 2026-02-20T13:28:35.136 WARNING:tasks.cbt_performance:Continuing without performance data collection 2026-02-20T13:28:35.137 DEBUG:teuthology.run_tasks:Unwinding manager ssh_keys 2026-02-20T13:28:35.148 INFO:teuthology.task.ssh_keys:Cleaning up SSH keys 2026-02-20T13:28:35.148 INFO:teuthology.task.ssh_keys:cleaning up keys added for testing 2026-02-20T13:28:35.148 INFO:teuthology.task.ssh_keys: cleaning up keys for user ubuntu on trial093.front.sepia.ceph.com 2026-02-20T13:28:35.148 DEBUG:teuthology.orchestra.run.trial093:> rm -- /home/ubuntu/.ssh/id_rsa 2026-02-20T13:28:35.187 DEBUG:teuthology.orchestra.run.trial093:> rm -- /home/ubuntu/.ssh/id_rsa.pub 2026-02-20T13:28:35.242 DEBUG:teuthology.orchestra.run.trial093:> stat -c '"%a"' /home/ubuntu/.ssh/authorized_keys 2026-02-20T13:28:35.299 INFO:teuthology.orchestra.run.trial093.stdout:"600" 2026-02-20T13:28:35.300 DEBUG:teuthology.orchestra.run.trial093:> mv -- /home/ubuntu/.ssh/authorized_keys_2026-02-20_13:03:38:329396 /home/ubuntu/.ssh/authorized_keys 2026-02-20T13:28:35.355 DEBUG:teuthology.orchestra.run.trial093:> chmod 600 /home/ubuntu/.ssh/authorized_keys 2026-02-20T13:28:35.411 DEBUG:teuthology.orchestra.run.trial093:> rm /home/ubuntu/.ssh/config 2026-02-20T13:28:35.467 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2026-02-20T13:28:35.483 DEBUG:teuthology.orchestra.run.trial093:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2026-02-20T13:28:35.582 INFO:teuthology.orchestra.run.trial093.stderr:2026-02-20T13:28:35.576+0000 7ff24725a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T13:28:35.583 INFO:teuthology.orchestra.run.trial093.stderr:2026-02-20T13:28:35.581+0000 7ff24725a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T13:28:35.706 INFO:teuthology.misc:Shutting down mds daemons... 2026-02-20T13:28:35.707 INFO:teuthology.misc:Shutting down osd daemons... 2026-02-20T13:28:35.707 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2026-02-20T13:28:35.707 INFO:teuthology.orchestra.run:waiting for 300 2026-02-20T13:28:35.801 INFO:tasks.ceph.osd.0:Stopped 2026-02-20T13:28:35.802 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2026-02-20T13:28:35.802 INFO:teuthology.orchestra.run:waiting for 300 2026-02-20T13:28:35.894 INFO:tasks.ceph.osd.1:Stopped 2026-02-20T13:28:35.894 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2026-02-20T13:28:35.894 INFO:teuthology.orchestra.run:waiting for 300 2026-02-20T13:28:35.988 INFO:tasks.ceph.osd.2:Stopped 2026-02-20T13:28:35.989 INFO:teuthology.misc:Shutting down mgr daemons... 2026-02-20T13:28:35.989 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2026-02-20T13:28:35.989 INFO:teuthology.orchestra.run:waiting for 300 2026-02-20T13:28:36.025 INFO:tasks.ceph.mgr.x:Stopped 2026-02-20T13:28:36.025 INFO:teuthology.misc:Shutting down mon daemons... 2026-02-20T13:28:36.025 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2026-02-20T13:28:36.026 INFO:teuthology.orchestra.run:waiting for 300 2026-02-20T13:28:36.074 INFO:tasks.ceph.mon.a:Stopped 2026-02-20T13:28:36.075 INFO:tasks.ceph:Checking cluster log for badness... 2026-02-20T13:28:36.075 DEBUG:teuthology.orchestra.run.trial093:> sudo grep -E '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | grep -E -v '\(MDS_ALL_DOWN\)' | grep -E -v '\(MDS_UP_LESS_THAN_MAX\)' | grep -E -v '\(POOL_APP_NOT_ENABLED\)' | grep -E -v '\(PG_' | grep -E -v '\(OSD_' | grep -E -v '\(OBJECT_' | grep -E -v 'overall HEALTH' | head -n 1 2026-02-20T13:28:36.141 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@trial093.front.sepia.ceph.com 2026-02-20T13:28:36.141 DEBUG:teuthology.orchestra.run.trial093:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2026-02-20T13:28:36.977 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@trial093.front.sepia.ceph.com 2026-02-20T13:28:36.977 DEBUG:teuthology.orchestra.run.trial093:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2026-02-20T13:28:37.012 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@trial093.front.sepia.ceph.com 2026-02-20T13:28:37.012 DEBUG:teuthology.orchestra.run.trial093:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2026-02-20T13:28:37.082 INFO:tasks.ceph:Archiving mon data... 2026-02-20T13:28:37.083 DEBUG:teuthology.misc:Transferring archived files from trial093:/var/lib/ceph/mon/ceph-a to /home/teuthworker/mnt/teuthology/jjperez-2026-02-20_10:29:19-crimson-rados-wip-perezjos-crimson-only-12-02-2026-PR66245-distro-debug-trial/60758/data/mon.a.tgz 2026-02-20T13:28:37.083 DEBUG:teuthology.orchestra.run.trial093:> mktemp 2026-02-20T13:28:37.138 INFO:teuthology.orchestra.run.trial093.stdout:/tmp/tmp.0tW9IfQ8df 2026-02-20T13:28:37.138 DEBUG:teuthology.orchestra.run.trial093:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.0tW9IfQ8df 2026-02-20T13:28:37.229 DEBUG:teuthology.orchestra.run.trial093:> sudo chmod 0666 /tmp/tmp.0tW9IfQ8df 2026-02-20T13:28:37.312 DEBUG:teuthology.orchestra.remote:trial093:/tmp/tmp.0tW9IfQ8df is 192KB 2026-02-20T13:28:37.364 DEBUG:teuthology.orchestra.run.trial093:> rm -fr /tmp/tmp.0tW9IfQ8df 2026-02-20T13:28:37.380 INFO:tasks.ceph:Cleaning ceph cluster... 2026-02-20T13:28:37.380 DEBUG:teuthology.orchestra.run.trial093:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2026-02-20T13:28:37.500 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2026-02-20T13:28:37.501 INFO:tasks.ceph:Archiving crash dumps... 2026-02-20T13:28:37.503 DEBUG:teuthology.misc:Transferring archived files from trial093:/var/lib/ceph/crash to /home/teuthworker/mnt/teuthology/jjperez-2026-02-20_10:29:19-crimson-rados-wip-perezjos-crimson-only-12-02-2026-PR66245-distro-debug-trial/60758/remote/trial093/crash 2026-02-20T13:28:37.504 DEBUG:teuthology.orchestra.run.trial093:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2026-02-20T13:28:37.564 INFO:tasks.ceph:Compressing logs... 2026-02-20T13:28:37.564 DEBUG:teuthology.orchestra.run.trial093:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2026-02-20T13:28:37.632 INFO:teuthology.orchestra.run.trial093.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2026-02-20T13:28:37.632 INFO:teuthology.orchestra.run.trial093.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2026-02-20T13:28:37.632 INFO:teuthology.orchestra.run.trial093.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2026-02-20T13:28:37.632 INFO:teuthology.orchestra.run.trial093.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47231.log 2026-02-20T13:28:37.633 INFO:teuthology.orchestra.run.trial093.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40911.log 2026-02-20T13:28:37.647 INFO:teuthology.orchestra.run.trial093.stderr:/var/log/ceph/ceph-client.admin.47231.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40748.log 2026-02-20T13:28:37.647 INFO:teuthology.orchestra.run.trial093.stderr: 66.9% -- replaced with /var/log/ceph/ceph-client.admin.47231.log.gz 2026-02-20T13:28:37.647 INFO:teuthology.orchestra.run.trial093.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40571.log 2026-02-20T13:28:37.647 INFO:teuthology.orchestra.run.trial093.stderr:/var/log/ceph/ceph-client.admin.40911.log: 92.0% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2026-02-20T13:28:37.647 INFO:teuthology.orchestra.run.trial093.stderr:/var/log/ceph/ceph-client.admin.40748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41123.log 2026-02-20T13:28:37.647 INFO:teuthology.orchestra.run.trial093.stderr: 66.4% -- replaced with /var/log/ceph/ceph-client.admin.40911.log.gz 2026-02-20T13:28:37.648 INFO:teuthology.orchestra.run.trial093.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40748.log.gz 2026-02-20T13:28:37.648 INFO:teuthology.orchestra.run.trial093.stderr:/var/log/ceph/ceph-client.admin.40571.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47456.log 2026-02-20T13:28:37.648 INFO:teuthology.orchestra.run.trial093.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.40571.log.gz 2026-02-20T13:28:37.648 INFO:teuthology.orchestra.run.trial093.stderr:/var/log/ceph/ceph-client.admin.41123.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41074.log 2026-02-20T13:28:37.648 INFO:teuthology.orchestra.run.trial093.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41123.log.gz 2026-02-20T13:28:37.648 INFO:teuthology.orchestra.run.trial093.stderr:/var/log/ceph/ceph-client.admin.47456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46936.log 2026-02-20T13:28:37.648 INFO:teuthology.orchestra.run.trial093.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.47456.log.gz 2026-02-20T13:28:37.648 INFO:teuthology.orchestra.run.trial093.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40816.log 2026-02-20T13:28:37.648 INFO:teuthology.orchestra.run.trial093.stderr:/var/log/ceph/ceph-client.admin.41074.log: /var/log/ceph/ceph-client.admin.46936.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41074.log.gz 2026-02-20T13:28:37.648 INFO:teuthology.orchestra.run.trial093.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47279.log 2026-02-20T13:28:37.648 INFO:teuthology.orchestra.run.trial093.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.46936.log.gz 2026-02-20T13:28:37.648 INFO:teuthology.orchestra.run.trial093.stderr:/var/log/ceph/ceph-client.admin.40816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48203.log 2026-02-20T13:28:37.649 INFO:teuthology.orchestra.run.trial093.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40816.log.gz 2026-02-20T13:28:37.649 INFO:teuthology.orchestra.run.trial093.stderr:/var/log/ceph/ceph-client.admin.47279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47090.log 2026-02-20T13:28:37.649 INFO:teuthology.orchestra.run.trial093.stderr:/var/log/ceph/ceph-client.admin.48203.log: 66.7% -- replaced with /var/log/ceph/ceph-client.admin.47279.log.gz 2026-02-20T13:28:37.649 INFO:teuthology.orchestra.run.trial093.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47144.log 2026-02-20T13:28:37.649 INFO:teuthology.orchestra.run.trial093.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48203.log.gz 2026-02-20T13:28:37.649 INFO:teuthology.orchestra.run.trial093.stderr:/var/log/ceph/ceph-client.admin.47090.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47635.log 2026-02-20T13:28:37.649 INFO:teuthology.orchestra.run.trial093.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47090.log.gz 2026-02-20T13:28:37.649 INFO:teuthology.orchestra.run.trial093.stderr:/var/log/ceph/ceph-client.admin.47144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40305.log 2026-02-20T13:28:37.649 INFO:teuthology.orchestra.run.trial093.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47144.log.gz 2026-02-20T13:28:37.649 INFO:teuthology.orchestra.run.trial093.stderr:/var/log/ceph/ceph-client.admin.47635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48204.log 2026-02-20T13:28:37.649 INFO:teuthology.orchestra.run.trial093.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.47635.log.gz 2026-02-20T13:28:37.649 INFO:teuthology.orchestra.run.trial093.stderr:/var/log/ceph/ceph-client.admin.40305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39979.log 2026-02-20T13:28:37.650 INFO:teuthology.orchestra.run.trial093.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40305.log.gz 2026-02-20T13:28:37.650 INFO:teuthology.orchestra.run.trial093.stderr:/var/log/ceph/ceph-client.admin.48204.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41371.log 2026-02-20T13:28:37.650 INFO:teuthology.orchestra.run.trial093.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48204.log.gz 2026-02-20T13:28:37.650 INFO:teuthology.orchestra.run.trial093.stderr:/var/log/ceph/ceph-client.admin.39979.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47054.log 2026-02-20T13:28:37.650 INFO:teuthology.orchestra.run.trial093.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39979.log.gz 2026-02-20T13:28:37.650 INFO:teuthology.orchestra.run.trial093.stderr:/var/log/ceph/ceph-client.admin.41371.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47633.log 2026-02-20T13:28:37.650 INFO:teuthology.orchestra.run.trial093.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41371.log.gz 2026-02-20T13:28:37.650 INFO:teuthology.orchestra.run.trial093.stderr:/var/log/ceph/ceph-client.admin.47054.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41466.log 2026-02-20T13:28:37.650 INFO:teuthology.orchestra.run.trial093.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.47054.log.gz 2026-02-20T13:28:37.650 INFO:teuthology.orchestra.run.trial093.stderr:/var/log/ceph/ceph-client.admin.47633.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2026-02-20T13:28:37.650 INFO:teuthology.orchestra.run.trial093.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.47633.log.gz 2026-02-20T13:28:37.650 INFO:teuthology.orchestra.run.trial093.stderr:/var/log/ceph/ceph-client.admin.41466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47457.log 2026-02-20T13:28:37.651 INFO:teuthology.orchestra.run.trial093.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41466.log.gz 2026-02-20T13:28:37.651 INFO:teuthology.orchestra.run.trial093.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40258.log 2026-02-20T13:28:37.651 INFO:teuthology.orchestra.run.trial093.stderr:/var/log/ceph/ceph-client.admin.47457.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2026-02-20T13:28:37.651 INFO:teuthology.orchestra.run.trial093.stderr: 86.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2026-02-20T13:28:37.651 INFO:teuthology.orchestra.run.trial093.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47457.log.gz 2026-02-20T13:28:37.651 INFO:teuthology.orchestra.run.trial093.stderr:/var/log/ceph/ceph-client.admin.40258.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40601.log 2026-02-20T13:28:37.651 INFO:teuthology.orchestra.run.trial093.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40258.log.gz 2026-02-20T13:28:37.651 INFO:teuthology.orchestra.run.trial093.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48881.log 2026-02-20T13:28:37.651 INFO:teuthology.orchestra.run.trial093.stderr:/var/log/ceph/ceph-osd.1.log: /var/log/ceph/ceph-client.admin.40601.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40352.log 2026-02-20T13:28:37.651 INFO:teuthology.orchestra.run.trial093.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40601.log.gz 2026-02-20T13:28:37.651 INFO:teuthology.orchestra.run.trial093.stderr:/var/log/ceph/ceph-client.admin.48881.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48881.log.gz 2026-02-20T13:28:37.651 INFO:teuthology.orchestra.run.trial093.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41515.log 2026-02-20T13:28:37.652 INFO:teuthology.orchestra.run.trial093.stderr:/var/log/ceph/ceph-client.admin.40352.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46994.log 2026-02-20T13:28:37.652 INFO:teuthology.orchestra.run.trial093.stderr:/var/log/ceph/ceph-client.admin.41515.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40352.log.gz 2026-02-20T13:28:37.652 INFO:teuthology.orchestra.run.trial093.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41224.log 2026-02-20T13:28:37.652 INFO:teuthology.orchestra.run.trial093.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41515.log.gz 2026-02-20T13:28:37.652 INFO:teuthology.orchestra.run.trial093.stderr:/var/log/ceph/ceph-client.admin.46994.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41373.log 2026-02-20T13:28:37.652 INFO:teuthology.orchestra.run.trial093.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46994.log.gz 2026-02-20T13:28:37.652 INFO:teuthology.orchestra.run.trial093.stderr:/var/log/ceph/ceph-client.admin.41224.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46821.log 2026-02-20T13:28:37.652 INFO:teuthology.orchestra.run.trial093.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41224.log.gz 2026-02-20T13:28:37.652 INFO:teuthology.orchestra.run.trial093.stderr:/var/log/ceph/ceph-client.admin.41373.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41372.log 2026-02-20T13:28:37.652 INFO:teuthology.orchestra.run.trial093.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41373.log.gz 2026-02-20T13:28:37.653 INFO:teuthology.orchestra.run.trial093.stderr:/var/log/ceph/ceph-client.admin.46821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40886.log 2026-02-20T13:28:37.653 INFO:teuthology.orchestra.run.trial093.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.46821.log.gz 2026-02-20T13:28:37.653 INFO:teuthology.orchestra.run.trial093.stderr:/var/log/ceph/ceph-client.admin.41372.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41564.log 2026-02-20T13:28:37.653 INFO:teuthology.orchestra.run.trial093.stderr:/var/log/ceph/ceph-client.admin.40886.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41372.log.gz 2026-02-20T13:28:37.653 INFO:teuthology.orchestra.run.trial093.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2026-02-20T13:28:37.653 INFO:teuthology.orchestra.run.trial093.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40886.log.gz 2026-02-20T13:28:37.653 INFO:teuthology.orchestra.run.trial093.stderr:/var/log/ceph/ceph-client.admin.41564.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41226.log 2026-02-20T13:28:37.653 INFO:teuthology.orchestra.run.trial093.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41564.log.gz 2026-02-20T13:28:37.653 INFO:teuthology.orchestra.run.trial093.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40976.log 2026-02-20T13:28:37.653 INFO:teuthology.orchestra.run.trial093.stderr:/var/log/ceph/ceph-client.admin.41226.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2026-02-20T13:28:37.653 INFO:teuthology.orchestra.run.trial093.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41226.log.gz 2026-02-20T13:28:37.654 INFO:teuthology.orchestra.run.trial093.stderr:/var/log/ceph/ceph-client.admin.40976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40121.log 2026-02-20T13:28:37.654 INFO:teuthology.orchestra.run.trial093.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.40976.log.gz 2026-02-20T13:28:37.654 INFO:teuthology.orchestra.run.trial093.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41225.log 2026-02-20T13:28:37.654 INFO:teuthology.orchestra.run.trial093.stderr:/var/log/ceph/ceph-client.admin.40121.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40074.log 2026-02-20T13:28:37.654 INFO:teuthology.orchestra.run.trial093.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40121.log.gz 2026-02-20T13:28:37.654 INFO:teuthology.orchestra.run.trial093.stderr:/var/log/ceph/ceph-client.admin.41225.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.38694.log 2026-02-20T13:28:37.654 INFO:teuthology.orchestra.run.trial093.stderr: 90.3% -- replaced with /var/log/ceph/ceph.log.gz 2026-02-20T13:28:37.654 INFO:teuthology.orchestra.run.trial093.stderr: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.41225.log.gz 2026-02-20T13:28:37.654 INFO:teuthology.orchestra.run.trial093.stderr:/var/log/ceph/ceph-client.admin.40074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47968.log 2026-02-20T13:28:37.654 INFO:teuthology.orchestra.run.trial093.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40074.log.gz 2026-02-20T13:28:37.654 INFO:teuthology.orchestra.run.trial093.stderr:/var/log/ceph/ceph.tmp-client.admin.38694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46877.log 2026-02-20T13:28:37.654 INFO:teuthology.orchestra.run.trial093.stderr:/var/log/ceph/ceph-client.admin.47968.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47968.log.gz 2026-02-20T13:28:37.654 INFO:teuthology.orchestra.run.trial093.stderr: 67.7% -- replaced with /var/log/ceph/ceph.tmp-client.admin.38694.log.gz 2026-02-20T13:28:37.655 INFO:teuthology.orchestra.run.trial093.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41025.log 2026-02-20T13:28:37.655 INFO:teuthology.orchestra.run.trial093.stderr:/var/log/ceph/ceph-client.admin.46877.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46877.log.gz 2026-02-20T13:28:37.655 INFO:teuthology.orchestra.run.trial093.stderr:/var/log/ceph/ceph-client.admin.41025.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41025.log.gz 2026-02-20T13:28:37.677 INFO:teuthology.orchestra.run.trial093.stderr: 93.1% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2026-02-20T13:28:46.639 INFO:teuthology.orchestra.run.trial093.stderr: 91.7% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2026-02-20T13:28:46.693 INFO:teuthology.orchestra.run.trial093.stderr: 91.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2026-02-20T13:28:46.852 INFO:teuthology.orchestra.run.trial093.stderr: 91.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2026-02-20T13:28:46.854 INFO:teuthology.orchestra.run.trial093.stderr: 2026-02-20T13:28:46.854 INFO:teuthology.orchestra.run.trial093.stderr:real 0m9.231s 2026-02-20T13:28:46.854 INFO:teuthology.orchestra.run.trial093.stderr:user 0m25.807s 2026-02-20T13:28:46.854 INFO:teuthology.orchestra.run.trial093.stderr:sys 0m1.487s 2026-02-20T13:28:46.854 INFO:tasks.ceph:Archiving logs... 2026-02-20T13:28:46.855 DEBUG:teuthology.misc:Transferring archived files from trial093:/var/log/ceph to /home/teuthworker/mnt/teuthology/jjperez-2026-02-20_10:29:19-crimson-rados-wip-perezjos-crimson-only-12-02-2026-PR66245-distro-debug-trial/60758/remote/trial093/log 2026-02-20T13:28:46.856 DEBUG:teuthology.orchestra.run.trial093:> sudo tar c -f - -C /var/log/ceph -- . 2026-02-20T13:28:49.203 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-20T13:28:49.285 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-20T13:28:49.296 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2026-02-20T13:28:49.296 DEBUG:teuthology.orchestra.run.trial093:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-20T13:28:49.323 DEBUG:teuthology.run_tasks:Unwinding manager clock 2026-02-20T13:28:49.334 INFO:teuthology.task.clock:Checking final clock skew... 2026-02-20T13:28:49.334 DEBUG:teuthology.orchestra.run.trial093:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-20T13:28:49.378 INFO:teuthology.orchestra.run.trial093.stderr:bash: line 1: ntpq: command not found 2026-02-20T13:28:49.782 INFO:teuthology.orchestra.run.trial093.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-20T13:28:49.783 INFO:teuthology.orchestra.run.trial093.stdout:=============================================================================== 2026-02-20T13:28:49.783 INFO:teuthology.orchestra.run.trial093.stdout:^+ blotch.image1tech.net 2 7 377 123 -703us[ -703us] +/- 22ms 2026-02-20T13:28:49.783 INFO:teuthology.orchestra.run.trial093.stdout:^+ 172-104-209-204.ip.linod> 4 7 377 58 +678us[ +678us] +/- 33ms 2026-02-20T13:28:49.783 INFO:teuthology.orchestra.run.trial093.stdout:^+ kjsl-fmt2-net.fmt2.kjsl.> 2 6 377 58 +53us[ +53us] +/- 33ms 2026-02-20T13:28:49.783 INFO:teuthology.orchestra.run.trial093.stdout:^* 38.45.64.130 2 8 377 190 -264us[ -230us] +/- 19ms 2026-02-20T13:28:49.784 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2026-02-20T13:28:49.801 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2026-02-20T13:28:49.801 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2026-02-20T13:28:49.819 DEBUG:teuthology.orchestra.run.trial093:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2026-02-20T13:28:49.887 DEBUG:teuthology.orchestra.run.trial093:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-20T13:28:49.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-20T13:28:49.914 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2026-02-20T13:28:49.931 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2026-02-20T13:28:49.951 INFO:teuthology.task.internal:Duration was 2067.394590 seconds 2026-02-20T13:28:49.951 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2026-02-20T13:28:49.967 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2026-02-20T13:28:49.968 DEBUG:teuthology.orchestra.run.trial093:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-20T13:28:50.007 INFO:teuthology.orchestra.run.trial093.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-20T13:28:50.292 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2026-02-20T13:28:50.292 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial093.front.sepia.ceph.com 2026-02-20T13:28:50.292 DEBUG:teuthology.orchestra.run.trial093:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-20T13:28:50.351 INFO:teuthology.task.internal.syslog:Gathering journactl... 2026-02-20T13:28:50.351 DEBUG:teuthology.orchestra.run.trial093:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-20T13:28:50.459 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2026-02-20T13:28:50.459 DEBUG:teuthology.orchestra.run.trial093:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-20T13:28:50.484 INFO:teuthology.orchestra.run.trial093.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-20T13:28:50.484 INFO:teuthology.orchestra.run.trial093.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-20T13:28:50.484 INFO:teuthology.orchestra.run.trial093.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-02-20T13:28:50.484 INFO:teuthology.orchestra.run.trial093.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-20T13:28:50.485 INFO:teuthology.orchestra.run.trial093.stderr:/home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-20T13:28:50.494 INFO:teuthology.orchestra.run.trial093.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: 83.1% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-20T13:28:50.496 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2026-02-20T13:28:50.515 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2026-02-20T13:28:50.515 DEBUG:teuthology.orchestra.run.trial093:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-20T13:28:50.561 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2026-02-20T13:28:50.579 DEBUG:teuthology.orchestra.run.trial093:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-20T13:28:50.627 INFO:teuthology.orchestra.run.trial093.stdout:kernel.core_pattern = core 2026-02-20T13:28:50.642 DEBUG:teuthology.orchestra.run.trial093:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-20T13:28:50.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-20T13:28:50.696 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2026-02-20T13:28:50.714 INFO:teuthology.task.internal:Transferring archived files... 2026-02-20T13:28:50.715 DEBUG:teuthology.misc:Transferring archived files from trial093:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/jjperez-2026-02-20_10:29:19-crimson-rados-wip-perezjos-crimson-only-12-02-2026-PR66245-distro-debug-trial/60758/remote/trial093 2026-02-20T13:28:50.715 DEBUG:teuthology.orchestra.run.trial093:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-20T13:28:50.799 INFO:teuthology.task.internal:Removing archive directory... 2026-02-20T13:28:50.799 DEBUG:teuthology.orchestra.run.trial093:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-20T13:28:50.829 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2026-02-20T13:28:50.850 INFO:teuthology.task.internal:Not uploading archives. 2026-02-20T13:28:50.851 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2026-02-20T13:28:50.869 INFO:teuthology.task.internal:Tidying up after the test... 2026-02-20T13:28:50.869 DEBUG:teuthology.orchestra.run.trial093:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-20T13:28:50.885 INFO:teuthology.orchestra.run.trial093.stdout: 917576 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 20 13:28 /home/ubuntu/cephtest 2026-02-20T13:28:50.887 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2026-02-20T13:28:50.905 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2026-02-20T13:28:50.944 INFO:teuthology.run:Summary data: description: crimson-rados/perf/{0-crimson_install clusters/{crimson-fixed-1} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/rbm$/{crimson_seastore_rbm_2q} perf_ignorelist settings/optimized workloads/sample_fio} duration: 2067.394589662552 flavor: debug owner: scheduled_jjperez@soko04.front.sepia.ceph.com success: true 2026-02-20T13:28:50.944 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-20T13:28:51.060 INFO:teuthology.run:pass