2026-02-24T15:35:26.667 INFO:root:teuthology version: 1.2.4.dev9+gfa17720d0 2026-02-24T15:35:26.682 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-24T15:35:26.761 INFO:teuthology.run:Config: archive_path: /home/teuthworker/mnt/teuthology/liucm-2026-02-24_15:28:57-crimson-rados-wip-liucm-shard-redesign-crimson-only-distro-debug-trial/68197 branch: wip-liucm-shard-redesign-crimson-only description: crimson-rados/perf/{0-crimson_install clusters/{crimson-fixed-1} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/segmented$/{crimson_seastore_segmented} perf_ignorelist settings/optimized workloads/radosbench_4M_seq_read} email: null first_in_suite: false flavor: debug job_id: '68197' kernel: branch: distro kdb: true sha1: distro last_in_suite: false machine_type: trial name: liucm-2026-02-24_15:28:57-crimson-rados-wip-liucm-shard-redesign-crimson-only-distro-debug-trial no_nested_subset: false os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-liucm-shard-redesign-crimson-only ceph: conf: global: auth client required: none auth cluster required: none auth service required: none auth supported: none debug asok: 0/0 debug auth: 0/0 debug bdev: 0/0 debug bluefs: 0/0 debug bluestore: 0/0 debug buffer: 0/0 debug civetweb: 0/0 debug client: 0/0 debug compressor: 0/0 debug context: 0/0 debug crush: 0/0 debug crypto: 0/0 debug dpdk: 0/0 debug eventtrace: 0/0 debug filer: 0/0 debug filestore: 0/0 debug finisher: 0/0 debug fuse: 0/0 debug heartbeatmap: 0/0 debug javaclient: 0/0 debug journal: 0/0 debug journaler: 0/0 debug kstore: 0/0 debug leveldb: 0/0 debug lockdep: 0/0 debug mds: 0/0 debug mds balancer: 0/0 debug mds locker: 0/0 debug mds log: 0/0 debug mds log expire: 0/0 debug mds migrator: 0/0 debug memdb: 0/0 debug mgr: 0/0 debug mgrc: 0/0 debug mon: 0/0 debug monc: 0/0 debug ms: 0/0 debug objclass: 0/0 debug objectcacher: 0/0 debug objecter: 0/0 debug optracker: 0/0 debug osd: 0/0 debug paxos: 0/0 debug perfcounter: 0/0 debug rados: 0/0 debug rbd: 0/0 debug rbd mirror: 0/0 debug rbd replay: 0/0 debug refs: 0/0 debug rgw: 0/0 debug rgw lifecycle: 0/0 debug rgw notification: 0/0 debug rgw sync: 0/0 debug rocksdb: 0/0 debug striper: 0/0 debug throttle: 0/0 debug timer: 0/0 debug tp: 0/0 enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 0/0 debug ms: 0/0 debug paxos: 0/0 osd pool default crimson: true osd: crimson cpu num: 4 crimson osd obc lru size: 10 debug filestore: 0/0 debug journal: 0/0 debug ms: 0/0 debug osd: 0/0 debug seastore: 20 debug seastore backref: 20 debug seastore cache: 20 debug seastore cleaner: 20 debug seastore device: 20 debug seastore epm: 20 debug seastore fixedkv tree: 20 debug seastore journal: 20 debug seastore lba: 20 debug seastore odata: 20 debug seastore omap: 20 debug seastore onode: 20 debug seastore t: 20 debug seastore tm: 20 osd objectstore: seastore osd shutdown pgref assert: true seastore cachepin size pershard: 64M seastore max concurrent transactions: 8 crimson_compat: true flavor: debug log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) - \(PG_ - \(OSD_ - \(OBJECT_ - overall HEALTH sha1: d26e5c228e3cdea4d585bcd077f8513a649ab5d1 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log global: osd crush chooseleaf type: 0 osd pool default pg num: 128 osd pool default pgp num: 128 osd pool default size: 2 mon: {} install: ceph: flavor: debug sha1: d26e5c228e3cdea4d585bcd077f8513a649ab5d1 workunit: branch: wip-liucm-shard-redesign-crimson-only env: CRIMSON_COMPAT: '1' sha1: d26e5c228e3cdea4d585bcd077f8513a649ab5d1 owner: scheduled_liucm@soko04.front.sepia.ceph.com priority: 120 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 9834 sha1: d26e5c228e3cdea4d585bcd077f8513a649ab5d1 sleep_before_teardown: 0 suite: crimson-rados suite_branch: wip-liucm-shard-redesign-crimson-only suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_d26e5c228e3cdea4d585bcd077f8513a649ab5d1/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: d26e5c228e3cdea4d585bcd077f8513a649ab5d1 targets: trial119.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGL3hFP2Fv6ElfoZ773NC+66AI3BSya1msosvrQ+0BjfOAmYaJPpCRRnt8IzVAiUDC9FbKBF0YwlhyfT9afRAwg= tasks: - install: extra_packages: deb: - ceph-osd-crimson - ceph-osd-crimson-dbg rpm: - ceph-osd-crimson - ceph-osd-crimson-debuginfo - pexec: all: - sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson - install: null - ceph: conf: osd: debug monc: 20 wait-for-scrub: false - ssh_keys: null - cbt: benchmarks: radosbench: concurrent_ops: 4 concurrent_procs: 2 op_size: - 4194304 pool_monitoring_list: - collectl pool_profile: replicated run_monitoring_list: - collectl time: 60 write_only: false cluster: iterations: 1 osds_per_node: 3 pool_profiles: replicated: pg_size: 256 pgp_size: 256 replication: replicated user: ubuntu monitoring_profiles: perf: args: stat -p {pid} -o {perf_dir}/perf_stat.{pid} nodes: - osds perf_cmd: perf pid_glob: crimson-osd.*.pid teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: fa17720d0088c3ac28e473468bfc79eeaff5cd38 timestamp: 2026-02-24_15:28:57 tube: trial user: liucm verbose: false worker_log: /home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.904699 2026-02-24T15:35:26.761 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_d26e5c228e3cdea4d585bcd077f8513a649ab5d1/qa; will attempt to use it 2026-02-24T15:35:26.761 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_d26e5c228e3cdea4d585bcd077f8513a649ab5d1/qa/tasks 2026-02-24T15:35:26.762 INFO:teuthology.run_tasks:Running task internal.check_packages... 2026-02-24T15:35:26.763 INFO:teuthology.task.internal:Checking packages... 2026-02-24T15:35:26.763 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'debug' and ceph hash 'd26e5c228e3cdea4d585bcd077f8513a649ab5d1' 2026-02-24T15:35:26.763 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2026-02-24T15:35:26.763 INFO:teuthology.packaging:ref: None 2026-02-24T15:35:26.763 INFO:teuthology.packaging:tag: None 2026-02-24T15:35:26.763 INFO:teuthology.packaging:branch: wip-liucm-shard-redesign-crimson-only 2026-02-24T15:35:26.763 INFO:teuthology.packaging:sha1: d26e5c228e3cdea4d585bcd077f8513a649ab5d1 2026-02-24T15:35:26.763 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&ref=wip-liucm-shard-redesign-crimson-only 2026-02-24T15:35:26.900 INFO:teuthology.task.internal:Found packages for ceph version 20.3.0-5343.gd26e5c22 2026-02-24T15:35:26.902 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2026-02-24T15:35:26.914 INFO:teuthology.task.internal:no buildpackages task found 2026-02-24T15:35:26.914 INFO:teuthology.run_tasks:Running task internal.save_config... 2026-02-24T15:35:26.928 INFO:teuthology.task.internal:Saving configuration 2026-02-24T15:35:26.945 INFO:teuthology.run_tasks:Running task internal.check_lock... 2026-02-24T15:35:26.957 INFO:teuthology.task.internal.check_lock:Checking locks... 2026-02-24T15:35:27.039 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial119.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/liucm-2026-02-24_15:28:57-crimson-rados-wip-liucm-shard-redesign-crimson-only-distro-debug-trial/68197', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-24 15:31:58.891630', 'locked_by': 'scheduled_liucm@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGL3hFP2Fv6ElfoZ773NC+66AI3BSya1msosvrQ+0BjfOAmYaJPpCRRnt8IzVAiUDC9FbKBF0YwlhyfT9afRAwg='} 2026-02-24T15:35:27.039 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2026-02-24T15:35:27.049 INFO:teuthology.task.internal:roles: ubuntu@trial119.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2026-02-24T15:35:27.050 INFO:teuthology.run_tasks:Running task console_log... 2026-02-24T15:35:27.133 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f8aef6959e0>, signals=[15]) 2026-02-24T15:35:27.134 INFO:teuthology.run_tasks:Running task internal.connect... 2026-02-24T15:35:27.144 INFO:teuthology.task.internal:Opening connections... 2026-02-24T15:35:27.144 DEBUG:teuthology.task.internal:connecting to ubuntu@trial119.front.sepia.ceph.com 2026-02-24T15:35:27.145 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-24T15:35:27.205 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2026-02-24T15:35:27.216 DEBUG:teuthology.orchestra.run.trial119:> uname -m 2026-02-24T15:35:27.276 INFO:teuthology.orchestra.run.trial119.stdout:x86_64 2026-02-24T15:35:27.277 DEBUG:teuthology.orchestra.run.trial119:> cat /etc/os-release 2026-02-24T15:35:27.331 INFO:teuthology.orchestra.run.trial119.stdout:NAME="CentOS Stream" 2026-02-24T15:35:27.331 INFO:teuthology.orchestra.run.trial119.stdout:VERSION="9" 2026-02-24T15:35:27.332 INFO:teuthology.orchestra.run.trial119.stdout:ID="centos" 2026-02-24T15:35:27.332 INFO:teuthology.orchestra.run.trial119.stdout:ID_LIKE="rhel fedora" 2026-02-24T15:35:27.332 INFO:teuthology.orchestra.run.trial119.stdout:VERSION_ID="9" 2026-02-24T15:35:27.332 INFO:teuthology.orchestra.run.trial119.stdout:PLATFORM_ID="platform:el9" 2026-02-24T15:35:27.332 INFO:teuthology.orchestra.run.trial119.stdout:PRETTY_NAME="CentOS Stream 9" 2026-02-24T15:35:27.332 INFO:teuthology.orchestra.run.trial119.stdout:ANSI_COLOR="0;31" 2026-02-24T15:35:27.332 INFO:teuthology.orchestra.run.trial119.stdout:LOGO="fedora-logo-icon" 2026-02-24T15:35:27.332 INFO:teuthology.orchestra.run.trial119.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2026-02-24T15:35:27.332 INFO:teuthology.orchestra.run.trial119.stdout:HOME_URL="https://centos.org/" 2026-02-24T15:35:27.332 INFO:teuthology.orchestra.run.trial119.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2026-02-24T15:35:27.332 INFO:teuthology.orchestra.run.trial119.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2026-02-24T15:35:27.332 INFO:teuthology.orchestra.run.trial119.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2026-02-24T15:35:27.333 INFO:teuthology.lock.ops:Updating trial119.front.sepia.ceph.com on lock server 2026-02-24T15:35:27.420 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2026-02-24T15:35:27.433 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2026-02-24T15:35:27.443 INFO:teuthology.task.internal:Checking for old test directory... 2026-02-24T15:35:27.443 DEBUG:teuthology.orchestra.run.trial119:> test '!' -e /home/ubuntu/cephtest 2026-02-24T15:35:27.458 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2026-02-24T15:35:27.469 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2026-02-24T15:35:27.469 DEBUG:teuthology.orchestra.run.trial119:> test -z $(ls -A /var/lib/ceph) 2026-02-24T15:35:27.515 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2026-02-24T15:35:27.619 INFO:teuthology.run_tasks:Running task kernel... 2026-02-24T15:35:27.639 INFO:teuthology.task.kernel:normalize config orig: {'branch': 'distro', 'kdb': True, 'sha1': 'distro'} 2026-02-24T15:35:27.639 INFO:teuthology.task.kernel:config {'mon.a': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}}, timeout 300 2026-02-24T15:35:27.639 DEBUG:teuthology.orchestra.run.trial119:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-24T15:35:27.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-24T15:35:27.654 DEBUG:teuthology.orchestra.run.trial119:> uname -r 2026-02-24T15:35:27.710 INFO:teuthology.orchestra.run.trial119.stdout:5.14.0-681.el9.x86_64 2026-02-24T15:35:27.710 INFO:teuthology.task.kernel:Running kernel on trial119: 5.14.0-681.el9.x86_64 2026-02-24T15:35:27.710 DEBUG:teuthology.orchestra.run.trial119:> sudo yum install -y kernel 2026-02-24T15:35:28.783 INFO:teuthology.orchestra.run.trial119.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.9 MB 00:00 2026-02-24T15:35:30.537 INFO:teuthology.orchestra.run.trial119.stdout:CentOS Stream 9 - AppStream 22 MB/s | 27 MB 00:01 2026-02-24T15:35:40.569 INFO:teuthology.orchestra.run.trial119.stdout:CentOS Stream 9 - CRB 1.0 MB/s | 8.0 MB 00:07 2026-02-24T15:35:41.522 INFO:teuthology.orchestra.run.trial119.stdout:CentOS Stream 9 - Extras packages 91 kB/s | 20 kB 00:00 2026-02-24T15:35:42.940 INFO:teuthology.orchestra.run.trial119.stdout:Extra Packages for Enterprise Linux 15 MB/s | 20 MB 00:01 2026-02-24T15:35:46.149 INFO:teuthology.orchestra.run.trial119.stdout:lab-extras 2.1 MB/s | 50 kB 00:00 2026-02-24T15:35:47.161 INFO:teuthology.orchestra.run.trial119.stdout:Package kernel-5.14.0-645.el9.x86_64 is already installed. 2026-02-24T15:35:47.162 INFO:teuthology.orchestra.run.trial119.stdout:Package kernel-5.14.0-681.el9.x86_64 is already installed. 2026-02-24T15:35:47.182 INFO:teuthology.orchestra.run.trial119.stdout:Dependencies resolved. 2026-02-24T15:35:47.183 INFO:teuthology.orchestra.run.trial119.stdout:================================================================================ 2026-02-24T15:35:47.183 INFO:teuthology.orchestra.run.trial119.stdout: Package Architecture Version Repository Size 2026-02-24T15:35:47.183 INFO:teuthology.orchestra.run.trial119.stdout:================================================================================ 2026-02-24T15:35:47.183 INFO:teuthology.orchestra.run.trial119.stdout:Installing: 2026-02-24T15:35:47.183 INFO:teuthology.orchestra.run.trial119.stdout: kernel x86_64 5.14.0-686.el9 baseos 901 k 2026-02-24T15:35:47.183 INFO:teuthology.orchestra.run.trial119.stdout:Installing dependencies: 2026-02-24T15:35:47.183 INFO:teuthology.orchestra.run.trial119.stdout: kernel-core x86_64 5.14.0-686.el9 baseos 17 M 2026-02-24T15:35:47.183 INFO:teuthology.orchestra.run.trial119.stdout: kernel-modules x86_64 5.14.0-686.el9 baseos 40 M 2026-02-24T15:35:47.184 INFO:teuthology.orchestra.run.trial119.stdout: kernel-modules-core x86_64 5.14.0-686.el9 baseos 31 M 2026-02-24T15:35:47.184 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:35:47.184 INFO:teuthology.orchestra.run.trial119.stdout:Transaction Summary 2026-02-24T15:35:47.184 INFO:teuthology.orchestra.run.trial119.stdout:================================================================================ 2026-02-24T15:35:47.184 INFO:teuthology.orchestra.run.trial119.stdout:Install 4 Packages 2026-02-24T15:35:47.184 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:35:47.184 INFO:teuthology.orchestra.run.trial119.stdout:Total download size: 89 M 2026-02-24T15:35:47.184 INFO:teuthology.orchestra.run.trial119.stdout:Installed size: 135 M 2026-02-24T15:35:47.184 INFO:teuthology.orchestra.run.trial119.stdout:Downloading Packages: 2026-02-24T15:35:47.853 INFO:teuthology.orchestra.run.trial119.stdout:(1/4): kernel-5.14.0-686.el9.x86_64.rpm 3.9 MB/s | 901 kB 00:00 2026-02-24T15:35:48.193 INFO:teuthology.orchestra.run.trial119.stdout:(2/4): kernel-core-5.14.0-686.el9.x86_64.rpm 30 MB/s | 17 MB 00:00 2026-02-24T15:35:48.493 INFO:teuthology.orchestra.run.trial119.stdout:(3/4): kernel-modules-core-5.14.0-686.el9.x86_6 48 MB/s | 31 MB 00:00 2026-02-24T15:35:48.616 INFO:teuthology.orchestra.run.trial119.stdout:(4/4): kernel-modules-5.14.0-686.el9.x86_64.rpm 40 MB/s | 40 MB 00:00 2026-02-24T15:35:48.616 INFO:teuthology.orchestra.run.trial119.stdout:-------------------------------------------------------------------------------- 2026-02-24T15:35:48.617 INFO:teuthology.orchestra.run.trial119.stdout:Total 62 MB/s | 89 MB 00:01 2026-02-24T15:35:48.809 INFO:teuthology.orchestra.run.trial119.stdout:Running transaction check 2026-02-24T15:35:49.019 INFO:teuthology.orchestra.run.trial119.stdout:Transaction check succeeded. 2026-02-24T15:35:49.019 INFO:teuthology.orchestra.run.trial119.stdout:Running transaction test 2026-02-24T15:35:49.405 INFO:teuthology.orchestra.run.trial119.stdout:Transaction test succeeded. 2026-02-24T15:35:49.405 INFO:teuthology.orchestra.run.trial119.stdout:Running transaction 2026-02-24T15:35:50.514 INFO:teuthology.orchestra.run.trial119.stdout: Preparing : 1/1 2026-02-24T15:35:50.806 INFO:teuthology.orchestra.run.trial119.stdout: Installing : kernel-modules-core-5.14.0-686.el9.x86_64 1/4 2026-02-24T15:35:50.828 INFO:teuthology.orchestra.run.trial119.stdout: Installing : kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-24T15:35:53.389 INFO:teuthology.orchestra.run.trial119.stdout: Running scriptlet: kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-24T15:35:53.423 INFO:teuthology.orchestra.run.trial119.stdout: Installing : kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-24T15:35:56.478 INFO:teuthology.orchestra.run.trial119.stdout: Running scriptlet: kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-24T15:35:56.485 INFO:teuthology.orchestra.run.trial119.stdout: Installing : kernel-5.14.0-686.el9.x86_64 4/4 2026-02-24T15:35:59.538 INFO:teuthology.orchestra.run.trial119.stdout: Running scriptlet: kernel-modules-core-5.14.0-686.el9.x86_64 4/4 2026-02-24T15:36:16.971 INFO:teuthology.orchestra.run.trial119.stdout: Running scriptlet: kernel-core-5.14.0-686.el9.x86_64 4/4 2026-02-24T15:36:16.991 INFO:teuthology.orchestra.run.trial119.stdout: Running scriptlet: kernel-modules-5.14.0-686.el9.x86_64 4/4 2026-02-24T15:36:17.180 INFO:teuthology.orchestra.run.trial119.stdout: Running scriptlet: kernel-5.14.0-686.el9.x86_64 4/4 2026-02-24T15:36:17.181 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : kernel-5.14.0-686.el9.x86_64 1/4 2026-02-24T15:36:17.181 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-24T15:36:17.181 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-24T15:36:17.238 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : kernel-modules-core-5.14.0-686.el9.x86_64 4/4 2026-02-24T15:36:17.238 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:36:17.238 INFO:teuthology.orchestra.run.trial119.stdout:Installed: 2026-02-24T15:36:17.238 INFO:teuthology.orchestra.run.trial119.stdout: kernel-5.14.0-686.el9.x86_64 2026-02-24T15:36:17.238 INFO:teuthology.orchestra.run.trial119.stdout: kernel-core-5.14.0-686.el9.x86_64 2026-02-24T15:36:17.238 INFO:teuthology.orchestra.run.trial119.stdout: kernel-modules-5.14.0-686.el9.x86_64 2026-02-24T15:36:17.238 INFO:teuthology.orchestra.run.trial119.stdout: kernel-modules-core-5.14.0-686.el9.x86_64 2026-02-24T15:36:17.238 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:36:17.239 INFO:teuthology.orchestra.run.trial119.stdout:Complete! 2026-02-24T15:36:17.300 DEBUG:teuthology.orchestra.run.trial119:> rpm -q kernel | sort -rV | head -n 1 2026-02-24T15:36:17.329 INFO:teuthology.orchestra.run.trial119.stdout:kernel-5.14.0-686.el9.x86_64 2026-02-24T15:36:17.330 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-686.el9.x86_64 2026-02-24T15:36:17.330 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-681.el9.x86_64 Expected: 5.14.0-686.el9.x86_64 2026-02-24T15:36:17.330 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2026-02-24T15:36:17.330 DEBUG:teuthology.task.kernel:src is distro, skipping download 2026-02-24T15:36:17.330 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2026-02-24T15:36:17.330 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@trial119.front.sepia.ceph.com, path=None, version=distro) 2026-02-24T15:36:17.331 DEBUG:teuthology.orchestra.run.trial119:> rpm -q kernel | sort -rV | head -n 1 2026-02-24T15:36:17.397 INFO:teuthology.orchestra.run.trial119.stdout:kernel-5.14.0-686.el9.x86_64 2026-02-24T15:36:17.398 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-686.el9.x86_64 2026-02-24T15:36:17.398 DEBUG:teuthology.orchestra.run.trial119:> sudo rpm -qi grub2-tools 2026-02-24T15:36:17.466 INFO:teuthology.orchestra.run.trial119.stdout:Name : grub2-tools 2026-02-24T15:36:17.467 INFO:teuthology.orchestra.run.trial119.stdout:Epoch : 1 2026-02-24T15:36:17.467 INFO:teuthology.orchestra.run.trial119.stdout:Version : 2.06 2026-02-24T15:36:17.467 INFO:teuthology.orchestra.run.trial119.stdout:Release : 120.el9 2026-02-24T15:36:17.467 INFO:teuthology.orchestra.run.trial119.stdout:Architecture: x86_64 2026-02-24T15:36:17.467 INFO:teuthology.orchestra.run.trial119.stdout:Install Date: Wed 18 Feb 2026 05:18:42 PM UTC 2026-02-24T15:36:17.467 INFO:teuthology.orchestra.run.trial119.stdout:Group : Unspecified 2026-02-24T15:36:17.467 INFO:teuthology.orchestra.run.trial119.stdout:Size : 9108611 2026-02-24T15:36:17.467 INFO:teuthology.orchestra.run.trial119.stdout:License : GPLv3+ 2026-02-24T15:36:17.467 INFO:teuthology.orchestra.run.trial119.stdout:Signature : RSA/SHA256, Thu 04 Dec 2025 09:42:59 AM UTC, Key ID 05b555b38483c65d 2026-02-24T15:36:17.467 INFO:teuthology.orchestra.run.trial119.stdout:Source RPM : grub2-2.06-120.el9.src.rpm 2026-02-24T15:36:17.468 INFO:teuthology.orchestra.run.trial119.stdout:Build Date : Wed 03 Dec 2025 12:10:38 PM UTC 2026-02-24T15:36:17.468 INFO:teuthology.orchestra.run.trial119.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2026-02-24T15:36:17.468 INFO:teuthology.orchestra.run.trial119.stdout:Packager : builder@centos.org 2026-02-24T15:36:17.468 INFO:teuthology.orchestra.run.trial119.stdout:Vendor : CentOS 2026-02-24T15:36:17.468 INFO:teuthology.orchestra.run.trial119.stdout:URL : http://www.gnu.org/software/grub/ 2026-02-24T15:36:17.468 INFO:teuthology.orchestra.run.trial119.stdout:Summary : Support tools for GRUB. 2026-02-24T15:36:17.468 INFO:teuthology.orchestra.run.trial119.stdout:Description : 2026-02-24T15:36:17.468 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:36:17.468 INFO:teuthology.orchestra.run.trial119.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2026-02-24T15:36:17.468 INFO:teuthology.orchestra.run.trial119.stdout:customizable bootloader with modular architecture. It supports a rich 2026-02-24T15:36:17.468 INFO:teuthology.orchestra.run.trial119.stdout:variety of kernel formats, file systems, computer architectures and 2026-02-24T15:36:17.468 INFO:teuthology.orchestra.run.trial119.stdout:hardware devices. 2026-02-24T15:36:17.469 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:36:17.469 INFO:teuthology.orchestra.run.trial119.stdout:This subpackage provides tools for support of all platforms. 2026-02-24T15:36:17.470 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2026-02-24T15:36:17.470 INFO:teuthology.task.kernel:Updating grub on trial119 to boot 5.14.0-686.el9.x86_64 2026-02-24T15:36:17.470 DEBUG:teuthology.orchestra.run.trial119:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f -print -quit 2026-02-24T15:36:17.535 INFO:teuthology.orchestra.run.trial119.stdout:/boot/loader/entries/b0373be789df49cdafedf423703d2566-5.14.0-681.el9.x86_64.conf 2026-02-24T15:36:17.537 DEBUG:teuthology.orchestra.run.trial119:> sudo command -v grubby && echo yes || echo no 2026-02-24T15:36:17.600 INFO:teuthology.orchestra.run.trial119.stdout:/sbin/grubby 2026-02-24T15:36:17.602 INFO:teuthology.orchestra.run.trial119.stdout:yes 2026-02-24T15:36:17.602 DEBUG:teuthology.orchestra.run.trial119:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f | grep -F "5.14.0-686.el9.x86_64" | head -n 1 2026-02-24T15:36:17.670 INFO:teuthology.orchestra.run.trial119.stdout:/boot/loader/entries/b0373be789df49cdafedf423703d2566-5.14.0-686.el9.x86_64.conf 2026-02-24T15:36:17.671 DEBUG:teuthology.orchestra.run.trial119:> sudo grub2-set-default b0373be789df49cdafedf423703d2566-5.14.0-686.el9.x86_64 2026-02-24T15:36:17.746 DEBUG:teuthology.orchestra.run.trial119:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2026-02-24T15:36:17.954 INFO:teuthology.orchestra.run.trial119.stderr:Generating grub configuration file ... 2026-02-24T15:36:18.021 INFO:teuthology.orchestra.run.trial119.stderr:Found linux image: /boot/vmlinuz-5.14.0-686.el9.x86_64 2026-02-24T15:36:18.028 INFO:teuthology.orchestra.run.trial119.stderr:Found initrd image: /boot/initramfs-5.14.0-686.el9.x86_64.img 2026-02-24T15:36:18.279 INFO:teuthology.orchestra.run.trial119.stderr:Found linux image: /boot/vmlinuz-5.14.0-681.el9.x86_64 2026-02-24T15:36:18.286 INFO:teuthology.orchestra.run.trial119.stderr:Found initrd image: /boot/initramfs-5.14.0-681.el9.x86_64.img 2026-02-24T15:36:18.320 INFO:teuthology.orchestra.run.trial119.stderr:Found linux image: /boot/vmlinuz-5.14.0-645.el9.x86_64 2026-02-24T15:36:18.327 INFO:teuthology.orchestra.run.trial119.stderr:Found initrd image: /boot/initramfs-5.14.0-645.el9.x86_64.img 2026-02-24T15:36:18.359 INFO:teuthology.orchestra.run.trial119.stderr:Found linux image: /boot/vmlinuz-0-rescue-b0373be789df49cdafedf423703d2566 2026-02-24T15:36:18.366 INFO:teuthology.orchestra.run.trial119.stderr:Found initrd image: /boot/initramfs-0-rescue-b0373be789df49cdafedf423703d2566.img 2026-02-24T15:36:18.389 INFO:teuthology.orchestra.run.trial119.stderr:Found linux image: /boot/vmlinuz-0-rescue-2734456005d743dca7f3dc77c35a2d0d 2026-02-24T15:36:18.395 INFO:teuthology.orchestra.run.trial119.stderr:Found initrd image: /boot/initramfs-0-rescue-2734456005d743dca7f3dc77c35a2d0d.img 2026-02-24T15:36:18.481 INFO:teuthology.orchestra.run.trial119.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2026-02-24T15:36:18.490 INFO:teuthology.orchestra.run.trial119.stderr:done 2026-02-24T15:36:18.492 DEBUG:teuthology.orchestra.run.trial119:> test -d /sys/firmware/efi 2026-02-24T15:36:18.508 DEBUG:teuthology.orchestra.run.trial119:> sudo find /boot/efi/EFI -mindepth 1 -maxdepth 1 -type d || true 2026-02-24T15:36:18.573 INFO:teuthology.orchestra.run.trial119.stdout:/boot/efi/EFI/BOOT 2026-02-24T15:36:18.574 INFO:teuthology.orchestra.run.trial119.stdout:/boot/efi/EFI/centos 2026-02-24T15:36:18.575 DEBUG:teuthology.orchestra.run.trial119:> sudo test -f /boot/efi/EFI/BOOT/grub.cfg && echo yes || echo no 2026-02-24T15:36:18.639 INFO:teuthology.orchestra.run.trial119.stdout:no 2026-02-24T15:36:18.640 DEBUG:teuthology.orchestra.run.trial119:> sudo test -f /boot/efi/EFI/centos/grub.cfg && echo yes || echo no 2026-02-24T15:36:18.703 INFO:teuthology.orchestra.run.trial119.stdout:yes 2026-02-24T15:36:18.703 DEBUG:teuthology.orchestra.run.trial119:> sudo cp -f /boot/grub2/grub.cfg /boot/efi/EFI/centos/grub.cfg 2026-02-24T15:36:18.767 INFO:teuthology.orchestra.remote:Triggering safe hard reboot via SysRq on trial119 2026-02-24T15:36:18.767 DEBUG:teuthology.orchestra.run.trial119:> sudo bash -c ' 2026-02-24T15:36:18.767 DEBUG:teuthology.orchestra.run.trial119:> echo pci > /sys/kernel/reboot/type && echo 1 > /sys/kernel/reboot/force && sync && echo s > /proc/sysrq-trigger && echo u > /proc/sysrq-trigger && sleep 2 && echo b > /proc/sysrq-trigger 2026-02-24T15:36:18.768 DEBUG:teuthology.orchestra.run.trial119:> ' 2026-02-24T15:36:48.810 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2026-02-24T15:36:48.811 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial119.front.sepia.ceph.com' 2026-02-24T15:36:48.812 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-24T15:37:48.874 DEBUG:teuthology.orchestra.remote:timed out 2026-02-24T15:37:57.883 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial119.front.sepia.ceph.com' 2026-02-24T15:37:57.884 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-24T15:37:57.984 DEBUG:teuthology.orchestra.run.trial119:> true 2026-02-24T15:37:58.147 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial119.front.sepia.ceph.com' 2026-02-24T15:37:58.147 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-686.el9.x86_64"... 2026-02-24T15:37:58.148 DEBUG:teuthology.orchestra.run.trial119:> uname -r 2026-02-24T15:37:58.203 INFO:teuthology.orchestra.run.trial119.stdout:5.14.0-686.el9.x86_64 2026-02-24T15:37:58.204 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-686.el9.x86_64 vs 5.14.0-686.el9.x86_64 2026-02-24T15:37:58.204 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2026-02-24T15:37:58.204 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2026-02-24T15:37:59.205 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2026-02-24T15:37:59.205 DEBUG:teuthology.orchestra.run.trial119:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-24T15:37:59.235 INFO:teuthology.orchestra.run.trial119.stdout:ttyS1 2026-02-24T15:37:59.261 DEBUG:teuthology.parallel:result is None 2026-02-24T15:37:59.262 INFO:teuthology.run_tasks:Running task internal.base... 2026-02-24T15:37:59.274 INFO:teuthology.task.internal:Creating test directory... 2026-02-24T15:37:59.275 DEBUG:teuthology.orchestra.run.trial119:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-24T15:37:59.317 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2026-02-24T15:37:59.327 INFO:teuthology.run_tasks:Running task internal.archive... 2026-02-24T15:37:59.339 INFO:teuthology.task.internal:Creating archive directory... 2026-02-24T15:37:59.339 DEBUG:teuthology.orchestra.run.trial119:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-24T15:37:59.378 INFO:teuthology.run_tasks:Running task internal.coredump... 2026-02-24T15:37:59.388 INFO:teuthology.task.internal:Enabling coredump saving... 2026-02-24T15:37:59.389 DEBUG:teuthology.orchestra.run.trial119:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-24T15:37:59.437 INFO:teuthology.orchestra.run.trial119.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-24T15:37:59.446 INFO:teuthology.orchestra.run.trial119.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-24T15:37:59.448 INFO:teuthology.run_tasks:Running task internal.sudo... 2026-02-24T15:37:59.462 INFO:teuthology.task.internal:Configuring sudo... 2026-02-24T15:37:59.463 DEBUG:teuthology.orchestra.run.trial119:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-24T15:37:59.511 INFO:teuthology.run_tasks:Running task internal.syslog... 2026-02-24T15:37:59.524 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2026-02-24T15:37:59.524 DEBUG:teuthology.orchestra.run.trial119:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-24T15:37:59.567 DEBUG:teuthology.orchestra.run.trial119:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-24T15:37:59.710 DEBUG:teuthology.orchestra.run.trial119:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-24T15:37:59.734 DEBUG:teuthology.orchestra.run.trial119:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-24T15:37:59.858 DEBUG:teuthology.orchestra.run.trial119:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-24T15:37:59.881 DEBUG:teuthology.orchestra.run.trial119:> set -ex 2026-02-24T15:37:59.881 DEBUG:teuthology.orchestra.run.trial119:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-24T15:37:59.944 DEBUG:teuthology.orchestra.run.trial119:> sudo service rsyslog restart 2026-02-24T15:38:00.011 INFO:teuthology.orchestra.run.trial119.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-24T15:38:00.192 INFO:teuthology.run_tasks:Running task internal.timer... 2026-02-24T15:38:00.204 INFO:teuthology.task.internal:Starting timer... 2026-02-24T15:38:00.204 INFO:teuthology.run_tasks:Running task pcp... 2026-02-24T15:38:00.218 INFO:teuthology.run_tasks:Running task selinux... 2026-02-24T15:38:00.231 DEBUG:teuthology.orchestra.run.trial119:> sudo service auditd rotate 2026-02-24T15:38:00.259 INFO:teuthology.orchestra.run.trial119.stdout:Rotating logs: 2026-02-24T15:38:00.261 DEBUG:teuthology.task.selinux:Getting current SELinux state 2026-02-24T15:38:00.261 DEBUG:teuthology.orchestra.run.trial119:> /usr/sbin/getenforce 2026-02-24T15:38:00.317 INFO:teuthology.orchestra.run.trial119.stdout:Permissive 2026-02-24T15:38:00.317 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@trial119.front.sepia.ceph.com': 'permissive'} 2026-02-24T15:38:00.318 DEBUG:teuthology.orchestra.run.trial119:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-24T15:38:00.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-24T15:38:00.382 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2026-02-24T15:38:00.382 DEBUG:teuthology.orchestra.run.trial119:> sudo /usr/sbin/setenforce permissive 2026-02-24T15:38:00.445 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2026-02-24T15:38:00.458 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2026-02-24T15:38:00.464 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2026-02-24T15:38:00.465 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit trial119.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2026-02-24T15:40:32.078 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@trial119.front.sepia.ceph.com')] 2026-02-24T15:40:32.078 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial119.front.sepia.ceph.com' 2026-02-24T15:40:32.079 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-24T15:40:32.183 DEBUG:teuthology.orchestra.run.trial119:> true 2026-02-24T15:40:32.269 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial119.front.sepia.ceph.com' 2026-02-24T15:40:32.269 INFO:teuthology.run_tasks:Running task clock... 2026-02-24T15:40:32.287 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2026-02-24T15:40:32.287 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-24T15:40:32.288 DEBUG:teuthology.orchestra.run.trial119:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-24T15:40:32.336 INFO:teuthology.orchestra.run.trial119.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2026-02-24T15:40:32.348 INFO:teuthology.orchestra.run.trial119.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2026-02-24T15:40:32.370 INFO:teuthology.orchestra.run.trial119.stderr:sudo: ntpd: command not found 2026-02-24T15:40:32.379 INFO:teuthology.orchestra.run.trial119.stdout:506 Cannot talk to daemon 2026-02-24T15:40:32.391 INFO:teuthology.orchestra.run.trial119.stderr:Failed to start ntp.service: Unit ntp.service not found. 2026-02-24T15:40:32.403 INFO:teuthology.orchestra.run.trial119.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2026-02-24T15:40:32.453 INFO:teuthology.orchestra.run.trial119.stderr:bash: line 1: ntpq: command not found 2026-02-24T15:40:32.478 INFO:teuthology.orchestra.run.trial119.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-24T15:40:32.478 INFO:teuthology.orchestra.run.trial119.stdout:=============================================================================== 2026-02-24T15:40:32.478 INFO:teuthology.orchestra.run.trial119.stdout:^? 23.186.168.133 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-24T15:40:32.478 INFO:teuthology.orchestra.run.trial119.stdout:^? 170-187-147-56.ip.linode> 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-24T15:40:32.478 INFO:teuthology.orchestra.run.trial119.stdout:^? owners.kjsl.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-24T15:40:32.478 INFO:teuthology.orchestra.run.trial119.stdout:^? time.cloudflare.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-24T15:40:32.479 INFO:teuthology.run_tasks:Running task install... 2026-02-24T15:40:32.491 DEBUG:teuthology.task.install:project ceph 2026-02-24T15:40:32.491 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'debug', 'sha1': 'd26e5c228e3cdea4d585bcd077f8513a649ab5d1'}} 2026-02-24T15:40:32.491 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'flavor': 'debug', 'sha1': 'd26e5c228e3cdea4d585bcd077f8513a649ab5d1'} 2026-02-24T15:40:32.491 INFO:teuthology.task.install:Using flavor: debug 2026-02-24T15:40:32.494 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-osd', 'ceph-osd-classic', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-osd', 'ceph-osd-classic', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-24T15:40:32.494 INFO:teuthology.task.install:extra packages: {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']} 2026-02-24T15:40:32.494 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'debug', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd26e5c228e3cdea4d585bcd077f8513a649ab5d1', 'tag': None, 'wait_for_package': False} 2026-02-24T15:40:32.495 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=d26e5c228e3cdea4d585bcd077f8513a649ab5d1 2026-02-24T15:40:32.634 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-liucm-shard-redesign-crimson-only/d26e5c228e3cdea4d585bcd077f8513a649ab5d1/centos/9/flavors/debug/ 2026-02-24T15:40:32.634 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5343.gd26e5c22 2026-02-24T15:40:32.976 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-liucm-shard-redesign-crimson-only/d26e5c228e3cdea4d585bcd077f8513a649ab5d1/centos/9/flavors/debug/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-liucm-shard-redesign-crimson-only/d26e5c228e3cdea4d585bcd077f8513a649ab5d1/centos/9/flavors/debug/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-liucm-shard-redesign-crimson-only/d26e5c228e3cdea4d585bcd077f8513a649ab5d1/centos/9/flavors/debug/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-24T15:40:32.976 DEBUG:teuthology.orchestra.run.trial119:> set -ex 2026-02-24T15:40:32.976 DEBUG:teuthology.orchestra.run.trial119:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-24T15:40:33.002 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-osd-crimson, ceph-osd-crimson-debuginfo on remote rpm x86_64 2026-02-24T15:40:33.003 DEBUG:teuthology.orchestra.run.trial119:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d26e5c228e3cdea4d585bcd077f8513a649ab5d1/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-24T15:40:33.067 DEBUG:teuthology.orchestra.run.trial119:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-24T15:40:33.140 DEBUG:teuthology.orchestra.run.trial119:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-24T15:40:33.203 INFO:teuthology.orchestra.run.trial119.stdout:check_obsoletes = 1 2026-02-24T15:40:33.204 DEBUG:teuthology.orchestra.run.trial119:> sudo yum clean all 2026-02-24T15:40:33.349 INFO:teuthology.orchestra.run.trial119.stdout:41 files removed 2026-02-24T15:40:33.363 DEBUG:teuthology.orchestra.run.trial119:> sudo yum -y install ceph-radosgw 2026-02-24T15:40:34.277 INFO:teuthology.orchestra.run.trial119.stdout:ceph packages for x86_64 120 kB/s | 92 kB 00:00 2026-02-24T15:40:34.926 INFO:teuthology.orchestra.run.trial119.stdout:ceph noarch packages 38 kB/s | 24 kB 00:00 2026-02-24T15:40:35.516 INFO:teuthology.orchestra.run.trial119.stdout:ceph source packages 3.6 kB/s | 2.1 kB 00:00 2026-02-24T15:40:38.924 INFO:teuthology.orchestra.run.trial119.stdout:CentOS Stream 9 - BaseOS 2.6 MB/s | 8.9 MB 00:03 2026-02-24T15:40:41.989 INFO:teuthology.orchestra.run.trial119.stdout:CentOS Stream 9 - AppStream 11 MB/s | 27 MB 00:02 2026-02-24T15:40:45.451 INFO:teuthology.orchestra.run.trial119.stdout:CentOS Stream 9 - CRB 7.4 MB/s | 8.0 MB 00:01 2026-02-24T15:40:46.618 INFO:teuthology.orchestra.run.trial119.stdout:CentOS Stream 9 - Extras packages 47 kB/s | 20 kB 00:00 2026-02-24T15:40:47.977 INFO:teuthology.orchestra.run.trial119.stdout:Extra Packages for Enterprise Linux 16 MB/s | 20 MB 00:01 2026-02-24T15:40:51.178 INFO:teuthology.orchestra.run.trial119.stdout:lab-extras 2.2 MB/s | 50 kB 00:00 2026-02-24T15:40:52.401 INFO:teuthology.orchestra.run.trial119.stdout:Dependencies resolved. 2026-02-24T15:40:52.403 INFO:teuthology.orchestra.run.trial119.stdout:================================================================================ 2026-02-24T15:40:52.403 INFO:teuthology.orchestra.run.trial119.stdout: Package Arch Version Repository Size 2026-02-24T15:40:52.403 INFO:teuthology.orchestra.run.trial119.stdout:================================================================================ 2026-02-24T15:40:52.404 INFO:teuthology.orchestra.run.trial119.stdout:Installing: 2026-02-24T15:40:52.404 INFO:teuthology.orchestra.run.trial119.stdout: ceph-radosgw x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 11 M 2026-02-24T15:40:52.404 INFO:teuthology.orchestra.run.trial119.stdout:Upgrading: 2026-02-24T15:40:52.404 INFO:teuthology.orchestra.run.trial119.stdout: librados2 x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 3.3 M 2026-02-24T15:40:52.404 INFO:teuthology.orchestra.run.trial119.stdout: librbd1 x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 2.7 M 2026-02-24T15:40:52.404 INFO:teuthology.orchestra.run.trial119.stdout:Installing dependencies: 2026-02-24T15:40:52.404 INFO:teuthology.orchestra.run.trial119.stdout: boost-program-options x86_64 1.75.0-13.el9 appstream 104 k 2026-02-24T15:40:52.404 INFO:teuthology.orchestra.run.trial119.stdout: ceph-base x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 5.1 M 2026-02-24T15:40:52.404 INFO:teuthology.orchestra.run.trial119.stdout: ceph-common x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 21 M 2026-02-24T15:40:52.404 INFO:teuthology.orchestra.run.trial119.stdout: ceph-selinux x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 25 k 2026-02-24T15:40:52.405 INFO:teuthology.orchestra.run.trial119.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2026-02-24T15:40:52.405 INFO:teuthology.orchestra.run.trial119.stdout: libarrow x86_64 9.0.0-15.el9 epel 4.4 M 2026-02-24T15:40:52.405 INFO:teuthology.orchestra.run.trial119.stdout: libarrow-doc noarch 9.0.0-15.el9 epel 25 k 2026-02-24T15:40:52.405 INFO:teuthology.orchestra.run.trial119.stdout: libcephfs2 x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 860 k 2026-02-24T15:40:52.405 INFO:teuthology.orchestra.run.trial119.stdout: libnbd x86_64 1.20.3-4.el9 appstream 164 k 2026-02-24T15:40:52.405 INFO:teuthology.orchestra.run.trial119.stdout: liboath x86_64 2.6.12-1.el9 epel 49 k 2026-02-24T15:40:52.405 INFO:teuthology.orchestra.run.trial119.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2026-02-24T15:40:52.405 INFO:teuthology.orchestra.run.trial119.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2026-02-24T15:40:52.405 INFO:teuthology.orchestra.run.trial119.stdout: libradosstriper1 x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 263 k 2026-02-24T15:40:52.405 INFO:teuthology.orchestra.run.trial119.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2026-02-24T15:40:52.405 INFO:teuthology.orchestra.run.trial119.stdout: librgw2 x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 5.6 M 2026-02-24T15:40:52.406 INFO:teuthology.orchestra.run.trial119.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2026-02-24T15:40:52.406 INFO:teuthology.orchestra.run.trial119.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2026-02-24T15:40:52.406 INFO:teuthology.orchestra.run.trial119.stdout: mailcap noarch 2.1.49-5.el9 baseos 33 k 2026-02-24T15:40:52.406 INFO:teuthology.orchestra.run.trial119.stdout: parquet-libs x86_64 9.0.0-15.el9 epel 838 k 2026-02-24T15:40:52.406 INFO:teuthology.orchestra.run.trial119.stdout: pciutils x86_64 3.7.0-7.el9 baseos 93 k 2026-02-24T15:40:52.406 INFO:teuthology.orchestra.run.trial119.stdout: python3-ceph-argparse x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 45 k 2026-02-24T15:40:52.406 INFO:teuthology.orchestra.run.trial119.stdout: python3-ceph-common x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 185 k 2026-02-24T15:40:52.406 INFO:teuthology.orchestra.run.trial119.stdout: python3-cephfs x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 203 k 2026-02-24T15:40:52.406 INFO:teuthology.orchestra.run.trial119.stdout: python3-rados x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 324 k 2026-02-24T15:40:52.406 INFO:teuthology.orchestra.run.trial119.stdout: python3-rbd x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 305 k 2026-02-24T15:40:52.406 INFO:teuthology.orchestra.run.trial119.stdout: python3-rgw x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 100 k 2026-02-24T15:40:52.406 INFO:teuthology.orchestra.run.trial119.stdout: qatlib x86_64 25.08.0-2.el9 appstream 240 k 2026-02-24T15:40:52.407 INFO:teuthology.orchestra.run.trial119.stdout: qatzip-libs x86_64 1.3.1-1.el9 appstream 66 k 2026-02-24T15:40:52.407 INFO:teuthology.orchestra.run.trial119.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2026-02-24T15:40:52.407 INFO:teuthology.orchestra.run.trial119.stdout: thrift x86_64 0.15.0-4.el9 epel 1.6 M 2026-02-24T15:40:52.407 INFO:teuthology.orchestra.run.trial119.stdout:Installing weak dependencies: 2026-02-24T15:40:52.407 INFO:teuthology.orchestra.run.trial119.stdout: qatlib-service x86_64 25.08.0-2.el9 appstream 37 k 2026-02-24T15:40:52.407 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:40:52.407 INFO:teuthology.orchestra.run.trial119.stdout:Transaction Summary 2026-02-24T15:40:52.408 INFO:teuthology.orchestra.run.trial119.stdout:================================================================================ 2026-02-24T15:40:52.408 INFO:teuthology.orchestra.run.trial119.stdout:Install 32 Packages 2026-02-24T15:40:52.408 INFO:teuthology.orchestra.run.trial119.stdout:Upgrade 2 Packages 2026-02-24T15:40:52.408 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:40:52.409 INFO:teuthology.orchestra.run.trial119.stdout:Total download size: 60 M 2026-02-24T15:40:52.409 INFO:teuthology.orchestra.run.trial119.stdout:Downloading Packages: 2026-02-24T15:40:57.902 INFO:teuthology.orchestra.run.trial119.stdout:(1/34): ceph-common-20.3.0-5343.gd26e5c22.el9.x 3.9 MB/s | 21 MB 00:05 2026-02-24T15:40:57.966 INFO:teuthology.orchestra.run.trial119.stdout:(2/34): ceph-selinux-20.3.0-5343.gd26e5c22.el9. 393 kB/s | 25 kB 00:00 2026-02-24T15:40:58.109 INFO:teuthology.orchestra.run.trial119.stdout:(3/34): libcephfs2-20.3.0-5343.gd26e5c22.el9.x8 5.9 MB/s | 860 kB 00:00 2026-02-24T15:40:58.177 INFO:teuthology.orchestra.run.trial119.stdout:(4/34): libradosstriper1-20.3.0-5343.gd26e5c22. 3.8 MB/s | 263 kB 00:00 2026-02-24T15:40:59.325 INFO:teuthology.orchestra.run.trial119.stdout:(5/34): librgw2-20.3.0-5343.gd26e5c22.el9.x86_6 4.9 MB/s | 5.6 MB 00:01 2026-02-24T15:40:59.388 INFO:teuthology.orchestra.run.trial119.stdout:(6/34): python3-ceph-argparse-20.3.0-5343.gd26e 711 kB/s | 45 kB 00:00 2026-02-24T15:40:59.453 INFO:teuthology.orchestra.run.trial119.stdout:(7/34): python3-ceph-common-20.3.0-5343.gd26e5c 2.8 MB/s | 185 kB 00:00 2026-02-24T15:40:59.518 INFO:teuthology.orchestra.run.trial119.stdout:(8/34): python3-cephfs-20.3.0-5343.gd26e5c22.el 3.0 MB/s | 203 kB 00:00 2026-02-24T15:40:59.587 INFO:teuthology.orchestra.run.trial119.stdout:(9/34): python3-rados-20.3.0-5343.gd26e5c22.el9 4.6 MB/s | 324 kB 00:00 2026-02-24T15:40:59.649 INFO:teuthology.orchestra.run.trial119.stdout:(10/34): ceph-base-20.3.0-5343.gd26e5c22.el9.x8 744 kB/s | 5.1 MB 00:07 2026-02-24T15:40:59.654 INFO:teuthology.orchestra.run.trial119.stdout:(11/34): python3-rbd-20.3.0-5343.gd26e5c22.el9. 4.4 MB/s | 305 kB 00:00 2026-02-24T15:40:59.713 INFO:teuthology.orchestra.run.trial119.stdout:(12/34): mailcap-2.1.49-5.el9.noarch.rpm 558 kB/s | 33 kB 00:00 2026-02-24T15:40:59.751 INFO:teuthology.orchestra.run.trial119.stdout:(13/34): pciutils-3.7.0-7.el9.x86_64.rpm 2.4 MB/s | 93 kB 00:00 2026-02-24T15:40:59.776 INFO:teuthology.orchestra.run.trial119.stdout:(14/34): python3-rgw-20.3.0-5343.gd26e5c22.el9. 788 kB/s | 100 kB 00:00 2026-02-24T15:40:59.796 INFO:teuthology.orchestra.run.trial119.stdout:(15/34): boost-program-options-1.75.0-13.el9.x8 2.3 MB/s | 104 kB 00:00 2026-02-24T15:40:59.814 INFO:teuthology.orchestra.run.trial119.stdout:(16/34): libpmemobj-1.12.1-1.el9.x86_64.rpm 8.9 MB/s | 160 kB 00:00 2026-02-24T15:40:59.819 INFO:teuthology.orchestra.run.trial119.stdout:(17/34): libnbd-1.20.3-4.el9.x86_64.rpm 3.7 MB/s | 164 kB 00:00 2026-02-24T15:40:59.823 INFO:teuthology.orchestra.run.trial119.stdout:(18/34): librabbitmq-0.11.0-7.el9.x86_64.rpm 4.8 MB/s | 45 kB 00:00 2026-02-24T15:40:59.841 INFO:teuthology.orchestra.run.trial119.stdout:(19/34): lttng-ust-2.12.0-6.el9.x86_64.rpm 16 MB/s | 292 kB 00:00 2026-02-24T15:40:59.851 INFO:teuthology.orchestra.run.trial119.stdout:(20/34): qatlib-25.08.0-2.el9.x86_64.rpm 22 MB/s | 240 kB 00:00 2026-02-24T15:40:59.853 INFO:teuthology.orchestra.run.trial119.stdout:(21/34): librdkafka-1.6.1-102.el9.x86_64.rpm 19 MB/s | 662 kB 00:00 2026-02-24T15:40:59.861 INFO:teuthology.orchestra.run.trial119.stdout:(22/34): qatlib-service-25.08.0-2.el9.x86_64.rp 4.0 MB/s | 37 kB 00:00 2026-02-24T15:40:59.863 INFO:teuthology.orchestra.run.trial119.stdout:(23/34): qatzip-libs-1.3.1-1.el9.x86_64.rpm 6.9 MB/s | 66 kB 00:00 2026-02-24T15:41:00.108 INFO:teuthology.orchestra.run.trial119.stdout:(24/34): gperftools-libs-2.9.1-3.el9.x86_64.rpm 1.2 MB/s | 308 kB 00:00 2026-02-24T15:41:00.144 INFO:teuthology.orchestra.run.trial119.stdout:(25/34): libarrow-doc-9.0.0-15.el9.noarch.rpm 679 kB/s | 25 kB 00:00 2026-02-24T15:41:00.183 INFO:teuthology.orchestra.run.trial119.stdout:(26/34): liboath-2.6.12-1.el9.x86_64.rpm 1.2 MB/s | 49 kB 00:00 2026-02-24T15:41:00.222 INFO:teuthology.orchestra.run.trial119.stdout:(27/34): libunwind-1.6.2-1.el9.x86_64.rpm 1.7 MB/s | 67 kB 00:00 2026-02-24T15:41:00.247 INFO:teuthology.orchestra.run.trial119.stdout:(28/34): libarrow-9.0.0-15.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2026-02-24T15:41:00.285 INFO:teuthology.orchestra.run.trial119.stdout:(29/34): re2-20211101-20.el9.x86_64.rpm 5.0 MB/s | 191 kB 00:00 2026-02-24T15:41:00.322 INFO:teuthology.orchestra.run.trial119.stdout:(30/34): parquet-libs-9.0.0-15.el9.x86_64.rpm 8.2 MB/s | 838 kB 00:00 2026-02-24T15:41:00.330 INFO:teuthology.orchestra.run.trial119.stdout:(31/34): thrift-0.15.0-4.el9.x86_64.rpm 35 MB/s | 1.6 MB 00:00 2026-02-24T15:41:01.985 INFO:teuthology.orchestra.run.trial119.stdout:(32/34): librbd1-20.3.0-5343.gd26e5c22.el9.x86_ 1.6 MB/s | 2.7 MB 00:01 2026-02-24T15:41:03.389 INFO:teuthology.orchestra.run.trial119.stdout:(33/34): librados2-20.3.0-5343.gd26e5c22.el9.x8 1.1 MB/s | 3.3 MB 00:03 2026-02-24T15:41:05.318 INFO:teuthology.orchestra.run.trial119.stdout:(34/34): ceph-radosgw-20.3.0-5343.gd26e5c22.el9 902 kB/s | 11 MB 00:12 2026-02-24T15:41:05.319 INFO:teuthology.orchestra.run.trial119.stdout:-------------------------------------------------------------------------------- 2026-02-24T15:41:05.319 INFO:teuthology.orchestra.run.trial119.stdout:Total 4.7 MB/s | 60 MB 00:12 2026-02-24T15:41:05.408 INFO:teuthology.orchestra.run.trial119.stdout:Running transaction check 2026-02-24T15:41:05.425 INFO:teuthology.orchestra.run.trial119.stdout:Transaction check succeeded. 2026-02-24T15:41:05.426 INFO:teuthology.orchestra.run.trial119.stdout:Running transaction test 2026-02-24T15:41:05.666 INFO:teuthology.orchestra.run.trial119.stdout:Transaction test succeeded. 2026-02-24T15:41:05.666 INFO:teuthology.orchestra.run.trial119.stdout:Running transaction 2026-02-24T15:41:05.946 INFO:teuthology.orchestra.run.trial119.stdout: Preparing : 1/1 2026-02-24T15:41:06.084 INFO:teuthology.orchestra.run.trial119.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/36 2026-02-24T15:41:06.089 INFO:teuthology.orchestra.run.trial119.stdout: Upgrading : librados2-2:20.3.0-5343.gd26e5c22.el9.x86_64 2/36 2026-02-24T15:41:06.120 INFO:teuthology.orchestra.run.trial119.stdout: Running scriptlet: librados2-2:20.3.0-5343.gd26e5c22.el9.x86_64 2/36 2026-02-24T15:41:06.156 INFO:teuthology.orchestra.run.trial119.stdout: Installing : python3-rados-2:20.3.0-5343.gd26e5c22.el9.x86_64 3/36 2026-02-24T15:41:06.157 INFO:teuthology.orchestra.run.trial119.stdout: Installing : libcephfs2-2:20.3.0-5343.gd26e5c22.el9.x86_64 4/36 2026-02-24T15:41:06.170 INFO:teuthology.orchestra.run.trial119.stdout: Running scriptlet: libcephfs2-2:20.3.0-5343.gd26e5c22.el9.x86_64 4/36 2026-02-24T15:41:06.178 INFO:teuthology.orchestra.run.trial119.stdout: Installing : liboath-2.6.12-1.el9.x86_64 5/36 2026-02-24T15:41:06.180 INFO:teuthology.orchestra.run.trial119.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/36 2026-02-24T15:41:06.183 INFO:teuthology.orchestra.run.trial119.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/36 2026-02-24T15:41:06.188 INFO:teuthology.orchestra.run.trial119.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 8/36 2026-02-24T15:41:06.199 INFO:teuthology.orchestra.run.trial119.stdout: Installing : python3-ceph-argparse-2:20.3.0-5343.gd26e5c22.el9. 9/36 2026-02-24T15:41:06.211 INFO:teuthology.orchestra.run.trial119.stdout: Installing : python3-cephfs-2:20.3.0-5343.gd26e5c22.el9.x86_64 10/36 2026-02-24T15:41:06.212 INFO:teuthology.orchestra.run.trial119.stdout: Installing : libradosstriper1-2:20.3.0-5343.gd26e5c22.el9.x86_6 11/36 2026-02-24T15:41:06.238 INFO:teuthology.orchestra.run.trial119.stdout: Running scriptlet: libradosstriper1-2:20.3.0-5343.gd26e5c22.el9.x86_6 11/36 2026-02-24T15:41:06.243 INFO:teuthology.orchestra.run.trial119.stdout: Installing : thrift-0.15.0-4.el9.x86_64 12/36 2026-02-24T15:41:06.276 INFO:teuthology.orchestra.run.trial119.stdout: Installing : re2-1:20211101-20.el9.x86_64 13/36 2026-02-24T15:41:06.280 INFO:teuthology.orchestra.run.trial119.stdout: Installing : libarrow-9.0.0-15.el9.x86_64 14/36 2026-02-24T15:41:06.288 INFO:teuthology.orchestra.run.trial119.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 15/36 2026-02-24T15:41:06.290 INFO:teuthology.orchestra.run.trial119.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 16/36 2026-02-24T15:41:06.295 INFO:teuthology.orchestra.run.trial119.stdout: Installing : libarrow-doc-9.0.0-15.el9.noarch 17/36 2026-02-24T15:41:06.410 INFO:teuthology.orchestra.run.trial119.stdout: Installing : libnbd-1.20.3-4.el9.x86_64 18/36 2026-02-24T15:41:06.412 INFO:teuthology.orchestra.run.trial119.stdout: Upgrading : librbd1-2:20.3.0-5343.gd26e5c22.el9.x86_64 19/36 2026-02-24T15:41:06.438 INFO:teuthology.orchestra.run.trial119.stdout: Running scriptlet: librbd1-2:20.3.0-5343.gd26e5c22.el9.x86_64 19/36 2026-02-24T15:41:06.440 INFO:teuthology.orchestra.run.trial119.stdout: Installing : python3-rbd-2:20.3.0-5343.gd26e5c22.el9.x86_64 20/36 2026-02-24T15:41:06.446 INFO:teuthology.orchestra.run.trial119.stdout: Installing : boost-program-options-1.75.0-13.el9.x86_64 21/36 2026-02-24T15:41:06.693 INFO:teuthology.orchestra.run.trial119.stdout: Installing : parquet-libs-9.0.0-15.el9.x86_64 22/36 2026-02-24T15:41:06.695 INFO:teuthology.orchestra.run.trial119.stdout: Installing : librgw2-2:20.3.0-5343.gd26e5c22.el9.x86_64 23/36 2026-02-24T15:41:06.711 INFO:teuthology.orchestra.run.trial119.stdout: Running scriptlet: librgw2-2:20.3.0-5343.gd26e5c22.el9.x86_64 23/36 2026-02-24T15:41:06.715 INFO:teuthology.orchestra.run.trial119.stdout: Installing : python3-rgw-2:20.3.0-5343.gd26e5c22.el9.x86_64 24/36 2026-02-24T15:41:06.721 INFO:teuthology.orchestra.run.trial119.stdout: Installing : pciutils-3.7.0-7.el9.x86_64 25/36 2026-02-24T15:41:06.723 INFO:teuthology.orchestra.run.trial119.stdout: Installing : qatlib-25.08.0-2.el9.x86_64 26/36 2026-02-24T15:41:06.724 INFO:teuthology.orchestra.run.trial119.stdout: Installing : qatlib-service-25.08.0-2.el9.x86_64 27/36 2026-02-24T15:41:06.742 INFO:teuthology.orchestra.run.trial119.stdout: Running scriptlet: qatlib-service-25.08.0-2.el9.x86_64 27/36 2026-02-24T15:41:06.745 INFO:teuthology.orchestra.run.trial119.stdout: Installing : qatzip-libs-1.3.1-1.el9.x86_64 28/36 2026-02-24T15:41:06.781 INFO:teuthology.orchestra.run.trial119.stdout: Installing : mailcap-2.1.49-5.el9.noarch 29/36 2026-02-24T15:41:06.782 INFO:teuthology.orchestra.run.trial119.stdout: Installing : python3-ceph-common-2:20.3.0-5343.gd26e5c22.el9.x8 30/36 2026-02-24T15:41:07.722 INFO:teuthology.orchestra.run.trial119.stdout: Running scriptlet: ceph-common-2:20.3.0-5343.gd26e5c22.el9.x86_64 31/36 2026-02-24T15:41:07.724 INFO:teuthology.orchestra.run.trial119.stdout: Installing : ceph-common-2:20.3.0-5343.gd26e5c22.el9.x86_64 31/36 2026-02-24T15:41:07.985 INFO:teuthology.orchestra.run.trial119.stdout: Running scriptlet: ceph-common-2:20.3.0-5343.gd26e5c22.el9.x86_64 31/36 2026-02-24T15:41:07.988 INFO:teuthology.orchestra.run.trial119.stdout: Installing : ceph-base-2:20.3.0-5343.gd26e5c22.el9.x86_64 32/36 2026-02-24T15:41:08.029 INFO:teuthology.orchestra.run.trial119.stdout: Running scriptlet: ceph-base-2:20.3.0-5343.gd26e5c22.el9.x86_64 32/36 2026-02-24T15:41:08.029 INFO:teuthology.orchestra.run.trial119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2026-02-24T15:41:08.029 INFO:teuthology.orchestra.run.trial119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2026-02-24T15:41:08.029 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:41:08.032 INFO:teuthology.orchestra.run.trial119.stdout: Installing : ceph-selinux-2:20.3.0-5343.gd26e5c22.el9.x86_64 33/36 2026-02-24T15:41:12.608 INFO:teuthology.orchestra.run.trial119.stdout: Running scriptlet: ceph-selinux-2:20.3.0-5343.gd26e5c22.el9.x86_64 33/36 2026-02-24T15:41:12.609 INFO:teuthology.orchestra.run.trial119.stdout:skipping the directory /sys 2026-02-24T15:41:12.609 INFO:teuthology.orchestra.run.trial119.stdout:skipping the directory /proc 2026-02-24T15:41:12.609 INFO:teuthology.orchestra.run.trial119.stdout:skipping the directory /mnt 2026-02-24T15:41:12.609 INFO:teuthology.orchestra.run.trial119.stdout:skipping the directory /var/tmp 2026-02-24T15:41:12.609 INFO:teuthology.orchestra.run.trial119.stdout:skipping the directory /home 2026-02-24T15:41:12.609 INFO:teuthology.orchestra.run.trial119.stdout:skipping the directory /root 2026-02-24T15:41:12.609 INFO:teuthology.orchestra.run.trial119.stdout:skipping the directory /tmp 2026-02-24T15:41:12.609 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:41:13.107 INFO:teuthology.orchestra.run.trial119.stdout: Installing : ceph-radosgw-2:20.3.0-5343.gd26e5c22.el9.x86_64 34/36 2026-02-24T15:41:13.130 INFO:teuthology.orchestra.run.trial119.stdout: Running scriptlet: ceph-radosgw-2:20.3.0-5343.gd26e5c22.el9.x86_64 34/36 2026-02-24T15:41:13.130 INFO:teuthology.orchestra.run.trial119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T15:41:13.130 INFO:teuthology.orchestra.run.trial119.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2026-02-24T15:41:13.130 INFO:teuthology.orchestra.run.trial119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-24T15:41:13.130 INFO:teuthology.orchestra.run.trial119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-24T15:41:13.130 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:41:13.131 INFO:teuthology.orchestra.run.trial119.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 35/36 2026-02-24T15:41:13.142 INFO:teuthology.orchestra.run.trial119.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 35/36 2026-02-24T15:41:13.142 INFO:teuthology.orchestra.run.trial119.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 36/36 2026-02-24T15:41:13.198 INFO:teuthology.orchestra.run.trial119.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 36/36 2026-02-24T15:41:13.199 INFO:teuthology.orchestra.run.trial119.stdout:Creating group 'qat' with GID 992. 2026-02-24T15:41:13.199 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:41:13.489 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : ceph-base-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/36 2026-02-24T15:41:13.490 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : ceph-common-2:20.3.0-5343.gd26e5c22.el9.x86_64 2/36 2026-02-24T15:41:13.490 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : ceph-radosgw-2:20.3.0-5343.gd26e5c22.el9.x86_64 3/36 2026-02-24T15:41:13.490 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : ceph-selinux-2:20.3.0-5343.gd26e5c22.el9.x86_64 4/36 2026-02-24T15:41:13.490 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : libcephfs2-2:20.3.0-5343.gd26e5c22.el9.x86_64 5/36 2026-02-24T15:41:13.490 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : libradosstriper1-2:20.3.0-5343.gd26e5c22.el9.x86_6 6/36 2026-02-24T15:41:13.490 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : librgw2-2:20.3.0-5343.gd26e5c22.el9.x86_64 7/36 2026-02-24T15:41:13.490 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : python3-ceph-argparse-2:20.3.0-5343.gd26e5c22.el9. 8/36 2026-02-24T15:41:13.490 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : python3-ceph-common-2:20.3.0-5343.gd26e5c22.el9.x8 9/36 2026-02-24T15:41:13.490 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : python3-cephfs-2:20.3.0-5343.gd26e5c22.el9.x86_64 10/36 2026-02-24T15:41:13.491 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : python3-rados-2:20.3.0-5343.gd26e5c22.el9.x86_64 11/36 2026-02-24T15:41:13.491 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : python3-rbd-2:20.3.0-5343.gd26e5c22.el9.x86_64 12/36 2026-02-24T15:41:13.491 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : python3-rgw-2:20.3.0-5343.gd26e5c22.el9.x86_64 13/36 2026-02-24T15:41:13.491 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : mailcap-2.1.49-5.el9.noarch 14/36 2026-02-24T15:41:13.491 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : pciutils-3.7.0-7.el9.x86_64 15/36 2026-02-24T15:41:13.491 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : boost-program-options-1.75.0-13.el9.x86_64 16/36 2026-02-24T15:41:13.491 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : libnbd-1.20.3-4.el9.x86_64 17/36 2026-02-24T15:41:13.491 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 18/36 2026-02-24T15:41:13.491 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 19/36 2026-02-24T15:41:13.491 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 20/36 2026-02-24T15:41:13.491 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 21/36 2026-02-24T15:41:13.491 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : qatlib-25.08.0-2.el9.x86_64 22/36 2026-02-24T15:41:13.491 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : qatlib-service-25.08.0-2.el9.x86_64 23/36 2026-02-24T15:41:13.492 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : qatzip-libs-1.3.1-1.el9.x86_64 24/36 2026-02-24T15:41:13.492 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 25/36 2026-02-24T15:41:13.492 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : libarrow-9.0.0-15.el9.x86_64 26/36 2026-02-24T15:41:13.492 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : libarrow-doc-9.0.0-15.el9.noarch 27/36 2026-02-24T15:41:13.492 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : liboath-2.6.12-1.el9.x86_64 28/36 2026-02-24T15:41:13.492 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 29/36 2026-02-24T15:41:13.492 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : parquet-libs-9.0.0-15.el9.x86_64 30/36 2026-02-24T15:41:13.492 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : re2-1:20211101-20.el9.x86_64 31/36 2026-02-24T15:41:13.492 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : thrift-0.15.0-4.el9.x86_64 32/36 2026-02-24T15:41:13.492 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : librados2-2:20.3.0-5343.gd26e5c22.el9.x86_64 33/36 2026-02-24T15:41:13.492 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 34/36 2026-02-24T15:41:13.492 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : librbd1-2:20.3.0-5343.gd26e5c22.el9.x86_64 35/36 2026-02-24T15:41:13.555 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 36/36 2026-02-24T15:41:13.555 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:41:13.555 INFO:teuthology.orchestra.run.trial119.stdout:Upgraded: 2026-02-24T15:41:13.555 INFO:teuthology.orchestra.run.trial119.stdout: librados2-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:41:13.555 INFO:teuthology.orchestra.run.trial119.stdout: librbd1-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:41:13.555 INFO:teuthology.orchestra.run.trial119.stdout:Installed: 2026-02-24T15:41:13.555 INFO:teuthology.orchestra.run.trial119.stdout: boost-program-options-1.75.0-13.el9.x86_64 2026-02-24T15:41:13.555 INFO:teuthology.orchestra.run.trial119.stdout: ceph-base-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:41:13.555 INFO:teuthology.orchestra.run.trial119.stdout: ceph-common-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:41:13.555 INFO:teuthology.orchestra.run.trial119.stdout: ceph-radosgw-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:41:13.556 INFO:teuthology.orchestra.run.trial119.stdout: ceph-selinux-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:41:13.556 INFO:teuthology.orchestra.run.trial119.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2026-02-24T15:41:13.556 INFO:teuthology.orchestra.run.trial119.stdout: libarrow-9.0.0-15.el9.x86_64 2026-02-24T15:41:13.556 INFO:teuthology.orchestra.run.trial119.stdout: libarrow-doc-9.0.0-15.el9.noarch 2026-02-24T15:41:13.556 INFO:teuthology.orchestra.run.trial119.stdout: libcephfs2-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:41:13.556 INFO:teuthology.orchestra.run.trial119.stdout: libnbd-1.20.3-4.el9.x86_64 2026-02-24T15:41:13.556 INFO:teuthology.orchestra.run.trial119.stdout: liboath-2.6.12-1.el9.x86_64 2026-02-24T15:41:13.556 INFO:teuthology.orchestra.run.trial119.stdout: libpmemobj-1.12.1-1.el9.x86_64 2026-02-24T15:41:13.556 INFO:teuthology.orchestra.run.trial119.stdout: librabbitmq-0.11.0-7.el9.x86_64 2026-02-24T15:41:13.556 INFO:teuthology.orchestra.run.trial119.stdout: libradosstriper1-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:41:13.556 INFO:teuthology.orchestra.run.trial119.stdout: librdkafka-1.6.1-102.el9.x86_64 2026-02-24T15:41:13.556 INFO:teuthology.orchestra.run.trial119.stdout: librgw2-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:41:13.556 INFO:teuthology.orchestra.run.trial119.stdout: libunwind-1.6.2-1.el9.x86_64 2026-02-24T15:41:13.557 INFO:teuthology.orchestra.run.trial119.stdout: lttng-ust-2.12.0-6.el9.x86_64 2026-02-24T15:41:13.557 INFO:teuthology.orchestra.run.trial119.stdout: mailcap-2.1.49-5.el9.noarch 2026-02-24T15:41:13.557 INFO:teuthology.orchestra.run.trial119.stdout: parquet-libs-9.0.0-15.el9.x86_64 2026-02-24T15:41:13.557 INFO:teuthology.orchestra.run.trial119.stdout: pciutils-3.7.0-7.el9.x86_64 2026-02-24T15:41:13.557 INFO:teuthology.orchestra.run.trial119.stdout: python3-ceph-argparse-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:41:13.557 INFO:teuthology.orchestra.run.trial119.stdout: python3-ceph-common-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:41:13.557 INFO:teuthology.orchestra.run.trial119.stdout: python3-cephfs-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:41:13.557 INFO:teuthology.orchestra.run.trial119.stdout: python3-rados-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:41:13.557 INFO:teuthology.orchestra.run.trial119.stdout: python3-rbd-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:41:13.557 INFO:teuthology.orchestra.run.trial119.stdout: python3-rgw-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:41:13.557 INFO:teuthology.orchestra.run.trial119.stdout: qatlib-25.08.0-2.el9.x86_64 2026-02-24T15:41:13.557 INFO:teuthology.orchestra.run.trial119.stdout: qatlib-service-25.08.0-2.el9.x86_64 2026-02-24T15:41:13.558 INFO:teuthology.orchestra.run.trial119.stdout: qatzip-libs-1.3.1-1.el9.x86_64 2026-02-24T15:41:13.558 INFO:teuthology.orchestra.run.trial119.stdout: re2-1:20211101-20.el9.x86_64 2026-02-24T15:41:13.558 INFO:teuthology.orchestra.run.trial119.stdout: thrift-0.15.0-4.el9.x86_64 2026-02-24T15:41:13.558 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:41:13.558 INFO:teuthology.orchestra.run.trial119.stdout:Complete! 2026-02-24T15:41:13.596 DEBUG:teuthology.orchestra.run.trial119:> sudo yum -y install ceph-test 2026-02-24T15:41:13.909 INFO:teuthology.orchestra.run.trial119.stdout:Last metadata expiration check: 0:00:22 ago on Tue 24 Feb 2026 03:40:51 PM UTC. 2026-02-24T15:41:13.992 INFO:teuthology.orchestra.run.trial119.stdout:Dependencies resolved. 2026-02-24T15:41:13.993 INFO:teuthology.orchestra.run.trial119.stdout:================================================================================ 2026-02-24T15:41:13.993 INFO:teuthology.orchestra.run.trial119.stdout: Package Arch Version Repository Size 2026-02-24T15:41:13.993 INFO:teuthology.orchestra.run.trial119.stdout:================================================================================ 2026-02-24T15:41:13.993 INFO:teuthology.orchestra.run.trial119.stdout:Installing: 2026-02-24T15:41:13.993 INFO:teuthology.orchestra.run.trial119.stdout: ceph-test x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 126 M 2026-02-24T15:41:13.993 INFO:teuthology.orchestra.run.trial119.stdout:Installing dependencies: 2026-02-24T15:41:13.993 INFO:teuthology.orchestra.run.trial119.stdout: hwloc-libs x86_64 2.4.1-6.el9 baseos 2.1 M 2026-02-24T15:41:13.993 INFO:teuthology.orchestra.run.trial119.stdout: libasan8 x86_64 15.2.1-7.el9 appstream 525 k 2026-02-24T15:41:13.993 INFO:teuthology.orchestra.run.trial119.stdout: libatomic x86_64 11.5.0-14.el9 baseos 23 k 2026-02-24T15:41:13.994 INFO:teuthology.orchestra.run.trial119.stdout: libcephsqlite x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 153 k 2026-02-24T15:41:13.994 INFO:teuthology.orchestra.run.trial119.stdout: libubsan x86_64 11.5.0-14.el9 appstream 176 k 2026-02-24T15:41:13.994 INFO:teuthology.orchestra.run.trial119.stdout: libxslt x86_64 1.1.34-12.el9 appstream 233 k 2026-02-24T15:41:13.994 INFO:teuthology.orchestra.run.trial119.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2026-02-24T15:41:13.994 INFO:teuthology.orchestra.run.trial119.stdout: protobuf x86_64 3.14.0-17.el9 appstream 1.0 M 2026-02-24T15:41:13.994 INFO:teuthology.orchestra.run.trial119.stdout: socat x86_64 1.7.4.1-8.el9 appstream 303 k 2026-02-24T15:41:13.994 INFO:teuthology.orchestra.run.trial119.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2026-02-24T15:41:13.994 INFO:teuthology.orchestra.run.trial119.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2026-02-24T15:41:13.994 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:41:13.994 INFO:teuthology.orchestra.run.trial119.stdout:Transaction Summary 2026-02-24T15:41:13.994 INFO:teuthology.orchestra.run.trial119.stdout:================================================================================ 2026-02-24T15:41:13.994 INFO:teuthology.orchestra.run.trial119.stdout:Install 12 Packages 2026-02-24T15:41:13.994 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:41:13.995 INFO:teuthology.orchestra.run.trial119.stdout:Total download size: 131 M 2026-02-24T15:41:13.995 INFO:teuthology.orchestra.run.trial119.stdout:Installed size: 981 M 2026-02-24T15:41:13.995 INFO:teuthology.orchestra.run.trial119.stdout:Downloading Packages: 2026-02-24T15:41:14.606 INFO:teuthology.orchestra.run.trial119.stdout:(1/12): hwloc-libs-2.4.1-6.el9.x86_64.rpm 12 MB/s | 2.1 MB 00:00 2026-02-24T15:41:14.628 INFO:teuthology.orchestra.run.trial119.stdout:(2/12): libatomic-11.5.0-14.el9.x86_64.rpm 1.0 MB/s | 23 kB 00:00 2026-02-24T15:41:14.651 INFO:teuthology.orchestra.run.trial119.stdout:(3/12): lksctp-tools-1.0.19-2.el9.x86_64.rpm 4.1 MB/s | 94 kB 00:00 2026-02-24T15:41:14.837 INFO:teuthology.orchestra.run.trial119.stdout:(4/12): libasan8-15.2.1-7.el9.x86_64.rpm 2.7 MB/s | 525 kB 00:00 2026-02-24T15:41:14.867 INFO:teuthology.orchestra.run.trial119.stdout:(5/12): libubsan-11.5.0-14.el9.x86_64.rpm 5.9 MB/s | 176 kB 00:00 2026-02-24T15:41:14.898 INFO:teuthology.orchestra.run.trial119.stdout:(6/12): libxslt-1.1.34-12.el9.x86_64.rpm 7.4 MB/s | 233 kB 00:00 2026-02-24T15:41:14.910 INFO:teuthology.orchestra.run.trial119.stdout:(7/12): libcephsqlite-20.3.0-5343.gd26e5c22.el9 323 kB/s | 153 kB 00:00 2026-02-24T15:41:14.956 INFO:teuthology.orchestra.run.trial119.stdout:(8/12): protobuf-3.14.0-17.el9.x86_64.rpm 17 MB/s | 1.0 MB 00:00 2026-02-24T15:41:14.982 INFO:teuthology.orchestra.run.trial119.stdout:(9/12): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.4 MB/s | 64 kB 00:00 2026-02-24T15:41:15.076 INFO:teuthology.orchestra.run.trial119.stdout:(10/12): socat-1.7.4.1-8.el9.x86_64.rpm 1.8 MB/s | 303 kB 00:00 2026-02-24T15:41:15.369 INFO:teuthology.orchestra.run.trial119.stdout:(11/12): yaml-cpp-0.6.3-5.el9.x86_64.rpm 315 kB/s | 122 kB 00:00 2026-02-24T15:43:38.175 INFO:teuthology.orchestra.run.trial119.stdout:(12/12): ceph-test-20.3.0-5343.gd26e5c22.el9.x8 900 kB/s | 126 MB 02:23 2026-02-24T15:43:38.176 INFO:teuthology.orchestra.run.trial119.stdout:-------------------------------------------------------------------------------- 2026-02-24T15:43:38.176 INFO:teuthology.orchestra.run.trial119.stdout:Total 931 kB/s | 131 MB 02:24 2026-02-24T15:43:38.243 INFO:teuthology.orchestra.run.trial119.stdout:Running transaction check 2026-02-24T15:43:38.255 INFO:teuthology.orchestra.run.trial119.stdout:Transaction check succeeded. 2026-02-24T15:43:38.255 INFO:teuthology.orchestra.run.trial119.stdout:Running transaction test 2026-02-24T15:43:38.686 INFO:teuthology.orchestra.run.trial119.stdout:Transaction test succeeded. 2026-02-24T15:43:38.687 INFO:teuthology.orchestra.run.trial119.stdout:Running transaction 2026-02-24T15:43:39.146 INFO:teuthology.orchestra.run.trial119.stdout: Preparing : 1/1 2026-02-24T15:43:39.154 INFO:teuthology.orchestra.run.trial119.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/12 2026-02-24T15:43:39.163 INFO:teuthology.orchestra.run.trial119.stdout: Installing : socat-1.7.4.1-8.el9.x86_64 2/12 2026-02-24T15:43:39.168 INFO:teuthology.orchestra.run.trial119.stdout: Installing : protobuf-3.14.0-17.el9.x86_64 3/12 2026-02-24T15:43:39.172 INFO:teuthology.orchestra.run.trial119.stdout: Installing : libxslt-1.1.34-12.el9.x86_64 4/12 2026-02-24T15:43:39.174 INFO:teuthology.orchestra.run.trial119.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 5/12 2026-02-24T15:43:39.178 INFO:teuthology.orchestra.run.trial119.stdout: Installing : libubsan-11.5.0-14.el9.x86_64 6/12 2026-02-24T15:43:39.179 INFO:teuthology.orchestra.run.trial119.stdout: Installing : libasan8-15.2.1-7.el9.x86_64 7/12 2026-02-24T15:43:39.194 INFO:teuthology.orchestra.run.trial119.stdout: Running scriptlet: libasan8-15.2.1-7.el9.x86_64 7/12 2026-02-24T15:43:39.196 INFO:teuthology.orchestra.run.trial119.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 8/12 2026-02-24T15:43:39.206 INFO:teuthology.orchestra.run.trial119.stdout: Installing : libatomic-11.5.0-14.el9.x86_64 9/12 2026-02-24T15:43:39.214 INFO:teuthology.orchestra.run.trial119.stdout: Installing : hwloc-libs-2.4.1-6.el9.x86_64 10/12 2026-02-24T15:43:39.215 INFO:teuthology.orchestra.run.trial119.stdout: Installing : libcephsqlite-2:20.3.0-5343.gd26e5c22.el9.x86_64 11/12 2026-02-24T15:43:46.098 INFO:teuthology.orchestra.run.trial119.stdout: Running scriptlet: libcephsqlite-2:20.3.0-5343.gd26e5c22.el9.x86_64 11/12 2026-02-24T15:43:46.117 INFO:teuthology.orchestra.run.trial119.stdout: Installing : ceph-test-2:20.3.0-5343.gd26e5c22.el9.x86_64 12/12 2026-02-24T15:43:46.884 INFO:teuthology.orchestra.run.trial119.stdout: Running scriptlet: ceph-test-2:20.3.0-5343.gd26e5c22.el9.x86_64 12/12 2026-02-24T15:43:46.884 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : ceph-test-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/12 2026-02-24T15:43:46.884 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : libcephsqlite-2:20.3.0-5343.gd26e5c22.el9.x86_64 2/12 2026-02-24T15:43:46.884 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : hwloc-libs-2.4.1-6.el9.x86_64 3/12 2026-02-24T15:43:46.884 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : libatomic-11.5.0-14.el9.x86_64 4/12 2026-02-24T15:43:46.884 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 5/12 2026-02-24T15:43:46.884 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : libasan8-15.2.1-7.el9.x86_64 6/12 2026-02-24T15:43:46.884 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : libubsan-11.5.0-14.el9.x86_64 7/12 2026-02-24T15:43:46.884 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : libxslt-1.1.34-12.el9.x86_64 8/12 2026-02-24T15:43:46.884 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : protobuf-3.14.0-17.el9.x86_64 9/12 2026-02-24T15:43:46.884 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : socat-1.7.4.1-8.el9.x86_64 10/12 2026-02-24T15:43:46.884 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 11/12 2026-02-24T15:43:46.945 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 12/12 2026-02-24T15:43:46.945 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:43:46.945 INFO:teuthology.orchestra.run.trial119.stdout:Installed: 2026-02-24T15:43:46.945 INFO:teuthology.orchestra.run.trial119.stdout: ceph-test-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:43:46.945 INFO:teuthology.orchestra.run.trial119.stdout: hwloc-libs-2.4.1-6.el9.x86_64 2026-02-24T15:43:46.945 INFO:teuthology.orchestra.run.trial119.stdout: libasan8-15.2.1-7.el9.x86_64 2026-02-24T15:43:46.946 INFO:teuthology.orchestra.run.trial119.stdout: libatomic-11.5.0-14.el9.x86_64 2026-02-24T15:43:46.946 INFO:teuthology.orchestra.run.trial119.stdout: libcephsqlite-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:43:46.946 INFO:teuthology.orchestra.run.trial119.stdout: libubsan-11.5.0-14.el9.x86_64 2026-02-24T15:43:46.946 INFO:teuthology.orchestra.run.trial119.stdout: libxslt-1.1.34-12.el9.x86_64 2026-02-24T15:43:46.946 INFO:teuthology.orchestra.run.trial119.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2026-02-24T15:43:46.946 INFO:teuthology.orchestra.run.trial119.stdout: protobuf-3.14.0-17.el9.x86_64 2026-02-24T15:43:46.946 INFO:teuthology.orchestra.run.trial119.stdout: socat-1.7.4.1-8.el9.x86_64 2026-02-24T15:43:46.946 INFO:teuthology.orchestra.run.trial119.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2026-02-24T15:43:46.946 INFO:teuthology.orchestra.run.trial119.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2026-02-24T15:43:46.946 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:43:46.946 INFO:teuthology.orchestra.run.trial119.stdout:Complete! 2026-02-24T15:43:46.987 DEBUG:teuthology.orchestra.run.trial119:> sudo yum -y install ceph 2026-02-24T15:43:47.302 INFO:teuthology.orchestra.run.trial119.stdout:Last metadata expiration check: 0:02:56 ago on Tue 24 Feb 2026 03:40:51 PM UTC. 2026-02-24T15:43:47.386 INFO:teuthology.orchestra.run.trial119.stdout:Dependencies resolved. 2026-02-24T15:43:47.388 INFO:teuthology.orchestra.run.trial119.stdout:================================================================================ 2026-02-24T15:43:47.388 INFO:teuthology.orchestra.run.trial119.stdout: Package Arch Version Repository Size 2026-02-24T15:43:47.388 INFO:teuthology.orchestra.run.trial119.stdout:================================================================================ 2026-02-24T15:43:47.388 INFO:teuthology.orchestra.run.trial119.stdout:Installing: 2026-02-24T15:43:47.388 INFO:teuthology.orchestra.run.trial119.stdout: ceph x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 6.5 k 2026-02-24T15:43:47.388 INFO:teuthology.orchestra.run.trial119.stdout:Installing dependencies: 2026-02-24T15:43:47.388 INFO:teuthology.orchestra.run.trial119.stdout: ceph-mds x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 2.0 M 2026-02-24T15:43:47.388 INFO:teuthology.orchestra.run.trial119.stdout: ceph-mgr x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 937 k 2026-02-24T15:43:47.388 INFO:teuthology.orchestra.run.trial119.stdout: ceph-mgr-modules-core noarch 2:20.3.0-5343.gd26e5c22.el9 ceph-noarch 297 k 2026-02-24T15:43:47.388 INFO:teuthology.orchestra.run.trial119.stdout: ceph-mon x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 4.9 M 2026-02-24T15:43:47.388 INFO:teuthology.orchestra.run.trial119.stdout: ceph-osd x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 163 k 2026-02-24T15:43:47.388 INFO:teuthology.orchestra.run.trial119.stdout: ceph-osd-classic x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 15 M 2026-02-24T15:43:47.388 INFO:teuthology.orchestra.run.trial119.stdout: ceph-osd-crimson x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 88 M 2026-02-24T15:43:47.388 INFO:teuthology.orchestra.run.trial119.stdout: ledmon-libs x86_64 1.1.0-3.el9 baseos 40 k 2026-02-24T15:43:47.389 INFO:teuthology.orchestra.run.trial119.stdout: libconfig x86_64 1.7.2-9.el9 baseos 72 k 2026-02-24T15:43:47.389 INFO:teuthology.orchestra.run.trial119.stdout: libstoragemgmt x86_64 1.10.1-1.el9 appstream 246 k 2026-02-24T15:43:47.389 INFO:teuthology.orchestra.run.trial119.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2026-02-24T15:43:47.389 INFO:teuthology.orchestra.run.trial119.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2026-02-24T15:43:47.389 INFO:teuthology.orchestra.run.trial119.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2026-02-24T15:43:47.389 INFO:teuthology.orchestra.run.trial119.stdout: python3-autocommand noarch 2.2.2-8.el9 epel 29 k 2026-02-24T15:43:47.389 INFO:teuthology.orchestra.run.trial119.stdout: python3-backports-tarfile noarch 1.2.0-1.el9 epel 60 k 2026-02-24T15:43:47.389 INFO:teuthology.orchestra.run.trial119.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2026-02-24T15:43:47.389 INFO:teuthology.orchestra.run.trial119.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2026-02-24T15:43:47.389 INFO:teuthology.orchestra.run.trial119.stdout: python3-cheroot noarch 10.0.1-4.el9 epel 173 k 2026-02-24T15:43:47.389 INFO:teuthology.orchestra.run.trial119.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2026-02-24T15:43:47.389 INFO:teuthology.orchestra.run.trial119.stdout: python3-cryptography x86_64 36.0.1-5.el9 baseos 1.2 M 2026-02-24T15:43:47.389 INFO:teuthology.orchestra.run.trial119.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2026-02-24T15:43:47.389 INFO:teuthology.orchestra.run.trial119.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2026-02-24T15:43:47.389 INFO:teuthology.orchestra.run.trial119.stdout: python3-jaraco-collections 2026-02-24T15:43:47.389 INFO:teuthology.orchestra.run.trial119.stdout: noarch 3.0.0-8.el9 epel 23 k 2026-02-24T15:43:47.389 INFO:teuthology.orchestra.run.trial119.stdout: python3-jaraco-context noarch 6.0.1-3.el9 epel 20 k 2026-02-24T15:43:47.389 INFO:teuthology.orchestra.run.trial119.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2026-02-24T15:43:47.389 INFO:teuthology.orchestra.run.trial119.stdout: python3-jaraco-text noarch 4.0.0-2.el9 epel 26 k 2026-02-24T15:43:47.389 INFO:teuthology.orchestra.run.trial119.stdout: python3-libstoragemgmt x86_64 1.10.1-1.el9 appstream 177 k 2026-02-24T15:43:47.389 INFO:teuthology.orchestra.run.trial119.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2026-02-24T15:43:47.389 INFO:teuthology.orchestra.run.trial119.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2026-02-24T15:43:47.389 INFO:teuthology.orchestra.run.trial119.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2026-02-24T15:43:47.389 INFO:teuthology.orchestra.run.trial119.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2026-02-24T15:43:47.390 INFO:teuthology.orchestra.run.trial119.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2026-02-24T15:43:47.390 INFO:teuthology.orchestra.run.trial119.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2026-02-24T15:43:47.390 INFO:teuthology.orchestra.run.trial119.stdout: python3-pyparsing noarch 2.4.7-9.el9 baseos 150 k 2026-02-24T15:43:47.390 INFO:teuthology.orchestra.run.trial119.stdout: python3-requests noarch 2.25.1-10.el9 baseos 126 k 2026-02-24T15:43:47.390 INFO:teuthology.orchestra.run.trial119.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2026-02-24T15:43:47.390 INFO:teuthology.orchestra.run.trial119.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2026-02-24T15:43:47.390 INFO:teuthology.orchestra.run.trial119.stdout: python3-urllib3 noarch 1.26.5-7.el9 baseos 218 k 2026-02-24T15:43:47.390 INFO:teuthology.orchestra.run.trial119.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2026-02-24T15:43:47.390 INFO:teuthology.orchestra.run.trial119.stdout: unzip x86_64 6.0-59.el9 baseos 182 k 2026-02-24T15:43:47.390 INFO:teuthology.orchestra.run.trial119.stdout: zip x86_64 3.0-35.el9 baseos 266 k 2026-02-24T15:43:47.390 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:43:47.390 INFO:teuthology.orchestra.run.trial119.stdout:Transaction Summary 2026-02-24T15:43:47.390 INFO:teuthology.orchestra.run.trial119.stdout:================================================================================ 2026-02-24T15:43:47.390 INFO:teuthology.orchestra.run.trial119.stdout:Install 42 Packages 2026-02-24T15:43:47.390 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:43:47.390 INFO:teuthology.orchestra.run.trial119.stdout:Total download size: 115 M 2026-02-24T15:43:47.391 INFO:teuthology.orchestra.run.trial119.stdout:Installed size: 974 M 2026-02-24T15:43:47.391 INFO:teuthology.orchestra.run.trial119.stdout:Downloading Packages: 2026-02-24T15:43:48.440 INFO:teuthology.orchestra.run.trial119.stdout:(1/42): ceph-20.3.0-5343.gd26e5c22.el9.x86_64.r 24 kB/s | 6.5 kB 00:00 2026-02-24T15:43:49.423 INFO:teuthology.orchestra.run.trial119.stdout:(2/42): ceph-mgr-20.3.0-5343.gd26e5c22.el9.x86_ 746 kB/s | 937 kB 00:01 2026-02-24T15:43:49.559 INFO:teuthology.orchestra.run.trial119.stdout:(3/42): ceph-osd-20.3.0-5343.gd26e5c22.el9.x86_ 1.2 MB/s | 163 kB 00:00 2026-02-24T15:43:50.824 INFO:teuthology.orchestra.run.trial119.stdout:(4/42): ceph-mds-20.3.0-5343.gd26e5c22.el9.x86_ 779 kB/s | 2.0 MB 00:02 2026-02-24T15:43:51.060 INFO:teuthology.orchestra.run.trial119.stdout:(5/42): ceph-mon-20.3.0-5343.gd26e5c22.el9.x86_ 1.9 MB/s | 4.9 MB 00:02 2026-02-24T15:43:51.191 INFO:teuthology.orchestra.run.trial119.stdout:(6/42): ceph-mgr-modules-core-20.3.0-5343.gd26e 2.2 MB/s | 297 kB 00:00 2026-02-24T15:43:51.529 INFO:teuthology.orchestra.run.trial119.stdout:[MIRROR] ledmon-libs-1.1.0-3.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/ledmon-libs-1.1.0-3.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T15:43:51.870 INFO:teuthology.orchestra.run.trial119.stdout:[MIRROR] ledmon-libs-1.1.0-3.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/ledmon-libs-1.1.0-3.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T15:43:52.242 INFO:teuthology.orchestra.run.trial119.stdout:[MIRROR] ledmon-libs-1.1.0-3.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/ledmon-libs-1.1.0-3.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T15:43:52.646 INFO:teuthology.orchestra.run.trial119.stdout:[MIRROR] ledmon-libs-1.1.0-3.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/ledmon-libs-1.1.0-3.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T15:43:52.924 INFO:teuthology.orchestra.run.trial119.stdout:(7/42): ledmon-libs-1.1.0-3.el9.x86_64.rpm 23 kB/s | 40 kB 00:01 2026-02-24T15:43:53.248 INFO:teuthology.orchestra.run.trial119.stdout:[MIRROR] libconfig-1.7.2-9.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/libconfig-1.7.2-9.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T15:43:53.571 INFO:teuthology.orchestra.run.trial119.stdout:[MIRROR] libconfig-1.7.2-9.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/libconfig-1.7.2-9.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T15:43:53.913 INFO:teuthology.orchestra.run.trial119.stdout:[MIRROR] libconfig-1.7.2-9.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/libconfig-1.7.2-9.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T15:43:54.253 INFO:teuthology.orchestra.run.trial119.stdout:[MIRROR] libconfig-1.7.2-9.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/libconfig-1.7.2-9.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T15:43:54.444 INFO:teuthology.orchestra.run.trial119.stdout:(8/42): libconfig-1.7.2-9.el9.x86_64.rpm 47 kB/s | 72 kB 00:01 2026-02-24T15:43:54.769 INFO:teuthology.orchestra.run.trial119.stdout:[MIRROR] python3-cffi-1.14.5-5.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/python3-cffi-1.14.5-5.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-24T15:43:55.480 INFO:teuthology.orchestra.run.trial119.stdout:(9/42): python3-cffi-1.14.5-5.el9.x86_64.rpm 244 kB/s | 253 kB 00:01 2026-02-24T15:43:56.977 INFO:teuthology.orchestra.run.trial119.stdout:(10/42): python3-cryptography-36.0.1-5.el9.x86_ 852 kB/s | 1.2 MB 00:01 2026-02-24T15:43:57.147 INFO:teuthology.orchestra.run.trial119.stdout:(11/42): python3-ply-3.11-14.el9.noarch.rpm 628 kB/s | 106 kB 00:00 2026-02-24T15:43:57.313 INFO:teuthology.orchestra.run.trial119.stdout:(12/42): python3-pycparser-2.20-6.el9.noarch.rp 812 kB/s | 135 kB 00:00 2026-02-24T15:43:57.563 INFO:teuthology.orchestra.run.trial119.stdout:(13/42): python3-pyparsing-2.4.7-9.el9.noarch.r 603 kB/s | 150 kB 00:00 2026-02-24T15:43:57.728 INFO:teuthology.orchestra.run.trial119.stdout:(14/42): python3-requests-2.25.1-10.el9.noarch. 763 kB/s | 126 kB 00:00 2026-02-24T15:43:58.050 INFO:teuthology.orchestra.run.trial119.stdout:(15/42): python3-urllib3-1.26.5-7.el9.noarch.rp 678 kB/s | 218 kB 00:00 2026-02-24T15:43:58.287 INFO:teuthology.orchestra.run.trial119.stdout:(16/42): unzip-6.0-59.el9.x86_64.rpm 768 kB/s | 182 kB 00:00 2026-02-24T15:43:58.601 INFO:teuthology.orchestra.run.trial119.stdout:(17/42): zip-3.0-35.el9.x86_64.rpm 845 kB/s | 266 kB 00:00 2026-02-24T15:43:58.651 INFO:teuthology.orchestra.run.trial119.stdout:(18/42): libstoragemgmt-1.10.1-1.el9.x86_64.rpm 4.8 MB/s | 246 kB 00:00 2026-02-24T15:43:58.663 INFO:teuthology.orchestra.run.trial119.stdout:(19/42): lua-5.4.4-4.el9.x86_64.rpm 15 MB/s | 188 kB 00:00 2026-02-24T15:43:58.675 INFO:teuthology.orchestra.run.trial119.stdout:(20/42): python3-libstoragemgmt-1.10.1-1.el9.x8 15 MB/s | 177 kB 00:00 2026-02-24T15:43:58.685 INFO:teuthology.orchestra.run.trial119.stdout:(21/42): python3-packaging-20.9-5.el9.noarch.rp 7.0 MB/s | 77 kB 00:00 2026-02-24T15:43:58.695 INFO:teuthology.orchestra.run.trial119.stdout:(22/42): python3-toml-0.10.2-6.el9.noarch.rpm 4.2 MB/s | 42 kB 00:00 2026-02-24T15:43:58.771 INFO:teuthology.orchestra.run.trial119.stdout:(23/42): lua-devel-5.4.4-4.el9.x86_64.rpm 294 kB/s | 22 kB 00:00 2026-02-24T15:43:58.816 INFO:teuthology.orchestra.run.trial119.stdout:(24/42): luarocks-3.9.2-5.el9.noarch.rpm 3.3 MB/s | 151 kB 00:00 2026-02-24T15:43:58.824 INFO:teuthology.orchestra.run.trial119.stdout:(25/42): python3-autocommand-2.2.2-8.el9.noarch 3.6 MB/s | 29 kB 00:00 2026-02-24T15:43:58.833 INFO:teuthology.orchestra.run.trial119.stdout:(26/42): python3-backports-tarfile-1.2.0-1.el9. 7.0 MB/s | 60 kB 00:00 2026-02-24T15:43:58.841 INFO:teuthology.orchestra.run.trial119.stdout:(27/42): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 5.2 MB/s | 43 kB 00:00 2026-02-24T15:43:58.858 INFO:teuthology.orchestra.run.trial119.stdout:(28/42): python3-cheroot-10.0.1-4.el9.noarch.rp 9.9 MB/s | 173 kB 00:00 2026-02-24T15:43:58.879 INFO:teuthology.orchestra.run.trial119.stdout:(29/42): python3-cherrypy-18.6.1-2.el9.noarch.r 17 MB/s | 358 kB 00:00 2026-02-24T15:43:58.887 INFO:teuthology.orchestra.run.trial119.stdout:(30/42): python3-jaraco-8.2.1-3.el9.noarch.rpm 1.4 MB/s | 11 kB 00:00 2026-02-24T15:43:58.894 INFO:teuthology.orchestra.run.trial119.stdout:(31/42): python3-jaraco-classes-3.2.1-5.el9.noa 2.3 MB/s | 18 kB 00:00 2026-02-24T15:43:58.905 INFO:teuthology.orchestra.run.trial119.stdout:(32/42): python3-jaraco-collections-3.0.0-8.el9 2.1 MB/s | 23 kB 00:00 2026-02-24T15:43:58.913 INFO:teuthology.orchestra.run.trial119.stdout:(33/42): python3-jaraco-context-6.0.1-3.el9.noa 2.4 MB/s | 20 kB 00:00 2026-02-24T15:43:58.921 INFO:teuthology.orchestra.run.trial119.stdout:(34/42): python3-jaraco-functools-3.5.0-2.el9.n 2.5 MB/s | 19 kB 00:00 2026-02-24T15:43:58.929 INFO:teuthology.orchestra.run.trial119.stdout:(35/42): python3-jaraco-text-4.0.0-2.el9.noarch 3.3 MB/s | 26 kB 00:00 2026-02-24T15:43:58.940 INFO:teuthology.orchestra.run.trial119.stdout:(36/42): python3-more-itertools-8.12.0-2.el9.no 7.0 MB/s | 79 kB 00:00 2026-02-24T15:43:58.948 INFO:teuthology.orchestra.run.trial119.stdout:(37/42): python3-portend-3.1.0-2.el9.noarch.rpm 2.1 MB/s | 16 kB 00:00 2026-02-24T15:43:58.957 INFO:teuthology.orchestra.run.trial119.stdout:(38/42): python3-pyOpenSSL-21.0.0-1.el9.noarch. 9.9 MB/s | 90 kB 00:00 2026-02-24T15:43:58.966 INFO:teuthology.orchestra.run.trial119.stdout:(39/42): python3-tempora-5.0.0-2.el9.noarch.rpm 4.2 MB/s | 36 kB 00:00 2026-02-24T15:43:58.974 INFO:teuthology.orchestra.run.trial119.stdout:(40/42): python3-zc-lockfile-2.0-10.el9.noarch. 2.5 MB/s | 20 kB 00:00 2026-02-24T15:44:04.976 INFO:teuthology.orchestra.run.trial119.stdout:(41/42): ceph-osd-classic-20.3.0-5343.gd26e5c22 981 kB/s | 15 MB 00:15 2026-02-24T15:46:47.351 INFO:teuthology.orchestra.run.trial119.stdout:(42/42): ceph-osd-crimson-20.3.0-5343.gd26e5c22 509 kB/s | 88 MB 02:56 2026-02-24T15:46:47.354 INFO:teuthology.orchestra.run.trial119.stdout:-------------------------------------------------------------------------------- 2026-02-24T15:46:47.354 INFO:teuthology.orchestra.run.trial119.stdout:Total 657 kB/s | 115 MB 02:59 2026-02-24T15:46:47.472 INFO:teuthology.orchestra.run.trial119.stdout:Running transaction check 2026-02-24T15:46:47.490 INFO:teuthology.orchestra.run.trial119.stdout:Transaction check succeeded. 2026-02-24T15:46:47.490 INFO:teuthology.orchestra.run.trial119.stdout:Running transaction test 2026-02-24T15:46:47.905 INFO:teuthology.orchestra.run.trial119.stdout:Transaction test succeeded. 2026-02-24T15:46:47.905 INFO:teuthology.orchestra.run.trial119.stdout:Running transaction 2026-02-24T15:46:48.335 INFO:teuthology.orchestra.run.trial119.stdout: Preparing : 1/1 2026-02-24T15:46:48.337 INFO:teuthology.orchestra.run.trial119.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/42 2026-02-24T15:46:48.342 INFO:teuthology.orchestra.run.trial119.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/42 2026-02-24T15:46:48.347 INFO:teuthology.orchestra.run.trial119.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/42 2026-02-24T15:46:48.349 INFO:teuthology.orchestra.run.trial119.stdout: Installing : unzip-6.0-59.el9.x86_64 4/42 2026-02-24T15:46:48.355 INFO:teuthology.orchestra.run.trial119.stdout: Installing : ledmon-libs-1.1.0-3.el9.x86_64 5/42 2026-02-24T15:46:48.379 INFO:teuthology.orchestra.run.trial119.stdout: Installing : zip-3.0-35.el9.x86_64 6/42 2026-02-24T15:46:48.383 INFO:teuthology.orchestra.run.trial119.stdout: Installing : luarocks-3.9.2-5.el9.noarch 7/42 2026-02-24T15:46:48.388 INFO:teuthology.orchestra.run.trial119.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 8/42 2026-02-24T15:46:48.393 INFO:teuthology.orchestra.run.trial119.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 9/42 2026-02-24T15:46:48.399 INFO:teuthology.orchestra.run.trial119.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 10/42 2026-02-24T15:46:48.403 INFO:teuthology.orchestra.run.trial119.stdout: Installing : python3-backports-tarfile-1.2.0-1.el9.noarch 11/42 2026-02-24T15:46:48.409 INFO:teuthology.orchestra.run.trial119.stdout: Installing : python3-jaraco-context-6.0.1-3.el9.noarch 12/42 2026-02-24T15:46:48.414 INFO:teuthology.orchestra.run.trial119.stdout: Installing : python3-autocommand-2.2.2-8.el9.noarch 13/42 2026-02-24T15:46:48.418 INFO:teuthology.orchestra.run.trial119.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 14/42 2026-02-24T15:46:48.423 INFO:teuthology.orchestra.run.trial119.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 15/42 2026-02-24T15:46:48.427 INFO:teuthology.orchestra.run.trial119.stdout: Installing : python3-jaraco-text-4.0.0-2.el9.noarch 16/42 2026-02-24T15:46:48.434 INFO:teuthology.orchestra.run.trial119.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 17/42 2026-02-24T15:46:48.437 INFO:teuthology.orchestra.run.trial119.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 18/42 2026-02-24T15:46:48.460 INFO:teuthology.orchestra.run.trial119.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 19/42 2026-02-24T15:46:48.473 INFO:teuthology.orchestra.run.trial119.stdout: Installing : python3-urllib3-1.26.5-7.el9.noarch 20/42 2026-02-24T15:46:48.477 INFO:teuthology.orchestra.run.trial119.stdout: Installing : python3-requests-2.25.1-10.el9.noarch 21/42 2026-02-24T15:46:48.487 INFO:teuthology.orchestra.run.trial119.stdout: Installing : python3-pyparsing-2.4.7-9.el9.noarch 22/42 2026-02-24T15:46:48.494 INFO:teuthology.orchestra.run.trial119.stdout: Installing : python3-packaging-20.9-5.el9.noarch 23/42 2026-02-24T15:46:48.507 INFO:teuthology.orchestra.run.trial119.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/42 2026-02-24T15:46:48.523 INFO:teuthology.orchestra.run.trial119.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/42 2026-02-24T15:46:48.598 INFO:teuthology.orchestra.run.trial119.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/42 2026-02-24T15:46:48.606 INFO:teuthology.orchestra.run.trial119.stdout: Installing : python3-cryptography-36.0.1-5.el9.x86_64 27/42 2026-02-24T15:46:48.629 INFO:teuthology.orchestra.run.trial119.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/42 2026-02-24T15:46:48.659 INFO:teuthology.orchestra.run.trial119.stdout: Installing : python3-cheroot-10.0.1-4.el9.noarch 29/42 2026-02-24T15:46:48.665 INFO:teuthology.orchestra.run.trial119.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/42 2026-02-24T15:46:48.717 INFO:teuthology.orchestra.run.trial119.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/42 2026-02-24T15:46:48.757 INFO:teuthology.orchestra.run.trial119.stdout: Installing : ceph-mgr-modules-core-2:20.3.0-5343.gd26e5c22.el9. 32/42 2026-02-24T15:46:48.758 INFO:teuthology.orchestra.run.trial119.stdout: Installing : ceph-mgr-2:20.3.0-5343.gd26e5c22.el9.x86_64 33/42 2026-02-24T15:46:48.781 INFO:teuthology.orchestra.run.trial119.stdout: Running scriptlet: ceph-mgr-2:20.3.0-5343.gd26e5c22.el9.x86_64 33/42 2026-02-24T15:46:48.781 INFO:teuthology.orchestra.run.trial119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T15:46:48.781 INFO:teuthology.orchestra.run.trial119.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2026-02-24T15:46:48.781 INFO:teuthology.orchestra.run.trial119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-24T15:46:48.781 INFO:teuthology.orchestra.run.trial119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-24T15:46:48.781 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:46:48.786 INFO:teuthology.orchestra.run.trial119.stdout: Installing : libconfig-1.7.2-9.el9.x86_64 34/42 2026-02-24T15:46:48.802 INFO:teuthology.orchestra.run.trial119.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 35/42 2026-02-24T15:46:48.802 INFO:teuthology.orchestra.run.trial119.stdout:Creating group 'libstoragemgmt' with GID 991. 2026-02-24T15:46:48.802 INFO:teuthology.orchestra.run.trial119.stdout:Creating user 'libstoragemgmt' (daemon account for libstoragemgmt) with UID 991 and GID 991. 2026-02-24T15:46:48.802 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:46:48.809 INFO:teuthology.orchestra.run.trial119.stdout: Installing : libstoragemgmt-1.10.1-1.el9.x86_64 35/42 2026-02-24T15:46:48.832 INFO:teuthology.orchestra.run.trial119.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 35/42 2026-02-24T15:46:48.832 INFO:teuthology.orchestra.run.trial119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/libstoragemgmt.service → /usr/lib/systemd/system/libstoragemgmt.service. 2026-02-24T15:46:48.832 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:46:49.428 INFO:teuthology.orchestra.run.trial119.stdout: Installing : python3-libstoragemgmt-1.10.1-1.el9.x86_64 36/42 2026-02-24T15:46:54.860 INFO:teuthology.orchestra.run.trial119.stdout: Installing : ceph-osd-classic-2:20.3.0-5343.gd26e5c22.el9.x86_6 37/42 2026-02-24T15:46:54.870 INFO:teuthology.orchestra.run.trial119.stdout: Installing : ceph-osd-crimson-2:20.3.0-5343.gd26e5c22.el9.x86_6 38/42 2026-02-24T15:46:54.871 INFO:teuthology.orchestra.run.trial119.stdout: Installing : ceph-osd-2:20.3.0-5343.gd26e5c22.el9.x86_64 39/42 2026-02-24T15:46:54.895 INFO:teuthology.orchestra.run.trial119.stdout: Running scriptlet: ceph-osd-2:20.3.0-5343.gd26e5c22.el9.x86_64 39/42 2026-02-24T15:46:54.895 INFO:teuthology.orchestra.run.trial119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T15:46:54.895 INFO:teuthology.orchestra.run.trial119.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2026-02-24T15:46:54.895 INFO:teuthology.orchestra.run.trial119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-24T15:46:54.895 INFO:teuthology.orchestra.run.trial119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-24T15:46:54.895 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:46:55.102 INFO:teuthology.orchestra.run.trial119.stdout: Installing : ceph-mon-2:20.3.0-5343.gd26e5c22.el9.x86_64 40/42 2026-02-24T15:46:55.123 INFO:teuthology.orchestra.run.trial119.stdout: Running scriptlet: ceph-mon-2:20.3.0-5343.gd26e5c22.el9.x86_64 40/42 2026-02-24T15:46:55.123 INFO:teuthology.orchestra.run.trial119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T15:46:55.123 INFO:teuthology.orchestra.run.trial119.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2026-02-24T15:46:55.124 INFO:teuthology.orchestra.run.trial119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-24T15:46:55.124 INFO:teuthology.orchestra.run.trial119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-24T15:46:55.124 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:46:55.206 INFO:teuthology.orchestra.run.trial119.stdout: Installing : ceph-mds-2:20.3.0-5343.gd26e5c22.el9.x86_64 41/42 2026-02-24T15:46:55.226 INFO:teuthology.orchestra.run.trial119.stdout: Running scriptlet: ceph-mds-2:20.3.0-5343.gd26e5c22.el9.x86_64 41/42 2026-02-24T15:46:55.227 INFO:teuthology.orchestra.run.trial119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T15:46:55.227 INFO:teuthology.orchestra.run.trial119.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2026-02-24T15:46:55.227 INFO:teuthology.orchestra.run.trial119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-24T15:46:55.227 INFO:teuthology.orchestra.run.trial119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-24T15:46:55.227 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:46:55.228 INFO:teuthology.orchestra.run.trial119.stdout: Installing : ceph-2:20.3.0-5343.gd26e5c22.el9.x86_64 42/42 2026-02-24T15:46:55.233 INFO:teuthology.orchestra.run.trial119.stdout: Running scriptlet: ceph-osd-classic-2:20.3.0-5343.gd26e5c22.el9.x86_6 42/42 2026-02-24T15:46:55.256 INFO:teuthology.orchestra.run.trial119.stdout: Running scriptlet: ceph-osd-crimson-2:20.3.0-5343.gd26e5c22.el9.x86_6 42/42 2026-02-24T15:46:56.221 INFO:teuthology.orchestra.run.trial119.stdout: Running scriptlet: ceph-2:20.3.0-5343.gd26e5c22.el9.x86_64 42/42 2026-02-24T15:46:56.221 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : ceph-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/42 2026-02-24T15:46:56.221 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : ceph-mds-2:20.3.0-5343.gd26e5c22.el9.x86_64 2/42 2026-02-24T15:46:56.221 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : ceph-mgr-2:20.3.0-5343.gd26e5c22.el9.x86_64 3/42 2026-02-24T15:46:56.221 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : ceph-mon-2:20.3.0-5343.gd26e5c22.el9.x86_64 4/42 2026-02-24T15:46:56.221 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : ceph-osd-2:20.3.0-5343.gd26e5c22.el9.x86_64 5/42 2026-02-24T15:46:56.221 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : ceph-osd-classic-2:20.3.0-5343.gd26e5c22.el9.x86_6 6/42 2026-02-24T15:46:56.221 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : ceph-osd-crimson-2:20.3.0-5343.gd26e5c22.el9.x86_6 7/42 2026-02-24T15:46:56.221 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : ceph-mgr-modules-core-2:20.3.0-5343.gd26e5c22.el9. 8/42 2026-02-24T15:46:56.221 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : ledmon-libs-1.1.0-3.el9.x86_64 9/42 2026-02-24T15:46:56.221 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : libconfig-1.7.2-9.el9.x86_64 10/42 2026-02-24T15:46:56.221 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/42 2026-02-24T15:46:56.221 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : python3-cryptography-36.0.1-5.el9.x86_64 12/42 2026-02-24T15:46:56.221 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/42 2026-02-24T15:46:56.221 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/42 2026-02-24T15:46:56.221 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : python3-pyparsing-2.4.7-9.el9.noarch 15/42 2026-02-24T15:46:56.222 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : python3-requests-2.25.1-10.el9.noarch 16/42 2026-02-24T15:46:56.222 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : python3-urllib3-1.26.5-7.el9.noarch 17/42 2026-02-24T15:46:56.222 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : unzip-6.0-59.el9.x86_64 18/42 2026-02-24T15:46:56.222 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : zip-3.0-35.el9.x86_64 19/42 2026-02-24T15:46:56.222 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : libstoragemgmt-1.10.1-1.el9.x86_64 20/42 2026-02-24T15:46:56.222 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : lua-5.4.4-4.el9.x86_64 21/42 2026-02-24T15:46:56.222 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : python3-libstoragemgmt-1.10.1-1.el9.x86_64 22/42 2026-02-24T15:46:56.222 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 23/42 2026-02-24T15:46:56.222 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 24/42 2026-02-24T15:46:56.222 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 25/42 2026-02-24T15:46:56.222 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 26/42 2026-02-24T15:46:56.222 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : python3-autocommand-2.2.2-8.el9.noarch 27/42 2026-02-24T15:46:56.222 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : python3-backports-tarfile-1.2.0-1.el9.noarch 28/42 2026-02-24T15:46:56.222 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 29/42 2026-02-24T15:46:56.222 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : python3-cheroot-10.0.1-4.el9.noarch 30/42 2026-02-24T15:46:56.222 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 31/42 2026-02-24T15:46:56.222 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 32/42 2026-02-24T15:46:56.222 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 33/42 2026-02-24T15:46:56.222 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 34/42 2026-02-24T15:46:56.222 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : python3-jaraco-context-6.0.1-3.el9.noarch 35/42 2026-02-24T15:46:56.222 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 36/42 2026-02-24T15:46:56.222 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : python3-jaraco-text-4.0.0-2.el9.noarch 37/42 2026-02-24T15:46:56.222 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 38/42 2026-02-24T15:46:56.223 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 39/42 2026-02-24T15:46:56.223 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 40/42 2026-02-24T15:46:56.223 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 41/42 2026-02-24T15:46:56.290 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 42/42 2026-02-24T15:46:56.290 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:46:56.290 INFO:teuthology.orchestra.run.trial119.stdout:Installed: 2026-02-24T15:46:56.291 INFO:teuthology.orchestra.run.trial119.stdout: ceph-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:46:56.291 INFO:teuthology.orchestra.run.trial119.stdout: ceph-mds-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:46:56.291 INFO:teuthology.orchestra.run.trial119.stdout: ceph-mgr-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:46:56.291 INFO:teuthology.orchestra.run.trial119.stdout: ceph-mgr-modules-core-2:20.3.0-5343.gd26e5c22.el9.noarch 2026-02-24T15:46:56.291 INFO:teuthology.orchestra.run.trial119.stdout: ceph-mon-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:46:56.291 INFO:teuthology.orchestra.run.trial119.stdout: ceph-osd-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:46:56.291 INFO:teuthology.orchestra.run.trial119.stdout: ceph-osd-classic-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:46:56.291 INFO:teuthology.orchestra.run.trial119.stdout: ceph-osd-crimson-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:46:56.291 INFO:teuthology.orchestra.run.trial119.stdout: ledmon-libs-1.1.0-3.el9.x86_64 2026-02-24T15:46:56.291 INFO:teuthology.orchestra.run.trial119.stdout: libconfig-1.7.2-9.el9.x86_64 2026-02-24T15:46:56.291 INFO:teuthology.orchestra.run.trial119.stdout: libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-24T15:46:56.291 INFO:teuthology.orchestra.run.trial119.stdout: lua-5.4.4-4.el9.x86_64 2026-02-24T15:46:56.291 INFO:teuthology.orchestra.run.trial119.stdout: lua-devel-5.4.4-4.el9.x86_64 2026-02-24T15:46:56.291 INFO:teuthology.orchestra.run.trial119.stdout: luarocks-3.9.2-5.el9.noarch 2026-02-24T15:46:56.291 INFO:teuthology.orchestra.run.trial119.stdout: python3-autocommand-2.2.2-8.el9.noarch 2026-02-24T15:46:56.291 INFO:teuthology.orchestra.run.trial119.stdout: python3-backports-tarfile-1.2.0-1.el9.noarch 2026-02-24T15:46:56.291 INFO:teuthology.orchestra.run.trial119.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2026-02-24T15:46:56.291 INFO:teuthology.orchestra.run.trial119.stdout: python3-cffi-1.14.5-5.el9.x86_64 2026-02-24T15:46:56.291 INFO:teuthology.orchestra.run.trial119.stdout: python3-cheroot-10.0.1-4.el9.noarch 2026-02-24T15:46:56.291 INFO:teuthology.orchestra.run.trial119.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2026-02-24T15:46:56.291 INFO:teuthology.orchestra.run.trial119.stdout: python3-cryptography-36.0.1-5.el9.x86_64 2026-02-24T15:46:56.291 INFO:teuthology.orchestra.run.trial119.stdout: python3-jaraco-8.2.1-3.el9.noarch 2026-02-24T15:46:56.291 INFO:teuthology.orchestra.run.trial119.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2026-02-24T15:46:56.291 INFO:teuthology.orchestra.run.trial119.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2026-02-24T15:46:56.291 INFO:teuthology.orchestra.run.trial119.stdout: python3-jaraco-context-6.0.1-3.el9.noarch 2026-02-24T15:46:56.291 INFO:teuthology.orchestra.run.trial119.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2026-02-24T15:46:56.291 INFO:teuthology.orchestra.run.trial119.stdout: python3-jaraco-text-4.0.0-2.el9.noarch 2026-02-24T15:46:56.291 INFO:teuthology.orchestra.run.trial119.stdout: python3-libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-24T15:46:56.291 INFO:teuthology.orchestra.run.trial119.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2026-02-24T15:46:56.291 INFO:teuthology.orchestra.run.trial119.stdout: python3-packaging-20.9-5.el9.noarch 2026-02-24T15:46:56.291 INFO:teuthology.orchestra.run.trial119.stdout: python3-ply-3.11-14.el9.noarch 2026-02-24T15:46:56.292 INFO:teuthology.orchestra.run.trial119.stdout: python3-portend-3.1.0-2.el9.noarch 2026-02-24T15:46:56.292 INFO:teuthology.orchestra.run.trial119.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2026-02-24T15:46:56.292 INFO:teuthology.orchestra.run.trial119.stdout: python3-pycparser-2.20-6.el9.noarch 2026-02-24T15:46:56.292 INFO:teuthology.orchestra.run.trial119.stdout: python3-pyparsing-2.4.7-9.el9.noarch 2026-02-24T15:46:56.292 INFO:teuthology.orchestra.run.trial119.stdout: python3-requests-2.25.1-10.el9.noarch 2026-02-24T15:46:56.292 INFO:teuthology.orchestra.run.trial119.stdout: python3-tempora-5.0.0-2.el9.noarch 2026-02-24T15:46:56.292 INFO:teuthology.orchestra.run.trial119.stdout: python3-toml-0.10.2-6.el9.noarch 2026-02-24T15:46:56.292 INFO:teuthology.orchestra.run.trial119.stdout: python3-urllib3-1.26.5-7.el9.noarch 2026-02-24T15:46:56.292 INFO:teuthology.orchestra.run.trial119.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2026-02-24T15:46:56.292 INFO:teuthology.orchestra.run.trial119.stdout: unzip-6.0-59.el9.x86_64 2026-02-24T15:46:56.292 INFO:teuthology.orchestra.run.trial119.stdout: zip-3.0-35.el9.x86_64 2026-02-24T15:46:56.292 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:46:56.292 INFO:teuthology.orchestra.run.trial119.stdout:Complete! 2026-02-24T15:46:56.333 DEBUG:teuthology.orchestra.run.trial119:> sudo yum -y install ceph-base 2026-02-24T15:46:56.652 INFO:teuthology.orchestra.run.trial119.stdout:Last metadata expiration check: 0:06:05 ago on Tue 24 Feb 2026 03:40:51 PM UTC. 2026-02-24T15:46:56.716 INFO:teuthology.orchestra.run.trial119.stdout:Package ceph-base-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T15:46:56.734 INFO:teuthology.orchestra.run.trial119.stdout:Dependencies resolved. 2026-02-24T15:46:56.734 INFO:teuthology.orchestra.run.trial119.stdout:Nothing to do. 2026-02-24T15:46:56.735 INFO:teuthology.orchestra.run.trial119.stdout:Complete! 2026-02-24T15:46:56.756 DEBUG:teuthology.orchestra.run.trial119:> sudo yum -y install cephadm 2026-02-24T15:46:57.065 INFO:teuthology.orchestra.run.trial119.stdout:Last metadata expiration check: 0:06:06 ago on Tue 24 Feb 2026 03:40:51 PM UTC. 2026-02-24T15:46:57.147 INFO:teuthology.orchestra.run.trial119.stdout:Dependencies resolved. 2026-02-24T15:46:57.148 INFO:teuthology.orchestra.run.trial119.stdout:================================================================================ 2026-02-24T15:46:57.148 INFO:teuthology.orchestra.run.trial119.stdout: Package Arch Version Repository Size 2026-02-24T15:46:57.148 INFO:teuthology.orchestra.run.trial119.stdout:================================================================================ 2026-02-24T15:46:57.148 INFO:teuthology.orchestra.run.trial119.stdout:Installing: 2026-02-24T15:46:57.148 INFO:teuthology.orchestra.run.trial119.stdout: cephadm noarch 2:20.3.0-5343.gd26e5c22.el9 ceph-noarch 1.0 M 2026-02-24T15:46:57.148 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:46:57.148 INFO:teuthology.orchestra.run.trial119.stdout:Transaction Summary 2026-02-24T15:46:57.149 INFO:teuthology.orchestra.run.trial119.stdout:================================================================================ 2026-02-24T15:46:57.149 INFO:teuthology.orchestra.run.trial119.stdout:Install 1 Package 2026-02-24T15:46:57.149 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:46:57.149 INFO:teuthology.orchestra.run.trial119.stdout:Total download size: 1.0 M 2026-02-24T15:46:57.149 INFO:teuthology.orchestra.run.trial119.stdout:Installed size: 1.0 M 2026-02-24T15:46:57.149 INFO:teuthology.orchestra.run.trial119.stdout:Downloading Packages: 2026-02-24T15:47:00.069 INFO:teuthology.orchestra.run.trial119.stdout:cephadm-20.3.0-5343.gd26e5c22.el9.noarch.rpm 356 kB/s | 1.0 MB 00:02 2026-02-24T15:47:00.069 INFO:teuthology.orchestra.run.trial119.stdout:-------------------------------------------------------------------------------- 2026-02-24T15:47:00.069 INFO:teuthology.orchestra.run.trial119.stdout:Total 356 kB/s | 1.0 MB 00:02 2026-02-24T15:47:00.069 INFO:teuthology.orchestra.run.trial119.stdout:Running transaction check 2026-02-24T15:47:00.073 INFO:teuthology.orchestra.run.trial119.stdout:Transaction check succeeded. 2026-02-24T15:47:00.073 INFO:teuthology.orchestra.run.trial119.stdout:Running transaction test 2026-02-24T15:47:00.077 INFO:teuthology.orchestra.run.trial119.stdout:Transaction test succeeded. 2026-02-24T15:47:00.077 INFO:teuthology.orchestra.run.trial119.stdout:Running transaction 2026-02-24T15:47:00.095 INFO:teuthology.orchestra.run.trial119.stdout: Preparing : 1/1 2026-02-24T15:47:00.153 INFO:teuthology.orchestra.run.trial119.stdout: Running scriptlet: cephadm-2:20.3.0-5343.gd26e5c22.el9.noarch 1/1 2026-02-24T15:47:00.168 INFO:teuthology.orchestra.run.trial119.stdout: Installing : cephadm-2:20.3.0-5343.gd26e5c22.el9.noarch 1/1 2026-02-24T15:47:00.238 INFO:teuthology.orchestra.run.trial119.stdout: Running scriptlet: cephadm-2:20.3.0-5343.gd26e5c22.el9.noarch 1/1 2026-02-24T15:47:00.301 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : cephadm-2:20.3.0-5343.gd26e5c22.el9.noarch 1/1 2026-02-24T15:47:00.301 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:47:00.301 INFO:teuthology.orchestra.run.trial119.stdout:Installed: 2026-02-24T15:47:00.301 INFO:teuthology.orchestra.run.trial119.stdout: cephadm-2:20.3.0-5343.gd26e5c22.el9.noarch 2026-02-24T15:47:00.301 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:47:00.301 INFO:teuthology.orchestra.run.trial119.stdout:Complete! 2026-02-24T15:47:00.330 DEBUG:teuthology.orchestra.run.trial119:> sudo yum -y install ceph-immutable-object-cache 2026-02-24T15:47:00.646 INFO:teuthology.orchestra.run.trial119.stdout:Last metadata expiration check: 0:06:09 ago on Tue 24 Feb 2026 03:40:51 PM UTC. 2026-02-24T15:47:00.729 INFO:teuthology.orchestra.run.trial119.stdout:Dependencies resolved. 2026-02-24T15:47:00.729 INFO:teuthology.orchestra.run.trial119.stdout:================================================================================ 2026-02-24T15:47:00.729 INFO:teuthology.orchestra.run.trial119.stdout: Package Arch Version Repo Size 2026-02-24T15:47:00.729 INFO:teuthology.orchestra.run.trial119.stdout:================================================================================ 2026-02-24T15:47:00.729 INFO:teuthology.orchestra.run.trial119.stdout:Installing: 2026-02-24T15:47:00.729 INFO:teuthology.orchestra.run.trial119.stdout: ceph-immutable-object-cache x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 149 k 2026-02-24T15:47:00.729 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:47:00.729 INFO:teuthology.orchestra.run.trial119.stdout:Transaction Summary 2026-02-24T15:47:00.729 INFO:teuthology.orchestra.run.trial119.stdout:================================================================================ 2026-02-24T15:47:00.729 INFO:teuthology.orchestra.run.trial119.stdout:Install 1 Package 2026-02-24T15:47:00.729 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:47:00.729 INFO:teuthology.orchestra.run.trial119.stdout:Total download size: 149 k 2026-02-24T15:47:00.729 INFO:teuthology.orchestra.run.trial119.stdout:Installed size: 455 k 2026-02-24T15:47:00.729 INFO:teuthology.orchestra.run.trial119.stdout:Downloading Packages: 2026-02-24T15:47:01.222 INFO:teuthology.orchestra.run.trial119.stdout:ceph-immutable-object-cache-20.3.0-5343.gd26e5c 304 kB/s | 149 kB 00:00 2026-02-24T15:47:01.222 INFO:teuthology.orchestra.run.trial119.stdout:-------------------------------------------------------------------------------- 2026-02-24T15:47:01.222 INFO:teuthology.orchestra.run.trial119.stdout:Total 303 kB/s | 149 kB 00:00 2026-02-24T15:47:01.222 INFO:teuthology.orchestra.run.trial119.stdout:Running transaction check 2026-02-24T15:47:01.227 INFO:teuthology.orchestra.run.trial119.stdout:Transaction check succeeded. 2026-02-24T15:47:01.227 INFO:teuthology.orchestra.run.trial119.stdout:Running transaction test 2026-02-24T15:47:01.252 INFO:teuthology.orchestra.run.trial119.stdout:Transaction test succeeded. 2026-02-24T15:47:01.252 INFO:teuthology.orchestra.run.trial119.stdout:Running transaction 2026-02-24T15:47:01.302 INFO:teuthology.orchestra.run.trial119.stdout: Preparing : 1/1 2026-02-24T15:47:01.303 INFO:teuthology.orchestra.run.trial119.stdout: Installing : ceph-immutable-object-cache-2:20.3.0-5343.gd26e5c22. 1/1 2026-02-24T15:47:01.326 INFO:teuthology.orchestra.run.trial119.stdout: Running scriptlet: ceph-immutable-object-cache-2:20.3.0-5343.gd26e5c22. 1/1 2026-02-24T15:47:01.327 INFO:teuthology.orchestra.run.trial119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T15:47:01.327 INFO:teuthology.orchestra.run.trial119.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2026-02-24T15:47:01.327 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:47:01.641 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : ceph-immutable-object-cache-2:20.3.0-5343.gd26e5c22. 1/1 2026-02-24T15:47:01.642 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:47:01.642 INFO:teuthology.orchestra.run.trial119.stdout:Installed: 2026-02-24T15:47:01.642 INFO:teuthology.orchestra.run.trial119.stdout: ceph-immutable-object-cache-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:47:01.642 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:47:01.642 INFO:teuthology.orchestra.run.trial119.stdout:Complete! 2026-02-24T15:47:01.669 DEBUG:teuthology.orchestra.run.trial119:> sudo yum -y install ceph-mgr 2026-02-24T15:47:01.978 INFO:teuthology.orchestra.run.trial119.stdout:Last metadata expiration check: 0:06:10 ago on Tue 24 Feb 2026 03:40:51 PM UTC. 2026-02-24T15:47:02.043 INFO:teuthology.orchestra.run.trial119.stdout:Package ceph-mgr-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T15:47:02.060 INFO:teuthology.orchestra.run.trial119.stdout:Dependencies resolved. 2026-02-24T15:47:02.061 INFO:teuthology.orchestra.run.trial119.stdout:Nothing to do. 2026-02-24T15:47:02.061 INFO:teuthology.orchestra.run.trial119.stdout:Complete! 2026-02-24T15:47:02.082 DEBUG:teuthology.orchestra.run.trial119:> sudo yum -y install ceph-mgr-dashboard 2026-02-24T15:47:02.396 INFO:teuthology.orchestra.run.trial119.stdout:Last metadata expiration check: 0:06:11 ago on Tue 24 Feb 2026 03:40:51 PM UTC. 2026-02-24T15:47:02.479 INFO:teuthology.orchestra.run.trial119.stdout:Dependencies resolved. 2026-02-24T15:47:02.479 INFO:teuthology.orchestra.run.trial119.stdout:================================================================================ 2026-02-24T15:47:02.480 INFO:teuthology.orchestra.run.trial119.stdout: Package Arch Version Repository Size 2026-02-24T15:47:02.480 INFO:teuthology.orchestra.run.trial119.stdout:================================================================================ 2026-02-24T15:47:02.480 INFO:teuthology.orchestra.run.trial119.stdout:Installing: 2026-02-24T15:47:02.480 INFO:teuthology.orchestra.run.trial119.stdout: ceph-mgr-dashboard noarch 2:20.3.0-5343.gd26e5c22.el9 ceph-noarch 51 M 2026-02-24T15:47:02.480 INFO:teuthology.orchestra.run.trial119.stdout:Installing dependencies: 2026-02-24T15:47:02.480 INFO:teuthology.orchestra.run.trial119.stdout: abseil-cpp x86_64 20211102.0-4.el9 epel 551 k 2026-02-24T15:47:02.480 INFO:teuthology.orchestra.run.trial119.stdout: ceph-grafana-dashboards noarch 2:20.3.0-5343.gd26e5c22.el9 ceph-noarch 57 k 2026-02-24T15:47:02.480 INFO:teuthology.orchestra.run.trial119.stdout: ceph-prometheus-alerts noarch 2:20.3.0-5343.gd26e5c22.el9 ceph-noarch 17 k 2026-02-24T15:47:02.480 INFO:teuthology.orchestra.run.trial119.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2026-02-24T15:47:02.480 INFO:teuthology.orchestra.run.trial119.stdout: protobuf-compiler x86_64 3.14.0-17.el9 crb 862 k 2026-02-24T15:47:02.480 INFO:teuthology.orchestra.run.trial119.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2026-02-24T15:47:02.480 INFO:teuthology.orchestra.run.trial119.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2026-02-24T15:47:02.480 INFO:teuthology.orchestra.run.trial119.stdout: python3-jmespath noarch 1.0.1-1.el9 appstream 48 k 2026-02-24T15:47:02.480 INFO:teuthology.orchestra.run.trial119.stdout: python3-protobuf noarch 3.14.0-17.el9 appstream 267 k 2026-02-24T15:47:02.480 INFO:teuthology.orchestra.run.trial119.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2026-02-24T15:47:02.480 INFO:teuthology.orchestra.run.trial119.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2026-02-24T15:47:02.481 INFO:teuthology.orchestra.run.trial119.stdout: python3-xmltodict noarch 0.12.0-15.el9 epel 22 k 2026-02-24T15:47:02.481 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:47:02.481 INFO:teuthology.orchestra.run.trial119.stdout:Transaction Summary 2026-02-24T15:47:02.481 INFO:teuthology.orchestra.run.trial119.stdout:================================================================================ 2026-02-24T15:47:02.481 INFO:teuthology.orchestra.run.trial119.stdout:Install 13 Packages 2026-02-24T15:47:02.481 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:47:02.481 INFO:teuthology.orchestra.run.trial119.stdout:Total download size: 55 M 2026-02-24T15:47:02.481 INFO:teuthology.orchestra.run.trial119.stdout:Installed size: 262 M 2026-02-24T15:47:02.481 INFO:teuthology.orchestra.run.trial119.stdout:Downloading Packages: 2026-02-24T15:47:03.574 INFO:teuthology.orchestra.run.trial119.stdout:(1/13): ceph-prometheus-alerts-20.3.0-5343.gd26 48 kB/s | 17 kB 00:00 2026-02-24T15:47:03.638 INFO:teuthology.orchestra.run.trial119.stdout:(2/13): ceph-grafana-dashboards-20.3.0-5343.gd2 134 kB/s | 57 kB 00:00 2026-02-24T15:47:03.655 INFO:teuthology.orchestra.run.trial119.stdout:(3/13): python3-jmespath-1.0.1-1.el9.noarch.rpm 588 kB/s | 48 kB 00:00 2026-02-24T15:47:03.689 INFO:teuthology.orchestra.run.trial119.stdout:(4/13): python3-protobuf-3.14.0-17.el9.noarch.r 5.2 MB/s | 267 kB 00:00 2026-02-24T15:47:03.782 INFO:teuthology.orchestra.run.trial119.stdout:(5/13): protobuf-compiler-3.14.0-17.el9.x86_64. 6.7 MB/s | 862 kB 00:00 2026-02-24T15:47:04.002 INFO:teuthology.orchestra.run.trial119.stdout:(6/13): grpc-data-1.46.7-10.el9.noarch.rpm 88 kB/s | 19 kB 00:00 2026-02-24T15:47:04.206 INFO:teuthology.orchestra.run.trial119.stdout:(7/13): abseil-cpp-20211102.0-4.el9.x86_64.rpm 1.0 MB/s | 551 kB 00:00 2026-02-24T15:47:04.570 INFO:teuthology.orchestra.run.trial119.stdout:(8/13): python3-grpcio-tools-1.46.7-10.el9.x86_ 398 kB/s | 144 kB 00:00 2026-02-24T15:47:04.725 INFO:teuthology.orchestra.run.trial119.stdout:(9/13): python3-grpcio-1.46.7-10.el9.x86_64.rpm 2.8 MB/s | 2.0 MB 00:00 2026-02-24T15:47:04.785 INFO:teuthology.orchestra.run.trial119.stdout:(10/13): python3-repoze-lru-0.7-16.el9.noarch.r 143 kB/s | 31 kB 00:00 2026-02-24T15:47:04.993 INFO:teuthology.orchestra.run.trial119.stdout:(11/13): python3-xmltodict-0.12.0-15.el9.noarch 106 kB/s | 22 kB 00:00 2026-02-24T15:47:05.102 INFO:teuthology.orchestra.run.trial119.stdout:(12/13): python3-routes-2.5.1-5.el9.noarch.rpm 500 kB/s | 188 kB 00:00 2026-02-24T15:49:14.667 INFO:teuthology.orchestra.run.trial119.stdout:(13/13): ceph-mgr-dashboard-20.3.0-5343.gd26e5c 395 kB/s | 51 MB 02:11 2026-02-24T15:49:14.669 INFO:teuthology.orchestra.run.trial119.stdout:-------------------------------------------------------------------------------- 2026-02-24T15:49:14.669 INFO:teuthology.orchestra.run.trial119.stdout:Total 425 kB/s | 55 MB 02:12 2026-02-24T15:49:14.692 INFO:teuthology.orchestra.run.trial119.stdout:Running transaction check 2026-02-24T15:49:14.702 INFO:teuthology.orchestra.run.trial119.stdout:Transaction check succeeded. 2026-02-24T15:49:14.702 INFO:teuthology.orchestra.run.trial119.stdout:Running transaction test 2026-02-24T15:49:14.915 INFO:teuthology.orchestra.run.trial119.stdout:Transaction test succeeded. 2026-02-24T15:49:14.915 INFO:teuthology.orchestra.run.trial119.stdout:Running transaction 2026-02-24T15:49:15.144 INFO:teuthology.orchestra.run.trial119.stdout: Preparing : 1/1 2026-02-24T15:49:15.150 INFO:teuthology.orchestra.run.trial119.stdout: Installing : python3-xmltodict-0.12.0-15.el9.noarch 1/13 2026-02-24T15:49:15.161 INFO:teuthology.orchestra.run.trial119.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 2/13 2026-02-24T15:49:15.163 INFO:teuthology.orchestra.run.trial119.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 3/13 2026-02-24T15:49:15.190 INFO:teuthology.orchestra.run.trial119.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 4/13 2026-02-24T15:49:15.235 INFO:teuthology.orchestra.run.trial119.stdout: Installing : abseil-cpp-20211102.0-4.el9.x86_64 5/13 2026-02-24T15:49:15.243 INFO:teuthology.orchestra.run.trial119.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 6/13 2026-02-24T15:49:15.271 INFO:teuthology.orchestra.run.trial119.stdout: Installing : protobuf-compiler-3.14.0-17.el9.x86_64 7/13 2026-02-24T15:49:15.280 INFO:teuthology.orchestra.run.trial119.stdout: Installing : python3-protobuf-3.14.0-17.el9.noarch 8/13 2026-02-24T15:49:15.287 INFO:teuthology.orchestra.run.trial119.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/13 2026-02-24T15:49:15.290 INFO:teuthology.orchestra.run.trial119.stdout: Installing : python3-jmespath-1.0.1-1.el9.noarch 10/13 2026-02-24T15:49:15.299 INFO:teuthology.orchestra.run.trial119.stdout: Installing : ceph-prometheus-alerts-2:20.3.0-5343.gd26e5c22.el9 11/13 2026-02-24T15:49:18.162 INFO:teuthology.orchestra.run.trial119.stdout: Installing : ceph-grafana-dashboards-2:20.3.0-5343.gd26e5c22.el 12/13 2026-02-24T15:49:18.168 INFO:teuthology.orchestra.run.trial119.stdout: Installing : ceph-mgr-dashboard-2:20.3.0-5343.gd26e5c22.el9.noa 13/13 2026-02-24T15:49:18.448 INFO:teuthology.orchestra.run.trial119.stdout: Running scriptlet: ceph-mgr-dashboard-2:20.3.0-5343.gd26e5c22.el9.noa 13/13 2026-02-24T15:49:18.449 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : ceph-grafana-dashboards-2:20.3.0-5343.gd26e5c22.el 1/13 2026-02-24T15:49:18.449 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : ceph-mgr-dashboard-2:20.3.0-5343.gd26e5c22.el9.noa 2/13 2026-02-24T15:49:18.449 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : ceph-prometheus-alerts-2:20.3.0-5343.gd26e5c22.el9 3/13 2026-02-24T15:49:18.449 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : python3-jmespath-1.0.1-1.el9.noarch 4/13 2026-02-24T15:49:18.449 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : python3-protobuf-3.14.0-17.el9.noarch 5/13 2026-02-24T15:49:18.449 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : protobuf-compiler-3.14.0-17.el9.x86_64 6/13 2026-02-24T15:49:18.450 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : abseil-cpp-20211102.0-4.el9.x86_64 7/13 2026-02-24T15:49:18.450 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/13 2026-02-24T15:49:18.450 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/13 2026-02-24T15:49:18.450 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/13 2026-02-24T15:49:18.450 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/13 2026-02-24T15:49:18.450 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/13 2026-02-24T15:49:18.511 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : python3-xmltodict-0.12.0-15.el9.noarch 13/13 2026-02-24T15:49:18.511 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:49:18.511 INFO:teuthology.orchestra.run.trial119.stdout:Installed: 2026-02-24T15:49:18.511 INFO:teuthology.orchestra.run.trial119.stdout: abseil-cpp-20211102.0-4.el9.x86_64 2026-02-24T15:49:18.511 INFO:teuthology.orchestra.run.trial119.stdout: ceph-grafana-dashboards-2:20.3.0-5343.gd26e5c22.el9.noarch 2026-02-24T15:49:18.512 INFO:teuthology.orchestra.run.trial119.stdout: ceph-mgr-dashboard-2:20.3.0-5343.gd26e5c22.el9.noarch 2026-02-24T15:49:18.512 INFO:teuthology.orchestra.run.trial119.stdout: ceph-prometheus-alerts-2:20.3.0-5343.gd26e5c22.el9.noarch 2026-02-24T15:49:18.512 INFO:teuthology.orchestra.run.trial119.stdout: grpc-data-1.46.7-10.el9.noarch 2026-02-24T15:49:18.512 INFO:teuthology.orchestra.run.trial119.stdout: protobuf-compiler-3.14.0-17.el9.x86_64 2026-02-24T15:49:18.512 INFO:teuthology.orchestra.run.trial119.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2026-02-24T15:49:18.512 INFO:teuthology.orchestra.run.trial119.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2026-02-24T15:49:18.512 INFO:teuthology.orchestra.run.trial119.stdout: python3-jmespath-1.0.1-1.el9.noarch 2026-02-24T15:49:18.512 INFO:teuthology.orchestra.run.trial119.stdout: python3-protobuf-3.14.0-17.el9.noarch 2026-02-24T15:49:18.512 INFO:teuthology.orchestra.run.trial119.stdout: python3-repoze-lru-0.7-16.el9.noarch 2026-02-24T15:49:18.512 INFO:teuthology.orchestra.run.trial119.stdout: python3-routes-2.5.1-5.el9.noarch 2026-02-24T15:49:18.512 INFO:teuthology.orchestra.run.trial119.stdout: python3-xmltodict-0.12.0-15.el9.noarch 2026-02-24T15:49:18.513 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:49:18.513 INFO:teuthology.orchestra.run.trial119.stdout:Complete! 2026-02-24T15:49:18.547 DEBUG:teuthology.orchestra.run.trial119:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-24T15:49:18.861 INFO:teuthology.orchestra.run.trial119.stdout:Last metadata expiration check: 0:08:27 ago on Tue 24 Feb 2026 03:40:51 PM UTC. 2026-02-24T15:49:18.945 INFO:teuthology.orchestra.run.trial119.stdout:Dependencies resolved. 2026-02-24T15:49:18.946 INFO:teuthology.orchestra.run.trial119.stdout:======================================================================================= 2026-02-24T15:49:18.946 INFO:teuthology.orchestra.run.trial119.stdout: Package Arch Version Repository Size 2026-02-24T15:49:18.946 INFO:teuthology.orchestra.run.trial119.stdout:======================================================================================= 2026-02-24T15:49:18.946 INFO:teuthology.orchestra.run.trial119.stdout:Installing: 2026-02-24T15:49:18.946 INFO:teuthology.orchestra.run.trial119.stdout: ceph-mgr-diskprediction-local noarch 2:20.3.0-5343.gd26e5c22.el9 ceph-noarch 7.4 M 2026-02-24T15:49:18.946 INFO:teuthology.orchestra.run.trial119.stdout:Installing dependencies: 2026-02-24T15:49:18.946 INFO:teuthology.orchestra.run.trial119.stdout: flexiblas x86_64 3.0.4-9.el9 appstream 30 k 2026-02-24T15:49:18.946 INFO:teuthology.orchestra.run.trial119.stdout: flexiblas-netlib x86_64 3.0.4-9.el9 appstream 3.0 M 2026-02-24T15:49:18.946 INFO:teuthology.orchestra.run.trial119.stdout: flexiblas-openblas-openmp x86_64 3.0.4-9.el9 appstream 15 k 2026-02-24T15:49:18.946 INFO:teuthology.orchestra.run.trial119.stdout: libgfortran x86_64 11.5.0-14.el9 baseos 794 k 2026-02-24T15:49:18.947 INFO:teuthology.orchestra.run.trial119.stdout: libquadmath x86_64 11.5.0-14.el9 baseos 184 k 2026-02-24T15:49:18.947 INFO:teuthology.orchestra.run.trial119.stdout: openblas x86_64 0.3.29-1.el9 appstream 42 k 2026-02-24T15:49:18.947 INFO:teuthology.orchestra.run.trial119.stdout: openblas-openmp x86_64 0.3.29-1.el9 appstream 5.3 M 2026-02-24T15:49:18.947 INFO:teuthology.orchestra.run.trial119.stdout: python3-devel x86_64 3.9.25-3.el9 appstream 244 k 2026-02-24T15:49:18.947 INFO:teuthology.orchestra.run.trial119.stdout: python3-numpy x86_64 1:1.23.5-2.el9 appstream 6.1 M 2026-02-24T15:49:18.947 INFO:teuthology.orchestra.run.trial119.stdout: python3-numpy-f2py x86_64 1:1.23.5-2.el9 appstream 442 k 2026-02-24T15:49:18.947 INFO:teuthology.orchestra.run.trial119.stdout: python3-scipy x86_64 1.9.3-2.el9 appstream 19 M 2026-02-24T15:49:18.947 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:49:18.947 INFO:teuthology.orchestra.run.trial119.stdout:Transaction Summary 2026-02-24T15:49:18.947 INFO:teuthology.orchestra.run.trial119.stdout:======================================================================================= 2026-02-24T15:49:18.947 INFO:teuthology.orchestra.run.trial119.stdout:Install 12 Packages 2026-02-24T15:49:18.947 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:49:18.948 INFO:teuthology.orchestra.run.trial119.stdout:Total download size: 43 M 2026-02-24T15:49:18.948 INFO:teuthology.orchestra.run.trial119.stdout:Installed size: 234 M 2026-02-24T15:49:18.948 INFO:teuthology.orchestra.run.trial119.stdout:Downloading Packages: 2026-02-24T15:49:19.834 INFO:teuthology.orchestra.run.trial119.stdout:(1/12): libquadmath-11.5.0-14.el9.x86_64.rpm 252 kB/s | 184 kB 00:00 2026-02-24T15:49:19.999 INFO:teuthology.orchestra.run.trial119.stdout:(2/12): libgfortran-11.5.0-14.el9.x86_64.rpm 886 kB/s | 794 kB 00:00 2026-02-24T15:49:20.348 INFO:teuthology.orchestra.run.trial119.stdout:(3/12): flexiblas-3.0.4-9.el9.x86_64.rpm 58 kB/s | 30 kB 00:00 2026-02-24T15:49:20.431 INFO:teuthology.orchestra.run.trial119.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-9.el9.x 180 kB/s | 15 kB 00:00 2026-02-24T15:49:20.591 INFO:teuthology.orchestra.run.trial119.stdout:(5/12): openblas-0.3.29-1.el9.x86_64.rpm 263 kB/s | 42 kB 00:00 2026-02-24T15:49:20.952 INFO:teuthology.orchestra.run.trial119.stdout:(6/12): flexiblas-netlib-3.0.4-9.el9.x86_64.rpm 3.1 MB/s | 3.0 MB 00:00 2026-02-24T15:49:21.037 INFO:teuthology.orchestra.run.trial119.stdout:(7/12): python3-devel-3.9.25-3.el9.x86_64.rpm 2.8 MB/s | 244 kB 00:00 2026-02-24T15:49:21.308 INFO:teuthology.orchestra.run.trial119.stdout:(8/12): openblas-openmp-0.3.29-1.el9.x86_64.rpm 7.4 MB/s | 5.3 MB 00:00 2026-02-24T15:49:21.394 INFO:teuthology.orchestra.run.trial119.stdout:(9/12): python3-numpy-f2py-1.23.5-2.el9.x86_64. 5.0 MB/s | 442 kB 00:00 2026-02-24T15:49:21.525 INFO:teuthology.orchestra.run.trial119.stdout:(10/12): python3-numpy-1.23.5-2.el9.x86_64.rpm 13 MB/s | 6.1 MB 00:00 2026-02-24T15:49:21.897 INFO:teuthology.orchestra.run.trial119.stdout:(11/12): python3-scipy-1.9.3-2.el9.x86_64.rpm 38 MB/s | 19 MB 00:00 2026-02-24T15:49:32.386 INFO:teuthology.orchestra.run.trial119.stdout:(12/12): ceph-mgr-diskprediction-local-20.3.0-5 570 kB/s | 7.4 MB 00:13 2026-02-24T15:49:32.388 INFO:teuthology.orchestra.run.trial119.stdout:-------------------------------------------------------------------------------- 2026-02-24T15:49:32.388 INFO:teuthology.orchestra.run.trial119.stdout:Total 3.2 MB/s | 43 MB 00:13 2026-02-24T15:49:32.544 INFO:teuthology.orchestra.run.trial119.stdout:Running transaction check 2026-02-24T15:49:32.557 INFO:teuthology.orchestra.run.trial119.stdout:Transaction check succeeded. 2026-02-24T15:49:32.557 INFO:teuthology.orchestra.run.trial119.stdout:Running transaction test 2026-02-24T15:49:32.759 INFO:teuthology.orchestra.run.trial119.stdout:Transaction test succeeded. 2026-02-24T15:49:32.759 INFO:teuthology.orchestra.run.trial119.stdout:Running transaction 2026-02-24T15:49:32.976 INFO:teuthology.orchestra.run.trial119.stdout: Preparing : 1/1 2026-02-24T15:49:32.978 INFO:teuthology.orchestra.run.trial119.stdout: Installing : flexiblas-3.0.4-9.el9.x86_64 1/12 2026-02-24T15:49:32.985 INFO:teuthology.orchestra.run.trial119.stdout: Installing : libquadmath-11.5.0-14.el9.x86_64 2/12 2026-02-24T15:49:33.018 INFO:teuthology.orchestra.run.trial119.stdout: Installing : libgfortran-11.5.0-14.el9.x86_64 3/12 2026-02-24T15:49:33.023 INFO:teuthology.orchestra.run.trial119.stdout: Installing : python3-devel-3.9.25-3.el9.x86_64 4/12 2026-02-24T15:49:33.081 INFO:teuthology.orchestra.run.trial119.stdout: Installing : openblas-0.3.29-1.el9.x86_64 5/12 2026-02-24T15:49:33.084 INFO:teuthology.orchestra.run.trial119.stdout: Installing : openblas-openmp-0.3.29-1.el9.x86_64 6/12 2026-02-24T15:49:33.105 INFO:teuthology.orchestra.run.trial119.stdout: Installing : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 7/12 2026-02-24T15:49:33.423 INFO:teuthology.orchestra.run.trial119.stdout: Installing : flexiblas-netlib-3.0.4-9.el9.x86_64 8/12 2026-02-24T15:49:33.495 INFO:teuthology.orchestra.run.trial119.stdout: Installing : python3-numpy-1:1.23.5-2.el9.x86_64 9/12 2026-02-24T15:49:34.149 INFO:teuthology.orchestra.run.trial119.stdout: Installing : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 10/12 2026-02-24T15:49:34.607 INFO:teuthology.orchestra.run.trial119.stdout: Installing : python3-scipy-1.9.3-2.el9.x86_64 11/12 2026-02-24T15:49:34.608 INFO:teuthology.orchestra.run.trial119.stdout: Installing : ceph-mgr-diskprediction-local-2:20.3.0-5343.gd26e5 12/12 2026-02-24T15:49:34.912 INFO:teuthology.orchestra.run.trial119.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:20.3.0-5343.gd26e5 12/12 2026-02-24T15:49:34.912 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : ceph-mgr-diskprediction-local-2:20.3.0-5343.gd26e5 1/12 2026-02-24T15:49:34.912 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : libgfortran-11.5.0-14.el9.x86_64 2/12 2026-02-24T15:49:34.912 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : libquadmath-11.5.0-14.el9.x86_64 3/12 2026-02-24T15:49:34.912 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : flexiblas-3.0.4-9.el9.x86_64 4/12 2026-02-24T15:49:34.912 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : flexiblas-netlib-3.0.4-9.el9.x86_64 5/12 2026-02-24T15:49:34.912 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 6/12 2026-02-24T15:49:34.912 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : openblas-0.3.29-1.el9.x86_64 7/12 2026-02-24T15:49:34.912 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : openblas-openmp-0.3.29-1.el9.x86_64 8/12 2026-02-24T15:49:34.912 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : python3-devel-3.9.25-3.el9.x86_64 9/12 2026-02-24T15:49:34.912 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : python3-numpy-1:1.23.5-2.el9.x86_64 10/12 2026-02-24T15:49:34.912 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 11/12 2026-02-24T15:49:34.976 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : python3-scipy-1.9.3-2.el9.x86_64 12/12 2026-02-24T15:49:34.976 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:49:34.976 INFO:teuthology.orchestra.run.trial119.stdout:Installed: 2026-02-24T15:49:34.976 INFO:teuthology.orchestra.run.trial119.stdout: ceph-mgr-diskprediction-local-2:20.3.0-5343.gd26e5c22.el9.noarch 2026-02-24T15:49:34.976 INFO:teuthology.orchestra.run.trial119.stdout: flexiblas-3.0.4-9.el9.x86_64 2026-02-24T15:49:34.976 INFO:teuthology.orchestra.run.trial119.stdout: flexiblas-netlib-3.0.4-9.el9.x86_64 2026-02-24T15:49:34.976 INFO:teuthology.orchestra.run.trial119.stdout: flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 2026-02-24T15:49:34.976 INFO:teuthology.orchestra.run.trial119.stdout: libgfortran-11.5.0-14.el9.x86_64 2026-02-24T15:49:34.976 INFO:teuthology.orchestra.run.trial119.stdout: libquadmath-11.5.0-14.el9.x86_64 2026-02-24T15:49:34.976 INFO:teuthology.orchestra.run.trial119.stdout: openblas-0.3.29-1.el9.x86_64 2026-02-24T15:49:34.976 INFO:teuthology.orchestra.run.trial119.stdout: openblas-openmp-0.3.29-1.el9.x86_64 2026-02-24T15:49:34.976 INFO:teuthology.orchestra.run.trial119.stdout: python3-devel-3.9.25-3.el9.x86_64 2026-02-24T15:49:34.976 INFO:teuthology.orchestra.run.trial119.stdout: python3-numpy-1:1.23.5-2.el9.x86_64 2026-02-24T15:49:34.976 INFO:teuthology.orchestra.run.trial119.stdout: python3-numpy-f2py-1:1.23.5-2.el9.x86_64 2026-02-24T15:49:34.976 INFO:teuthology.orchestra.run.trial119.stdout: python3-scipy-1.9.3-2.el9.x86_64 2026-02-24T15:49:34.977 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:49:34.977 INFO:teuthology.orchestra.run.trial119.stdout:Complete! 2026-02-24T15:49:35.015 DEBUG:teuthology.orchestra.run.trial119:> sudo yum -y install ceph-mgr-rook 2026-02-24T15:49:35.332 INFO:teuthology.orchestra.run.trial119.stdout:Last metadata expiration check: 0:08:44 ago on Tue 24 Feb 2026 03:40:51 PM UTC. 2026-02-24T15:49:35.416 INFO:teuthology.orchestra.run.trial119.stdout:Dependencies resolved. 2026-02-24T15:49:35.416 INFO:teuthology.orchestra.run.trial119.stdout:================================================================================ 2026-02-24T15:49:35.417 INFO:teuthology.orchestra.run.trial119.stdout: Package Arch Version Repository Size 2026-02-24T15:49:35.417 INFO:teuthology.orchestra.run.trial119.stdout:================================================================================ 2026-02-24T15:49:35.417 INFO:teuthology.orchestra.run.trial119.stdout:Installing: 2026-02-24T15:49:35.417 INFO:teuthology.orchestra.run.trial119.stdout: ceph-mgr-rook noarch 2:20.3.0-5343.gd26e5c22.el9 ceph-noarch 50 k 2026-02-24T15:49:35.417 INFO:teuthology.orchestra.run.trial119.stdout:Installing dependencies: 2026-02-24T15:49:35.417 INFO:teuthology.orchestra.run.trial119.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2026-02-24T15:49:35.417 INFO:teuthology.orchestra.run.trial119.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2026-02-24T15:49:35.417 INFO:teuthology.orchestra.run.trial119.stdout: python3-google-auth noarch 1:2.45.0-1.el9 epel 254 k 2026-02-24T15:49:35.417 INFO:teuthology.orchestra.run.trial119.stdout: python3-kubernetes noarch 1:26.1.0-3.el9 epel 1.0 M 2026-02-24T15:49:35.417 INFO:teuthology.orchestra.run.trial119.stdout: python3-pyasn1 noarch 0.4.8-7.el9 appstream 157 k 2026-02-24T15:49:35.418 INFO:teuthology.orchestra.run.trial119.stdout: python3-pyasn1-modules noarch 0.4.8-7.el9 appstream 277 k 2026-02-24T15:49:35.418 INFO:teuthology.orchestra.run.trial119.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2026-02-24T15:49:35.418 INFO:teuthology.orchestra.run.trial119.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2026-02-24T15:49:35.418 INFO:teuthology.orchestra.run.trial119.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2026-02-24T15:49:35.418 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:49:35.418 INFO:teuthology.orchestra.run.trial119.stdout:Transaction Summary 2026-02-24T15:49:35.418 INFO:teuthology.orchestra.run.trial119.stdout:================================================================================ 2026-02-24T15:49:35.418 INFO:teuthology.orchestra.run.trial119.stdout:Install 10 Packages 2026-02-24T15:49:35.418 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:49:35.418 INFO:teuthology.orchestra.run.trial119.stdout:Total download size: 2.0 M 2026-02-24T15:49:35.418 INFO:teuthology.orchestra.run.trial119.stdout:Installed size: 25 M 2026-02-24T15:49:35.419 INFO:teuthology.orchestra.run.trial119.stdout:Downloading Packages: 2026-02-24T15:49:35.859 INFO:teuthology.orchestra.run.trial119.stdout:(1/10): python3-pyasn1-0.4.8-7.el9.noarch.rpm 2.9 MB/s | 157 kB 00:00 2026-02-24T15:49:35.870 INFO:teuthology.orchestra.run.trial119.stdout:(2/10): python3-requests-oauthlib-1.3.0-12.el9. 4.5 MB/s | 54 kB 00:00 2026-02-24T15:49:35.883 INFO:teuthology.orchestra.run.trial119.stdout:(3/10): python3-pyasn1-modules-0.4.8-7.el9.noar 3.5 MB/s | 277 kB 00:00 2026-02-24T15:49:35.959 INFO:teuthology.orchestra.run.trial119.stdout:(4/10): python3-certifi-2023.05.07-4.el9.noarch 186 kB/s | 14 kB 00:00 2026-02-24T15:49:35.976 INFO:teuthology.orchestra.run.trial119.stdout:(5/10): python3-cachetools-4.2.4-1.el9.noarch.r 305 kB/s | 32 kB 00:00 2026-02-24T15:49:36.074 INFO:teuthology.orchestra.run.trial119.stdout:(6/10): python3-google-auth-2.45.0-1.el9.noarch 2.2 MB/s | 254 kB 00:00 2026-02-24T15:49:36.104 INFO:teuthology.orchestra.run.trial119.stdout:(7/10): python3-rsa-4.9-2.el9.noarch.rpm 2.0 MB/s | 59 kB 00:00 2026-02-24T15:49:36.139 INFO:teuthology.orchestra.run.trial119.stdout:(8/10): python3-websocket-client-1.2.3-2.el9.no 2.5 MB/s | 90 kB 00:00 2026-02-24T15:49:36.157 INFO:teuthology.orchestra.run.trial119.stdout:(9/10): python3-kubernetes-26.1.0-3.el9.noarch. 5.6 MB/s | 1.0 MB 00:00 2026-02-24T15:49:36.299 INFO:teuthology.orchestra.run.trial119.stdout:(10/10): ceph-mgr-rook-20.3.0-5343.gd26e5c22.el 102 kB/s | 50 kB 00:00 2026-02-24T15:49:36.300 INFO:teuthology.orchestra.run.trial119.stdout:-------------------------------------------------------------------------------- 2026-02-24T15:49:36.301 INFO:teuthology.orchestra.run.trial119.stdout:Total 2.2 MB/s | 2.0 MB 00:00 2026-02-24T15:49:36.328 INFO:teuthology.orchestra.run.trial119.stdout:Running transaction check 2026-02-24T15:49:36.335 INFO:teuthology.orchestra.run.trial119.stdout:Transaction check succeeded. 2026-02-24T15:49:36.336 INFO:teuthology.orchestra.run.trial119.stdout:Running transaction test 2026-02-24T15:49:36.374 INFO:teuthology.orchestra.run.trial119.stdout:Transaction test succeeded. 2026-02-24T15:49:36.375 INFO:teuthology.orchestra.run.trial119.stdout:Running transaction 2026-02-24T15:49:36.450 INFO:teuthology.orchestra.run.trial119.stdout: Preparing : 1/1 2026-02-24T15:49:36.501 INFO:teuthology.orchestra.run.trial119.stdout: Installing : python3-pyasn1-0.4.8-7.el9.noarch 1/10 2026-02-24T15:49:36.515 INFO:teuthology.orchestra.run.trial119.stdout: Installing : python3-pyasn1-modules-0.4.8-7.el9.noarch 2/10 2026-02-24T15:49:36.530 INFO:teuthology.orchestra.run.trial119.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/10 2026-02-24T15:49:36.535 INFO:teuthology.orchestra.run.trial119.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/10 2026-02-24T15:49:36.542 INFO:teuthology.orchestra.run.trial119.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/10 2026-02-24T15:49:36.582 INFO:teuthology.orchestra.run.trial119.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/10 2026-02-24T15:49:36.594 INFO:teuthology.orchestra.run.trial119.stdout: Installing : python3-google-auth-1:2.45.0-1.el9.noarch 7/10 2026-02-24T15:49:36.913 INFO:teuthology.orchestra.run.trial119.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 8/10 2026-02-24T15:49:36.928 INFO:teuthology.orchestra.run.trial119.stdout: Installing : python3-kubernetes-1:26.1.0-3.el9.noarch 9/10 2026-02-24T15:49:36.929 INFO:teuthology.orchestra.run.trial119.stdout: Installing : ceph-mgr-rook-2:20.3.0-5343.gd26e5c22.el9.noarch 10/10 2026-02-24T15:49:37.040 INFO:teuthology.orchestra.run.trial119.stdout: Running scriptlet: ceph-mgr-rook-2:20.3.0-5343.gd26e5c22.el9.noarch 10/10 2026-02-24T15:49:37.040 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : ceph-mgr-rook-2:20.3.0-5343.gd26e5c22.el9.noarch 1/10 2026-02-24T15:49:37.041 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : python3-pyasn1-0.4.8-7.el9.noarch 2/10 2026-02-24T15:49:37.041 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : python3-pyasn1-modules-0.4.8-7.el9.noarch 3/10 2026-02-24T15:49:37.041 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 4/10 2026-02-24T15:49:37.041 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 5/10 2026-02-24T15:49:37.041 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 6/10 2026-02-24T15:49:37.041 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : python3-google-auth-1:2.45.0-1.el9.noarch 7/10 2026-02-24T15:49:37.041 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : python3-kubernetes-1:26.1.0-3.el9.noarch 8/10 2026-02-24T15:49:37.041 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 9/10 2026-02-24T15:49:37.104 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 10/10 2026-02-24T15:49:37.104 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:49:37.104 INFO:teuthology.orchestra.run.trial119.stdout:Installed: 2026-02-24T15:49:37.104 INFO:teuthology.orchestra.run.trial119.stdout: ceph-mgr-rook-2:20.3.0-5343.gd26e5c22.el9.noarch 2026-02-24T15:49:37.105 INFO:teuthology.orchestra.run.trial119.stdout: python3-cachetools-4.2.4-1.el9.noarch 2026-02-24T15:49:37.105 INFO:teuthology.orchestra.run.trial119.stdout: python3-certifi-2023.05.07-4.el9.noarch 2026-02-24T15:49:37.105 INFO:teuthology.orchestra.run.trial119.stdout: python3-google-auth-1:2.45.0-1.el9.noarch 2026-02-24T15:49:37.105 INFO:teuthology.orchestra.run.trial119.stdout: python3-kubernetes-1:26.1.0-3.el9.noarch 2026-02-24T15:49:37.105 INFO:teuthology.orchestra.run.trial119.stdout: python3-pyasn1-0.4.8-7.el9.noarch 2026-02-24T15:49:37.105 INFO:teuthology.orchestra.run.trial119.stdout: python3-pyasn1-modules-0.4.8-7.el9.noarch 2026-02-24T15:49:37.105 INFO:teuthology.orchestra.run.trial119.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2026-02-24T15:49:37.105 INFO:teuthology.orchestra.run.trial119.stdout: python3-rsa-4.9-2.el9.noarch 2026-02-24T15:49:37.105 INFO:teuthology.orchestra.run.trial119.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2026-02-24T15:49:37.105 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:49:37.105 INFO:teuthology.orchestra.run.trial119.stdout:Complete! 2026-02-24T15:49:37.136 DEBUG:teuthology.orchestra.run.trial119:> sudo yum -y install ceph-mgr-cephadm 2026-02-24T15:49:37.452 INFO:teuthology.orchestra.run.trial119.stdout:Last metadata expiration check: 0:08:46 ago on Tue 24 Feb 2026 03:40:51 PM UTC. 2026-02-24T15:49:37.537 INFO:teuthology.orchestra.run.trial119.stdout:Dependencies resolved. 2026-02-24T15:49:37.538 INFO:teuthology.orchestra.run.trial119.stdout:================================================================================ 2026-02-24T15:49:37.538 INFO:teuthology.orchestra.run.trial119.stdout: Package Arch Version Repository Size 2026-02-24T15:49:37.538 INFO:teuthology.orchestra.run.trial119.stdout:================================================================================ 2026-02-24T15:49:37.538 INFO:teuthology.orchestra.run.trial119.stdout:Installing: 2026-02-24T15:49:37.538 INFO:teuthology.orchestra.run.trial119.stdout: ceph-mgr-cephadm noarch 2:20.3.0-5343.gd26e5c22.el9 ceph-noarch 184 k 2026-02-24T15:49:37.539 INFO:teuthology.orchestra.run.trial119.stdout:Installing dependencies: 2026-02-24T15:49:37.539 INFO:teuthology.orchestra.run.trial119.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2026-02-24T15:49:37.539 INFO:teuthology.orchestra.run.trial119.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2026-02-24T15:49:37.539 INFO:teuthology.orchestra.run.trial119.stdout: python3-jinja2 noarch 2.11.3-8.el9 appstream 249 k 2026-02-24T15:49:37.539 INFO:teuthology.orchestra.run.trial119.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2026-02-24T15:49:37.539 INFO:teuthology.orchestra.run.trial119.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2026-02-24T15:49:37.539 INFO:teuthology.orchestra.run.trial119.stdout: python3-typing-extensions noarch 4.15.0-1.el9 epel 86 k 2026-02-24T15:49:37.539 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:49:37.539 INFO:teuthology.orchestra.run.trial119.stdout:Transaction Summary 2026-02-24T15:49:37.539 INFO:teuthology.orchestra.run.trial119.stdout:================================================================================ 2026-02-24T15:49:37.539 INFO:teuthology.orchestra.run.trial119.stdout:Install 7 Packages 2026-02-24T15:49:37.539 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:49:37.540 INFO:teuthology.orchestra.run.trial119.stdout:Total download size: 7.1 M 2026-02-24T15:49:37.540 INFO:teuthology.orchestra.run.trial119.stdout:Installed size: 34 M 2026-02-24T15:49:37.540 INFO:teuthology.orchestra.run.trial119.stdout:Downloading Packages: 2026-02-24T15:49:38.244 INFO:teuthology.orchestra.run.trial119.stdout:(1/7): python3-jinja2-2.11.3-8.el9.noarch.rpm 496 kB/s | 249 kB 00:00 2026-02-24T15:49:38.330 INFO:teuthology.orchestra.run.trial119.stdout:(2/7): python3-markupsafe-1.1.1-12.el9.x86_64.r 403 kB/s | 35 kB 00:00 2026-02-24T15:49:38.519 INFO:teuthology.orchestra.run.trial119.stdout:(3/7): python3-asyncssh-2.13.2-5.el9.noarch.rpm 2.8 MB/s | 548 kB 00:00 2026-02-24T15:49:38.538 INFO:teuthology.orchestra.run.trial119.stdout:(4/7): python3-natsort-7.1.1-5.el9.noarch.rpm 2.9 MB/s | 58 kB 00:00 2026-02-24T15:49:38.558 INFO:teuthology.orchestra.run.trial119.stdout:(5/7): python3-typing-extensions-4.15.0-1.el9.n 4.4 MB/s | 86 kB 00:00 2026-02-24T15:49:38.599 INFO:teuthology.orchestra.run.trial119.stdout:(6/7): python3-babel-2.9.1-2.el9.noarch.rpm 7.0 MB/s | 6.0 MB 00:00 2026-02-24T15:49:39.252 INFO:teuthology.orchestra.run.trial119.stdout:(7/7): ceph-mgr-cephadm-20.3.0-5343.gd26e5c22.e 122 kB/s | 184 kB 00:01 2026-02-24T15:49:39.255 INFO:teuthology.orchestra.run.trial119.stdout:-------------------------------------------------------------------------------- 2026-02-24T15:49:39.255 INFO:teuthology.orchestra.run.trial119.stdout:Total 4.1 MB/s | 7.1 MB 00:01 2026-02-24T15:49:39.295 INFO:teuthology.orchestra.run.trial119.stdout:Running transaction check 2026-02-24T15:49:39.302 INFO:teuthology.orchestra.run.trial119.stdout:Transaction check succeeded. 2026-02-24T15:49:39.302 INFO:teuthology.orchestra.run.trial119.stdout:Running transaction test 2026-02-24T15:49:39.375 INFO:teuthology.orchestra.run.trial119.stdout:Transaction test succeeded. 2026-02-24T15:49:39.375 INFO:teuthology.orchestra.run.trial119.stdout:Running transaction 2026-02-24T15:49:39.466 INFO:teuthology.orchestra.run.trial119.stdout: Preparing : 1/1 2026-02-24T15:49:39.515 INFO:teuthology.orchestra.run.trial119.stdout: Installing : python3-typing-extensions-4.15.0-1.el9.noarch 1/7 2026-02-24T15:49:39.525 INFO:teuthology.orchestra.run.trial119.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/7 2026-02-24T15:49:39.530 INFO:teuthology.orchestra.run.trial119.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/7 2026-02-24T15:49:39.749 INFO:teuthology.orchestra.run.trial119.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 4/7 2026-02-24T15:49:39.772 INFO:teuthology.orchestra.run.trial119.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 5/7 2026-02-24T15:49:39.797 INFO:teuthology.orchestra.run.trial119.stdout: Installing : python3-jinja2-2.11.3-8.el9.noarch 6/7 2026-02-24T15:49:39.798 INFO:teuthology.orchestra.run.trial119.stdout: Installing : ceph-mgr-cephadm-2:20.3.0-5343.gd26e5c22.el9.noarch 7/7 2026-02-24T15:49:39.911 INFO:teuthology.orchestra.run.trial119.stdout: Running scriptlet: ceph-mgr-cephadm-2:20.3.0-5343.gd26e5c22.el9.noarch 7/7 2026-02-24T15:49:39.912 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : ceph-mgr-cephadm-2:20.3.0-5343.gd26e5c22.el9.noarch 1/7 2026-02-24T15:49:39.912 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/7 2026-02-24T15:49:39.912 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : python3-jinja2-2.11.3-8.el9.noarch 3/7 2026-02-24T15:49:39.912 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 4/7 2026-02-24T15:49:39.912 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 5/7 2026-02-24T15:49:39.912 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 6/7 2026-02-24T15:49:39.975 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : python3-typing-extensions-4.15.0-1.el9.noarch 7/7 2026-02-24T15:49:39.976 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:49:39.976 INFO:teuthology.orchestra.run.trial119.stdout:Installed: 2026-02-24T15:49:39.976 INFO:teuthology.orchestra.run.trial119.stdout: ceph-mgr-cephadm-2:20.3.0-5343.gd26e5c22.el9.noarch 2026-02-24T15:49:39.976 INFO:teuthology.orchestra.run.trial119.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2026-02-24T15:49:39.976 INFO:teuthology.orchestra.run.trial119.stdout: python3-babel-2.9.1-2.el9.noarch 2026-02-24T15:49:39.977 INFO:teuthology.orchestra.run.trial119.stdout: python3-jinja2-2.11.3-8.el9.noarch 2026-02-24T15:49:39.977 INFO:teuthology.orchestra.run.trial119.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2026-02-24T15:49:39.977 INFO:teuthology.orchestra.run.trial119.stdout: python3-natsort-7.1.1-5.el9.noarch 2026-02-24T15:49:39.977 INFO:teuthology.orchestra.run.trial119.stdout: python3-typing-extensions-4.15.0-1.el9.noarch 2026-02-24T15:49:39.977 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:49:39.977 INFO:teuthology.orchestra.run.trial119.stdout:Complete! 2026-02-24T15:49:40.007 DEBUG:teuthology.orchestra.run.trial119:> sudo yum -y install ceph-osd 2026-02-24T15:49:40.326 INFO:teuthology.orchestra.run.trial119.stdout:Last metadata expiration check: 0:08:49 ago on Tue 24 Feb 2026 03:40:51 PM UTC. 2026-02-24T15:49:40.391 INFO:teuthology.orchestra.run.trial119.stdout:Package ceph-osd-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T15:49:40.410 INFO:teuthology.orchestra.run.trial119.stdout:Dependencies resolved. 2026-02-24T15:49:40.411 INFO:teuthology.orchestra.run.trial119.stdout:Nothing to do. 2026-02-24T15:49:40.411 INFO:teuthology.orchestra.run.trial119.stdout:Complete! 2026-02-24T15:49:40.432 DEBUG:teuthology.orchestra.run.trial119:> sudo yum -y install ceph-osd-classic 2026-02-24T15:49:40.746 INFO:teuthology.orchestra.run.trial119.stdout:Last metadata expiration check: 0:08:49 ago on Tue 24 Feb 2026 03:40:51 PM UTC. 2026-02-24T15:49:40.811 INFO:teuthology.orchestra.run.trial119.stdout:Package ceph-osd-classic-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T15:49:40.830 INFO:teuthology.orchestra.run.trial119.stdout:Dependencies resolved. 2026-02-24T15:49:40.831 INFO:teuthology.orchestra.run.trial119.stdout:Nothing to do. 2026-02-24T15:49:40.831 INFO:teuthology.orchestra.run.trial119.stdout:Complete! 2026-02-24T15:49:40.852 DEBUG:teuthology.orchestra.run.trial119:> sudo yum -y install ceph-fuse 2026-02-24T15:49:41.172 INFO:teuthology.orchestra.run.trial119.stdout:Last metadata expiration check: 0:08:50 ago on Tue 24 Feb 2026 03:40:51 PM UTC. 2026-02-24T15:49:41.256 INFO:teuthology.orchestra.run.trial119.stdout:Dependencies resolved. 2026-02-24T15:49:41.256 INFO:teuthology.orchestra.run.trial119.stdout:================================================================================ 2026-02-24T15:49:41.256 INFO:teuthology.orchestra.run.trial119.stdout: Package Arch Version Repo Size 2026-02-24T15:49:41.256 INFO:teuthology.orchestra.run.trial119.stdout:================================================================================ 2026-02-24T15:49:41.256 INFO:teuthology.orchestra.run.trial119.stdout:Installing: 2026-02-24T15:49:41.256 INFO:teuthology.orchestra.run.trial119.stdout: ceph-fuse x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 914 k 2026-02-24T15:49:41.256 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:49:41.257 INFO:teuthology.orchestra.run.trial119.stdout:Transaction Summary 2026-02-24T15:49:41.257 INFO:teuthology.orchestra.run.trial119.stdout:================================================================================ 2026-02-24T15:49:41.257 INFO:teuthology.orchestra.run.trial119.stdout:Install 1 Package 2026-02-24T15:49:41.257 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:49:41.257 INFO:teuthology.orchestra.run.trial119.stdout:Total download size: 914 k 2026-02-24T15:49:41.257 INFO:teuthology.orchestra.run.trial119.stdout:Installed size: 2.9 M 2026-02-24T15:49:41.257 INFO:teuthology.orchestra.run.trial119.stdout:Downloading Packages: 2026-02-24T15:49:45.415 INFO:teuthology.orchestra.run.trial119.stdout:ceph-fuse-20.3.0-5343.gd26e5c22.el9.x86_64.rpm 220 kB/s | 914 kB 00:04 2026-02-24T15:49:45.415 INFO:teuthology.orchestra.run.trial119.stdout:-------------------------------------------------------------------------------- 2026-02-24T15:49:45.415 INFO:teuthology.orchestra.run.trial119.stdout:Total 220 kB/s | 914 kB 00:04 2026-02-24T15:49:45.416 INFO:teuthology.orchestra.run.trial119.stdout:Running transaction check 2026-02-24T15:49:45.421 INFO:teuthology.orchestra.run.trial119.stdout:Transaction check succeeded. 2026-02-24T15:49:45.421 INFO:teuthology.orchestra.run.trial119.stdout:Running transaction test 2026-02-24T15:49:45.453 INFO:teuthology.orchestra.run.trial119.stdout:Transaction test succeeded. 2026-02-24T15:49:45.454 INFO:teuthology.orchestra.run.trial119.stdout:Running transaction 2026-02-24T15:49:45.543 INFO:teuthology.orchestra.run.trial119.stdout: Preparing : 1/1 2026-02-24T15:49:45.562 INFO:teuthology.orchestra.run.trial119.stdout: Installing : ceph-fuse-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/1 2026-02-24T15:49:45.789 INFO:teuthology.orchestra.run.trial119.stdout: Running scriptlet: ceph-fuse-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/1 2026-02-24T15:49:45.853 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : ceph-fuse-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/1 2026-02-24T15:49:45.853 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:49:45.853 INFO:teuthology.orchestra.run.trial119.stdout:Installed: 2026-02-24T15:49:45.853 INFO:teuthology.orchestra.run.trial119.stdout: ceph-fuse-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:49:45.853 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:49:45.853 INFO:teuthology.orchestra.run.trial119.stdout:Complete! 2026-02-24T15:49:45.882 DEBUG:teuthology.orchestra.run.trial119:> sudo yum -y install ceph-volume 2026-02-24T15:49:46.204 INFO:teuthology.orchestra.run.trial119.stdout:Last metadata expiration check: 0:08:55 ago on Tue 24 Feb 2026 03:40:51 PM UTC. 2026-02-24T15:49:46.287 INFO:teuthology.orchestra.run.trial119.stdout:Dependencies resolved. 2026-02-24T15:49:46.287 INFO:teuthology.orchestra.run.trial119.stdout:================================================================================ 2026-02-24T15:49:46.287 INFO:teuthology.orchestra.run.trial119.stdout: Package Arch Version Repository Size 2026-02-24T15:49:46.288 INFO:teuthology.orchestra.run.trial119.stdout:================================================================================ 2026-02-24T15:49:46.288 INFO:teuthology.orchestra.run.trial119.stdout:Installing: 2026-02-24T15:49:46.288 INFO:teuthology.orchestra.run.trial119.stdout: ceph-volume noarch 2:20.3.0-5343.gd26e5c22.el9 ceph-noarch 301 k 2026-02-24T15:49:46.288 INFO:teuthology.orchestra.run.trial119.stdout:Installing dependencies: 2026-02-24T15:49:46.288 INFO:teuthology.orchestra.run.trial119.stdout: cryptsetup x86_64 2.8.1-3.el9 baseos 351 k 2026-02-24T15:49:46.288 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:49:46.288 INFO:teuthology.orchestra.run.trial119.stdout:Transaction Summary 2026-02-24T15:49:46.288 INFO:teuthology.orchestra.run.trial119.stdout:================================================================================ 2026-02-24T15:49:46.288 INFO:teuthology.orchestra.run.trial119.stdout:Install 2 Packages 2026-02-24T15:49:46.288 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:49:46.288 INFO:teuthology.orchestra.run.trial119.stdout:Total download size: 652 k 2026-02-24T15:49:46.288 INFO:teuthology.orchestra.run.trial119.stdout:Installed size: 2.2 M 2026-02-24T15:49:46.288 INFO:teuthology.orchestra.run.trial119.stdout:Downloading Packages: 2026-02-24T15:49:47.204 INFO:teuthology.orchestra.run.trial119.stdout:(1/2): cryptsetup-2.8.1-3.el9.x86_64.rpm 409 kB/s | 351 kB 00:00 2026-02-24T15:49:48.238 INFO:teuthology.orchestra.run.trial119.stdout:(2/2): ceph-volume-20.3.0-5343.gd26e5c22.el9.no 159 kB/s | 301 kB 00:01 2026-02-24T15:49:48.239 INFO:teuthology.orchestra.run.trial119.stdout:-------------------------------------------------------------------------------- 2026-02-24T15:49:48.239 INFO:teuthology.orchestra.run.trial119.stdout:Total 334 kB/s | 652 kB 00:01 2026-02-24T15:49:48.250 INFO:teuthology.orchestra.run.trial119.stdout:Running transaction check 2026-02-24T15:49:48.256 INFO:teuthology.orchestra.run.trial119.stdout:Transaction check succeeded. 2026-02-24T15:49:48.256 INFO:teuthology.orchestra.run.trial119.stdout:Running transaction test 2026-02-24T15:49:48.301 INFO:teuthology.orchestra.run.trial119.stdout:Transaction test succeeded. 2026-02-24T15:49:48.301 INFO:teuthology.orchestra.run.trial119.stdout:Running transaction 2026-02-24T15:49:48.382 INFO:teuthology.orchestra.run.trial119.stdout: Preparing : 1/1 2026-02-24T15:49:48.444 INFO:teuthology.orchestra.run.trial119.stdout: Installing : cryptsetup-2.8.1-3.el9.x86_64 1/2 2026-02-24T15:49:48.446 INFO:teuthology.orchestra.run.trial119.stdout: Installing : ceph-volume-2:20.3.0-5343.gd26e5c22.el9.noarch 2/2 2026-02-24T15:49:48.461 INFO:teuthology.orchestra.run.trial119.stdout: Running scriptlet: ceph-volume-2:20.3.0-5343.gd26e5c22.el9.noarch 2/2 2026-02-24T15:49:48.461 INFO:teuthology.orchestra.run.trial119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T15:49:48.461 INFO:teuthology.orchestra.run.trial119.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2026-02-24T15:49:48.461 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:49:48.696 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : ceph-volume-2:20.3.0-5343.gd26e5c22.el9.noarch 1/2 2026-02-24T15:49:48.760 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : cryptsetup-2.8.1-3.el9.x86_64 2/2 2026-02-24T15:49:48.760 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:49:48.760 INFO:teuthology.orchestra.run.trial119.stdout:Installed: 2026-02-24T15:49:48.760 INFO:teuthology.orchestra.run.trial119.stdout: ceph-volume-2:20.3.0-5343.gd26e5c22.el9.noarch cryptsetup-2.8.1-3.el9.x86_64 2026-02-24T15:49:48.760 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:49:48.760 INFO:teuthology.orchestra.run.trial119.stdout:Complete! 2026-02-24T15:49:48.790 DEBUG:teuthology.orchestra.run.trial119:> sudo yum -y install librados-devel 2026-02-24T15:49:49.111 INFO:teuthology.orchestra.run.trial119.stdout:Last metadata expiration check: 0:08:58 ago on Tue 24 Feb 2026 03:40:51 PM UTC. 2026-02-24T15:49:49.198 INFO:teuthology.orchestra.run.trial119.stdout:Dependencies resolved. 2026-02-24T15:49:49.199 INFO:teuthology.orchestra.run.trial119.stdout:================================================================================ 2026-02-24T15:49:49.200 INFO:teuthology.orchestra.run.trial119.stdout: Package Arch Version Repo Size 2026-02-24T15:49:49.200 INFO:teuthology.orchestra.run.trial119.stdout:================================================================================ 2026-02-24T15:49:49.200 INFO:teuthology.orchestra.run.trial119.stdout:Installing: 2026-02-24T15:49:49.200 INFO:teuthology.orchestra.run.trial119.stdout: librados-devel x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 124 k 2026-02-24T15:49:49.200 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:49:49.200 INFO:teuthology.orchestra.run.trial119.stdout:Transaction Summary 2026-02-24T15:49:49.200 INFO:teuthology.orchestra.run.trial119.stdout:================================================================================ 2026-02-24T15:49:49.200 INFO:teuthology.orchestra.run.trial119.stdout:Install 1 Package 2026-02-24T15:49:49.200 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:49:49.200 INFO:teuthology.orchestra.run.trial119.stdout:Total download size: 124 k 2026-02-24T15:49:49.201 INFO:teuthology.orchestra.run.trial119.stdout:Installed size: 434 k 2026-02-24T15:49:49.201 INFO:teuthology.orchestra.run.trial119.stdout:Downloading Packages: 2026-02-24T15:49:49.974 INFO:teuthology.orchestra.run.trial119.stdout:librados-devel-20.3.0-5343.gd26e5c22.el9.x86_64 160 kB/s | 124 kB 00:00 2026-02-24T15:49:49.974 INFO:teuthology.orchestra.run.trial119.stdout:-------------------------------------------------------------------------------- 2026-02-24T15:49:49.975 INFO:teuthology.orchestra.run.trial119.stdout:Total 159 kB/s | 124 kB 00:00 2026-02-24T15:49:49.975 INFO:teuthology.orchestra.run.trial119.stdout:Running transaction check 2026-02-24T15:49:49.978 INFO:teuthology.orchestra.run.trial119.stdout:Transaction check succeeded. 2026-02-24T15:49:49.978 INFO:teuthology.orchestra.run.trial119.stdout:Running transaction test 2026-02-24T15:49:50.009 INFO:teuthology.orchestra.run.trial119.stdout:Transaction test succeeded. 2026-02-24T15:49:50.009 INFO:teuthology.orchestra.run.trial119.stdout:Running transaction 2026-02-24T15:49:50.064 INFO:teuthology.orchestra.run.trial119.stdout: Preparing : 1/1 2026-02-24T15:49:50.081 INFO:teuthology.orchestra.run.trial119.stdout: Installing : librados-devel-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/1 2026-02-24T15:49:50.167 INFO:teuthology.orchestra.run.trial119.stdout: Running scriptlet: librados-devel-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/1 2026-02-24T15:49:50.230 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : librados-devel-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/1 2026-02-24T15:49:50.230 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:49:50.230 INFO:teuthology.orchestra.run.trial119.stdout:Installed: 2026-02-24T15:49:50.230 INFO:teuthology.orchestra.run.trial119.stdout: librados-devel-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:49:50.230 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:49:50.230 INFO:teuthology.orchestra.run.trial119.stdout:Complete! 2026-02-24T15:49:50.258 DEBUG:teuthology.orchestra.run.trial119:> sudo yum -y install libcephfs2 2026-02-24T15:49:50.576 INFO:teuthology.orchestra.run.trial119.stdout:Last metadata expiration check: 0:08:59 ago on Tue 24 Feb 2026 03:40:51 PM UTC. 2026-02-24T15:49:50.640 INFO:teuthology.orchestra.run.trial119.stdout:Package libcephfs2-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T15:49:50.659 INFO:teuthology.orchestra.run.trial119.stdout:Dependencies resolved. 2026-02-24T15:49:50.659 INFO:teuthology.orchestra.run.trial119.stdout:Nothing to do. 2026-02-24T15:49:50.660 INFO:teuthology.orchestra.run.trial119.stdout:Complete! 2026-02-24T15:49:50.680 DEBUG:teuthology.orchestra.run.trial119:> sudo yum -y install libcephfs-devel 2026-02-24T15:49:50.993 INFO:teuthology.orchestra.run.trial119.stdout:Last metadata expiration check: 0:08:59 ago on Tue 24 Feb 2026 03:40:51 PM UTC. 2026-02-24T15:49:51.077 INFO:teuthology.orchestra.run.trial119.stdout:Dependencies resolved. 2026-02-24T15:49:51.078 INFO:teuthology.orchestra.run.trial119.stdout:================================================================================ 2026-02-24T15:49:51.078 INFO:teuthology.orchestra.run.trial119.stdout: Package Arch Version Repo Size 2026-02-24T15:49:51.078 INFO:teuthology.orchestra.run.trial119.stdout:================================================================================ 2026-02-24T15:49:51.078 INFO:teuthology.orchestra.run.trial119.stdout:Installing: 2026-02-24T15:49:51.078 INFO:teuthology.orchestra.run.trial119.stdout: libcephfs-devel x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 37 k 2026-02-24T15:49:51.078 INFO:teuthology.orchestra.run.trial119.stdout:Installing dependencies: 2026-02-24T15:49:51.078 INFO:teuthology.orchestra.run.trial119.stdout: libcephfs-proxy2 x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 26 k 2026-02-24T15:49:51.078 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:49:51.079 INFO:teuthology.orchestra.run.trial119.stdout:Transaction Summary 2026-02-24T15:49:51.079 INFO:teuthology.orchestra.run.trial119.stdout:================================================================================ 2026-02-24T15:49:51.079 INFO:teuthology.orchestra.run.trial119.stdout:Install 2 Packages 2026-02-24T15:49:51.079 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:49:51.079 INFO:teuthology.orchestra.run.trial119.stdout:Total download size: 63 k 2026-02-24T15:49:51.079 INFO:teuthology.orchestra.run.trial119.stdout:Installed size: 227 k 2026-02-24T15:49:51.079 INFO:teuthology.orchestra.run.trial119.stdout:Downloading Packages: 2026-02-24T15:49:51.434 INFO:teuthology.orchestra.run.trial119.stdout:(1/2): libcephfs-proxy2-20.3.0-5343.gd26e5c22.e 72 kB/s | 26 kB 00:00 2026-02-24T15:49:51.797 INFO:teuthology.orchestra.run.trial119.stdout:(2/2): libcephfs-devel-20.3.0-5343.gd26e5c22.el 52 kB/s | 37 kB 00:00 2026-02-24T15:49:51.798 INFO:teuthology.orchestra.run.trial119.stdout:-------------------------------------------------------------------------------- 2026-02-24T15:49:51.798 INFO:teuthology.orchestra.run.trial119.stdout:Total 87 kB/s | 63 kB 00:00 2026-02-24T15:49:51.798 INFO:teuthology.orchestra.run.trial119.stdout:Running transaction check 2026-02-24T15:49:51.801 INFO:teuthology.orchestra.run.trial119.stdout:Transaction check succeeded. 2026-02-24T15:49:51.801 INFO:teuthology.orchestra.run.trial119.stdout:Running transaction test 2026-02-24T15:49:51.832 INFO:teuthology.orchestra.run.trial119.stdout:Transaction test succeeded. 2026-02-24T15:49:51.832 INFO:teuthology.orchestra.run.trial119.stdout:Running transaction 2026-02-24T15:49:51.881 INFO:teuthology.orchestra.run.trial119.stdout: Preparing : 1/1 2026-02-24T15:49:51.882 INFO:teuthology.orchestra.run.trial119.stdout: Installing : libcephfs-proxy2-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/2 2026-02-24T15:49:51.897 INFO:teuthology.orchestra.run.trial119.stdout: Running scriptlet: libcephfs-proxy2-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/2 2026-02-24T15:49:51.914 INFO:teuthology.orchestra.run.trial119.stdout: Installing : libcephfs-devel-2:20.3.0-5343.gd26e5c22.el9.x86_64 2/2 2026-02-24T15:49:51.978 INFO:teuthology.orchestra.run.trial119.stdout: Running scriptlet: libcephfs-devel-2:20.3.0-5343.gd26e5c22.el9.x86_64 2/2 2026-02-24T15:49:51.978 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : libcephfs-devel-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/2 2026-02-24T15:49:52.041 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : libcephfs-proxy2-2:20.3.0-5343.gd26e5c22.el9.x86_64 2/2 2026-02-24T15:49:52.041 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:49:52.041 INFO:teuthology.orchestra.run.trial119.stdout:Installed: 2026-02-24T15:49:52.041 INFO:teuthology.orchestra.run.trial119.stdout: libcephfs-devel-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:49:52.042 INFO:teuthology.orchestra.run.trial119.stdout: libcephfs-proxy2-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:49:52.042 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:49:52.042 INFO:teuthology.orchestra.run.trial119.stdout:Complete! 2026-02-24T15:49:52.068 DEBUG:teuthology.orchestra.run.trial119:> sudo yum -y install librados2 2026-02-24T15:49:52.380 INFO:teuthology.orchestra.run.trial119.stdout:Last metadata expiration check: 0:09:01 ago on Tue 24 Feb 2026 03:40:51 PM UTC. 2026-02-24T15:49:52.444 INFO:teuthology.orchestra.run.trial119.stdout:Package librados2-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T15:49:52.463 INFO:teuthology.orchestra.run.trial119.stdout:Dependencies resolved. 2026-02-24T15:49:52.464 INFO:teuthology.orchestra.run.trial119.stdout:Nothing to do. 2026-02-24T15:49:52.464 INFO:teuthology.orchestra.run.trial119.stdout:Complete! 2026-02-24T15:49:52.484 DEBUG:teuthology.orchestra.run.trial119:> sudo yum -y install librbd1 2026-02-24T15:49:52.800 INFO:teuthology.orchestra.run.trial119.stdout:Last metadata expiration check: 0:09:01 ago on Tue 24 Feb 2026 03:40:51 PM UTC. 2026-02-24T15:49:52.864 INFO:teuthology.orchestra.run.trial119.stdout:Package librbd1-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T15:49:52.883 INFO:teuthology.orchestra.run.trial119.stdout:Dependencies resolved. 2026-02-24T15:49:52.884 INFO:teuthology.orchestra.run.trial119.stdout:Nothing to do. 2026-02-24T15:49:52.884 INFO:teuthology.orchestra.run.trial119.stdout:Complete! 2026-02-24T15:49:52.905 DEBUG:teuthology.orchestra.run.trial119:> sudo yum -y install python3-rados 2026-02-24T15:49:53.219 INFO:teuthology.orchestra.run.trial119.stdout:Last metadata expiration check: 0:09:02 ago on Tue 24 Feb 2026 03:40:51 PM UTC. 2026-02-24T15:49:53.283 INFO:teuthology.orchestra.run.trial119.stdout:Package python3-rados-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T15:49:53.302 INFO:teuthology.orchestra.run.trial119.stdout:Dependencies resolved. 2026-02-24T15:49:53.303 INFO:teuthology.orchestra.run.trial119.stdout:Nothing to do. 2026-02-24T15:49:53.303 INFO:teuthology.orchestra.run.trial119.stdout:Complete! 2026-02-24T15:49:53.323 DEBUG:teuthology.orchestra.run.trial119:> sudo yum -y install python3-rgw 2026-02-24T15:49:53.633 INFO:teuthology.orchestra.run.trial119.stdout:Last metadata expiration check: 0:09:02 ago on Tue 24 Feb 2026 03:40:51 PM UTC. 2026-02-24T15:49:53.697 INFO:teuthology.orchestra.run.trial119.stdout:Package python3-rgw-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T15:49:53.716 INFO:teuthology.orchestra.run.trial119.stdout:Dependencies resolved. 2026-02-24T15:49:53.717 INFO:teuthology.orchestra.run.trial119.stdout:Nothing to do. 2026-02-24T15:49:53.717 INFO:teuthology.orchestra.run.trial119.stdout:Complete! 2026-02-24T15:49:53.737 DEBUG:teuthology.orchestra.run.trial119:> sudo yum -y install python3-cephfs 2026-02-24T15:49:54.051 INFO:teuthology.orchestra.run.trial119.stdout:Last metadata expiration check: 0:09:03 ago on Tue 24 Feb 2026 03:40:51 PM UTC. 2026-02-24T15:49:54.116 INFO:teuthology.orchestra.run.trial119.stdout:Package python3-cephfs-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T15:49:54.135 INFO:teuthology.orchestra.run.trial119.stdout:Dependencies resolved. 2026-02-24T15:49:54.136 INFO:teuthology.orchestra.run.trial119.stdout:Nothing to do. 2026-02-24T15:49:54.136 INFO:teuthology.orchestra.run.trial119.stdout:Complete! 2026-02-24T15:49:54.157 DEBUG:teuthology.orchestra.run.trial119:> sudo yum -y install python3-rbd 2026-02-24T15:49:54.474 INFO:teuthology.orchestra.run.trial119.stdout:Last metadata expiration check: 0:09:03 ago on Tue 24 Feb 2026 03:40:51 PM UTC. 2026-02-24T15:49:54.539 INFO:teuthology.orchestra.run.trial119.stdout:Package python3-rbd-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T15:49:54.557 INFO:teuthology.orchestra.run.trial119.stdout:Dependencies resolved. 2026-02-24T15:49:54.558 INFO:teuthology.orchestra.run.trial119.stdout:Nothing to do. 2026-02-24T15:49:54.558 INFO:teuthology.orchestra.run.trial119.stdout:Complete! 2026-02-24T15:49:54.580 DEBUG:teuthology.orchestra.run.trial119:> sudo yum -y install rbd-fuse 2026-02-24T15:49:54.900 INFO:teuthology.orchestra.run.trial119.stdout:Last metadata expiration check: 0:09:03 ago on Tue 24 Feb 2026 03:40:51 PM UTC. 2026-02-24T15:49:54.984 INFO:teuthology.orchestra.run.trial119.stdout:Dependencies resolved. 2026-02-24T15:49:54.985 INFO:teuthology.orchestra.run.trial119.stdout:================================================================================ 2026-02-24T15:49:54.985 INFO:teuthology.orchestra.run.trial119.stdout: Package Architecture Version Repository Size 2026-02-24T15:49:54.985 INFO:teuthology.orchestra.run.trial119.stdout:================================================================================ 2026-02-24T15:49:54.985 INFO:teuthology.orchestra.run.trial119.stdout:Installing: 2026-02-24T15:49:54.986 INFO:teuthology.orchestra.run.trial119.stdout: rbd-fuse x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 86 k 2026-02-24T15:49:54.986 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:49:54.986 INFO:teuthology.orchestra.run.trial119.stdout:Transaction Summary 2026-02-24T15:49:54.986 INFO:teuthology.orchestra.run.trial119.stdout:================================================================================ 2026-02-24T15:49:54.986 INFO:teuthology.orchestra.run.trial119.stdout:Install 1 Package 2026-02-24T15:49:54.986 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:49:54.986 INFO:teuthology.orchestra.run.trial119.stdout:Total download size: 86 k 2026-02-24T15:49:54.986 INFO:teuthology.orchestra.run.trial119.stdout:Installed size: 236 k 2026-02-24T15:49:54.986 INFO:teuthology.orchestra.run.trial119.stdout:Downloading Packages: 2026-02-24T15:49:55.889 INFO:teuthology.orchestra.run.trial119.stdout:rbd-fuse-20.3.0-5343.gd26e5c22.el9.x86_64.rpm 96 kB/s | 86 kB 00:00 2026-02-24T15:49:55.889 INFO:teuthology.orchestra.run.trial119.stdout:-------------------------------------------------------------------------------- 2026-02-24T15:49:55.890 INFO:teuthology.orchestra.run.trial119.stdout:Total 96 kB/s | 86 kB 00:00 2026-02-24T15:49:55.890 INFO:teuthology.orchestra.run.trial119.stdout:Running transaction check 2026-02-24T15:49:55.894 INFO:teuthology.orchestra.run.trial119.stdout:Transaction check succeeded. 2026-02-24T15:49:55.894 INFO:teuthology.orchestra.run.trial119.stdout:Running transaction test 2026-02-24T15:49:55.924 INFO:teuthology.orchestra.run.trial119.stdout:Transaction test succeeded. 2026-02-24T15:49:55.924 INFO:teuthology.orchestra.run.trial119.stdout:Running transaction 2026-02-24T15:49:55.975 INFO:teuthology.orchestra.run.trial119.stdout: Preparing : 1/1 2026-02-24T15:49:55.993 INFO:teuthology.orchestra.run.trial119.stdout: Installing : rbd-fuse-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/1 2026-02-24T15:49:56.079 INFO:teuthology.orchestra.run.trial119.stdout: Running scriptlet: rbd-fuse-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/1 2026-02-24T15:49:56.143 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : rbd-fuse-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/1 2026-02-24T15:49:56.143 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:49:56.144 INFO:teuthology.orchestra.run.trial119.stdout:Installed: 2026-02-24T15:49:56.144 INFO:teuthology.orchestra.run.trial119.stdout: rbd-fuse-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:49:56.144 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:49:56.144 INFO:teuthology.orchestra.run.trial119.stdout:Complete! 2026-02-24T15:49:56.172 DEBUG:teuthology.orchestra.run.trial119:> sudo yum -y install rbd-mirror 2026-02-24T15:49:56.489 INFO:teuthology.orchestra.run.trial119.stdout:Last metadata expiration check: 0:09:05 ago on Tue 24 Feb 2026 03:40:51 PM UTC. 2026-02-24T15:49:56.573 INFO:teuthology.orchestra.run.trial119.stdout:Dependencies resolved. 2026-02-24T15:49:56.574 INFO:teuthology.orchestra.run.trial119.stdout:================================================================================ 2026-02-24T15:49:56.574 INFO:teuthology.orchestra.run.trial119.stdout: Package Arch Version Repo Size 2026-02-24T15:49:56.574 INFO:teuthology.orchestra.run.trial119.stdout:================================================================================ 2026-02-24T15:49:56.574 INFO:teuthology.orchestra.run.trial119.stdout:Installing: 2026-02-24T15:49:56.574 INFO:teuthology.orchestra.run.trial119.stdout: rbd-mirror x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 2.6 M 2026-02-24T15:49:56.574 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:49:56.574 INFO:teuthology.orchestra.run.trial119.stdout:Transaction Summary 2026-02-24T15:49:56.574 INFO:teuthology.orchestra.run.trial119.stdout:================================================================================ 2026-02-24T15:49:56.574 INFO:teuthology.orchestra.run.trial119.stdout:Install 1 Package 2026-02-24T15:49:56.575 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:49:56.575 INFO:teuthology.orchestra.run.trial119.stdout:Total download size: 2.6 M 2026-02-24T15:49:56.575 INFO:teuthology.orchestra.run.trial119.stdout:Installed size: 11 M 2026-02-24T15:49:56.575 INFO:teuthology.orchestra.run.trial119.stdout:Downloading Packages: 2026-02-24T15:50:06.655 INFO:teuthology.orchestra.run.trial119.stdout:rbd-mirror-20.3.0-5343.gd26e5c22.el9.x86_64.rpm 263 kB/s | 2.6 MB 00:10 2026-02-24T15:50:06.656 INFO:teuthology.orchestra.run.trial119.stdout:-------------------------------------------------------------------------------- 2026-02-24T15:50:06.657 INFO:teuthology.orchestra.run.trial119.stdout:Total 263 kB/s | 2.6 MB 00:10 2026-02-24T15:50:06.657 INFO:teuthology.orchestra.run.trial119.stdout:Running transaction check 2026-02-24T15:50:06.662 INFO:teuthology.orchestra.run.trial119.stdout:Transaction check succeeded. 2026-02-24T15:50:06.662 INFO:teuthology.orchestra.run.trial119.stdout:Running transaction test 2026-02-24T15:50:06.700 INFO:teuthology.orchestra.run.trial119.stdout:Transaction test succeeded. 2026-02-24T15:50:06.700 INFO:teuthology.orchestra.run.trial119.stdout:Running transaction 2026-02-24T15:50:06.863 INFO:teuthology.orchestra.run.trial119.stdout: Preparing : 1/1 2026-02-24T15:50:06.864 INFO:teuthology.orchestra.run.trial119.stdout: Installing : rbd-mirror-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/1 2026-02-24T15:50:06.896 INFO:teuthology.orchestra.run.trial119.stdout: Running scriptlet: rbd-mirror-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/1 2026-02-24T15:50:06.896 INFO:teuthology.orchestra.run.trial119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-24T15:50:06.896 INFO:teuthology.orchestra.run.trial119.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2026-02-24T15:50:06.896 INFO:teuthology.orchestra.run.trial119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-24T15:50:06.896 INFO:teuthology.orchestra.run.trial119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-24T15:50:06.896 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:50:07.209 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : rbd-mirror-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/1 2026-02-24T15:50:07.209 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:50:07.209 INFO:teuthology.orchestra.run.trial119.stdout:Installed: 2026-02-24T15:50:07.209 INFO:teuthology.orchestra.run.trial119.stdout: rbd-mirror-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:50:07.209 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:50:07.209 INFO:teuthology.orchestra.run.trial119.stdout:Complete! 2026-02-24T15:50:07.237 DEBUG:teuthology.orchestra.run.trial119:> sudo yum -y install rbd-nbd 2026-02-24T15:50:07.560 INFO:teuthology.orchestra.run.trial119.stdout:Last metadata expiration check: 0:09:16 ago on Tue 24 Feb 2026 03:40:51 PM UTC. 2026-02-24T15:50:07.644 INFO:teuthology.orchestra.run.trial119.stdout:Dependencies resolved. 2026-02-24T15:50:07.645 INFO:teuthology.orchestra.run.trial119.stdout:================================================================================ 2026-02-24T15:50:07.645 INFO:teuthology.orchestra.run.trial119.stdout: Package Architecture Version Repository Size 2026-02-24T15:50:07.645 INFO:teuthology.orchestra.run.trial119.stdout:================================================================================ 2026-02-24T15:50:07.646 INFO:teuthology.orchestra.run.trial119.stdout:Installing: 2026-02-24T15:50:07.646 INFO:teuthology.orchestra.run.trial119.stdout: rbd-nbd x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 167 k 2026-02-24T15:50:07.646 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:50:07.646 INFO:teuthology.orchestra.run.trial119.stdout:Transaction Summary 2026-02-24T15:50:07.646 INFO:teuthology.orchestra.run.trial119.stdout:================================================================================ 2026-02-24T15:50:07.646 INFO:teuthology.orchestra.run.trial119.stdout:Install 1 Package 2026-02-24T15:50:07.646 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:50:07.646 INFO:teuthology.orchestra.run.trial119.stdout:Total download size: 167 k 2026-02-24T15:50:07.646 INFO:teuthology.orchestra.run.trial119.stdout:Installed size: 496 k 2026-02-24T15:50:07.646 INFO:teuthology.orchestra.run.trial119.stdout:Downloading Packages: 2026-02-24T15:50:08.840 INFO:teuthology.orchestra.run.trial119.stdout:rbd-nbd-20.3.0-5343.gd26e5c22.el9.x86_64.rpm 139 kB/s | 167 kB 00:01 2026-02-24T15:50:08.840 INFO:teuthology.orchestra.run.trial119.stdout:-------------------------------------------------------------------------------- 2026-02-24T15:50:08.840 INFO:teuthology.orchestra.run.trial119.stdout:Total 139 kB/s | 167 kB 00:01 2026-02-24T15:50:08.840 INFO:teuthology.orchestra.run.trial119.stdout:Running transaction check 2026-02-24T15:50:08.845 INFO:teuthology.orchestra.run.trial119.stdout:Transaction check succeeded. 2026-02-24T15:50:08.845 INFO:teuthology.orchestra.run.trial119.stdout:Running transaction test 2026-02-24T15:50:08.875 INFO:teuthology.orchestra.run.trial119.stdout:Transaction test succeeded. 2026-02-24T15:50:08.875 INFO:teuthology.orchestra.run.trial119.stdout:Running transaction 2026-02-24T15:50:08.931 INFO:teuthology.orchestra.run.trial119.stdout: Preparing : 1/1 2026-02-24T15:50:08.949 INFO:teuthology.orchestra.run.trial119.stdout: Installing : rbd-nbd-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/1 2026-02-24T15:50:09.035 INFO:teuthology.orchestra.run.trial119.stdout: Running scriptlet: rbd-nbd-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/1 2026-02-24T15:50:09.098 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : rbd-nbd-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/1 2026-02-24T15:50:09.098 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:50:09.098 INFO:teuthology.orchestra.run.trial119.stdout:Installed: 2026-02-24T15:50:09.098 INFO:teuthology.orchestra.run.trial119.stdout: rbd-nbd-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T15:50:09.098 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:50:09.099 INFO:teuthology.orchestra.run.trial119.stdout:Complete! 2026-02-24T15:50:09.127 DEBUG:teuthology.orchestra.run.trial119:> sudo yum -y install ceph-osd-crimson 2026-02-24T15:50:09.452 INFO:teuthology.orchestra.run.trial119.stdout:Last metadata expiration check: 0:09:18 ago on Tue 24 Feb 2026 03:40:51 PM UTC. 2026-02-24T15:50:09.517 INFO:teuthology.orchestra.run.trial119.stdout:Package ceph-osd-crimson-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T15:50:09.536 INFO:teuthology.orchestra.run.trial119.stdout:Dependencies resolved. 2026-02-24T15:50:09.537 INFO:teuthology.orchestra.run.trial119.stdout:Nothing to do. 2026-02-24T15:50:09.537 INFO:teuthology.orchestra.run.trial119.stdout:Complete! 2026-02-24T15:50:09.558 DEBUG:teuthology.orchestra.run.trial119:> sudo yum -y install ceph-osd-crimson-debuginfo 2026-02-24T15:50:09.875 INFO:teuthology.orchestra.run.trial119.stdout:Last metadata expiration check: 0:09:18 ago on Tue 24 Feb 2026 03:40:51 PM UTC. 2026-02-24T15:50:09.960 INFO:teuthology.orchestra.run.trial119.stdout:Dependencies resolved. 2026-02-24T15:50:09.960 INFO:teuthology.orchestra.run.trial119.stdout:================================================================================ 2026-02-24T15:50:09.960 INFO:teuthology.orchestra.run.trial119.stdout: Package Arch Version Repo Size 2026-02-24T15:50:09.960 INFO:teuthology.orchestra.run.trial119.stdout:================================================================================ 2026-02-24T15:50:09.960 INFO:teuthology.orchestra.run.trial119.stdout:Installing: 2026-02-24T15:50:09.960 INFO:teuthology.orchestra.run.trial119.stdout: ceph-osd-crimson-debuginfo x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 433 M 2026-02-24T15:50:09.960 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:50:09.960 INFO:teuthology.orchestra.run.trial119.stdout:Transaction Summary 2026-02-24T15:50:09.960 INFO:teuthology.orchestra.run.trial119.stdout:================================================================================ 2026-02-24T15:50:09.960 INFO:teuthology.orchestra.run.trial119.stdout:Install 1 Package 2026-02-24T15:50:09.960 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T15:50:09.960 INFO:teuthology.orchestra.run.trial119.stdout:Total download size: 433 M 2026-02-24T15:50:09.961 INFO:teuthology.orchestra.run.trial119.stdout:Installed size: 2.6 G 2026-02-24T15:50:09.961 INFO:teuthology.orchestra.run.trial119.stdout:Downloading Packages: 2026-02-24T16:04:57.567 INFO:teuthology.orchestra.run.trial119.stdout:ceph-osd-crimson-debuginfo-20.3.0-5343.gd26e5c2 500 kB/s | 433 MB 14:47 2026-02-24T16:04:57.568 INFO:teuthology.orchestra.run.trial119.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:04:57.568 INFO:teuthology.orchestra.run.trial119.stdout:Total 500 kB/s | 433 MB 14:47 2026-02-24T16:04:57.568 INFO:teuthology.orchestra.run.trial119.stdout:Running transaction check 2026-02-24T16:04:57.570 INFO:teuthology.orchestra.run.trial119.stdout:Transaction check succeeded. 2026-02-24T16:04:57.570 INFO:teuthology.orchestra.run.trial119.stdout:Running transaction test 2026-02-24T16:04:58.860 INFO:teuthology.orchestra.run.trial119.stdout:Transaction test succeeded. 2026-02-24T16:04:58.861 INFO:teuthology.orchestra.run.trial119.stdout:Running transaction 2026-02-24T16:05:22.901 INFO:teuthology.orchestra.run.trial119.stdout: Preparing : 1/1 2026-02-24T16:05:22.918 INFO:teuthology.orchestra.run.trial119.stdout: Installing : ceph-osd-crimson-debuginfo-2:20.3.0-5343.gd26e5c22.e 1/1 2026-02-24T16:05:24.869 INFO:teuthology.orchestra.run.trial119.stdout: Running scriptlet: ceph-osd-crimson-debuginfo-2:20.3.0-5343.gd26e5c22.e 1/1 2026-02-24T16:05:24.932 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : ceph-osd-crimson-debuginfo-2:20.3.0-5343.gd26e5c22.e 1/1 2026-02-24T16:05:24.932 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T16:05:24.932 INFO:teuthology.orchestra.run.trial119.stdout:Installed: 2026-02-24T16:05:24.933 INFO:teuthology.orchestra.run.trial119.stdout: ceph-osd-crimson-debuginfo-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T16:05:24.933 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T16:05:24.933 INFO:teuthology.orchestra.run.trial119.stdout:Complete! 2026-02-24T16:05:25.002 DEBUG:teuthology.parallel:result is None 2026-02-24T16:05:25.002 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=d26e5c228e3cdea4d585bcd077f8513a649ab5d1 2026-02-24T16:05:25.169 DEBUG:teuthology.orchestra.run.trial119:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-24T16:05:25.190 INFO:teuthology.orchestra.run.trial119.stdout:20.3.0-5343.gd26e5c22.el9 2026-02-24T16:05:25.191 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5343.gd26e5c22.el9 2026-02-24T16:05:25.191 INFO:teuthology.task.install:The correct ceph version 20.3.0-5343.gd26e5c22 is installed. 2026-02-24T16:05:25.194 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2026-02-24T16:05:25.194 DEBUG:teuthology.orchestra.run.trial119:> set -ex 2026-02-24T16:05:25.194 DEBUG:teuthology.orchestra.run.trial119:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-24T16:05:25.256 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2026-02-24T16:05:25.256 DEBUG:teuthology.orchestra.run.trial119:> set -ex 2026-02-24T16:05:25.256 DEBUG:teuthology.orchestra.run.trial119:> sudo dd of=/usr/bin/daemon-helper 2026-02-24T16:05:25.319 DEBUG:teuthology.orchestra.run.trial119:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-24T16:05:25.384 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2026-02-24T16:05:25.384 DEBUG:teuthology.orchestra.run.trial119:> set -ex 2026-02-24T16:05:25.384 DEBUG:teuthology.orchestra.run.trial119:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-24T16:05:25.448 DEBUG:teuthology.orchestra.run.trial119:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-24T16:05:25.511 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2026-02-24T16:05:25.512 DEBUG:teuthology.orchestra.run.trial119:> set -ex 2026-02-24T16:05:25.512 DEBUG:teuthology.orchestra.run.trial119:> sudo dd of=/usr/bin/stdin-killer 2026-02-24T16:05:25.574 DEBUG:teuthology.orchestra.run.trial119:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-24T16:05:25.637 INFO:teuthology.run_tasks:Running task pexec... 2026-02-24T16:05:25.656 INFO:teuthology.task.pexec:Executing custom commands... 2026-02-24T16:05:25.657 DEBUG:teuthology.orchestra.run.trial119:> TESTDIR=/home/ubuntu/cephtest bash -s 2026-02-24T16:05:25.679 INFO:teuthology.task.pexec:Running commands on host ubuntu@trial119.front.sepia.ceph.com 2026-02-24T16:05:25.680 INFO:teuthology.task.pexec:sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson 2026-02-24T16:05:25.731 DEBUG:teuthology.parallel:result is None 2026-02-24T16:05:25.731 INFO:teuthology.run_tasks:Running task install... 2026-02-24T16:05:25.745 DEBUG:teuthology.task.install:project ceph 2026-02-24T16:05:25.745 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'debug', 'sha1': 'd26e5c228e3cdea4d585bcd077f8513a649ab5d1'}} 2026-02-24T16:05:25.745 DEBUG:teuthology.task.install:config {'flavor': 'debug', 'sha1': 'd26e5c228e3cdea4d585bcd077f8513a649ab5d1'} 2026-02-24T16:05:25.745 INFO:teuthology.task.install:Using flavor: debug 2026-02-24T16:05:25.754 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-osd', 'ceph-osd-classic', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-osd', 'ceph-osd-classic', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-24T16:05:25.754 INFO:teuthology.task.install:extra packages: [] 2026-02-24T16:05:25.754 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'debug', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd26e5c228e3cdea4d585bcd077f8513a649ab5d1', 'tag': None, 'wait_for_package': False} 2026-02-24T16:05:25.755 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=d26e5c228e3cdea4d585bcd077f8513a649ab5d1 2026-02-24T16:05:25.912 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-liucm-shard-redesign-crimson-only/d26e5c228e3cdea4d585bcd077f8513a649ab5d1/centos/9/flavors/debug/ 2026-02-24T16:05:25.912 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5343.gd26e5c22 2026-02-24T16:05:26.294 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-liucm-shard-redesign-crimson-only/d26e5c228e3cdea4d585bcd077f8513a649ab5d1/centos/9/flavors/debug/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-liucm-shard-redesign-crimson-only/d26e5c228e3cdea4d585bcd077f8513a649ab5d1/centos/9/flavors/debug/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-liucm-shard-redesign-crimson-only/d26e5c228e3cdea4d585bcd077f8513a649ab5d1/centos/9/flavors/debug/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-24T16:05:26.295 DEBUG:teuthology.orchestra.run.trial119:> set -ex 2026-02-24T16:05:26.295 DEBUG:teuthology.orchestra.run.trial119:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-24T16:05:26.321 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2026-02-24T16:05:26.321 DEBUG:teuthology.orchestra.run.trial119:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d26e5c228e3cdea4d585bcd077f8513a649ab5d1/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-24T16:05:26.386 DEBUG:teuthology.orchestra.run.trial119:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-24T16:05:26.450 DEBUG:teuthology.orchestra.run.trial119:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-24T16:05:26.506 INFO:teuthology.orchestra.run.trial119.stdout:check_obsoletes = 1 2026-02-24T16:05:26.516 DEBUG:teuthology.orchestra.run.trial119:> sudo yum clean all 2026-02-24T16:05:26.676 INFO:teuthology.orchestra.run.trial119.stdout:56 files removed 2026-02-24T16:05:26.691 DEBUG:teuthology.orchestra.run.trial119:> sudo yum -y install ceph-radosgw 2026-02-24T16:05:27.568 INFO:teuthology.orchestra.run.trial119.stdout:ceph packages for x86_64 127 kB/s | 92 kB 00:00 2026-02-24T16:05:28.237 INFO:teuthology.orchestra.run.trial119.stdout:ceph noarch packages 37 kB/s | 24 kB 00:00 2026-02-24T16:05:29.013 INFO:teuthology.orchestra.run.trial119.stdout:ceph source packages 2.7 kB/s | 2.1 kB 00:00 2026-02-24T16:05:30.720 INFO:teuthology.orchestra.run.trial119.stdout:CentOS Stream 9 - BaseOS 5.3 MB/s | 8.9 MB 00:01 2026-02-24T16:05:33.894 INFO:teuthology.orchestra.run.trial119.stdout:CentOS Stream 9 - AppStream 10 MB/s | 27 MB 00:02 2026-02-24T16:05:42.505 INFO:teuthology.orchestra.run.trial119.stdout:CentOS Stream 9 - CRB 1.3 MB/s | 8.0 MB 00:06 2026-02-24T16:05:43.636 INFO:teuthology.orchestra.run.trial119.stdout:CentOS Stream 9 - Extras packages 51 kB/s | 20 kB 00:00 2026-02-24T16:05:44.162 INFO:teuthology.orchestra.run.trial119.stdout:Extra Packages for Enterprise Linux 44 MB/s | 20 MB 00:00 2026-02-24T16:05:47.342 INFO:teuthology.orchestra.run.trial119.stdout:lab-extras 1.3 MB/s | 50 kB 00:00 2026-02-24T16:05:48.535 INFO:teuthology.orchestra.run.trial119.stdout:Package ceph-radosgw-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:05:48.554 INFO:teuthology.orchestra.run.trial119.stdout:Dependencies resolved. 2026-02-24T16:05:48.555 INFO:teuthology.orchestra.run.trial119.stdout:Nothing to do. 2026-02-24T16:05:48.555 INFO:teuthology.orchestra.run.trial119.stdout:Complete! 2026-02-24T16:05:48.578 DEBUG:teuthology.orchestra.run.trial119:> sudo yum -y install ceph-test 2026-02-24T16:05:48.902 INFO:teuthology.orchestra.run.trial119.stdout:Last metadata expiration check: 0:00:01 ago on Tue 24 Feb 2026 04:05:47 PM UTC. 2026-02-24T16:05:48.967 INFO:teuthology.orchestra.run.trial119.stdout:Package ceph-test-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:05:48.986 INFO:teuthology.orchestra.run.trial119.stdout:Dependencies resolved. 2026-02-24T16:05:48.987 INFO:teuthology.orchestra.run.trial119.stdout:Nothing to do. 2026-02-24T16:05:48.987 INFO:teuthology.orchestra.run.trial119.stdout:Complete! 2026-02-24T16:05:49.009 DEBUG:teuthology.orchestra.run.trial119:> sudo yum -y install ceph 2026-02-24T16:05:49.322 INFO:teuthology.orchestra.run.trial119.stdout:Last metadata expiration check: 0:00:02 ago on Tue 24 Feb 2026 04:05:47 PM UTC. 2026-02-24T16:05:49.386 INFO:teuthology.orchestra.run.trial119.stdout:Package ceph-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:05:49.405 INFO:teuthology.orchestra.run.trial119.stdout:Dependencies resolved. 2026-02-24T16:05:49.405 INFO:teuthology.orchestra.run.trial119.stdout:Nothing to do. 2026-02-24T16:05:49.405 INFO:teuthology.orchestra.run.trial119.stdout:Complete! 2026-02-24T16:05:49.427 DEBUG:teuthology.orchestra.run.trial119:> sudo yum -y install ceph-base 2026-02-24T16:05:49.746 INFO:teuthology.orchestra.run.trial119.stdout:Last metadata expiration check: 0:00:02 ago on Tue 24 Feb 2026 04:05:47 PM UTC. 2026-02-24T16:05:49.811 INFO:teuthology.orchestra.run.trial119.stdout:Package ceph-base-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:05:49.829 INFO:teuthology.orchestra.run.trial119.stdout:Dependencies resolved. 2026-02-24T16:05:49.830 INFO:teuthology.orchestra.run.trial119.stdout:Nothing to do. 2026-02-24T16:05:49.830 INFO:teuthology.orchestra.run.trial119.stdout:Complete! 2026-02-24T16:05:49.853 DEBUG:teuthology.orchestra.run.trial119:> sudo yum -y install cephadm 2026-02-24T16:05:50.172 INFO:teuthology.orchestra.run.trial119.stdout:Last metadata expiration check: 0:00:03 ago on Tue 24 Feb 2026 04:05:47 PM UTC. 2026-02-24T16:05:50.237 INFO:teuthology.orchestra.run.trial119.stdout:Package cephadm-2:20.3.0-5343.gd26e5c22.el9.noarch is already installed. 2026-02-24T16:05:50.256 INFO:teuthology.orchestra.run.trial119.stdout:Dependencies resolved. 2026-02-24T16:05:50.257 INFO:teuthology.orchestra.run.trial119.stdout:Nothing to do. 2026-02-24T16:05:50.257 INFO:teuthology.orchestra.run.trial119.stdout:Complete! 2026-02-24T16:05:50.279 DEBUG:teuthology.orchestra.run.trial119:> sudo yum -y install ceph-immutable-object-cache 2026-02-24T16:05:50.595 INFO:teuthology.orchestra.run.trial119.stdout:Last metadata expiration check: 0:00:03 ago on Tue 24 Feb 2026 04:05:47 PM UTC. 2026-02-24T16:05:50.662 INFO:teuthology.orchestra.run.trial119.stdout:Package ceph-immutable-object-cache-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:05:50.681 INFO:teuthology.orchestra.run.trial119.stdout:Dependencies resolved. 2026-02-24T16:05:50.681 INFO:teuthology.orchestra.run.trial119.stdout:Nothing to do. 2026-02-24T16:05:50.681 INFO:teuthology.orchestra.run.trial119.stdout:Complete! 2026-02-24T16:05:50.703 DEBUG:teuthology.orchestra.run.trial119:> sudo yum -y install ceph-mgr 2026-02-24T16:05:51.021 INFO:teuthology.orchestra.run.trial119.stdout:Last metadata expiration check: 0:00:04 ago on Tue 24 Feb 2026 04:05:47 PM UTC. 2026-02-24T16:05:51.086 INFO:teuthology.orchestra.run.trial119.stdout:Package ceph-mgr-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:05:51.105 INFO:teuthology.orchestra.run.trial119.stdout:Dependencies resolved. 2026-02-24T16:05:51.106 INFO:teuthology.orchestra.run.trial119.stdout:Nothing to do. 2026-02-24T16:05:51.106 INFO:teuthology.orchestra.run.trial119.stdout:Complete! 2026-02-24T16:05:51.128 DEBUG:teuthology.orchestra.run.trial119:> sudo yum -y install ceph-mgr-dashboard 2026-02-24T16:05:51.446 INFO:teuthology.orchestra.run.trial119.stdout:Last metadata expiration check: 0:00:04 ago on Tue 24 Feb 2026 04:05:47 PM UTC. 2026-02-24T16:05:51.511 INFO:teuthology.orchestra.run.trial119.stdout:Package ceph-mgr-dashboard-2:20.3.0-5343.gd26e5c22.el9.noarch is already installed. 2026-02-24T16:05:51.530 INFO:teuthology.orchestra.run.trial119.stdout:Dependencies resolved. 2026-02-24T16:05:51.530 INFO:teuthology.orchestra.run.trial119.stdout:Nothing to do. 2026-02-24T16:05:51.530 INFO:teuthology.orchestra.run.trial119.stdout:Complete! 2026-02-24T16:05:51.552 DEBUG:teuthology.orchestra.run.trial119:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-24T16:05:51.868 INFO:teuthology.orchestra.run.trial119.stdout:Last metadata expiration check: 0:00:04 ago on Tue 24 Feb 2026 04:05:47 PM UTC. 2026-02-24T16:05:51.933 INFO:teuthology.orchestra.run.trial119.stdout:Package ceph-mgr-diskprediction-local-2:20.3.0-5343.gd26e5c22.el9.noarch is already installed. 2026-02-24T16:05:51.951 INFO:teuthology.orchestra.run.trial119.stdout:Dependencies resolved. 2026-02-24T16:05:51.952 INFO:teuthology.orchestra.run.trial119.stdout:Nothing to do. 2026-02-24T16:05:51.952 INFO:teuthology.orchestra.run.trial119.stdout:Complete! 2026-02-24T16:05:51.973 DEBUG:teuthology.orchestra.run.trial119:> sudo yum -y install ceph-mgr-rook 2026-02-24T16:05:52.295 INFO:teuthology.orchestra.run.trial119.stdout:Last metadata expiration check: 0:00:05 ago on Tue 24 Feb 2026 04:05:47 PM UTC. 2026-02-24T16:05:52.360 INFO:teuthology.orchestra.run.trial119.stdout:Package ceph-mgr-rook-2:20.3.0-5343.gd26e5c22.el9.noarch is already installed. 2026-02-24T16:05:52.379 INFO:teuthology.orchestra.run.trial119.stdout:Dependencies resolved. 2026-02-24T16:05:52.379 INFO:teuthology.orchestra.run.trial119.stdout:Nothing to do. 2026-02-24T16:05:52.380 INFO:teuthology.orchestra.run.trial119.stdout:Complete! 2026-02-24T16:05:52.401 DEBUG:teuthology.orchestra.run.trial119:> sudo yum -y install ceph-mgr-cephadm 2026-02-24T16:05:52.725 INFO:teuthology.orchestra.run.trial119.stdout:Last metadata expiration check: 0:00:05 ago on Tue 24 Feb 2026 04:05:47 PM UTC. 2026-02-24T16:05:52.790 INFO:teuthology.orchestra.run.trial119.stdout:Package ceph-mgr-cephadm-2:20.3.0-5343.gd26e5c22.el9.noarch is already installed. 2026-02-24T16:05:52.809 INFO:teuthology.orchestra.run.trial119.stdout:Dependencies resolved. 2026-02-24T16:05:52.810 INFO:teuthology.orchestra.run.trial119.stdout:Nothing to do. 2026-02-24T16:05:52.810 INFO:teuthology.orchestra.run.trial119.stdout:Complete! 2026-02-24T16:05:52.832 DEBUG:teuthology.orchestra.run.trial119:> sudo yum -y install ceph-osd 2026-02-24T16:05:53.150 INFO:teuthology.orchestra.run.trial119.stdout:Last metadata expiration check: 0:00:06 ago on Tue 24 Feb 2026 04:05:47 PM UTC. 2026-02-24T16:05:53.216 INFO:teuthology.orchestra.run.trial119.stdout:Package ceph-osd-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:05:53.235 INFO:teuthology.orchestra.run.trial119.stdout:Dependencies resolved. 2026-02-24T16:05:53.236 INFO:teuthology.orchestra.run.trial119.stdout:Nothing to do. 2026-02-24T16:05:53.236 INFO:teuthology.orchestra.run.trial119.stdout:Complete! 2026-02-24T16:05:53.258 DEBUG:teuthology.orchestra.run.trial119:> sudo yum -y install ceph-osd-classic 2026-02-24T16:05:53.577 INFO:teuthology.orchestra.run.trial119.stdout:Last metadata expiration check: 0:00:06 ago on Tue 24 Feb 2026 04:05:47 PM UTC. 2026-02-24T16:05:53.642 INFO:teuthology.orchestra.run.trial119.stdout:Package ceph-osd-classic-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:05:53.661 INFO:teuthology.orchestra.run.trial119.stdout:Dependencies resolved. 2026-02-24T16:05:53.662 INFO:teuthology.orchestra.run.trial119.stdout:Nothing to do. 2026-02-24T16:05:53.662 INFO:teuthology.orchestra.run.trial119.stdout:Complete! 2026-02-24T16:05:53.683 DEBUG:teuthology.orchestra.run.trial119:> sudo yum -y install ceph-fuse 2026-02-24T16:05:54.003 INFO:teuthology.orchestra.run.trial119.stdout:Last metadata expiration check: 0:00:07 ago on Tue 24 Feb 2026 04:05:47 PM UTC. 2026-02-24T16:05:54.068 INFO:teuthology.orchestra.run.trial119.stdout:Package ceph-fuse-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:05:54.087 INFO:teuthology.orchestra.run.trial119.stdout:Dependencies resolved. 2026-02-24T16:05:54.088 INFO:teuthology.orchestra.run.trial119.stdout:Nothing to do. 2026-02-24T16:05:54.088 INFO:teuthology.orchestra.run.trial119.stdout:Complete! 2026-02-24T16:05:54.110 DEBUG:teuthology.orchestra.run.trial119:> sudo yum -y install ceph-volume 2026-02-24T16:05:54.427 INFO:teuthology.orchestra.run.trial119.stdout:Last metadata expiration check: 0:00:07 ago on Tue 24 Feb 2026 04:05:47 PM UTC. 2026-02-24T16:05:54.491 INFO:teuthology.orchestra.run.trial119.stdout:Package ceph-volume-2:20.3.0-5343.gd26e5c22.el9.noarch is already installed. 2026-02-24T16:05:54.510 INFO:teuthology.orchestra.run.trial119.stdout:Dependencies resolved. 2026-02-24T16:05:54.511 INFO:teuthology.orchestra.run.trial119.stdout:Nothing to do. 2026-02-24T16:05:54.511 INFO:teuthology.orchestra.run.trial119.stdout:Complete! 2026-02-24T16:05:54.532 DEBUG:teuthology.orchestra.run.trial119:> sudo yum -y install librados-devel 2026-02-24T16:05:54.848 INFO:teuthology.orchestra.run.trial119.stdout:Last metadata expiration check: 0:00:07 ago on Tue 24 Feb 2026 04:05:47 PM UTC. 2026-02-24T16:05:54.914 INFO:teuthology.orchestra.run.trial119.stdout:Package librados-devel-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:05:54.933 INFO:teuthology.orchestra.run.trial119.stdout:Dependencies resolved. 2026-02-24T16:05:54.933 INFO:teuthology.orchestra.run.trial119.stdout:Nothing to do. 2026-02-24T16:05:54.933 INFO:teuthology.orchestra.run.trial119.stdout:Complete! 2026-02-24T16:05:54.955 DEBUG:teuthology.orchestra.run.trial119:> sudo yum -y install libcephfs2 2026-02-24T16:05:55.273 INFO:teuthology.orchestra.run.trial119.stdout:Last metadata expiration check: 0:00:08 ago on Tue 24 Feb 2026 04:05:47 PM UTC. 2026-02-24T16:05:55.338 INFO:teuthology.orchestra.run.trial119.stdout:Package libcephfs2-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:05:55.357 INFO:teuthology.orchestra.run.trial119.stdout:Dependencies resolved. 2026-02-24T16:05:55.358 INFO:teuthology.orchestra.run.trial119.stdout:Nothing to do. 2026-02-24T16:05:55.358 INFO:teuthology.orchestra.run.trial119.stdout:Complete! 2026-02-24T16:05:55.380 DEBUG:teuthology.orchestra.run.trial119:> sudo yum -y install libcephfs-devel 2026-02-24T16:05:55.697 INFO:teuthology.orchestra.run.trial119.stdout:Last metadata expiration check: 0:00:08 ago on Tue 24 Feb 2026 04:05:47 PM UTC. 2026-02-24T16:05:55.762 INFO:teuthology.orchestra.run.trial119.stdout:Package libcephfs-devel-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:05:55.781 INFO:teuthology.orchestra.run.trial119.stdout:Dependencies resolved. 2026-02-24T16:05:55.782 INFO:teuthology.orchestra.run.trial119.stdout:Nothing to do. 2026-02-24T16:05:55.782 INFO:teuthology.orchestra.run.trial119.stdout:Complete! 2026-02-24T16:05:55.804 DEBUG:teuthology.orchestra.run.trial119:> sudo yum -y install librados2 2026-02-24T16:05:56.121 INFO:teuthology.orchestra.run.trial119.stdout:Last metadata expiration check: 0:00:09 ago on Tue 24 Feb 2026 04:05:47 PM UTC. 2026-02-24T16:05:56.187 INFO:teuthology.orchestra.run.trial119.stdout:Package librados2-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:05:56.206 INFO:teuthology.orchestra.run.trial119.stdout:Dependencies resolved. 2026-02-24T16:05:56.206 INFO:teuthology.orchestra.run.trial119.stdout:Nothing to do. 2026-02-24T16:05:56.206 INFO:teuthology.orchestra.run.trial119.stdout:Complete! 2026-02-24T16:05:56.229 DEBUG:teuthology.orchestra.run.trial119:> sudo yum -y install librbd1 2026-02-24T16:05:56.544 INFO:teuthology.orchestra.run.trial119.stdout:Last metadata expiration check: 0:00:09 ago on Tue 24 Feb 2026 04:05:47 PM UTC. 2026-02-24T16:05:56.609 INFO:teuthology.orchestra.run.trial119.stdout:Package librbd1-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:05:56.628 INFO:teuthology.orchestra.run.trial119.stdout:Dependencies resolved. 2026-02-24T16:05:56.629 INFO:teuthology.orchestra.run.trial119.stdout:Nothing to do. 2026-02-24T16:05:56.629 INFO:teuthology.orchestra.run.trial119.stdout:Complete! 2026-02-24T16:05:56.650 DEBUG:teuthology.orchestra.run.trial119:> sudo yum -y install python3-rados 2026-02-24T16:05:56.966 INFO:teuthology.orchestra.run.trial119.stdout:Last metadata expiration check: 0:00:09 ago on Tue 24 Feb 2026 04:05:47 PM UTC. 2026-02-24T16:05:57.030 INFO:teuthology.orchestra.run.trial119.stdout:Package python3-rados-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:05:57.049 INFO:teuthology.orchestra.run.trial119.stdout:Dependencies resolved. 2026-02-24T16:05:57.050 INFO:teuthology.orchestra.run.trial119.stdout:Nothing to do. 2026-02-24T16:05:57.050 INFO:teuthology.orchestra.run.trial119.stdout:Complete! 2026-02-24T16:05:57.072 DEBUG:teuthology.orchestra.run.trial119:> sudo yum -y install python3-rgw 2026-02-24T16:05:57.391 INFO:teuthology.orchestra.run.trial119.stdout:Last metadata expiration check: 0:00:10 ago on Tue 24 Feb 2026 04:05:47 PM UTC. 2026-02-24T16:05:57.455 INFO:teuthology.orchestra.run.trial119.stdout:Package python3-rgw-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:05:57.474 INFO:teuthology.orchestra.run.trial119.stdout:Dependencies resolved. 2026-02-24T16:05:57.475 INFO:teuthology.orchestra.run.trial119.stdout:Nothing to do. 2026-02-24T16:05:57.475 INFO:teuthology.orchestra.run.trial119.stdout:Complete! 2026-02-24T16:05:57.497 DEBUG:teuthology.orchestra.run.trial119:> sudo yum -y install python3-cephfs 2026-02-24T16:05:57.816 INFO:teuthology.orchestra.run.trial119.stdout:Last metadata expiration check: 0:00:10 ago on Tue 24 Feb 2026 04:05:47 PM UTC. 2026-02-24T16:05:57.881 INFO:teuthology.orchestra.run.trial119.stdout:Package python3-cephfs-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:05:57.900 INFO:teuthology.orchestra.run.trial119.stdout:Dependencies resolved. 2026-02-24T16:05:57.901 INFO:teuthology.orchestra.run.trial119.stdout:Nothing to do. 2026-02-24T16:05:57.901 INFO:teuthology.orchestra.run.trial119.stdout:Complete! 2026-02-24T16:05:57.923 DEBUG:teuthology.orchestra.run.trial119:> sudo yum -y install python3-rbd 2026-02-24T16:05:58.242 INFO:teuthology.orchestra.run.trial119.stdout:Last metadata expiration check: 0:00:11 ago on Tue 24 Feb 2026 04:05:47 PM UTC. 2026-02-24T16:05:58.306 INFO:teuthology.orchestra.run.trial119.stdout:Package python3-rbd-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:05:58.325 INFO:teuthology.orchestra.run.trial119.stdout:Dependencies resolved. 2026-02-24T16:05:58.326 INFO:teuthology.orchestra.run.trial119.stdout:Nothing to do. 2026-02-24T16:05:58.326 INFO:teuthology.orchestra.run.trial119.stdout:Complete! 2026-02-24T16:05:58.347 DEBUG:teuthology.orchestra.run.trial119:> sudo yum -y install rbd-fuse 2026-02-24T16:05:58.662 INFO:teuthology.orchestra.run.trial119.stdout:Last metadata expiration check: 0:00:11 ago on Tue 24 Feb 2026 04:05:47 PM UTC. 2026-02-24T16:05:58.727 INFO:teuthology.orchestra.run.trial119.stdout:Package rbd-fuse-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:05:58.746 INFO:teuthology.orchestra.run.trial119.stdout:Dependencies resolved. 2026-02-24T16:05:58.747 INFO:teuthology.orchestra.run.trial119.stdout:Nothing to do. 2026-02-24T16:05:58.747 INFO:teuthology.orchestra.run.trial119.stdout:Complete! 2026-02-24T16:05:58.768 DEBUG:teuthology.orchestra.run.trial119:> sudo yum -y install rbd-mirror 2026-02-24T16:05:59.090 INFO:teuthology.orchestra.run.trial119.stdout:Last metadata expiration check: 0:00:12 ago on Tue 24 Feb 2026 04:05:47 PM UTC. 2026-02-24T16:05:59.154 INFO:teuthology.orchestra.run.trial119.stdout:Package rbd-mirror-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:05:59.173 INFO:teuthology.orchestra.run.trial119.stdout:Dependencies resolved. 2026-02-24T16:05:59.174 INFO:teuthology.orchestra.run.trial119.stdout:Nothing to do. 2026-02-24T16:05:59.174 INFO:teuthology.orchestra.run.trial119.stdout:Complete! 2026-02-24T16:05:59.196 DEBUG:teuthology.orchestra.run.trial119:> sudo yum -y install rbd-nbd 2026-02-24T16:05:59.518 INFO:teuthology.orchestra.run.trial119.stdout:Last metadata expiration check: 0:00:12 ago on Tue 24 Feb 2026 04:05:47 PM UTC. 2026-02-24T16:05:59.583 INFO:teuthology.orchestra.run.trial119.stdout:Package rbd-nbd-2:20.3.0-5343.gd26e5c22.el9.x86_64 is already installed. 2026-02-24T16:05:59.602 INFO:teuthology.orchestra.run.trial119.stdout:Dependencies resolved. 2026-02-24T16:05:59.602 INFO:teuthology.orchestra.run.trial119.stdout:Nothing to do. 2026-02-24T16:05:59.602 INFO:teuthology.orchestra.run.trial119.stdout:Complete! 2026-02-24T16:05:59.627 DEBUG:teuthology.parallel:result is None 2026-02-24T16:05:59.628 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=d26e5c228e3cdea4d585bcd077f8513a649ab5d1 2026-02-24T16:05:59.792 DEBUG:teuthology.orchestra.run.trial119:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-24T16:05:59.813 INFO:teuthology.orchestra.run.trial119.stdout:20.3.0-5343.gd26e5c22.el9 2026-02-24T16:05:59.814 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5343.gd26e5c22.el9 2026-02-24T16:05:59.814 INFO:teuthology.task.install:The correct ceph version 20.3.0-5343.gd26e5c22 is installed. 2026-02-24T16:05:59.816 INFO:teuthology.task.install.util:Utilities already shipped, skip it... 2026-02-24T16:05:59.816 INFO:teuthology.run_tasks:Running task ceph... 2026-02-24T16:05:59.920 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2026-02-24T16:05:59.920 DEBUG:teuthology.orchestra.run.trial119:> sudo chmod 777 /var/log/ceph 2026-02-24T16:05:59.952 INFO:tasks.ceph:Disabling ceph logrotate... 2026-02-24T16:05:59.952 DEBUG:teuthology.orchestra.run.trial119:> sudo rm -f -- /etc/logrotate.d/ceph 2026-02-24T16:06:00.015 INFO:tasks.ceph:Creating extra log directories... 2026-02-24T16:06:00.015 DEBUG:teuthology.orchestra.run.trial119:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2026-02-24T16:06:00.087 INFO:tasks.ceph:Creating ceph cluster ceph... 2026-02-24T16:06:00.087 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore max concurrent transactions': 8}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2026-02-24T16:06:00.087 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/mnt/teuthology/liucm-2026-02-24_15:28:57-crimson-rados-wip-liucm-shard-redesign-crimson-only-distro-debug-trial/68197', 'branch': 'wip-liucm-shard-redesign-crimson-only', 'description': 'crimson-rados/perf/{0-crimson_install clusters/{crimson-fixed-1} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/segmented$/{crimson_seastore_segmented} perf_ignorelist settings/optimized workloads/radosbench_4M_seq_read}', 'email': None, 'first_in_suite': False, 'flavor': 'debug', 'job_id': '68197', 'kernel': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'trial', 'name': 'liucm-2026-02-24_15:28:57-crimson-rados-wip-liucm-shard-redesign-crimson-only-distro-debug-trial', 'no_nested_subset': False, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-liucm-shard-redesign-crimson-only'}, 'ceph': {'conf': {'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd': {'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore max concurrent transactions': 8}}, 'crimson_compat': True, 'flavor': 'debug', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH'], 'sha1': 'd26e5c228e3cdea4d585bcd077f8513a649ab5d1'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'global': {'osd crush chooseleaf type': 0, 'osd pool default pg num': 128, 'osd pool default pgp num': 128, 'osd pool default size': 2}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'debug', 'sha1': 'd26e5c228e3cdea4d585bcd077f8513a649ab5d1'}}, 'workunit': {'branch': 'wip-liucm-shard-redesign-crimson-only', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': 'd26e5c228e3cdea4d585bcd077f8513a649ab5d1'}}, 'owner': 'scheduled_liucm@soko04.front.sepia.ceph.com', 'priority': 120, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 9834, 'sha1': 'd26e5c228e3cdea4d585bcd077f8513a649ab5d1', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'wip-liucm-shard-redesign-crimson-only', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_d26e5c228e3cdea4d585bcd077f8513a649ab5d1/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': 'd26e5c228e3cdea4d585bcd077f8513a649ab5d1', 'targets': {'trial119.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGL3hFP2Fv6ElfoZ773NC+66AI3BSya1msosvrQ+0BjfOAmYaJPpCRRnt8IzVAiUDC9FbKBF0YwlhyfT9afRAwg='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'flavor': 'debug', 'sha1': 'd26e5c228e3cdea4d585bcd077f8513a649ab5d1'}}, {'pexec': {'all': ['sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson']}}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore max concurrent transactions': 8}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}}, 'wait-for-scrub': False, 'crimson_compat': True, 'flavor': 'debug', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH'], 'sha1': 'd26e5c228e3cdea4d585bcd077f8513a649ab5d1', 'cluster': 'ceph'}}, {'ssh_keys': None}, {'cbt': {'benchmarks': {'radosbench': {'concurrent_ops': 4, 'concurrent_procs': 2, 'op_size': [4194304], 'pool_monitoring_list': ['collectl'], 'pool_profile': 'replicated', 'run_monitoring_list': ['collectl'], 'time': 60, 'write_only': False}}, 'cluster': {'iterations': 1, 'osds_per_node': 3, 'pool_profiles': {'replicated': {'pg_size': 256, 'pgp_size': 256, 'replication': 'replicated'}}, 'user': 'ubuntu'}, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'fa17720d0088c3ac28e473468bfc79eeaff5cd38', 'timestamp': '2026-02-24_15:28:57', 'tube': 'trial', 'user': 'liucm', 'verbose': False, 'worker_log': '/home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.904699'} 2026-02-24T16:06:00.088 DEBUG:teuthology.orchestra.run.trial119:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2026-02-24T16:06:00.135 DEBUG:teuthology.orchestra.run.trial119:> sudo install -d -m0777 -- /var/run/ceph 2026-02-24T16:06:00.198 DEBUG:teuthology.orchestra.run.trial119:> set -ex 2026-02-24T16:06:00.198 DEBUG:teuthology.orchestra.run.trial119:> dd if=/scratch_devs of=/dev/stdout 2026-02-24T16:06:00.255 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2026-02-24T16:06:00.256 DEBUG:teuthology.orchestra.run.trial119:> stat /dev/vg_nvme/lv_1 2026-02-24T16:06:00.312 INFO:teuthology.orchestra.run.trial119.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2026-02-24T16:06:00.313 INFO:teuthology.orchestra.run.trial119.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-24T16:06:00.313 INFO:teuthology.orchestra.run.trial119.stdout:Device: 6h/6d Inode: 918 Links: 1 2026-02-24T16:06:00.313 INFO:teuthology.orchestra.run.trial119.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-24T16:06:00.313 INFO:teuthology.orchestra.run.trial119.stdout:Context: system_u:object_r:device_t:s0 2026-02-24T16:06:00.313 INFO:teuthology.orchestra.run.trial119.stdout:Access: 2026-02-24 15:50:07.043407240 +0000 2026-02-24T16:06:00.313 INFO:teuthology.orchestra.run.trial119.stdout:Modify: 2026-02-24 15:40:21.709515435 +0000 2026-02-24T16:06:00.313 INFO:teuthology.orchestra.run.trial119.stdout:Change: 2026-02-24 15:40:21.709515435 +0000 2026-02-24T16:06:00.313 INFO:teuthology.orchestra.run.trial119.stdout: Birth: 2026-02-24 15:40:21.709515435 +0000 2026-02-24T16:06:00.314 DEBUG:teuthology.orchestra.run.trial119:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2026-02-24T16:06:00.375 INFO:teuthology.orchestra.run.trial119.stderr:1+0 records in 2026-02-24T16:06:00.375 INFO:teuthology.orchestra.run.trial119.stderr:1+0 records out 2026-02-24T16:06:00.376 INFO:teuthology.orchestra.run.trial119.stderr:512 bytes copied, 0.00015629 s, 3.3 MB/s 2026-02-24T16:06:00.377 DEBUG:teuthology.orchestra.run.trial119:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2026-02-24T16:06:00.434 DEBUG:teuthology.orchestra.run.trial119:> stat /dev/vg_nvme/lv_2 2026-02-24T16:06:00.490 INFO:teuthology.orchestra.run.trial119.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2026-02-24T16:06:00.490 INFO:teuthology.orchestra.run.trial119.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-24T16:06:00.490 INFO:teuthology.orchestra.run.trial119.stdout:Device: 6h/6d Inode: 943 Links: 1 2026-02-24T16:06:00.490 INFO:teuthology.orchestra.run.trial119.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-24T16:06:00.490 INFO:teuthology.orchestra.run.trial119.stdout:Context: system_u:object_r:device_t:s0 2026-02-24T16:06:00.490 INFO:teuthology.orchestra.run.trial119.stdout:Access: 2026-02-24 15:50:07.044407204 +0000 2026-02-24T16:06:00.491 INFO:teuthology.orchestra.run.trial119.stdout:Modify: 2026-02-24 15:40:21.950506052 +0000 2026-02-24T16:06:00.491 INFO:teuthology.orchestra.run.trial119.stdout:Change: 2026-02-24 15:40:21.950506052 +0000 2026-02-24T16:06:00.491 INFO:teuthology.orchestra.run.trial119.stdout: Birth: 2026-02-24 15:40:21.950506052 +0000 2026-02-24T16:06:00.491 DEBUG:teuthology.orchestra.run.trial119:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2026-02-24T16:06:00.552 INFO:teuthology.orchestra.run.trial119.stderr:1+0 records in 2026-02-24T16:06:00.552 INFO:teuthology.orchestra.run.trial119.stderr:1+0 records out 2026-02-24T16:06:00.552 INFO:teuthology.orchestra.run.trial119.stderr:512 bytes copied, 0.000143868 s, 3.6 MB/s 2026-02-24T16:06:00.554 DEBUG:teuthology.orchestra.run.trial119:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2026-02-24T16:06:00.611 DEBUG:teuthology.orchestra.run.trial119:> stat /dev/vg_nvme/lv_3 2026-02-24T16:06:00.667 INFO:teuthology.orchestra.run.trial119.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2026-02-24T16:06:00.667 INFO:teuthology.orchestra.run.trial119.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-24T16:06:00.667 INFO:teuthology.orchestra.run.trial119.stdout:Device: 6h/6d Inode: 971 Links: 1 2026-02-24T16:06:00.667 INFO:teuthology.orchestra.run.trial119.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-24T16:06:00.667 INFO:teuthology.orchestra.run.trial119.stdout:Context: system_u:object_r:device_t:s0 2026-02-24T16:06:00.667 INFO:teuthology.orchestra.run.trial119.stdout:Access: 2026-02-24 15:50:07.044407204 +0000 2026-02-24T16:06:00.667 INFO:teuthology.orchestra.run.trial119.stdout:Modify: 2026-02-24 15:40:22.198496395 +0000 2026-02-24T16:06:00.667 INFO:teuthology.orchestra.run.trial119.stdout:Change: 2026-02-24 15:40:22.198496395 +0000 2026-02-24T16:06:00.668 INFO:teuthology.orchestra.run.trial119.stdout: Birth: 2026-02-24 15:40:22.198496395 +0000 2026-02-24T16:06:00.668 DEBUG:teuthology.orchestra.run.trial119:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2026-02-24T16:06:00.729 INFO:teuthology.orchestra.run.trial119.stderr:1+0 records in 2026-02-24T16:06:00.729 INFO:teuthology.orchestra.run.trial119.stderr:1+0 records out 2026-02-24T16:06:00.729 INFO:teuthology.orchestra.run.trial119.stderr:512 bytes copied, 0.000155979 s, 3.3 MB/s 2026-02-24T16:06:00.731 DEBUG:teuthology.orchestra.run.trial119:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2026-02-24T16:06:00.788 DEBUG:teuthology.orchestra.run.trial119:> stat /dev/vg_nvme/lv_4 2026-02-24T16:06:00.844 INFO:teuthology.orchestra.run.trial119.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2026-02-24T16:06:00.845 INFO:teuthology.orchestra.run.trial119.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-24T16:06:00.845 INFO:teuthology.orchestra.run.trial119.stdout:Device: 6h/6d Inode: 1002 Links: 1 2026-02-24T16:06:00.845 INFO:teuthology.orchestra.run.trial119.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-24T16:06:00.845 INFO:teuthology.orchestra.run.trial119.stdout:Context: system_u:object_r:device_t:s0 2026-02-24T16:06:00.845 INFO:teuthology.orchestra.run.trial119.stdout:Access: 2026-02-24 15:50:07.044407204 +0000 2026-02-24T16:06:00.845 INFO:teuthology.orchestra.run.trial119.stdout:Modify: 2026-02-24 15:40:22.433487245 +0000 2026-02-24T16:06:00.845 INFO:teuthology.orchestra.run.trial119.stdout:Change: 2026-02-24 15:40:22.433487245 +0000 2026-02-24T16:06:00.845 INFO:teuthology.orchestra.run.trial119.stdout: Birth: 2026-02-24 15:40:22.433487245 +0000 2026-02-24T16:06:00.846 DEBUG:teuthology.orchestra.run.trial119:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2026-02-24T16:06:00.908 INFO:teuthology.orchestra.run.trial119.stderr:1+0 records in 2026-02-24T16:06:00.908 INFO:teuthology.orchestra.run.trial119.stderr:1+0 records out 2026-02-24T16:06:00.909 INFO:teuthology.orchestra.run.trial119.stderr:512 bytes copied, 0.00012914 s, 4.0 MB/s 2026-02-24T16:06:00.910 DEBUG:teuthology.orchestra.run.trial119:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2026-02-24T16:06:00.967 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-24T16:06:00.967 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@trial119.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2026-02-24T16:06:00.967 INFO:tasks.ceph:Generating config... 2026-02-24T16:06:00.969 INFO:tasks.ceph:[osd] debug monc = 20 2026-02-24T16:06:00.970 INFO:tasks.ceph:[osd] crimson cpu num = 4 2026-02-24T16:06:00.970 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2026-02-24T16:06:00.970 INFO:tasks.ceph:[osd] debug filestore = 0/0 2026-02-24T16:06:00.970 INFO:tasks.ceph:[osd] debug journal = 0/0 2026-02-24T16:06:00.970 INFO:tasks.ceph:[osd] debug ms = 0/0 2026-02-24T16:06:00.970 INFO:tasks.ceph:[osd] debug osd = 0/0 2026-02-24T16:06:00.970 INFO:tasks.ceph:[osd] debug seastore = 20 2026-02-24T16:06:00.970 INFO:tasks.ceph:[osd] debug seastore backref = 20 2026-02-24T16:06:00.970 INFO:tasks.ceph:[osd] debug seastore cache = 20 2026-02-24T16:06:00.970 INFO:tasks.ceph:[osd] debug seastore cleaner = 20 2026-02-24T16:06:00.970 INFO:tasks.ceph:[osd] debug seastore device = 20 2026-02-24T16:06:00.970 INFO:tasks.ceph:[osd] debug seastore epm = 20 2026-02-24T16:06:00.971 INFO:tasks.ceph:[osd] debug seastore fixedkv tree = 20 2026-02-24T16:06:00.971 INFO:tasks.ceph:[osd] debug seastore journal = 20 2026-02-24T16:06:00.971 INFO:tasks.ceph:[osd] debug seastore lba = 20 2026-02-24T16:06:00.971 INFO:tasks.ceph:[osd] debug seastore odata = 20 2026-02-24T16:06:00.971 INFO:tasks.ceph:[osd] debug seastore omap = 20 2026-02-24T16:06:00.971 INFO:tasks.ceph:[osd] debug seastore onode = 20 2026-02-24T16:06:00.971 INFO:tasks.ceph:[osd] debug seastore t = 20 2026-02-24T16:06:00.971 INFO:tasks.ceph:[osd] debug seastore tm = 20 2026-02-24T16:06:00.971 INFO:tasks.ceph:[osd] osd objectstore = seastore 2026-02-24T16:06:00.971 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2026-02-24T16:06:00.972 INFO:tasks.ceph:[osd] seastore cachepin size pershard = 64M 2026-02-24T16:06:00.972 INFO:tasks.ceph:[osd] seastore max concurrent transactions = 8 2026-02-24T16:06:00.972 INFO:tasks.ceph:[global] auth client required = none 2026-02-24T16:06:00.972 INFO:tasks.ceph:[global] auth cluster required = none 2026-02-24T16:06:00.972 INFO:tasks.ceph:[global] auth service required = none 2026-02-24T16:06:00.972 INFO:tasks.ceph:[global] auth supported = none 2026-02-24T16:06:00.972 INFO:tasks.ceph:[global] debug asok = 0/0 2026-02-24T16:06:00.972 INFO:tasks.ceph:[global] debug auth = 0/0 2026-02-24T16:06:00.972 INFO:tasks.ceph:[global] debug bdev = 0/0 2026-02-24T16:06:00.972 INFO:tasks.ceph:[global] debug bluefs = 0/0 2026-02-24T16:06:00.972 INFO:tasks.ceph:[global] debug bluestore = 0/0 2026-02-24T16:06:00.973 INFO:tasks.ceph:[global] debug buffer = 0/0 2026-02-24T16:06:00.973 INFO:tasks.ceph:[global] debug civetweb = 0/0 2026-02-24T16:06:00.973 INFO:tasks.ceph:[global] debug client = 0/0 2026-02-24T16:06:00.973 INFO:tasks.ceph:[global] debug compressor = 0/0 2026-02-24T16:06:00.973 INFO:tasks.ceph:[global] debug context = 0/0 2026-02-24T16:06:00.973 INFO:tasks.ceph:[global] debug crush = 0/0 2026-02-24T16:06:00.973 INFO:tasks.ceph:[global] debug crypto = 0/0 2026-02-24T16:06:00.973 INFO:tasks.ceph:[global] debug dpdk = 0/0 2026-02-24T16:06:00.973 INFO:tasks.ceph:[global] debug eventtrace = 0/0 2026-02-24T16:06:00.973 INFO:tasks.ceph:[global] debug filer = 0/0 2026-02-24T16:06:00.973 INFO:tasks.ceph:[global] debug filestore = 0/0 2026-02-24T16:06:00.973 INFO:tasks.ceph:[global] debug finisher = 0/0 2026-02-24T16:06:00.973 INFO:tasks.ceph:[global] debug fuse = 0/0 2026-02-24T16:06:00.974 INFO:tasks.ceph:[global] debug heartbeatmap = 0/0 2026-02-24T16:06:00.974 INFO:tasks.ceph:[global] debug javaclient = 0/0 2026-02-24T16:06:00.974 INFO:tasks.ceph:[global] debug journal = 0/0 2026-02-24T16:06:00.974 INFO:tasks.ceph:[global] debug journaler = 0/0 2026-02-24T16:06:00.974 INFO:tasks.ceph:[global] debug kstore = 0/0 2026-02-24T16:06:00.974 INFO:tasks.ceph:[global] debug leveldb = 0/0 2026-02-24T16:06:00.974 INFO:tasks.ceph:[global] debug lockdep = 0/0 2026-02-24T16:06:00.974 INFO:tasks.ceph:[global] debug mds = 0/0 2026-02-24T16:06:00.974 INFO:tasks.ceph:[global] debug mds balancer = 0/0 2026-02-24T16:06:00.974 INFO:tasks.ceph:[global] debug mds locker = 0/0 2026-02-24T16:06:00.974 INFO:tasks.ceph:[global] debug mds log = 0/0 2026-02-24T16:06:00.974 INFO:tasks.ceph:[global] debug mds log expire = 0/0 2026-02-24T16:06:00.974 INFO:tasks.ceph:[global] debug mds migrator = 0/0 2026-02-24T16:06:00.975 INFO:tasks.ceph:[global] debug memdb = 0/0 2026-02-24T16:06:00.975 INFO:tasks.ceph:[global] debug mgr = 0/0 2026-02-24T16:06:00.975 INFO:tasks.ceph:[global] debug mgrc = 0/0 2026-02-24T16:06:00.975 INFO:tasks.ceph:[global] debug mon = 0/0 2026-02-24T16:06:00.975 INFO:tasks.ceph:[global] debug monc = 0/0 2026-02-24T16:06:00.975 INFO:tasks.ceph:[global] debug ms = 0/0 2026-02-24T16:06:00.975 INFO:tasks.ceph:[global] debug objclass = 0/0 2026-02-24T16:06:00.975 INFO:tasks.ceph:[global] debug objectcacher = 0/0 2026-02-24T16:06:00.975 INFO:tasks.ceph:[global] debug objecter = 0/0 2026-02-24T16:06:00.975 INFO:tasks.ceph:[global] debug optracker = 0/0 2026-02-24T16:06:00.975 INFO:tasks.ceph:[global] debug osd = 0/0 2026-02-24T16:06:00.975 INFO:tasks.ceph:[global] debug paxos = 0/0 2026-02-24T16:06:00.975 INFO:tasks.ceph:[global] debug perfcounter = 0/0 2026-02-24T16:06:00.976 INFO:tasks.ceph:[global] debug rados = 0/0 2026-02-24T16:06:00.976 INFO:tasks.ceph:[global] debug rbd = 0/0 2026-02-24T16:06:00.976 INFO:tasks.ceph:[global] debug rbd mirror = 0/0 2026-02-24T16:06:00.976 INFO:tasks.ceph:[global] debug rbd replay = 0/0 2026-02-24T16:06:00.976 INFO:tasks.ceph:[global] debug refs = 0/0 2026-02-24T16:06:00.976 INFO:tasks.ceph:[global] debug rgw = 0/0 2026-02-24T16:06:00.976 INFO:tasks.ceph:[global] debug rgw lifecycle = 0/0 2026-02-24T16:06:00.976 INFO:tasks.ceph:[global] debug rgw notification = 0/0 2026-02-24T16:06:00.976 INFO:tasks.ceph:[global] debug rgw sync = 0/0 2026-02-24T16:06:00.976 INFO:tasks.ceph:[global] debug rocksdb = 0/0 2026-02-24T16:06:00.976 INFO:tasks.ceph:[global] debug striper = 0/0 2026-02-24T16:06:00.976 INFO:tasks.ceph:[global] debug throttle = 0/0 2026-02-24T16:06:00.976 INFO:tasks.ceph:[global] debug timer = 0/0 2026-02-24T16:06:00.977 INFO:tasks.ceph:[global] debug tp = 0/0 2026-02-24T16:06:00.977 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2026-02-24T16:06:00.977 INFO:tasks.ceph:[global] ms client mode = crc 2026-02-24T16:06:00.977 INFO:tasks.ceph:[global] ms cluster mode = crc 2026-02-24T16:06:00.977 INFO:tasks.ceph:[global] ms mon client mode = crc 2026-02-24T16:06:00.977 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2026-02-24T16:06:00.977 INFO:tasks.ceph:[global] ms mon service mode = crc 2026-02-24T16:06:00.977 INFO:tasks.ceph:[global] ms service mode = crc 2026-02-24T16:06:00.977 INFO:tasks.ceph:[mgr] debug mgr = 20 2026-02-24T16:06:00.977 INFO:tasks.ceph:[mgr] debug ms = 1 2026-02-24T16:06:00.977 INFO:tasks.ceph:[mon] debug mon = 0/0 2026-02-24T16:06:00.977 INFO:tasks.ceph:[mon] debug ms = 0/0 2026-02-24T16:06:00.978 INFO:tasks.ceph:[mon] debug paxos = 0/0 2026-02-24T16:06:00.978 INFO:tasks.ceph:[mon] osd pool default crimson = True 2026-02-24T16:06:00.978 INFO:tasks.ceph:Setting up mon.a... 2026-02-24T16:06:00.978 DEBUG:teuthology.orchestra.run.trial119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2026-02-24T16:06:01.046 INFO:teuthology.orchestra.run.trial119.stdout:creating /etc/ceph/ceph.keyring 2026-02-24T16:06:01.049 DEBUG:teuthology.orchestra.run.trial119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2026-02-24T16:06:01.133 DEBUG:teuthology.orchestra.run.trial119:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-24T16:06:01.198 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '10.20.193.119')] 2026-02-24T16:06:01.198 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'none', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '10.20.193.119', 'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': 'true', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore max concurrent transactions': 8}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': '0/0', 'debug mon': '0/0', 'debug paxos': '0/0', 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}} 2026-02-24T16:06:01.200 DEBUG:teuthology.orchestra.run.trial119:> set -ex 2026-02-24T16:06:01.200 DEBUG:teuthology.orchestra.run.trial119:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2026-02-24T16:06:01.254 DEBUG:teuthology.orchestra.run.trial119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 10.20.193.119 --print /home/ubuntu/cephtest/ceph.monmap 2026-02-24T16:06:01.333 INFO:teuthology.orchestra.run.trial119.stderr:2026-02-24T16:06:01.332+0000 7fc1d9445f40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:06:01.333 INFO:teuthology.orchestra.run.trial119.stderr:2026-02-24T16:06:01.332+0000 7fc1d9445f40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:06:01.333 INFO:teuthology.orchestra.run.trial119.stderr:2026-02-24T16:06:01.332+0000 7fc1d9445f40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:06:01.335 INFO:teuthology.orchestra.run.trial119.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2026-02-24T16:06:01.335 INFO:teuthology.orchestra.run.trial119.stdout:monmaptool: generated fsid f14cc7f1-24ff-452a-853d-38c3f5542203 2026-02-24T16:06:01.335 INFO:teuthology.orchestra.run.trial119.stdout:setting min_mon_release = tentacle 2026-02-24T16:06:01.335 INFO:teuthology.orchestra.run.trial119.stdout:epoch 0 2026-02-24T16:06:01.335 INFO:teuthology.orchestra.run.trial119.stdout:fsid f14cc7f1-24ff-452a-853d-38c3f5542203 2026-02-24T16:06:01.335 INFO:teuthology.orchestra.run.trial119.stdout:last_changed 2026-02-24T16:06:01.333096+0000 2026-02-24T16:06:01.335 INFO:teuthology.orchestra.run.trial119.stdout:created 2026-02-24T16:06:01.333096+0000 2026-02-24T16:06:01.335 INFO:teuthology.orchestra.run.trial119.stdout:min_mon_release 20 (tentacle) 2026-02-24T16:06:01.336 INFO:teuthology.orchestra.run.trial119.stdout:election_strategy: 1 2026-02-24T16:06:01.336 INFO:teuthology.orchestra.run.trial119.stdout:0: [v2:10.20.193.119:3300/0,v1:10.20.193.119:6789/0] mon.a 2026-02-24T16:06:01.336 INFO:teuthology.orchestra.run.trial119.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2026-02-24T16:06:01.336 DEBUG:teuthology.orchestra.run.trial119:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2026-02-24T16:06:01.391 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID f14cc7f1-24ff-452a-853d-38c3f5542203... 2026-02-24T16:06:01.393 DEBUG:teuthology.orchestra.run.trial119:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2026-02-24T16:06:01.471 INFO:teuthology.orchestra.run.trial119.stdout:[global] 2026-02-24T16:06:01.471 INFO:teuthology.orchestra.run.trial119.stdout: chdir = "" 2026-02-24T16:06:01.471 INFO:teuthology.orchestra.run.trial119.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2026-02-24T16:06:01.471 INFO:teuthology.orchestra.run.trial119.stdout: auth supported = none 2026-02-24T16:06:01.471 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T16:06:01.471 INFO:teuthology.orchestra.run.trial119.stdout: filestore xattr use omap = true 2026-02-24T16:06:01.472 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T16:06:01.472 INFO:teuthology.orchestra.run.trial119.stdout: mon clock drift allowed = 1.000 2026-02-24T16:06:01.472 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T16:06:01.472 INFO:teuthology.orchestra.run.trial119.stdout: osd crush chooseleaf type = 0 2026-02-24T16:06:01.472 INFO:teuthology.orchestra.run.trial119.stdout: auth debug = true 2026-02-24T16:06:01.472 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T16:06:01.472 INFO:teuthology.orchestra.run.trial119.stdout: ms die on old message = true 2026-02-24T16:06:01.472 INFO:teuthology.orchestra.run.trial119.stdout: ms die on bug = true 2026-02-24T16:06:01.472 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T16:06:01.472 INFO:teuthology.orchestra.run.trial119.stdout: mon max pg per osd = 10000 # >= luminous 2026-02-24T16:06:01.472 INFO:teuthology.orchestra.run.trial119.stdout: mon pg warn max object skew = 0 2026-02-24T16:06:01.472 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T16:06:01.473 INFO:teuthology.orchestra.run.trial119.stdout: # disable pg_autoscaler by default for new pools 2026-02-24T16:06:01.473 INFO:teuthology.orchestra.run.trial119.stdout: osd_pool_default_pg_autoscale_mode = off 2026-02-24T16:06:01.473 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T16:06:01.473 INFO:teuthology.orchestra.run.trial119.stdout: osd pool default size = 2 2026-02-24T16:06:01.473 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T16:06:01.473 INFO:teuthology.orchestra.run.trial119.stdout: mon osd allow primary affinity = true 2026-02-24T16:06:01.473 INFO:teuthology.orchestra.run.trial119.stdout: mon osd allow pg remap = true 2026-02-24T16:06:01.473 INFO:teuthology.orchestra.run.trial119.stdout: mon warn on legacy crush tunables = false 2026-02-24T16:06:01.473 INFO:teuthology.orchestra.run.trial119.stdout: mon warn on crush straw calc version zero = false 2026-02-24T16:06:01.473 INFO:teuthology.orchestra.run.trial119.stdout: mon warn on no sortbitwise = false 2026-02-24T16:06:01.473 INFO:teuthology.orchestra.run.trial119.stdout: mon warn on osd down out interval zero = false 2026-02-24T16:06:01.473 INFO:teuthology.orchestra.run.trial119.stdout: mon warn on too few osds = false 2026-02-24T16:06:01.474 INFO:teuthology.orchestra.run.trial119.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2026-02-24T16:06:01.474 INFO:teuthology.orchestra.run.trial119.stdout: mon_warn_on_pool_no_redundancy = false 2026-02-24T16:06:01.474 INFO:teuthology.orchestra.run.trial119.stdout: mon_allow_pool_size_one = true 2026-02-24T16:06:01.474 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T16:06:01.474 INFO:teuthology.orchestra.run.trial119.stdout: osd pool default erasure code profile = plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2026-02-24T16:06:01.474 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T16:06:01.474 INFO:teuthology.orchestra.run.trial119.stdout: osd default data pool replay window = 5 2026-02-24T16:06:01.474 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T16:06:01.474 INFO:teuthology.orchestra.run.trial119.stdout: mon allow pool delete = true 2026-02-24T16:06:01.474 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T16:06:01.475 INFO:teuthology.orchestra.run.trial119.stdout: mon cluster log file level = debug 2026-02-24T16:06:01.475 INFO:teuthology.orchestra.run.trial119.stdout: debug asserts on shutdown = true 2026-02-24T16:06:01.475 INFO:teuthology.orchestra.run.trial119.stdout: mon health detail to clog = false 2026-02-24T16:06:01.475 INFO:teuthology.orchestra.run.trial119.stdout: mon host = 10.20.193.119 2026-02-24T16:06:01.475 INFO:teuthology.orchestra.run.trial119.stdout: auth client required = none 2026-02-24T16:06:01.475 INFO:teuthology.orchestra.run.trial119.stdout: auth cluster required = none 2026-02-24T16:06:01.475 INFO:teuthology.orchestra.run.trial119.stdout: auth service required = none 2026-02-24T16:06:01.475 INFO:teuthology.orchestra.run.trial119.stdout: debug asok = 0/0 2026-02-24T16:06:01.475 INFO:teuthology.orchestra.run.trial119.stdout: debug auth = 0/0 2026-02-24T16:06:01.475 INFO:teuthology.orchestra.run.trial119.stdout: debug bdev = 0/0 2026-02-24T16:06:01.475 INFO:teuthology.orchestra.run.trial119.stdout: debug bluefs = 0/0 2026-02-24T16:06:01.475 INFO:teuthology.orchestra.run.trial119.stdout: debug bluestore = 0/0 2026-02-24T16:06:01.476 INFO:teuthology.orchestra.run.trial119.stdout: debug buffer = 0/0 2026-02-24T16:06:01.476 INFO:teuthology.orchestra.run.trial119.stdout: debug civetweb = 0/0 2026-02-24T16:06:01.476 INFO:teuthology.orchestra.run.trial119.stdout: debug client = 0/0 2026-02-24T16:06:01.476 INFO:teuthology.orchestra.run.trial119.stdout: debug compressor = 0/0 2026-02-24T16:06:01.476 INFO:teuthology.orchestra.run.trial119.stdout: debug context = 0/0 2026-02-24T16:06:01.476 INFO:teuthology.orchestra.run.trial119.stdout: debug crush = 0/0 2026-02-24T16:06:01.476 INFO:teuthology.orchestra.run.trial119.stdout: debug crypto = 0/0 2026-02-24T16:06:01.476 INFO:teuthology.orchestra.run.trial119.stdout: debug dpdk = 0/0 2026-02-24T16:06:01.476 INFO:teuthology.orchestra.run.trial119.stdout: debug eventtrace = 0/0 2026-02-24T16:06:01.476 INFO:teuthology.orchestra.run.trial119.stdout: debug filer = 0/0 2026-02-24T16:06:01.476 INFO:teuthology.orchestra.run.trial119.stdout: debug filestore = 0/0 2026-02-24T16:06:01.476 INFO:teuthology.orchestra.run.trial119.stdout: debug finisher = 0/0 2026-02-24T16:06:01.476 INFO:teuthology.orchestra.run.trial119.stdout: debug fuse = 0/0 2026-02-24T16:06:01.477 INFO:teuthology.orchestra.run.trial119.stdout: debug heartbeatmap = 0/0 2026-02-24T16:06:01.477 INFO:teuthology.orchestra.run.trial119.stdout: debug javaclient = 0/0 2026-02-24T16:06:01.477 INFO:teuthology.orchestra.run.trial119.stdout: debug journal = 0/0 2026-02-24T16:06:01.477 INFO:teuthology.orchestra.run.trial119.stdout: debug journaler = 0/0 2026-02-24T16:06:01.477 INFO:teuthology.orchestra.run.trial119.stdout: debug kstore = 0/0 2026-02-24T16:06:01.477 INFO:teuthology.orchestra.run.trial119.stdout: debug leveldb = 0/0 2026-02-24T16:06:01.477 INFO:teuthology.orchestra.run.trial119.stdout: debug lockdep = 0/0 2026-02-24T16:06:01.477 INFO:teuthology.orchestra.run.trial119.stdout: debug mds = 0/0 2026-02-24T16:06:01.477 INFO:teuthology.orchestra.run.trial119.stdout: debug mds balancer = 0/0 2026-02-24T16:06:01.477 INFO:teuthology.orchestra.run.trial119.stdout: debug mds locker = 0/0 2026-02-24T16:06:01.477 INFO:teuthology.orchestra.run.trial119.stdout: debug mds log = 0/0 2026-02-24T16:06:01.477 INFO:teuthology.orchestra.run.trial119.stdout: debug mds log expire = 0/0 2026-02-24T16:06:01.477 INFO:teuthology.orchestra.run.trial119.stdout: debug mds migrator = 0/0 2026-02-24T16:06:01.478 INFO:teuthology.orchestra.run.trial119.stdout: debug memdb = 0/0 2026-02-24T16:06:01.478 INFO:teuthology.orchestra.run.trial119.stdout: debug mgr = 0/0 2026-02-24T16:06:01.478 INFO:teuthology.orchestra.run.trial119.stdout: debug mgrc = 0/0 2026-02-24T16:06:01.478 INFO:teuthology.orchestra.run.trial119.stdout: debug mon = 0/0 2026-02-24T16:06:01.478 INFO:teuthology.orchestra.run.trial119.stdout: debug monc = 0/0 2026-02-24T16:06:01.478 INFO:teuthology.orchestra.run.trial119.stdout: debug ms = 0/0 2026-02-24T16:06:01.478 INFO:teuthology.orchestra.run.trial119.stdout: debug objclass = 0/0 2026-02-24T16:06:01.478 INFO:teuthology.orchestra.run.trial119.stdout: debug objectcacher = 0/0 2026-02-24T16:06:01.478 INFO:teuthology.orchestra.run.trial119.stdout: debug objecter = 0/0 2026-02-24T16:06:01.478 INFO:teuthology.orchestra.run.trial119.stdout: debug optracker = 0/0 2026-02-24T16:06:01.478 INFO:teuthology.orchestra.run.trial119.stdout: debug osd = 0/0 2026-02-24T16:06:01.478 INFO:teuthology.orchestra.run.trial119.stdout: debug paxos = 0/0 2026-02-24T16:06:01.478 INFO:teuthology.orchestra.run.trial119.stdout: debug perfcounter = 0/0 2026-02-24T16:06:01.479 INFO:teuthology.orchestra.run.trial119.stdout: debug rados = 0/0 2026-02-24T16:06:01.479 INFO:teuthology.orchestra.run.trial119.stdout: debug rbd = 0/0 2026-02-24T16:06:01.479 INFO:teuthology.orchestra.run.trial119.stdout: debug rbd mirror = 0/0 2026-02-24T16:06:01.479 INFO:teuthology.orchestra.run.trial119.stdout: debug rbd replay = 0/0 2026-02-24T16:06:01.479 INFO:teuthology.orchestra.run.trial119.stdout: debug refs = 0/0 2026-02-24T16:06:01.479 INFO:teuthology.orchestra.run.trial119.stdout: debug rgw = 0/0 2026-02-24T16:06:01.479 INFO:teuthology.orchestra.run.trial119.stdout: debug rgw lifecycle = 0/0 2026-02-24T16:06:01.479 INFO:teuthology.orchestra.run.trial119.stdout: debug rgw notification = 0/0 2026-02-24T16:06:01.479 INFO:teuthology.orchestra.run.trial119.stdout: debug rgw sync = 0/0 2026-02-24T16:06:01.479 INFO:teuthology.orchestra.run.trial119.stdout: debug rocksdb = 0/0 2026-02-24T16:06:01.479 INFO:teuthology.orchestra.run.trial119.stdout: debug striper = 0/0 2026-02-24T16:06:01.479 INFO:teuthology.orchestra.run.trial119.stdout: debug throttle = 0/0 2026-02-24T16:06:01.479 INFO:teuthology.orchestra.run.trial119.stdout: debug timer = 0/0 2026-02-24T16:06:01.480 INFO:teuthology.orchestra.run.trial119.stdout: debug tp = 0/0 2026-02-24T16:06:01.480 INFO:teuthology.orchestra.run.trial119.stdout: enable experimental unrecoverable data corrupting features = crimson 2026-02-24T16:06:01.480 INFO:teuthology.orchestra.run.trial119.stdout: ms client mode = crc 2026-02-24T16:06:01.480 INFO:teuthology.orchestra.run.trial119.stdout: ms cluster mode = crc 2026-02-24T16:06:01.480 INFO:teuthology.orchestra.run.trial119.stdout: ms mon client mode = crc 2026-02-24T16:06:01.480 INFO:teuthology.orchestra.run.trial119.stdout: ms mon cluster mode = crc 2026-02-24T16:06:01.480 INFO:teuthology.orchestra.run.trial119.stdout: ms mon service mode = crc 2026-02-24T16:06:01.480 INFO:teuthology.orchestra.run.trial119.stdout: ms service mode = crc 2026-02-24T16:06:01.480 INFO:teuthology.orchestra.run.trial119.stdout: fsid = f14cc7f1-24ff-452a-853d-38c3f5542203 2026-02-24T16:06:01.480 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T16:06:01.480 INFO:teuthology.orchestra.run.trial119.stdout:[osd] 2026-02-24T16:06:01.480 INFO:teuthology.orchestra.run.trial119.stdout: osd journal size = 100 2026-02-24T16:06:01.481 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T16:06:01.481 INFO:teuthology.orchestra.run.trial119.stdout: osd scrub load threshold = 5.0 2026-02-24T16:06:01.481 INFO:teuthology.orchestra.run.trial119.stdout: osd scrub max interval = 600 2026-02-24T16:06:01.481 INFO:teuthology.orchestra.run.trial119.stdout: osd mclock profile = high_recovery_ops 2026-02-24T16:06:01.481 INFO:teuthology.orchestra.run.trial119.stdout: osd mclock skip benchmark = true 2026-02-24T16:06:01.481 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T16:06:01.481 INFO:teuthology.orchestra.run.trial119.stdout: osd recover clone overlap = true 2026-02-24T16:06:01.481 INFO:teuthology.orchestra.run.trial119.stdout: osd recovery max chunk = 1048576 2026-02-24T16:06:01.481 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T16:06:01.481 INFO:teuthology.orchestra.run.trial119.stdout: osd debug shutdown = true 2026-02-24T16:06:01.481 INFO:teuthology.orchestra.run.trial119.stdout: osd debug op order = true 2026-02-24T16:06:01.481 INFO:teuthology.orchestra.run.trial119.stdout: osd debug verify stray on activate = true 2026-02-24T16:06:01.482 INFO:teuthology.orchestra.run.trial119.stdout: osd debug trim objects = true 2026-02-24T16:06:01.482 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T16:06:01.482 INFO:teuthology.orchestra.run.trial119.stdout: osd open classes on start = true 2026-02-24T16:06:01.482 INFO:teuthology.orchestra.run.trial119.stdout: osd debug pg log writeout = true 2026-02-24T16:06:01.482 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T16:06:01.482 INFO:teuthology.orchestra.run.trial119.stdout: osd deep scrub update digest min age = 30 2026-02-24T16:06:01.482 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T16:06:01.482 INFO:teuthology.orchestra.run.trial119.stdout: osd map max advance = 10 2026-02-24T16:06:01.482 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T16:06:01.482 INFO:teuthology.orchestra.run.trial119.stdout: journal zero on create = true 2026-02-24T16:06:01.482 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T16:06:01.482 INFO:teuthology.orchestra.run.trial119.stdout: filestore ondisk finisher threads = 3 2026-02-24T16:06:01.482 INFO:teuthology.orchestra.run.trial119.stdout: filestore apply finisher threads = 3 2026-02-24T16:06:01.483 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T16:06:01.483 INFO:teuthology.orchestra.run.trial119.stdout: bdev debug aio = true 2026-02-24T16:06:01.483 INFO:teuthology.orchestra.run.trial119.stdout: osd debug misdirected ops = true 2026-02-24T16:06:01.483 INFO:teuthology.orchestra.run.trial119.stdout: debug monc = 20 2026-02-24T16:06:01.483 INFO:teuthology.orchestra.run.trial119.stdout: crimson cpu num = 4 2026-02-24T16:06:01.483 INFO:teuthology.orchestra.run.trial119.stdout: crimson osd obc lru size = 10 2026-02-24T16:06:01.483 INFO:teuthology.orchestra.run.trial119.stdout: debug filestore = 0/0 2026-02-24T16:06:01.483 INFO:teuthology.orchestra.run.trial119.stdout: debug journal = 0/0 2026-02-24T16:06:01.483 INFO:teuthology.orchestra.run.trial119.stdout: debug ms = 0/0 2026-02-24T16:06:01.483 INFO:teuthology.orchestra.run.trial119.stdout: debug osd = 0/0 2026-02-24T16:06:01.483 INFO:teuthology.orchestra.run.trial119.stdout: debug seastore = 20 2026-02-24T16:06:01.483 INFO:teuthology.orchestra.run.trial119.stdout: debug seastore backref = 20 2026-02-24T16:06:01.483 INFO:teuthology.orchestra.run.trial119.stdout: debug seastore cache = 20 2026-02-24T16:06:01.484 INFO:teuthology.orchestra.run.trial119.stdout: debug seastore cleaner = 20 2026-02-24T16:06:01.484 INFO:teuthology.orchestra.run.trial119.stdout: debug seastore device = 20 2026-02-24T16:06:01.484 INFO:teuthology.orchestra.run.trial119.stdout: debug seastore epm = 20 2026-02-24T16:06:01.484 INFO:teuthology.orchestra.run.trial119.stdout: debug seastore fixedkv tree = 20 2026-02-24T16:06:01.484 INFO:teuthology.orchestra.run.trial119.stdout: debug seastore journal = 20 2026-02-24T16:06:01.484 INFO:teuthology.orchestra.run.trial119.stdout: debug seastore lba = 20 2026-02-24T16:06:01.484 INFO:teuthology.orchestra.run.trial119.stdout: debug seastore odata = 20 2026-02-24T16:06:01.484 INFO:teuthology.orchestra.run.trial119.stdout: debug seastore omap = 20 2026-02-24T16:06:01.484 INFO:teuthology.orchestra.run.trial119.stdout: debug seastore onode = 20 2026-02-24T16:06:01.484 INFO:teuthology.orchestra.run.trial119.stdout: debug seastore t = 20 2026-02-24T16:06:01.484 INFO:teuthology.orchestra.run.trial119.stdout: debug seastore tm = 20 2026-02-24T16:06:01.484 INFO:teuthology.orchestra.run.trial119.stdout: osd objectstore = seastore 2026-02-24T16:06:01.484 INFO:teuthology.orchestra.run.trial119.stdout: osd shutdown pgref assert = True 2026-02-24T16:06:01.485 INFO:teuthology.orchestra.run.trial119.stdout: seastore cachepin size pershard = 64M 2026-02-24T16:06:01.485 INFO:teuthology.orchestra.run.trial119.stdout: seastore max concurrent transactions = 8 2026-02-24T16:06:01.485 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T16:06:01.485 INFO:teuthology.orchestra.run.trial119.stdout:[mgr] 2026-02-24T16:06:01.485 INFO:teuthology.orchestra.run.trial119.stdout: debug ms = 1 2026-02-24T16:06:01.485 INFO:teuthology.orchestra.run.trial119.stdout: debug mgr = 20 2026-02-24T16:06:01.485 INFO:teuthology.orchestra.run.trial119.stdout: debug mon = 20 2026-02-24T16:06:01.485 INFO:teuthology.orchestra.run.trial119.stdout: debug auth = 20 2026-02-24T16:06:01.485 INFO:teuthology.orchestra.run.trial119.stdout: mon reweight min pgs per osd = 4 2026-02-24T16:06:01.485 INFO:teuthology.orchestra.run.trial119.stdout: mon reweight min bytes per osd = 10 2026-02-24T16:06:01.485 INFO:teuthology.orchestra.run.trial119.stdout: mgr/telemetry/nag = false 2026-02-24T16:06:01.485 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T16:06:01.485 INFO:teuthology.orchestra.run.trial119.stdout:[mon] 2026-02-24T16:06:01.486 INFO:teuthology.orchestra.run.trial119.stdout: debug ms = 0/0 2026-02-24T16:06:01.486 INFO:teuthology.orchestra.run.trial119.stdout: debug mon = 0/0 2026-02-24T16:06:01.486 INFO:teuthology.orchestra.run.trial119.stdout: debug paxos = 0/0 2026-02-24T16:06:01.486 INFO:teuthology.orchestra.run.trial119.stdout: debug auth = 20 2026-02-24T16:06:01.486 INFO:teuthology.orchestra.run.trial119.stdout: mon data avail warn = 5 2026-02-24T16:06:01.486 INFO:teuthology.orchestra.run.trial119.stdout: mon mgr mkfs grace = 240 2026-02-24T16:06:01.486 INFO:teuthology.orchestra.run.trial119.stdout: mon reweight min pgs per osd = 4 2026-02-24T16:06:01.486 INFO:teuthology.orchestra.run.trial119.stdout: mon osd reporter subtree level = osd 2026-02-24T16:06:01.486 INFO:teuthology.orchestra.run.trial119.stdout: mon osd prime pg temp = true 2026-02-24T16:06:01.486 INFO:teuthology.orchestra.run.trial119.stdout: mon reweight min bytes per osd = 10 2026-02-24T16:06:01.486 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T16:06:01.486 INFO:teuthology.orchestra.run.trial119.stdout: # rotate auth tickets quickly to exercise renewal paths 2026-02-24T16:06:01.486 INFO:teuthology.orchestra.run.trial119.stdout: auth mon ticket ttl = 660 # 11m 2026-02-24T16:06:01.487 INFO:teuthology.orchestra.run.trial119.stdout: auth service ticket ttl = 240 # 4m 2026-02-24T16:06:01.487 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T16:06:01.487 INFO:teuthology.orchestra.run.trial119.stdout: # don't complain about insecure global_id in the test suite 2026-02-24T16:06:01.487 INFO:teuthology.orchestra.run.trial119.stdout: mon_warn_on_insecure_global_id_reclaim = false 2026-02-24T16:06:01.487 INFO:teuthology.orchestra.run.trial119.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2026-02-24T16:06:01.487 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T16:06:01.487 INFO:teuthology.orchestra.run.trial119.stdout: # 1m isn't quite enough 2026-02-24T16:06:01.487 INFO:teuthology.orchestra.run.trial119.stdout: mon_down_mkfs_grace = 2m 2026-02-24T16:06:01.487 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T16:06:01.487 INFO:teuthology.orchestra.run.trial119.stdout: mon_warn_on_filestore_osds = false 2026-02-24T16:06:01.487 INFO:teuthology.orchestra.run.trial119.stdout: osd pool default crimson = True 2026-02-24T16:06:01.487 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T16:06:01.488 INFO:teuthology.orchestra.run.trial119.stdout:[client] 2026-02-24T16:06:01.488 INFO:teuthology.orchestra.run.trial119.stdout: rgw cache enabled = true 2026-02-24T16:06:01.488 INFO:teuthology.orchestra.run.trial119.stdout: rgw enable ops log = true 2026-02-24T16:06:01.488 INFO:teuthology.orchestra.run.trial119.stdout: rgw enable usage log = true 2026-02-24T16:06:01.488 INFO:teuthology.orchestra.run.trial119.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2026-02-24T16:06:01.488 INFO:teuthology.orchestra.run.trial119.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2026-02-24T16:06:01.488 INFO:teuthology.orchestra.run.trial119.stdout:[mon.a] 2026-02-24T16:06:01.489 INFO:tasks.ceph:Creating admin key on mon.a... 2026-02-24T16:06:01.490 DEBUG:teuthology.orchestra.run.trial119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2026-02-24T16:06:01.578 INFO:tasks.ceph:Copying monmap to all nodes... 2026-02-24T16:06:01.579 DEBUG:teuthology.orchestra.run.trial119:> set -ex 2026-02-24T16:06:01.579 DEBUG:teuthology.orchestra.run.trial119:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2026-02-24T16:06:01.596 DEBUG:teuthology.orchestra.run.trial119:> set -ex 2026-02-24T16:06:01.596 DEBUG:teuthology.orchestra.run.trial119:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2026-02-24T16:06:01.652 INFO:tasks.ceph:Sending monmap to node ubuntu@trial119.front.sepia.ceph.com 2026-02-24T16:06:01.652 DEBUG:teuthology.orchestra.run.trial119:> set -ex 2026-02-24T16:06:01.652 DEBUG:teuthology.orchestra.run.trial119:> sudo dd of=/etc/ceph/ceph.keyring 2026-02-24T16:06:01.653 DEBUG:teuthology.orchestra.run.trial119:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-24T16:06:01.724 DEBUG:teuthology.orchestra.run.trial119:> set -ex 2026-02-24T16:06:01.724 DEBUG:teuthology.orchestra.run.trial119:> dd of=/home/ubuntu/cephtest/ceph.monmap 2026-02-24T16:06:01.780 INFO:tasks.ceph:Setting up mon nodes... 2026-02-24T16:06:01.780 INFO:tasks.ceph:Setting up mgr nodes... 2026-02-24T16:06:01.780 DEBUG:teuthology.orchestra.run.trial119:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2026-02-24T16:06:01.869 INFO:teuthology.orchestra.run.trial119.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2026-02-24T16:06:01.872 INFO:tasks.ceph:Setting up mds nodes... 2026-02-24T16:06:01.872 INFO:tasks.ceph_client:Setting up client nodes... 2026-02-24T16:06:01.873 DEBUG:teuthology.orchestra.run.trial119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2026-02-24T16:06:01.914 INFO:teuthology.orchestra.run.trial119.stdout:creating /etc/ceph/ceph.client.0.keyring 2026-02-24T16:06:01.926 INFO:tasks.ceph:Running mkfs on osd nodes... 2026-02-24T16:06:01.927 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@trial119.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2026-02-24T16:06:01.927 DEBUG:teuthology.orchestra.run.trial119:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2026-02-24T16:06:01.991 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-24T16:06:01.991 INFO:tasks.ceph:role: osd.0 2026-02-24T16:06:01.991 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@trial119.front.sepia.ceph.com 2026-02-24T16:06:01.991 DEBUG:teuthology.orchestra.run.trial119:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2026-02-24T16:06:02.056 INFO:teuthology.orchestra.run.trial119.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=45875200 blks 2026-02-24T16:06:02.056 INFO:teuthology.orchestra.run.trial119.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-24T16:06:02.056 INFO:teuthology.orchestra.run.trial119.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-24T16:06:02.056 INFO:teuthology.orchestra.run.trial119.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-24T16:06:02.056 INFO:teuthology.orchestra.run.trial119.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-24T16:06:02.056 INFO:teuthology.orchestra.run.trial119.stdout: = sunit=0 swidth=0 blks 2026-02-24T16:06:02.056 INFO:teuthology.orchestra.run.trial119.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-24T16:06:02.056 INFO:teuthology.orchestra.run.trial119.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-24T16:06:02.057 INFO:teuthology.orchestra.run.trial119.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-24T16:06:02.057 INFO:teuthology.orchestra.run.trial119.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-24T16:06:02.633 INFO:teuthology.orchestra.run.trial119.stdout:Discarding blocks...Done. 2026-02-24T16:06:02.636 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@trial119.front.sepia.ceph.com -o noatime 2026-02-24T16:06:02.636 DEBUG:teuthology.orchestra.run.trial119:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2026-02-24T16:06:02.704 DEBUG:teuthology.orchestra.run.trial119:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2026-02-24T16:06:02.773 DEBUG:teuthology.orchestra.run.trial119:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2026-02-24T16:06:02.836 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-24T16:06:02.837 INFO:tasks.ceph:role: osd.1 2026-02-24T16:06:02.837 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@trial119.front.sepia.ceph.com 2026-02-24T16:06:02.837 DEBUG:teuthology.orchestra.run.trial119:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2026-02-24T16:06:02.901 INFO:teuthology.orchestra.run.trial119.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=45875200 blks 2026-02-24T16:06:02.901 INFO:teuthology.orchestra.run.trial119.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-24T16:06:02.902 INFO:teuthology.orchestra.run.trial119.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-24T16:06:02.902 INFO:teuthology.orchestra.run.trial119.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-24T16:06:02.902 INFO:teuthology.orchestra.run.trial119.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-24T16:06:02.902 INFO:teuthology.orchestra.run.trial119.stdout: = sunit=0 swidth=0 blks 2026-02-24T16:06:02.902 INFO:teuthology.orchestra.run.trial119.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-24T16:06:02.902 INFO:teuthology.orchestra.run.trial119.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-24T16:06:02.902 INFO:teuthology.orchestra.run.trial119.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-24T16:06:02.902 INFO:teuthology.orchestra.run.trial119.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-24T16:06:03.459 INFO:teuthology.orchestra.run.trial119.stdout:Discarding blocks...Done. 2026-02-24T16:06:03.461 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@trial119.front.sepia.ceph.com -o noatime 2026-02-24T16:06:03.461 DEBUG:teuthology.orchestra.run.trial119:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2026-02-24T16:06:03.528 DEBUG:teuthology.orchestra.run.trial119:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2026-02-24T16:06:03.594 DEBUG:teuthology.orchestra.run.trial119:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2026-02-24T16:06:03.657 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-24T16:06:03.658 INFO:tasks.ceph:role: osd.2 2026-02-24T16:06:03.658 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@trial119.front.sepia.ceph.com 2026-02-24T16:06:03.658 DEBUG:teuthology.orchestra.run.trial119:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2026-02-24T16:06:03.723 INFO:teuthology.orchestra.run.trial119.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=45875200 blks 2026-02-24T16:06:03.723 INFO:teuthology.orchestra.run.trial119.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-24T16:06:03.723 INFO:teuthology.orchestra.run.trial119.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-24T16:06:03.723 INFO:teuthology.orchestra.run.trial119.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-24T16:06:03.723 INFO:teuthology.orchestra.run.trial119.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-24T16:06:03.723 INFO:teuthology.orchestra.run.trial119.stdout: = sunit=0 swidth=0 blks 2026-02-24T16:06:03.723 INFO:teuthology.orchestra.run.trial119.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-24T16:06:03.723 INFO:teuthology.orchestra.run.trial119.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-24T16:06:03.723 INFO:teuthology.orchestra.run.trial119.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-24T16:06:03.724 INFO:teuthology.orchestra.run.trial119.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-24T16:06:04.279 INFO:teuthology.orchestra.run.trial119.stdout:Discarding blocks...Done. 2026-02-24T16:06:04.281 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@trial119.front.sepia.ceph.com -o noatime 2026-02-24T16:06:04.281 DEBUG:teuthology.orchestra.run.trial119:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2026-02-24T16:06:04.349 DEBUG:teuthology.orchestra.run.trial119:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2026-02-24T16:06:04.418 DEBUG:teuthology.orchestra.run.trial119:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-24T16:06:04.571 INFO:teuthology.orchestra.run.trial119.stderr:INFO  osd - OSD::main: parsing early config 2026-02-24T16:06:04.624 INFO:teuthology.orchestra.run.trial119.stderr:INFO 2026-02-24 16:06:04,623 seastar - Reactor backend: linux-aio 2026-02-24T16:06:04.633 INFO:teuthology.orchestra.run.trial119.stderr:INFO 2026-02-24 16:06:04,633 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-24T16:06:04.634 INFO:teuthology.orchestra.run.trial119.stderr:INFO 2026-02-24 16:06:04,633 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-24T16:06:04.636 INFO:teuthology.orchestra.run.trial119.stderr:==39703==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-24T16:06:04.695 INFO:teuthology.orchestra.run.trial119.stderr:DEBUG 2026-02-24 16:06:04,695 [shard 0:main] seastar - reactor::drain 2026-02-24T16:06:06.737 INFO:teuthology.orchestra.run.trial119.stderr:----------------------------------------------------- 2026-02-24T16:06:06.737 INFO:teuthology.orchestra.run.trial119.stderr:Suppressions used: 2026-02-24T16:06:06.737 INFO:teuthology.orchestra.run.trial119.stderr: count bytes template 2026-02-24T16:06:06.737 INFO:teuthology.orchestra.run.trial119.stderr: 1 8 MallocExtension::Register 2026-02-24T16:06:06.737 INFO:teuthology.orchestra.run.trial119.stderr:----------------------------------------------------- 2026-02-24T16:06:06.737 INFO:teuthology.orchestra.run.trial119.stderr: 2026-02-24T16:06:06.756 INFO:teuthology.orchestra.run.trial119.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-24T16:06:06.756 INFO:teuthology.orchestra.run.trial119.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-24T16:06:06.805 INFO:teuthology.orchestra.run.trial119.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-24T16:06:06.805 INFO:teuthology.orchestra.run.trial119.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-24T16:06:06.812 INFO:teuthology.orchestra.run.trial119.stderr:INFO 2026-02-24 16:06:06,812 seastar - Reactor backend: linux-aio 2026-02-24T16:06:06.827 INFO:teuthology.orchestra.run.trial119.stderr:INFO 2026-02-24 16:06:06,826 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-24T16:06:06.827 INFO:teuthology.orchestra.run.trial119.stderr:INFO 2026-02-24 16:06:06,826 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-24T16:06:06.831 INFO:teuthology.orchestra.run.trial119.stderr:==39701==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-24T16:06:06.831 INFO:teuthology.orchestra.run.trial119.stderr:INFO 2026-02-24 16:06:06,830 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-24T16:06:06.989 INFO:teuthology.orchestra.run.trial119.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc597 /lib64/libasan.so.8+0x91c1 /lib64/libasan.so.8+0xee7dc 0x776e7c7 0x776e866 0x755bcdb 0x763e0a6 0x7bc7238 0x7efedf5 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7272fae 0x7276b31 0x72f0839 0x7747f18 0x755fef8 0x7585796 0x752dc84 0x72cd4a6 0x7632ed7 0x7534252 0x72cd53c 0x76d5c2f 0x7747fa1 0x755fffa 0x75600f2 0x7560178 0x75601fd 0x758594e 0x7748022 0x756027a 0x75602ff 0x7585b55 0x76e815e 0x76e81e3 0x76e82c1 0x77428e2 0x7699a68 0x784409e 0x787bf26 0x76e6210 0x77b2de2 0x78d6e12 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:06:06.990 INFO:teuthology.orchestra.run.trial119.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffbb3ca691 0xffffffffbb3d4bab 0xffffffffbb3d500b 0xffffffffbb0910d2 0xffffffffbbd50451 0xffffffffbbe00ba2 2026-02-24T16:06:06.990 INFO:teuthology.orchestra.run.trial119.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x191c1b /lib64/libasan.so.8+0xc748 /lib64/libasan.so.8+0x91c1 /lib64/libasan.so.8+0xee7dc 0x9ec9adb 0x9f2f2d6 0x9f2f3ec 0x9f2f4c8 0x9f2f744 0x9f2fceb 0x9f300ef 0x9f303ac 0xa0d4a95 0xa096e39 0xa095e39 0x7eff1a0 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7272fae 0x7276b31 0x72f0839 0x7747f18 0x755fef8 0x7585796 0x752dc84 0x72cd4a6 0x7632ed7 0x7534252 0x72cd53c 0x76d5c2f 0x7747fa1 0x755fffa 0x75600f2 0x7560178 0x75601fd 0x758594e 0x7748022 0x756027a 0x75602ff 0x7585b55 0x76e815e 0x76e81e3 0x76e82c1 0x77428e2 0x7699a68 0x784409e 0x787bf26 0x76e6210 0x77b2de2 0x78d6e12 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:06:06.990 INFO:teuthology.orchestra.run.trial119.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffbbd636e6 0xffffffffbbd634a5 0xffffffffbb3e7985 0xffffffffbb3cc97a 0xffffffffbb3d4bab 0xffffffffbb3d500b 0xffffffffbb0910d2 0xffffffffbbd50451 0xffffffffbbe00ba2 2026-02-24T16:06:06.990 INFO:teuthology.orchestra.run.trial119.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x191c1b /lib64/libasan.so.8+0xc748 /lib64/libasan.so.8+0x91c1 /lib64/libasan.so.8+0xee7dc 0x9ec9adb 0x9f2f2d6 0x9f2f3ec 0x9f2f4c8 0x9f2fb84 0x9f2fceb 0x9f300ef 0x9f303ac 0xa0c0757 0xa0c0935 0xa0c0b91 0xa0c0c82 0xa0c1822 0xa0c1e26 0xa0c21b9 0xa0d4e40 0xa096e39 0xa095e39 0x7eff1a0 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7272fae 0x7276b31 0x72f0839 0x7747f18 0x755fef8 0x7585796 0x752dc84 0x72cd4a6 0x7632ed7 0x7534252 0x72cd53c 0x76d5c2f 0x7747fa1 0x755fffa 0x75600f2 0x7560178 0x75601fd 0x758594e 0x7748022 0x756027a 0x75602ff 0x7585b55 0x76e815e 0x76e81e3 0x76e82c1 0x77428e2 0x7699a68 0x784409e 0x787bf26 0x76e6210 0x77b2de2 0x78d6e12 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:06:06.990 INFO:teuthology.orchestra.run.trial119.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffbbd6387d 0xffffffffbbd634a5 0xffffffffbb3e7985 0xffffffffbb3cc97a 0xffffffffbb3d4bab 0xffffffffbb3d500b 0xffffffffbb0910d2 0xffffffffbbd50451 0xffffffffbbe00ba2 2026-02-24T16:06:07.022 INFO:teuthology.orchestra.run.trial119.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xf4e8f /lib64/libasan.so.8+0x10069c /lib64/libasan.so.8+0xc65b9 /lib64/libasan.so.8+0x157c7 0x10d1d95 0xf741d0 0xa0c299d 0xa0c2f8a 0xa0c330d 0xa0d3464 0xa0d68c7 0xa096e39 0xa095e39 0x7eff1a0 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7273c89 0x781ca8c 0x781cbe0 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fbc5e 0x9f82d83 0x9f835be 0xf2905b /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb83e94 2026-02-24T16:06:07.083 INFO:teuthology.orchestra.run.trial119.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1297e4 /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x157c7 0x10229ef 0x9f31044 0xa0a9b3b 0xa0a9c35 0x7efee32 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7273c89 0x781ca8c 0x781cbe0 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:06:07.083 INFO:teuthology.orchestra.run.trial119.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1589a 0x10d1d95 0xf741d0 0x9fe4d11 0x9fe500e 0x9fe5281 0x9fe5537 0x9fe5fe5 0x9fe64e3 0x9fe6b13 0x9fe6c7d 0xa0c203d 0xa0d4e40 0xa096e39 0xa095e39 0x7eff1a0 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7273c89 0x781ca8c 0x781cbe0 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:06:07.084 INFO:teuthology.orchestra.run.trial119.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1588f 0x10d1d95 0xf741d0 0x9fe882e 0x9fe916d 0xa0ce105 0xa0d87eb 0x7bc72d4 0x7eff0d5 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7273c89 0x781ca8c 0x781cbe0 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:06:07.109 INFO:teuthology.orchestra.run.trial119.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f 0x2ce9c0a 0x80977e7 0x81685ee 0x8168e55 0x8166af7 0x7b56e2f 0x7b57b24 0x7c6a1c6 0x7b2c212 0x7b8b1ca 0x7c1c577 0x7c1c7f0 0x7c56199 0x7c562f8 0x7c6a329 0x7b52948 0x7de7e59 0x7c77990 0x7db8dca 0x76ae866 0x76bcc3b 0x776718f 0x790351c 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fbc5e 0x9f82d83 0x9f835be 0xf2905b /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb83e94 2026-02-24T16:06:07.167 INFO:teuthology.orchestra.run.trial119.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1297e4 /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x15197 0x2cea8e9 0x80977b9 0x81685ee 0x8168e55 0x8166af7 0x7b56e2f 0x7b57b24 0x7c6a1c6 0x7b2c212 0x7b8b1ca 0x7c1c577 0x7c1c7f0 0x7c56199 0x7c562f8 0x7c6a329 0x7b52948 0x7de7e59 0x7c77990 0x7db8dca 0x76ae866 0x76bcc3b 0x776718f 0x790351c 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:06:07.167 INFO:teuthology.orchestra.run.trial119.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f 0x2cea6e3 0x80977b9 0x81685ee 0x8168e55 0x8166af7 0x7b56e2f 0x7b57b24 0x7c6a1c6 0x7b2c212 0x7b8b1ca 0x7c1c577 0x7c1c7f0 0x7c56199 0x7c562f8 0x7c6a329 0x7b52948 0x7de7e59 0x7c77990 0x7db8dca 0x76ae866 0x76bcc3b 0x776718f 0x790351c 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:06:07.168 INFO:teuthology.orchestra.run.trial119.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f 0x2ce98d8 0x80977d2 0x81685ee 0x8168e55 0x8166af7 0x7b56e2f 0x7b57b24 0x7c6a1c6 0x7b2c212 0x7b8b1ca 0x7c1c577 0x7c1c7f0 0x7c56199 0x7c562f8 0x7c6a329 0x7b52948 0x7de7e59 0x7c77990 0x7db8dca 0x76ae866 0x76bcc3b 0x776718f 0x790351c 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:06:07.305 INFO:teuthology.orchestra.run.trial119.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1f57f 0xe5adfd 0xa0b4811 0xa0d1bdf 0xa0d68c7 0xa096e39 0xa095e39 0x7eff1a0 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x722be9f 0x722c8e3 0x77424df 0x755e078 0x757f093 0x752da6e 0x71c9494 0x763159f 0x7533811 0x71c952a 0x76d525e 0x77426e1 0x755e1fe 0x755e2f6 0x755e37c 0x755e401 0x757f24a 0x7742762 0x755e47e 0x755e503 0x757f451 0x76e5fc1 0x76e6046 0x76e6124 0x77428e2 0x7699a68 0x784409e 0x787bf26 0x76e6210 0x77b2de2 0x78d6e12 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:06:07.305 INFO:teuthology.orchestra.run.trial119.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc597 /lib64/libasan.so.8+0x91c1 /lib64/libasan.so.8+0xee7dc 0x9ec9adb 0x9f2f2d6 0x9f2f3ec 0x9f2f4c8 0x9f2f744 0x9f2f7fd 0x9f2fceb 0x9f300ef 0x9f303ac 0x9f311f4 0xa0a9b3b 0xa0a9c35 0x7efee32 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x722be9f 0x722c8e3 0x77424df 0x755e078 0x757f093 0x752da6e 0x71c9494 0x763159f 0x7533811 0x71c952a 0x76d525e 0x77426e1 0x755e1fe 0x755e2f6 0x755e37c 0x755e401 0x757f24a 0x7742762 0x755e47e 0x755e503 0x757f451 0x76e5fc1 0x76e6046 0x76e6124 0x77428e2 0x7699a68 0x784409e 0x787bf26 0x76e6210 0x77b2de2 0x78d6e12 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:06:07.305 INFO:teuthology.orchestra.run.trial119.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffbb72a157 0xffffffffbb3f9006 0xffffffffbb3fc9c0 0xffffffffbb3fdd21 0xffffffffbb425a53 0xffffffffbb425c15 0xffffffffbb3ca888 0xffffffffbb3d4bab 0xffffffffbb3d500b 0xffffffffbb0910d2 0xffffffffbbd50451 0xffffffffbbe00ba2 2026-02-24T16:06:07.306 INFO:teuthology.orchestra.run.trial119.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x8cc0f /lib64/libasan.so.8+0x10069c /lib64/libasan.so.8+0xc65b9 /lib64/libasan.so.8+0x15197 0x7ebcff1 0xa0d8806 0x7bc72d4 0x7efedf5 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x722be9f 0x722c8e3 0x77424df 0x755e078 0x757f093 0x752da6e 0x71c9494 0x763159f 0x7533811 0x71c952a 0x76d525e 0x77426e1 0x755e1fe 0x755e2f6 0x755e37c 0x755e401 0x757f24a 0x7742762 0x755e47e 0x755e503 0x757f451 0x76e5fc1 0x76e6046 0x76e6124 0x77428e2 0x7699a68 0x784409e 0x787bf26 0x76e6210 0x77b2de2 0x78d6e12 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:06:07.387 INFO:teuthology.orchestra.run.trial119.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f 0x2c95a69 0x2cea88a 0x80977b9 0x81685ee 0x8168e55 0x8166af7 0x7b56e2f 0x7b57b24 0x7c6a1c6 0x7b2c212 0x7b8b1ca 0x7c1c577 0x7c1c7f0 0x7c56199 0x7c562f8 0x7c6a329 0x7b52948 0x7de7e59 0x7c77990 0x7db8dca 0x76ae866 0x76bcc3b 0x776718f 0x790351c 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fbc5e 0x9f82d83 0x9f835be 0xf2905b /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb83e94 2026-02-24T16:06:07.390 INFO:teuthology.orchestra.run.trial119.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f 0x2cea26b 0x80977f4 0x81685ee 0x8168e55 0x8166af7 0x7b56e2f 0x7b57b24 0x7c6a1c6 0x7b2c212 0x7b8b1ca 0x7c1c577 0x7c1c7f0 0x7c56199 0x7c562f8 0x7c6a329 0x7b52948 0x7de7e59 0x7c77990 0x7db8dca 0x76ae866 0x76bcc3b 0x776718f 0x790351c 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:06:07.422 INFO:teuthology.orchestra.run.trial119.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f 0x2cea26b 0x80977f4 0x81685ee 0x8168e55 0x8166af7 0x7b56e2f 0x7b57b24 0x7c6a1c6 0x7b2c212 0x7b8b1ca 0x7c1c577 0x7c1c7f0 0x7c56199 0x7c562f8 0x7c6a329 0x7b52948 0x7de7e59 0x7c77990 0x7db8dca 0x76ae866 0x76bcc3b 0x776718f 0x790351c 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:06:07.422 INFO:teuthology.orchestra.run.trial119.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f 0x2cea8e9 0x80977b9 0x81685ee 0x8168e55 0x8166af7 0x7b56e2f 0x7b57b24 0x7c6a1c6 0x7b2c212 0x7b8b1ca 0x7c1c577 0x7c1c7f0 0x7c56199 0x7c562f8 0x7c6a329 0x7b52948 0x7de7e59 0x7c77990 0x7db8dca 0x76ae866 0x76bcc3b 0x776718f 0x790351c 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:06:09.622 INFO:teuthology.orchestra.run.trial119.stderr:----------------------------------------------------- 2026-02-24T16:06:09.622 INFO:teuthology.orchestra.run.trial119.stderr:Suppressions used: 2026-02-24T16:06:09.623 INFO:teuthology.orchestra.run.trial119.stderr: count bytes template 2026-02-24T16:06:09.623 INFO:teuthology.orchestra.run.trial119.stderr: 1 8 MallocExtension::Register 2026-02-24T16:06:09.623 INFO:teuthology.orchestra.run.trial119.stderr:----------------------------------------------------- 2026-02-24T16:06:09.623 INFO:teuthology.orchestra.run.trial119.stderr: 2026-02-24T16:06:09.662 DEBUG:teuthology.orchestra.run.trial119:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2026-02-24T16:06:09.688 DEBUG:teuthology.orchestra.run.trial119:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-24T16:06:09.841 INFO:teuthology.orchestra.run.trial119.stderr:INFO  osd - OSD::main: parsing early config 2026-02-24T16:06:09.892 INFO:teuthology.orchestra.run.trial119.stderr:INFO 2026-02-24 16:06:09,891 seastar - Reactor backend: linux-aio 2026-02-24T16:06:09.902 INFO:teuthology.orchestra.run.trial119.stderr:INFO 2026-02-24 16:06:09,901 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-24T16:06:09.902 INFO:teuthology.orchestra.run.trial119.stderr:INFO 2026-02-24 16:06:09,901 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-24T16:06:09.904 INFO:teuthology.orchestra.run.trial119.stderr:==39769==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-24T16:06:09.964 INFO:teuthology.orchestra.run.trial119.stderr:DEBUG 2026-02-24 16:06:09,964 [shard 0:main] seastar - reactor::drain 2026-02-24T16:06:12.019 INFO:teuthology.orchestra.run.trial119.stderr:----------------------------------------------------- 2026-02-24T16:06:12.019 INFO:teuthology.orchestra.run.trial119.stderr:Suppressions used: 2026-02-24T16:06:12.019 INFO:teuthology.orchestra.run.trial119.stderr: count bytes template 2026-02-24T16:06:12.020 INFO:teuthology.orchestra.run.trial119.stderr: 1 8 MallocExtension::Register 2026-02-24T16:06:12.020 INFO:teuthology.orchestra.run.trial119.stderr:----------------------------------------------------- 2026-02-24T16:06:12.020 INFO:teuthology.orchestra.run.trial119.stderr: 2026-02-24T16:06:12.039 INFO:teuthology.orchestra.run.trial119.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-24T16:06:12.039 INFO:teuthology.orchestra.run.trial119.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-24T16:06:12.081 INFO:teuthology.orchestra.run.trial119.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-24T16:06:12.082 INFO:teuthology.orchestra.run.trial119.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-24T16:06:12.088 INFO:teuthology.orchestra.run.trial119.stderr:INFO 2026-02-24 16:06:12,088 seastar - Reactor backend: linux-aio 2026-02-24T16:06:12.097 INFO:teuthology.orchestra.run.trial119.stderr:INFO 2026-02-24 16:06:12,097 [shard 3:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-24T16:06:12.098 INFO:teuthology.orchestra.run.trial119.stderr:INFO 2026-02-24 16:06:12,097 [shard 3:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-24T16:06:12.101 INFO:teuthology.orchestra.run.trial119.stderr:==39767==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-24T16:06:12.101 INFO:teuthology.orchestra.run.trial119.stderr:INFO 2026-02-24 16:06:12,101 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-24T16:06:12.260 INFO:teuthology.orchestra.run.trial119.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f 0x341e225 0x9fe4f97 0x9fe5281 0x9fe6fa7 0x9fe74f4 0x9fe76f3 0xa0c7a65 0xa0c81c0 0xa0d244e 0xa0d68c7 0xa096e39 0xa095e39 0x7eff1a0 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7272fae 0x7276b31 0x72f0839 0x7747f18 0x755fef8 0x7585796 0x752dc84 0x72cd4a6 0x7632ed7 0x7534252 0x72cd53c 0x76d5c2f 0x7747fa1 0x755fffa 0x75600f2 0x7560178 0x75601fd 0x758594e 0x7748022 0x756027a 0x75602ff 0x7585b55 0x76e815e 0x76e81e3 0x76e82c1 0x77428e2 0x7699a68 0x784409e 0x787bf26 0x76e6210 0x77b2de2 0x78d6e12 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:06:12.260 INFO:teuthology.orchestra.run.trial119.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x104e4a /lib64/libasan.so.8+0xea9a /lib64/libasan.so.8+0xec776 0xcee0e1 0x7242d24 0x7eff2ec 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7272fae 0x7276b31 0x72f0839 0x7747f18 0x755fef8 0x7585796 0x752dc84 0x72cd4a6 0x7632ed7 0x7534252 0x72cd53c 0x76d5c2f 0x7747fa1 0x755fffa 0x75600f2 0x7560178 0x75601fd 0x758594e 0x7748022 0x756027a 0x75602ff 0x7585b55 0x76e815e 0x76e81e3 0x76e82c1 0x77428e2 0x7699a68 0x784409e 0x787bf26 0x76e6210 0x77b2de2 0x78d6e12 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:06:12.260 INFO:teuthology.orchestra.run.trial119.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f 0xf74212 0x9fe4d11 0x9fe500e 0x9fe5281 0x9fe6fa7 0x9fe74f4 0x9fe76f3 0xa0c7a65 0xa0c81c0 0xa0d244e 0xa0d68c7 0xa096e39 0xa095e39 0x7eff1a0 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7272fae 0x7276b31 0x72f0839 0x7747f18 0x755fef8 0x7585796 0x752dc84 0x72cd4a6 0x7632ed7 0x7534252 0x72cd53c 0x76d5c2f 0x7747fa1 0x755fffa 0x75600f2 0x7560178 0x75601fd 0x758594e 0x7748022 0x756027a 0x75602ff 0x7585b55 0x76e815e 0x76e81e3 0x76e82c1 0x77428e2 0x7699a68 0x784409e 0x787bf26 0x76e6210 0x77b2de2 0x78d6e12 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:06:12.292 INFO:teuthology.orchestra.run.trial119.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x191cbf /lib64/libasan.so.8+0xb9ea /lib64/libasan.so.8+0xbf98 /lib64/libasan.so.8+0xc08c /lib64/libasan.so.8+0xc360 /lib64/libasan.so.8+0xc553 /lib64/libasan.so.8+0x91c1 /lib64/libasan.so.8+0xee7dc 0x776e7c7 0x776e866 0x755bcdb 0x763e0a6 0x7efeea0 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7273c89 0x781ca8c 0x781cbe0 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fbc5e 0x9f82d83 0x9f835be 0xf2905b /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb83e94 2026-02-24T16:06:12.293 INFO:teuthology.orchestra.run.trial119.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffbbd62f0b 0xffffffffbb3f9b78 0xffffffffbb3fa3e3 0xffffffffbb3fa864 0xffffffffbb3fc7f9 0xffffffffbb3fdd21 0xffffffffbb425a53 0xffffffffbb425c15 0xffffffffbb3cc8ce 0xffffffffbb3d4bab 0xffffffffbb3d500b 0xffffffffbb0910d2 0xffffffffbbd50451 0xffffffffbbe00ba2 2026-02-24T16:06:12.321 INFO:teuthology.orchestra.run.trial119.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f 0x7f1b6f7 0x801910a 0x80191f2 0x8019516 0x801a135 0x7ee8e78 0x7eff6a2 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7273c89 0x781ca8c 0x781cbe0 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:06:12.353 INFO:teuthology.orchestra.run.trial119.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x191d45 /lib64/libasan.so.8+0xc748 /lib64/libasan.so.8+0x8eea /lib64/libasan.so.8+0xedc45 0xe6ca27 0xe6cd08 0xa0d88c9 0x7bc72d4 0x7eff0d5 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7273c89 0x781ca8c 0x781cbe0 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:06:12.354 INFO:teuthology.orchestra.run.trial119.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffbb6c7ef1 0xffffffffbb3fa253 0xffffffffbb3fa864 0xffffffffbb3fc7f9 0xffffffffbb3fdd21 0xffffffffbb425a53 0xffffffffbb425c15 0xffffffffbb3cc8ce 0xffffffffbb3d4bab 0xffffffffbb3d500b 0xffffffffbb0910d2 0xffffffffbbd50451 0xffffffffbbe00ba2 2026-02-24T16:06:12.354 INFO:teuthology.orchestra.run.trial119.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xa35e /lib64/libasan.so.8+0xc478 /lib64/libasan.so.8+0x8eea /lib64/libasan.so.8+0xedc45 0xe6ca27 0xa0d87b8 0x7bc72d4 0x7efedf5 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7273c89 0x781ca8c 0x781cbe0 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:06:12.378 INFO:teuthology.orchestra.run.trial119.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc65d9 /lib64/libasan.so.8+0x15197 0x2cea8e9 0x80977b9 0x81685ee 0x8168e55 0x8166af7 0x7b56e2f 0x7b57b24 0x7c6a1c6 0x7b2c212 0x7b8b1ca 0x7c1c577 0x7c1c7f0 0x7c56199 0x7c562f8 0x7c6a329 0x7b52948 0x7de7e59 0x7c77990 0x7db8dca 0x76ae866 0x76bcc3b 0x776718f 0x790351c 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fbc5e 0x9f82d83 0x9f835be 0xf2905b /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb83e94 2026-02-24T16:06:12.436 INFO:teuthology.orchestra.run.trial119.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f 0x2cea20d 0x80977f4 0x81685ee 0x8168e55 0x8166af7 0x7b56e2f 0x7b57b24 0x7c6a1c6 0x7b2c212 0x7b8b1ca 0x7c1c577 0x7c1c7f0 0x7c56199 0x7c562f8 0x7c6a329 0x7b52948 0x7de7e59 0x7c77990 0x7db8dca 0x76ae866 0x76bcc3b 0x776718f 0x790351c 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:06:12.437 INFO:teuthology.orchestra.run.trial119.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x12985d /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x15197 0x2cea8e9 0x80977b9 0x81685ee 0x8168e55 0x8166af7 0x7b56e2f 0x7b57b24 0x7c6a1c6 0x7b2c212 0x7b8b1ca 0x7c1c577 0x7c1c7f0 0x7c56199 0x7c562f8 0x7c6a329 0x7b52948 0x7de7e59 0x7c77990 0x7db8dca 0x76ae866 0x76bcc3b 0x776718f 0x790351c 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:06:12.440 INFO:teuthology.orchestra.run.trial119.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc657f /lib64/libasan.so.8+0x15197 0x2cea8e9 0x80977b9 0x81685ee 0x8168e55 0x8166af7 0x7b56e2f 0x7b57b24 0x7c6a1c6 0x7b2c212 0x7b8b1ca 0x7c1c577 0x7c1c7f0 0x7c56199 0x7c562f8 0x7c6a329 0x7b52948 0x7de7e59 0x7c77990 0x7db8dca 0x76ae866 0x76bcc3b 0x776718f 0x790351c 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:06:12.538 INFO:teuthology.orchestra.run.trial119.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f 0x38f04a6 0x9ff8673 0x9ff8735 0xa0b171c 0xa0b18b1 0xa0b1c43 0xa0b3229 0xa0b4544 0xa0b4964 0xa0d1bdf 0xa0d68c7 0xa096e39 0xa095e39 0x7eec254 0x7f0412c 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x722be9f 0x722c8e3 0x77424df 0x755e078 0x757f093 0x752da6e 0x71c9494 0x763159f 0x7533811 0x71c952a 0x76d525e 0x77426e1 0x755e1fe 0x755e2f6 0x755e37c 0x755e401 0x757f24a 0x7742762 0x755e47e 0x755e503 0x757f451 0x76e5fc1 0x76e6046 0x76e6124 0x77428e2 0x7699a68 0x784409e 0x787bf26 0x76e6210 0x77b2de2 0x77b6764 0x76973b7 0x787c3c0 0x76e6d14 0x76e6e2d 0xa3a2446 0xa3a2649 0xa3a26e7 0xa3a276c 0xa3f5b6e 0xa3f5c27 0xa427569 0xa427910 0xa32dc97 0xa32dd50 0xa32dddf 0xa3f56d6 0xa3f5799 0xa3f58a6 0xa3f5a61 0x763eb45 0x7697625 0x769785c 0x7538bad 0x726516c 0x781cf48 0x781d09c 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fbc5e 0x9f82d83 0x9f835be 0xf2905b /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb83e94 2026-02-24T16:06:12.575 INFO:teuthology.orchestra.run.trial119.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f 0x38d44dc 0x38d45db 0x38f0458 0x9ff8673 0x9ff8756 0xa0b171c 0xa0b18b1 0xa0b1c43 0xa0b3229 0xa0b4544 0xa0b4964 0xa0d1bdf 0xa0d68c7 0xa096e39 0xa095e39 0x7eff1a0 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x722be9f 0x722c8e3 0x77424df 0x755e078 0x757f093 0x752da6e 0x71c9494 0x763159f 0x7533811 0x71c952a 0x76d525e 0x77426e1 0x755e1fe 0x755e2f6 0x755e37c 0x755e401 0x757f24a 0x7742762 0x755e47e 0x755e503 0x757f451 0x76e5fc1 0x76e6046 0x76e6124 0x77428e2 0x7699a68 0x784409e 0x787bf26 0x76e6210 0x77b2de2 0x78d6e12 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:06:12.575 INFO:teuthology.orchestra.run.trial119.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1195e8 /lib64/libasan.so.8+0xfd239 /lib64/libasan.so.8+0xee859 0x9ec9adb 0xa09aab6 0xa09abe4 0xa0cdb7b 0xa0ce31e 0xa0d87eb 0x7bc72d4 0x7efedf5 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x722be9f 0x722c8e3 0x77424df 0x755e078 0x757f093 0x752da6e 0x71c9494 0x763159f 0x7533811 0x71c952a 0x76d525e 0x77426e1 0x755e1fe 0x755e2f6 0x755e37c 0x755e401 0x757f24a 0x7742762 0x755e47e 0x755e503 0x757f451 0x76e5fc1 0x76e6046 0x76e6124 0x77428e2 0x7699a68 0x784409e 0x787bf26 0x76e6210 0x77b2de2 0x78d6e12 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:06:12.576 INFO:teuthology.orchestra.run.trial119.stderr:Reactor stalled for 33 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x119122 /lib64/libasan.so.8+0xc5f4 /lib64/libasan.so.8+0x8eea /lib64/libasan.so.8+0xedc45 0xe6ca27 0xe6cd08 0xa0d88c9 0x7bc72d4 0x7efedf5 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x722be9f 0x722c8e3 0x77424df 0x755e078 0x757f093 0x752da6e 0x71c9494 0x763159f 0x7533811 0x71c952a 0x76d525e 0x77426e1 0x755e1fe 0x755e2f6 0x755e37c 0x755e401 0x757f24a 0x7742762 0x755e47e 0x755e503 0x757f451 0x76e5fc1 0x76e6046 0x76e6124 0x77428e2 0x7699a68 0x784409e 0x787bf26 0x76e6210 0x77b2de2 0x78d6e12 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:06:12.658 INFO:teuthology.orchestra.run.trial119.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x151af 0x2cea8e9 0x80977b9 0x81685ee 0x8168e55 0x8166af7 0x7b56e2f 0x7b57b24 0x7c6a1c6 0x7b2c212 0x7b8b1ca 0x7c1c577 0x7c1c7f0 0x7c56199 0x7c562f8 0x7c6a329 0x7b52948 0x7de7e59 0x7c77990 0x7db8dca 0x76ae866 0x76bcc3b 0x776718f 0x790351c 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fbc5e 0x9f82d83 0x9f835be 0xf2905b /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb83e94 2026-02-24T16:06:12.695 INFO:teuthology.orchestra.run.trial119.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc6592 /lib64/libasan.so.8+0x15197 0x2cea8e9 0x80977b9 0x81685ee 0x8168e55 0x8166af7 0x7b56e2f 0x7b57b24 0x7c6a1c6 0x7b2c212 0x7b8b1ca 0x7c1c577 0x7c1c7f0 0x7c56199 0x7c562f8 0x7c6a329 0x7b52948 0x7de7e59 0x7c77990 0x7db8dca 0x76ae866 0x76bcc3b 0x776718f 0x790351c 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:06:12.695 INFO:teuthology.orchestra.run.trial119.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f 0x2cea71f 0x80977b9 0x81685ee 0x8168e55 0x8166af7 0x7b56e2f 0x7b57b24 0x7c6a1c6 0x7b2c212 0x7b8b1ca 0x7c1c577 0x7c1c7f0 0x7c56199 0x7c562f8 0x7c6a329 0x7b52948 0x7de7e59 0x7c77990 0x7db8dca 0x76ae866 0x76bcc3b 0x776718f 0x790351c 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:06:12.697 INFO:teuthology.orchestra.run.trial119.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f 0x2ce9be0 0x80977e7 0x81685ee 0x8168e55 0x8166af7 0x7b56e2f 0x7b57b24 0x7c6a1c6 0x7b2c212 0x7b8b1ca 0x7c1c577 0x7c1c7f0 0x7c56199 0x7c562f8 0x7c6a329 0x7b52948 0x7de7e59 0x7c77990 0x7db8dca 0x76ae866 0x76bcc3b 0x776718f 0x790351c 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:06:14.887 INFO:teuthology.orchestra.run.trial119.stderr:----------------------------------------------------- 2026-02-24T16:06:14.888 INFO:teuthology.orchestra.run.trial119.stderr:Suppressions used: 2026-02-24T16:06:14.888 INFO:teuthology.orchestra.run.trial119.stderr: count bytes template 2026-02-24T16:06:14.888 INFO:teuthology.orchestra.run.trial119.stderr: 1 8 MallocExtension::Register 2026-02-24T16:06:14.888 INFO:teuthology.orchestra.run.trial119.stderr:----------------------------------------------------- 2026-02-24T16:06:14.888 INFO:teuthology.orchestra.run.trial119.stderr: 2026-02-24T16:06:14.927 DEBUG:teuthology.orchestra.run.trial119:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2026-02-24T16:06:14.952 DEBUG:teuthology.orchestra.run.trial119:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-24T16:06:15.107 INFO:teuthology.orchestra.run.trial119.stderr:INFO  osd - OSD::main: parsing early config 2026-02-24T16:06:15.164 INFO:teuthology.orchestra.run.trial119.stderr:INFO 2026-02-24 16:06:15,163 seastar - Reactor backend: linux-aio 2026-02-24T16:06:15.177 INFO:teuthology.orchestra.run.trial119.stderr:INFO 2026-02-24 16:06:15,177 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-24T16:06:15.178 INFO:teuthology.orchestra.run.trial119.stderr:INFO 2026-02-24 16:06:15,177 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-24T16:06:15.180 INFO:teuthology.orchestra.run.trial119.stderr:==39836==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-24T16:06:15.239 INFO:teuthology.orchestra.run.trial119.stderr:DEBUG 2026-02-24 16:06:15,239 [shard 0:main] seastar - reactor::drain 2026-02-24T16:06:17.307 INFO:teuthology.orchestra.run.trial119.stderr:----------------------------------------------------- 2026-02-24T16:06:17.308 INFO:teuthology.orchestra.run.trial119.stderr:Suppressions used: 2026-02-24T16:06:17.308 INFO:teuthology.orchestra.run.trial119.stderr: count bytes template 2026-02-24T16:06:17.308 INFO:teuthology.orchestra.run.trial119.stderr: 1 8 MallocExtension::Register 2026-02-24T16:06:17.308 INFO:teuthology.orchestra.run.trial119.stderr:----------------------------------------------------- 2026-02-24T16:06:17.308 INFO:teuthology.orchestra.run.trial119.stderr: 2026-02-24T16:06:17.328 INFO:teuthology.orchestra.run.trial119.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-24T16:06:17.328 INFO:teuthology.orchestra.run.trial119.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-24T16:06:17.379 INFO:teuthology.orchestra.run.trial119.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-24T16:06:17.379 INFO:teuthology.orchestra.run.trial119.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-24T16:06:17.386 INFO:teuthology.orchestra.run.trial119.stderr:INFO 2026-02-24 16:06:17,386 seastar - Reactor backend: linux-aio 2026-02-24T16:06:17.398 INFO:teuthology.orchestra.run.trial119.stderr:INFO 2026-02-24 16:06:17,397 [shard 3:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-24T16:06:17.398 INFO:teuthology.orchestra.run.trial119.stderr:INFO 2026-02-24 16:06:17,397 [shard 3:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-24T16:06:17.402 INFO:teuthology.orchestra.run.trial119.stderr:==39834==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-24T16:06:17.402 INFO:teuthology.orchestra.run.trial119.stderr:INFO 2026-02-24 16:06:17,402 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-24T16:06:17.562 INFO:teuthology.orchestra.run.trial119.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc65b1 /lib64/libasan.so.8+0x157c7 0xa09a6fe 0xa09a86f 0xa09aadf 0xa09abe4 0xa0cdb7b 0xa0ce31e 0xa0d8c14 0x7bc72d4 0x7eff0d5 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7272fae 0x7276b31 0x72f0839 0x7747f18 0x755fef8 0x7585796 0x752dc84 0x72cd4a6 0x7632ed7 0x7534252 0x72cd53c 0x76d5c2f 0x7747fa1 0x755fffa 0x75600f2 0x7560178 0x75601fd 0x758594e 0x7748022 0x756027a 0x75602ff 0x7585b55 0x76e815e 0x76e81e3 0x76e82c1 0x77428e2 0x7699a68 0x784409e 0x787bf26 0x76e6210 0x77b2de2 0x78d6e12 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:06:17.562 INFO:teuthology.orchestra.run.trial119.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f 0x341e2c1 0x9fe4f7e 0x9fe5281 0x9fe6fa7 0xa0c7f51 0xa0c81c0 0xa0d244e 0xa0d68c7 0xa096e39 0xa095e39 0x7eff1a0 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7272fae 0x7276b31 0x72f0839 0x7747f18 0x755fef8 0x7585796 0x752dc84 0x72cd4a6 0x7632ed7 0x7534252 0x72cd53c 0x76d5c2f 0x7747fa1 0x755fffa 0x75600f2 0x7560178 0x75601fd 0x758594e 0x7748022 0x756027a 0x75602ff 0x7585b55 0x76e815e 0x76e81e3 0x76e82c1 0x77428e2 0x7699a68 0x784409e 0x787bf26 0x76e6210 0x77b2de2 0x78d6e12 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:06:17.562 INFO:teuthology.orchestra.run.trial119.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x15829 0x10d1d95 0xf741d0 0x9fe882e 0x9fe916d 0xa0ce105 0xa0d87eb 0x7bc72d4 0x7eff0d5 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7272fae 0x7276b31 0x72f0839 0x7747f18 0x755fef8 0x7585796 0x752dc84 0x72cd4a6 0x7632ed7 0x7534252 0x72cd53c 0x76d5c2f 0x7747fa1 0x755fffa 0x75600f2 0x7560178 0x75601fd 0x758594e 0x7748022 0x756027a 0x75602ff 0x7585b55 0x76e815e 0x76e81e3 0x76e82c1 0x77428e2 0x7699a68 0x784409e 0x787bf26 0x76e6210 0x77b2de2 0x78d6e12 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:06:17.594 INFO:teuthology.orchestra.run.trial119.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc3e0 /lib64/libasan.so.8+0x8eea /lib64/libasan.so.8+0xedc45 0xe6ca27 0x9f30e27 0xa0a9b3b 0xa0a9c35 0x7efee32 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7273c89 0x781ca8c 0x781cbe0 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fbc5e 0x9f82d83 0x9f835be 0xf2905b /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb83e94 2026-02-24T16:06:17.624 INFO:teuthology.orchestra.run.trial119.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc597 /lib64/libasan.so.8+0x8eea /lib64/libasan.so.8+0xedc45 0xe6ca27 0x7bc7122 0x7eff0d5 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7273c89 0x781ca8c 0x781cbe0 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:06:17.624 INFO:teuthology.orchestra.run.trial119.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffbb3fc751 0xffffffffbb3fdd21 0xffffffffbb425a53 0xffffffffbb425c15 0xffffffffbb3ca888 0xffffffffbb3d4bab 0xffffffffbb3d500b 0xffffffffbb0910d2 0xffffffffbbd50451 0xffffffffbbe00ba2 2026-02-24T16:06:17.647 INFO:teuthology.orchestra.run.trial119.stderr:Reactor stalled for 32 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc6592 /lib64/libasan.so.8+0x15197 0x2cea8e9 0x80977b9 0x81685ee 0x8168e55 0x8166af7 0x7b56e2f 0x7b57b24 0x7c6a1c6 0x7b2c212 0x7b8b1ca 0x7c1c577 0x7c1c7f0 0x7c56199 0x7c562f8 0x7c6a329 0x7b52948 0x7de7e59 0x7c77990 0x7db8dca 0x76ae866 0x76bcc3b 0x776718f 0x790351c 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fbc5e 0x9f82d83 0x9f835be 0xf2905b /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb83e94 2026-02-24T16:06:17.655 INFO:teuthology.orchestra.run.trial119.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xb13e6 0xba66a7 0x10d1d1e 0xf741d0 0xa0ce5c9 0xa0d87eb 0x7bc72d4 0x7efedf5 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7273c89 0x781ca8c 0x781cbe0 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:06:17.655 INFO:teuthology.orchestra.run.trial119.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f 0x9ff6cc2 0xa0cdd4c 0xa0ce31e 0xa0d87eb 0x7bc72d4 0x7eff0d5 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x7273c89 0x781ca8c 0x781cbe0 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:06:17.740 INFO:teuthology.orchestra.run.trial119.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1297e4 /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x15197 0x2cea8e9 0x80977b9 0x81685ee 0x8168e55 0x8166af7 0x7b56e2f 0x7b57b24 0x7c6a1c6 0x7b2c212 0x7b8b1ca 0x7c1c577 0x7c1c7f0 0x7c56199 0x7c562f8 0x7c6a329 0x7b52948 0x7de7e59 0x7c77990 0x7db8dca 0x76ae866 0x76bcc3b 0x776718f 0x790351c 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:06:17.740 INFO:teuthology.orchestra.run.trial119.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f 0x2c95a46 0x2cea88a 0x80977b9 0x81685ee 0x8168e55 0x8166af7 0x7b56e2f 0x7b57b24 0x7c6a1c6 0x7b2c212 0x7b8b1ca 0x7c1c577 0x7c1c7f0 0x7c56199 0x7c562f8 0x7c6a329 0x7b52948 0x7de7e59 0x7c77990 0x7db8dca 0x76ae866 0x76bcc3b 0x776718f 0x790351c 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:06:17.741 INFO:teuthology.orchestra.run.trial119.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f 0x2c95a94 0x2cea88a 0x80977b9 0x81685ee 0x8168e55 0x8166af7 0x7b56e2f 0x7b57b24 0x7c6a1c6 0x7b2c212 0x7b8b1ca 0x7c1c577 0x7c1c7f0 0x7c56199 0x7c562f8 0x7c6a329 0x7b52948 0x7de7e59 0x7c77990 0x7db8dca 0x76ae866 0x76bcc3b 0x776718f 0x790351c 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:06:17.839 INFO:teuthology.orchestra.run.trial119.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xfcb03 /lib64/libasan.so.8+0xb13ef 0xba66a7 0x10d1d1e 0xf741d0 0x9fe4d11 0x9fe500e 0x9fe5281 0x9fe5537 0x9fe5af8 0x9fec307 0x9fec9e7 0xa0d221c 0xa0d68c7 0xa096e39 0xa095e39 0x7eff1a0 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x722be9f 0x722c8e3 0x77424df 0x755e078 0x757f093 0x752da6e 0x71c9494 0x763159f 0x7533811 0x71c952a 0x76d525e 0x77426e1 0x755e1fe 0x755e2f6 0x755e37c 0x755e401 0x757f24a 0x7742762 0x755e47e 0x755e503 0x757f451 0x76e5fc1 0x76e6046 0x76e6124 0x77428e2 0x7699a68 0x784409e 0x787bf26 0x76e6210 0x77b2de2 0x77b6764 0x76973b7 0x787c3c0 0x76e6d14 0x76e6e2d 0xa3a2446 0xa3a2649 0xa3a26e7 0xa3a276c 0xa3f5b6e 0xa3f5c27 0xa427569 0xa427910 0xa32dc97 0xa32dd50 0xa32dddf 0xa3f56d6 0xa3f5799 0xa3f58a6 0xa3f5a61 0x763eb45 0x7697625 0x769785c 0x7538bad 0x726516c 0x781cf48 0x781d09c 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fbc5e 0x9f82d83 0x9f835be 0xf2905b /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb83e94 2026-02-24T16:06:17.883 INFO:teuthology.orchestra.run.trial119.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f 0xa0d8769 0x7bc72d4 0x7efedf5 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x722be9f 0x722c8e3 0x77424df 0x755e078 0x757f093 0x752da6e 0x71c9494 0x763159f 0x7533811 0x71c952a 0x76d525e 0x77426e1 0x755e1fe 0x755e2f6 0x755e37c 0x755e401 0x757f24a 0x7742762 0x755e47e 0x755e503 0x757f451 0x76e5fc1 0x76e6046 0x76e6124 0x77428e2 0x7699a68 0x784409e 0x787bf26 0x76e6210 0x77b2de2 0x78d6e12 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:06:17.883 INFO:teuthology.orchestra.run.trial119.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc65c2 /lib64/libasan.so.8+0x15197 0x9fe4bad 0x9fe500e 0x9fe5281 0x9fe6fa7 0x9fe74f4 0x9fe76f3 0xa0c7a65 0xa0c81c0 0xa0d244e 0xa0d68c7 0xa096e39 0xa095e39 0x7eff1a0 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x722be9f 0x722c8e3 0x77424df 0x755e078 0x757f093 0x752da6e 0x71c9494 0x763159f 0x7533811 0x71c952a 0x76d525e 0x77426e1 0x755e1fe 0x755e2f6 0x755e37c 0x755e401 0x757f24a 0x7742762 0x755e47e 0x755e503 0x757f451 0x76e5fc1 0x76e6046 0x76e6124 0x77428e2 0x7699a68 0x784409e 0x787bf26 0x76e6210 0x77b2de2 0x78d6e12 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:06:17.884 INFO:teuthology.orchestra.run.trial119.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x8cc0c /lib64/libasan.so.8+0x10069c /lib64/libasan.so.8+0xc65b9 /lib64/libasan.so.8+0x157c7 0x10d1d95 0xf741d0 0x9fe882e 0x9fe916d 0xa0ce105 0xa0d87eb 0x7bc72d4 0x7efedf5 0x7f0addf 0x7d0df14 0x7d43a07 0x7492021 0x722be9f 0x722c8e3 0x77424df 0x755e078 0x757f093 0x752da6e 0x71c9494 0x763159f 0x7533811 0x71c952a 0x76d525e 0x77426e1 0x755e1fe 0x755e2f6 0x755e37c 0x755e401 0x757f24a 0x7742762 0x755e47e 0x755e503 0x757f451 0x76e5fc1 0x76e6046 0x76e6124 0x77428e2 0x7699a68 0x784409e 0x787bf26 0x76e6210 0x77b2de2 0x78d6e12 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:06:17.966 INFO:teuthology.orchestra.run.trial119.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x151d9 0x2cea8e9 0x80977b9 0x81685ee 0x8168e55 0x8166af7 0x7b56e2f 0x7b57b24 0x7c6a1c6 0x7b2c212 0x7b8b1ca 0x7c1c577 0x7c1c7f0 0x7c56199 0x7c562f8 0x7c6a329 0x7b52948 0x7de7e59 0x7c77990 0x7db8dca 0x76ae866 0x76bcc3b 0x776718f 0x790351c 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fbc5e 0x9f82d83 0x9f835be 0xf2905b /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb83e94 2026-02-24T16:06:18.000 INFO:teuthology.orchestra.run.trial119.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f 0x2cea1a2 0x80977f4 0x81685ee 0x8168e55 0x8166af7 0x7b56e2f 0x7b57b24 0x7c6a1c6 0x7b2c212 0x7b8b1ca 0x7c1c577 0x7c1c7f0 0x7c56199 0x7c562f8 0x7c6a329 0x7b52948 0x7de7e59 0x7c77990 0x7db8dca 0x76ae866 0x76bcc3b 0x776718f 0x790351c 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:06:18.001 INFO:teuthology.orchestra.run.trial119.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x8cc25 /lib64/libasan.so.8+0x10069c /lib64/libasan.so.8+0xc65b9 /lib64/libasan.so.8+0x15197 0x2cea8e9 0x80977b9 0x81685ee 0x8168e55 0x8166af7 0x7b56e2f 0x7b57b24 0x7c6a1c6 0x7b2c212 0x7b8b1ca 0x7c1c577 0x7c1c7f0 0x7c56199 0x7c562f8 0x7c6a329 0x7b52948 0x7de7e59 0x7c77990 0x7db8dca 0x76ae866 0x76bcc3b 0x776718f 0x790351c 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:06:18.001 INFO:teuthology.orchestra.run.trial119.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa2130ab 0xa0226c9 0xa06f8ed 0xa246ca6 0xa246f14 0xa24705c 0xa24751b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x151be 0x2cea8e9 0x80977b9 0x81685ee 0x8168e55 0x8166af7 0x7b56e2f 0x7b57b24 0x7c6a1c6 0x7b2c212 0x7b8b1ca 0x7c1c577 0x7c1c7f0 0x7c56199 0x7c562f8 0x7c6a329 0x7b52948 0x7de7e59 0x7c77990 0x7db8dca 0x76ae866 0x76bcc3b 0x776718f 0x790351c 0xa25f109 0xa1c54ec 0xa28b9f3 0xa2fa441 0xa2fdd0a 0xa2fe386 0x2967283 0xa91b790 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-24T16:06:20.197 INFO:teuthology.orchestra.run.trial119.stderr:----------------------------------------------------- 2026-02-24T16:06:20.197 INFO:teuthology.orchestra.run.trial119.stderr:Suppressions used: 2026-02-24T16:06:20.197 INFO:teuthology.orchestra.run.trial119.stderr: count bytes template 2026-02-24T16:06:20.197 INFO:teuthology.orchestra.run.trial119.stderr: 1 8 MallocExtension::Register 2026-02-24T16:06:20.198 INFO:teuthology.orchestra.run.trial119.stderr:----------------------------------------------------- 2026-02-24T16:06:20.198 INFO:teuthology.orchestra.run.trial119.stderr: 2026-02-24T16:06:20.236 DEBUG:teuthology.orchestra.run.trial119:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2026-02-24T16:06:20.261 INFO:tasks.ceph:Reading keys from all nodes... 2026-02-24T16:06:20.262 DEBUG:teuthology.orchestra.run.trial119:> set -ex 2026-02-24T16:06:20.262 DEBUG:teuthology.orchestra.run.trial119:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2026-02-24T16:06:20.325 DEBUG:teuthology.orchestra.run.trial119:> set -ex 2026-02-24T16:06:20.325 DEBUG:teuthology.orchestra.run.trial119:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2026-02-24T16:06:20.389 DEBUG:teuthology.orchestra.run.trial119:> set -ex 2026-02-24T16:06:20.389 DEBUG:teuthology.orchestra.run.trial119:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2026-02-24T16:06:20.453 DEBUG:teuthology.orchestra.run.trial119:> set -ex 2026-02-24T16:06:20.453 DEBUG:teuthology.orchestra.run.trial119:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2026-02-24T16:06:20.517 DEBUG:teuthology.orchestra.run.trial119:> set -ex 2026-02-24T16:06:20.517 DEBUG:teuthology.orchestra.run.trial119:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2026-02-24T16:06:20.573 INFO:tasks.ceph:Adding keys to all mons... 2026-02-24T16:06:20.574 DEBUG:teuthology.orchestra.run.trial119:> sudo tee -a /etc/ceph/ceph.keyring 2026-02-24T16:06:20.636 INFO:teuthology.orchestra.run.trial119.stdout:[mgr.x] 2026-02-24T16:06:20.636 INFO:teuthology.orchestra.run.trial119.stdout: key = AQBpzJ1p0U7RMxAAXyxOD2hWOR3yIwCRoyTXaA== 2026-02-24T16:06:20.636 INFO:teuthology.orchestra.run.trial119.stdout:[osd.0] 2026-02-24T16:06:20.636 INFO:teuthology.orchestra.run.trial119.stdout: key = AQBuzJ1pN5tTNRAAxkF8W6QtO+g2dGX/rKl0/Q== 2026-02-24T16:06:20.636 INFO:teuthology.orchestra.run.trial119.stdout:[osd.1] 2026-02-24T16:06:20.636 INFO:teuthology.orchestra.run.trial119.stdout: key = AQB0zJ1prangCRAA9tMFdJVM1F0kDizYsB9asg== 2026-02-24T16:06:20.636 INFO:teuthology.orchestra.run.trial119.stdout:[osd.2] 2026-02-24T16:06:20.636 INFO:teuthology.orchestra.run.trial119.stdout: key = AQB5zJ1pHYPgGxAAIXkpCJ6N7LjJFqZvsIYCtQ== 2026-02-24T16:06:20.636 INFO:teuthology.orchestra.run.trial119.stdout:[client.0] 2026-02-24T16:06:20.637 INFO:teuthology.orchestra.run.trial119.stdout: key = AQBpzJ1p50p9NhAAiPEMoGGeSBiagVS0TWJwVw== 2026-02-24T16:06:20.638 DEBUG:teuthology.orchestra.run.trial119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2026-02-24T16:06:20.725 DEBUG:teuthology.orchestra.run.trial119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-24T16:06:20.773 DEBUG:teuthology.orchestra.run.trial119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-24T16:06:20.820 DEBUG:teuthology.orchestra.run.trial119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-24T16:06:20.869 DEBUG:teuthology.orchestra.run.trial119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-24T16:06:20.918 INFO:tasks.ceph:Running mkfs on mon nodes... 2026-02-24T16:06:20.919 DEBUG:teuthology.orchestra.run.trial119:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2026-02-24T16:06:20.943 DEBUG:teuthology.orchestra.run.trial119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2026-02-24T16:06:21.028 INFO:teuthology.orchestra.run.trial119.stderr:2026-02-24T16:06:21.026+0000 7f72e0c98c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:06:21.028 INFO:teuthology.orchestra.run.trial119.stderr:2026-02-24T16:06:21.026+0000 7f72e0c98c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:06:21.035 DEBUG:teuthology.orchestra.run.trial119:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2026-02-24T16:06:21.061 DEBUG:teuthology.orchestra.run.trial119:> rm -- /home/ubuntu/cephtest/ceph.monmap 2026-02-24T16:06:21.117 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2026-02-24T16:06:21.117 INFO:tasks.ceph.mon.a:Restarting daemon 2026-02-24T16:06:21.117 DEBUG:teuthology.orchestra.run.trial119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2026-02-24T16:06:21.159 INFO:tasks.ceph.mon.a:Started 2026-02-24T16:06:21.159 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2026-02-24T16:06:21.160 INFO:tasks.ceph.mgr.x:Restarting daemon 2026-02-24T16:06:21.160 DEBUG:teuthology.orchestra.run.trial119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2026-02-24T16:06:21.162 INFO:tasks.ceph.mgr.x:Started 2026-02-24T16:06:21.162 DEBUG:tasks.ceph:set 0 configs 2026-02-24T16:06:21.163 DEBUG:teuthology.orchestra.run.trial119:> sudo ceph --cluster ceph config dump 2026-02-24T16:06:21.222 INFO:tasks.ceph.mon.a.trial119.stderr:2026-02-24T16:06:21.219+0000 7fe3f5295c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:06:21.222 INFO:tasks.ceph.mon.a.trial119.stderr:2026-02-24T16:06:21.219+0000 7fe3f5295c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:06:21.222 INFO:tasks.ceph.mon.a.trial119.stderr:2026-02-24T16:06:21.220+0000 7fe3f5295c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:06:21.225 INFO:tasks.ceph.mgr.x.trial119.stderr:2026-02-24T16:06:21.223+0000 7f4f695c6fc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:06:21.238 INFO:teuthology.orchestra.run.trial119.stderr:2026-02-24T16:06:21.234+0000 7f088c435640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:06:21.239 INFO:teuthology.orchestra.run.trial119.stderr:2026-02-24T16:06:21.238+0000 7f088c435640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:06:21.271 INFO:teuthology.orchestra.run.trial119.stdout:WHO MASK LEVEL OPTION VALUE RO 2026-02-24T16:06:21.277 INFO:tasks.ceph:Setting crush tunables to default 2026-02-24T16:06:21.277 DEBUG:teuthology.orchestra.run.trial119:> sudo ceph --cluster ceph osd crush tunables default 2026-02-24T16:06:21.352 INFO:teuthology.orchestra.run.trial119.stderr:2026-02-24T16:06:21.347+0000 7f000628b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:06:21.353 INFO:teuthology.orchestra.run.trial119.stderr:2026-02-24T16:06:21.351+0000 7f000628b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:06:21.386 INFO:teuthology.orchestra.run.trial119.stderr:adjusted tunables profile to default 2026-02-24T16:06:21.393 INFO:tasks.ceph:check_enable_crimson: True 2026-02-24T16:06:21.394 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2026-02-24T16:06:21.394 DEBUG:teuthology.orchestra.run.trial119:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2026-02-24T16:06:21.428 INFO:tasks.ceph.mgr.x.trial119.stderr:2026-02-24T16:06:21.426+0000 7f4f695c6fc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:06:21.429 INFO:tasks.ceph.mgr.x.trial119.stderr:2026-02-24T16:06:21.427+0000 7f4f695c6fc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:06:21.468 INFO:teuthology.orchestra.run.trial119.stderr:2026-02-24T16:06:21.463+0000 7fe2d27b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:06:21.469 INFO:teuthology.orchestra.run.trial119.stderr:2026-02-24T16:06:21.467+0000 7fe2d27b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:06:21.507 INFO:tasks.ceph.mon.a.trial119.stderr:2026-02-24T16:06:21.505+0000 7fe3ec9f1640 -1 WARNING: experimental feature 'crimson' is enabled 2026-02-24T16:06:21.507 INFO:tasks.ceph.mon.a.trial119.stderr:Please be aware that this feature is experimental, untested, 2026-02-24T16:06:21.507 INFO:tasks.ceph.mon.a.trial119.stderr:unsupported, and may result in data corruption, data loss, 2026-02-24T16:06:21.507 INFO:tasks.ceph.mon.a.trial119.stderr:and/or irreparable damage to your cluster. Do not use 2026-02-24T16:06:21.507 INFO:tasks.ceph.mon.a.trial119.stderr:feature with important data. 2026-02-24T16:06:21.507 INFO:tasks.ceph.mon.a.trial119.stderr: 2026-02-24T16:06:22.258 INFO:tasks.ceph.mgr.x.trial119.stderr:/usr/lib64/python3.9/site-packages/scipy/__init__.py:73: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2026-02-24T16:06:22.258 INFO:tasks.ceph.mgr.x.trial119.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2026-02-24T16:06:22.258 INFO:tasks.ceph.mgr.x.trial119.stderr: from numpy import show_config as show_numpy_config 2026-02-24T16:06:22.397 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2026-02-24T16:06:22.397 DEBUG:teuthology.orchestra.run.trial119:> set -ex 2026-02-24T16:06:22.397 DEBUG:teuthology.orchestra.run.trial119:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2026-02-24T16:06:22.462 DEBUG:teuthology.orchestra.run.trial119:> set -ex 2026-02-24T16:06:22.462 DEBUG:teuthology.orchestra.run.trial119:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2026-02-24T16:06:22.527 DEBUG:teuthology.orchestra.run.trial119:> set -ex 2026-02-24T16:06:22.527 DEBUG:teuthology.orchestra.run.trial119:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2026-02-24T16:06:22.591 DEBUG:teuthology.orchestra.run.trial119:> sudo ceph --cluster ceph osd new c3a8cb24-f200-43b1-8e44-76361d6e46d1 0 2026-02-24T16:06:22.704 INFO:teuthology.orchestra.run.trial119.stderr:2026-02-24T16:06:22.700+0000 7f1d77f0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:06:22.705 INFO:teuthology.orchestra.run.trial119.stderr:2026-02-24T16:06:22.704+0000 7f1d77f0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:06:22.743 INFO:teuthology.orchestra.run.trial119.stdout:0 2026-02-24T16:06:22.752 DEBUG:teuthology.orchestra.run.trial119:> sudo ceph --cluster ceph osd new 14bf0912-2f52-462c-9324-d07ba316d88d 1 2026-02-24T16:06:22.826 INFO:teuthology.orchestra.run.trial119.stderr:2026-02-24T16:06:22.821+0000 7fae93cc2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:06:22.827 INFO:teuthology.orchestra.run.trial119.stderr:2026-02-24T16:06:22.826+0000 7fae93cc2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:06:22.863 INFO:teuthology.orchestra.run.trial119.stdout:1 2026-02-24T16:06:22.872 DEBUG:teuthology.orchestra.run.trial119:> sudo ceph --cluster ceph osd new 10e51252-d390-405a-9b5b-7871e0b28d40 2 2026-02-24T16:06:22.946 INFO:teuthology.orchestra.run.trial119.stderr:2026-02-24T16:06:22.942+0000 7fb6e4c46640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:06:22.947 INFO:teuthology.orchestra.run.trial119.stderr:2026-02-24T16:06:22.946+0000 7fb6e4c46640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:06:22.984 INFO:teuthology.orchestra.run.trial119.stdout:2 2026-02-24T16:06:22.993 DEBUG:teuthology.orchestra.run.trial119:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2026-02-24T16:06:23.020 INFO:tasks.ceph.osd.0:Restarting daemon 2026-02-24T16:06:23.020 DEBUG:teuthology.orchestra.run.trial119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2026-02-24T16:06:23.062 INFO:tasks.ceph.osd.0:Started 2026-02-24T16:06:23.062 DEBUG:teuthology.orchestra.run.trial119:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2026-02-24T16:06:23.091 INFO:tasks.ceph.osd.1:Restarting daemon 2026-02-24T16:06:23.091 DEBUG:teuthology.orchestra.run.trial119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2026-02-24T16:06:23.134 INFO:tasks.ceph.osd.1:Started 2026-02-24T16:06:23.134 DEBUG:teuthology.orchestra.run.trial119:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2026-02-24T16:06:23.163 INFO:tasks.ceph.osd.2:Restarting daemon 2026-02-24T16:06:23.163 DEBUG:teuthology.orchestra.run.trial119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2026-02-24T16:06:23.206 INFO:tasks.ceph.osd.2:Started 2026-02-24T16:06:23.206 DEBUG:teuthology.orchestra.run.trial119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-24T16:06:23.281 INFO:teuthology.orchestra.run.trial119.stderr:2026-02-24T16:06:23.278+0000 7ff67a4c2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:06:23.282 INFO:teuthology.orchestra.run.trial119.stderr:2026-02-24T16:06:23.281+0000 7ff67a4c2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:06:23.314 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T16:06:23.314 INFO:teuthology.orchestra.run.trial119.stdout:{"epoch":6,"fsid":"f14cc7f1-24ff-452a-853d-38c3f5542203","created":"2026-02-24T16:06:21.231844+0000","modified":"2026-02-24T16:06:22.983588+0000","last_up_change":"0.000000","last_in_change":"2026-02-24T16:06:22.983588+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"c3a8cb24-f200-43b1-8e44-76361d6e46d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14bf0912-2f52-462c-9324-d07ba316d88d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"10e51252-d390-405a-9b5b-7871e0b28d40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-24T16:06:23.326 INFO:tasks.ceph.ceph_manager.ceph:[] 2026-02-24T16:06:23.326 INFO:tasks.ceph:Waiting for OSDs to come up 2026-02-24T16:06:23.628 DEBUG:teuthology.orchestra.run.trial119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-24T16:06:23.695 INFO:teuthology.misc.health.trial119.stderr:2026-02-24T16:06:23.693+0000 7f56e0e06640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:06:23.696 INFO:teuthology.misc.health.trial119.stderr:2026-02-24T16:06:23.696+0000 7f56e0e06640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:06:23.728 INFO:teuthology.misc.health.trial119.stdout: 2026-02-24T16:06:23.728 INFO:teuthology.misc.health.trial119.stdout:{"epoch":6,"fsid":"f14cc7f1-24ff-452a-853d-38c3f5542203","created":"2026-02-24T16:06:21.231844+0000","modified":"2026-02-24T16:06:22.983588+0000","last_up_change":"0.000000","last_in_change":"2026-02-24T16:06:22.983588+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"c3a8cb24-f200-43b1-8e44-76361d6e46d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14bf0912-2f52-462c-9324-d07ba316d88d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"10e51252-d390-405a-9b5b-7871e0b28d40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-24T16:06:23.735 DEBUG:teuthology.misc:0 of 3 OSDs are up 2026-02-24T16:06:24.480 INFO:tasks.ceph.mgr.x.trial119.stderr:2026-02-24T16:06:24.479+0000 7f4e864e7640 -1 mgr.server handle_report got status from non-daemon mon.a 2026-02-24T16:06:30.037 DEBUG:teuthology.orchestra.run.trial119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-24T16:06:30.125 INFO:teuthology.misc.health.trial119.stderr:2026-02-24T16:06:30.122+0000 7f496330b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:06:30.128 INFO:teuthology.misc.health.trial119.stderr:2026-02-24T16:06:30.127+0000 7f496330b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:06:30.262 INFO:teuthology.misc.health.trial119.stdout: 2026-02-24T16:06:30.262 INFO:teuthology.misc.health.trial119.stdout:{"epoch":7,"fsid":"f14cc7f1-24ff-452a-853d-38c3f5542203","created":"2026-02-24T16:06:21.231844+0000","modified":"2026-02-24T16:06:29.440705+0000","last_up_change":"0.000000","last_in_change":"2026-02-24T16:06:22.983588+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"c3a8cb24-f200-43b1-8e44-76361d6e46d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14bf0912-2f52-462c-9324-d07ba316d88d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"10e51252-d390-405a-9b5b-7871e0b28d40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-24T16:06:30.275 DEBUG:teuthology.misc:0 of 3 OSDs are up 2026-02-24T16:06:36.577 DEBUG:teuthology.orchestra.run.trial119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-24T16:06:36.646 INFO:teuthology.misc.health.trial119.stderr:2026-02-24T16:06:36.643+0000 7f4bdd2cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:06:36.647 INFO:teuthology.misc.health.trial119.stderr:2026-02-24T16:06:36.646+0000 7f4bdd2cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:06:36.756 INFO:teuthology.misc.health.trial119.stdout: 2026-02-24T16:06:36.756 INFO:teuthology.misc.health.trial119.stdout:{"epoch":13,"fsid":"f14cc7f1-24ff-452a-853d-38c3f5542203","created":"2026-02-24T16:06:21.231844+0000","modified":"2026-02-24T16:06:36.456939+0000","last_up_change":"2026-02-24T16:06:32.447676+0000","last_in_change":"2026-02-24T16:06:22.983588+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-24T16:06:33.491070+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c3a8cb24-f200-43b1-8e44-76361d6e46d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.119:6803","nonce":3602667223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.119:6804","nonce":3602667223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.119:6810","nonce":3602667223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.119:6809","nonce":3602667223}]},"public_addr":"10.20.193.119:6803/3602667223","cluster_addr":"10.20.193.119:6804/3602667223","heartbeat_back_addr":"10.20.193.119:6810/3602667223","heartbeat_front_addr":"10.20.193.119:6809/3602667223","state":["exists","up"]},{"osd":1,"uuid":"14bf0912-2f52-462c-9324-d07ba316d88d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.119:6806","nonce":1741939387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.119:6807","nonce":1741939387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.119:6808","nonce":1741939387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.119:6811","nonce":1741939387}]},"public_addr":"10.20.193.119:6806/1741939387","cluster_addr":"10.20.193.119:6807/1741939387","heartbeat_back_addr":"10.20.193.119:6808/1741939387","heartbeat_front_addr":"10.20.193.119:6811/1741939387","state":["exists","up"]},{"osd":2,"uuid":"10e51252-d390-405a-9b5b-7871e0b28d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.119:6805","nonce":3552598003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.119:6802","nonce":3552598003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.119:6813","nonce":3552598003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.119:6812","nonce":3552598003}]},"public_addr":"10.20.193.119:6805/3552598003","cluster_addr":"10.20.193.119:6802/3552598003","heartbeat_back_addr":"10.20.193.119:6813/3552598003","heartbeat_front_addr":"10.20.193.119:6812/3552598003","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-24T16:06:36.763 DEBUG:teuthology.misc:3 of 3 OSDs are up 2026-02-24T16:06:36.764 INFO:tasks.ceph:Creating RBD pool 2026-02-24T16:06:36.764 DEBUG:teuthology.orchestra.run.trial119:> sudo ceph --cluster ceph osd pool create rbd 8 2026-02-24T16:06:36.836 INFO:teuthology.orchestra.run.trial119.stderr:2026-02-24T16:06:36.832+0000 7fe1aad9c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:06:36.837 INFO:teuthology.orchestra.run.trial119.stderr:2026-02-24T16:06:36.836+0000 7fe1aad9c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:06:37.463 INFO:teuthology.orchestra.run.trial119.stderr:pool 'rbd' created 2026-02-24T16:06:37.476 DEBUG:teuthology.orchestra.run.trial119:> rbd --cluster ceph pool init rbd 2026-02-24T16:06:37.522 INFO:teuthology.orchestra.run.trial119.stderr:2026-02-24T16:06:37.520+0000 7f45868fb240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:06:37.523 INFO:teuthology.orchestra.run.trial119.stderr:2026-02-24T16:06:37.520+0000 7f45868fb240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:06:37.524 INFO:teuthology.orchestra.run.trial119.stderr:2026-02-24T16:06:37.522+0000 7f45868fb240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:06:40.502 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2026-02-24T16:06:40.503 DEBUG:teuthology.orchestra.run.trial119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2026-02-24T16:06:40.504 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2026-02-24T16:06:40.581 INFO:teuthology.orchestra.run.trial119.stderr:2026-02-24T16:06:40.577+0000 7f4c03df5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:06:40.583 INFO:teuthology.orchestra.run.trial119.stderr:2026-02-24T16:06:40.581+0000 7f4c03df5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:06:40.695 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T16:06:40.703 INFO:teuthology.orchestra.run.trial119.stdout:[{"version":1,"timestamp":"0.000000","name":"","changes":[]}] 2026-02-24T16:06:40.704 INFO:tasks.ceph_manager:config epoch is 1 2026-02-24T16:06:40.704 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2026-02-24T16:06:40.704 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2026-02-24T16:06:40.704 DEBUG:teuthology.orchestra.run.trial119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2026-02-24T16:06:40.779 INFO:teuthology.orchestra.run.trial119.stderr:2026-02-24T16:06:40.775+0000 7f88e63a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:06:40.780 INFO:teuthology.orchestra.run.trial119.stderr:2026-02-24T16:06:40.778+0000 7f88e63a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:06:40.920 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T16:06:40.928 INFO:teuthology.orchestra.run.trial119.stdout:{"epoch":5,"flags":0,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.119:6800","nonce":3470947457},{"type":"v1","addr":"10.20.193.119:6801","nonce":3470947457}]},"active_addr":"10.20.193.119:6801/3470947457","active_change":"2026-02-24T16:06:23.474244+0000","active_mgr_features":4544132024016699391,"available":true,"standbys":[],"modules":["iostat","nfs"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_pg_upmap_activity":{"name":"update_pg_upmap_activity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Updates pg_upmap activity stats to be used in `balancer status detail`","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often the agent on each host will gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"The first TCP port the agent will try to bind to. Up to 1000 subsequent ports will be attempted if this port cannot be bound","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this option can facilitate debugging daemons that encounter runtime problems.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"How frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"Fraction of total system memory to divide among autotuned daemons. Converged systems hosting both Ceph and compute should set a lower value; systems with more than 10 GiB of memory per OSD may benefit from a slightly larger value so that the abundance of RAM can be fully utilized","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command persistent logging","long_desc":"","tags":[],"see_also":[]},"certificate_automated_rotation_enabled":{"name":"certificate_automated_rotation_enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"This flag controls whether cephadm automatically rotates certificates upon expiration.","long_desc":"","tags":[],"see_also":[]},"certificate_check_debug_mode":{"name":"certificate_check_debug_mode","type":"bool","level":"dev","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"FOR TESTING ONLY: This flag forces the certificate check instead of waiting for certificate_check_period.","long_desc":"","tags":[],"see_also":[]},"certificate_check_period":{"name":"certificate_check_period","type":"int","level":"advanced","flags":0,"default_value":"1","min":"0","max":"30","enum_allowed":[],"desc":"Specifies how often (in days) the certificate should be checked for validity.","long_desc":"","tags":[],"see_also":[]},"certificate_duration_days":{"name":"certificate_duration_days","type":"int","level":"advanced","flags":0,"default_value":"1095","min":"90","max":"3650","enum_allowed":[],"desc":"Specifies the duration of self-signed certificates generated and signed by the cephadm root CA","long_desc":"","tags":[],"see_also":[]},"certificate_renewal_threshold_days":{"name":"certificate_renewal_threshold_days","type":"int","level":"advanced","flags":0,"default_value":"30","min":"10","max":"90","enum_allowed":[],"desc":"Specifies the lead time in days to initiate certificate renewal before expiration.","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently Podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.28.1","min":"","max":"","enum_allowed":[],"desc":"Alertmanager container image","long_desc":"","tags":[],"see_also":[]},"container_image_alloy":{"name":"container_image_alloy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/alloy:latest","min":"","max":"","enum_allowed":[],"desc":"Alloy container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"Elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:12.3.1","min":"","max":"","enum_allowed":[],"desc":"Grafana container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"Haproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.9.1","min":"","max":"","enum_allowed":[],"desc":"Node exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.6","min":"","max":"","enum_allowed":[],"desc":"Nvmeof container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"Oauth2 proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v3.6.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba metrics container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"Snmp gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"quay.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"Location of dashboards to include in Grafana","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"How frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether the hardware monitoring daemon be deployed on every host?","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at the debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"Max number of daemons per service per host: guard against mistakes and runaways","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"Max number of OSDs that will be drained simultaneously when OSDs are removed","long_desc":"","tags":[],"see_also":[]},"max_parallel_osd_upgrades":{"name":"max_parallel_osd_upgrades","type":"int","level":"advanced","flags":0,"default_value":"16","min":"","max":"","enum_allowed":[],"desc":"Maximum number of OSD daemons upgraded in parallel.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"Mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default IP address for RedFish API (OOB management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"Location of alerts to include in Prometheus Alertmanager","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry URL for login purposes. This is not the default registry.","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"Cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Customized SSH config file to distribute to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many sequential SSH connection liveness check failures before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often SSH connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"stray_daemon_check_interval":{"name":"stray_daemon_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"How frequently cephadm should check for the presence of stray daemons","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use the cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest to ensure that all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROM_ALERT_CREDENTIAL_CACHE_TTL":{"name":"PROM_ALERT_CREDENTIAL_CACHE_TTL","type":"int","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_HOSTNAME_PER_DAEMON":{"name":"RGW_HOSTNAME_PER_DAEMON","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crypto_caller":{"name":"crypto_caller","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"healthcheck_history_max_entries":{"name":"healthcheck_history_max_entries","type":"int","level":"advanced","flags":1,"default_value":"1000","min":"","max":"","enum_allowed":[],"desc":"Maximum number of health check history entries to keep","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"prometheus_tls_secret_name":{"name":"prometheus_tls_secret_name","type":"str","level":"advanced","flags":0,"default_value":"rook-ceph-prometheus-server-tls","min":"","max":"","enum_allowed":[],"desc":"name of tls secret in k8s for prometheus","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_cloning":{"name":"pause_cloning","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_purging":{"name":"pause_purging","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous subvolume purge threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"tentacle":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"force_disabled_modules":{},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"10.20.193.119:0","nonce":2738849717}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"10.20.193.119:0","nonce":799713185}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"10.20.193.119:0","nonce":2082394440}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"10.20.193.119:0","nonce":2429041839}]}]} 2026-02-24T16:06:40.932 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2026-02-24T16:06:40.932 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2026-02-24T16:06:40.932 DEBUG:teuthology.orchestra.run.trial119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-24T16:06:41.009 INFO:teuthology.orchestra.run.trial119.stderr:2026-02-24T16:06:41.005+0000 7f008b05a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:06:41.010 INFO:teuthology.orchestra.run.trial119.stderr:2026-02-24T16:06:41.008+0000 7f008b05a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:06:41.123 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T16:06:41.123 INFO:teuthology.orchestra.run.trial119.stdout:{"epoch":17,"fsid":"f14cc7f1-24ff-452a-853d-38c3f5542203","created":"2026-02-24T16:06:21.231844+0000","modified":"2026-02-24T16:06:40.472244+0000","last_up_change":"2026-02-24T16:06:32.447676+0000","last_in_change":"2026-02-24T16:06:22.983588+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-24T16:06:33.491070+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-24T16:06:36.947623+0000","flags":270337,"flags_names":"hashpspool,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c3a8cb24-f200-43b1-8e44-76361d6e46d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.119:6803","nonce":3602667223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.119:6804","nonce":3602667223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.119:6810","nonce":3602667223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.119:6809","nonce":3602667223}]},"public_addr":"10.20.193.119:6803/3602667223","cluster_addr":"10.20.193.119:6804/3602667223","heartbeat_back_addr":"10.20.193.119:6810/3602667223","heartbeat_front_addr":"10.20.193.119:6809/3602667223","state":["exists","up"]},{"osd":1,"uuid":"14bf0912-2f52-462c-9324-d07ba316d88d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.119:6806","nonce":1741939387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.119:6807","nonce":1741939387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.119:6808","nonce":1741939387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.119:6811","nonce":1741939387}]},"public_addr":"10.20.193.119:6806/1741939387","cluster_addr":"10.20.193.119:6807/1741939387","heartbeat_back_addr":"10.20.193.119:6808/1741939387","heartbeat_front_addr":"10.20.193.119:6811/1741939387","state":["exists","up"]},{"osd":2,"uuid":"10e51252-d390-405a-9b5b-7871e0b28d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.119:6805","nonce":3552598003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.119:6802","nonce":3552598003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.119:6813","nonce":3552598003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.119:6812","nonce":3552598003}]},"public_addr":"10.20.193.119:6805/3552598003","cluster_addr":"10.20.193.119:6802/3552598003","heartbeat_back_addr":"10.20.193.119:6813/3552598003","heartbeat_front_addr":"10.20.193.119:6812/3552598003","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-24T16:06:41.133 INFO:tasks.ceph.ceph_manager.ceph:all up! 2026-02-24T16:06:41.133 DEBUG:teuthology.orchestra.run.trial119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-24T16:06:41.209 INFO:teuthology.orchestra.run.trial119.stderr:2026-02-24T16:06:41.205+0000 7ffb0510b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:06:41.210 INFO:teuthology.orchestra.run.trial119.stderr:2026-02-24T16:06:41.209+0000 7ffb0510b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:06:41.325 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T16:06:41.325 INFO:teuthology.orchestra.run.trial119.stdout:{"epoch":17,"fsid":"f14cc7f1-24ff-452a-853d-38c3f5542203","created":"2026-02-24T16:06:21.231844+0000","modified":"2026-02-24T16:06:40.472244+0000","last_up_change":"2026-02-24T16:06:32.447676+0000","last_in_change":"2026-02-24T16:06:22.983588+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-24T16:06:33.491070+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-24T16:06:36.947623+0000","flags":270337,"flags_names":"hashpspool,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c3a8cb24-f200-43b1-8e44-76361d6e46d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.119:6803","nonce":3602667223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.119:6804","nonce":3602667223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.119:6810","nonce":3602667223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.119:6809","nonce":3602667223}]},"public_addr":"10.20.193.119:6803/3602667223","cluster_addr":"10.20.193.119:6804/3602667223","heartbeat_back_addr":"10.20.193.119:6810/3602667223","heartbeat_front_addr":"10.20.193.119:6809/3602667223","state":["exists","up"]},{"osd":1,"uuid":"14bf0912-2f52-462c-9324-d07ba316d88d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.119:6806","nonce":1741939387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.119:6807","nonce":1741939387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.119:6808","nonce":1741939387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.119:6811","nonce":1741939387}]},"public_addr":"10.20.193.119:6806/1741939387","cluster_addr":"10.20.193.119:6807/1741939387","heartbeat_back_addr":"10.20.193.119:6808/1741939387","heartbeat_front_addr":"10.20.193.119:6811/1741939387","state":["exists","up"]},{"osd":2,"uuid":"10e51252-d390-405a-9b5b-7871e0b28d40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.119:6805","nonce":3552598003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.119:6802","nonce":3552598003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.119:6813","nonce":3552598003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.119:6812","nonce":3552598003}]},"public_addr":"10.20.193.119:6805/3552598003","cluster_addr":"10.20.193.119:6802/3552598003","heartbeat_back_addr":"10.20.193.119:6813/3552598003","heartbeat_front_addr":"10.20.193.119:6812/3552598003","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-24T16:06:41.336 DEBUG:teuthology.orchestra.run.trial119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-24T16:06:41.336 DEBUG:teuthology.orchestra.run.trial119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-24T16:06:41.337 DEBUG:teuthology.orchestra.run.trial119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-24T16:06:41.418 INFO:teuthology.orchestra.run.trial119.stderr:2026-02-24T16:06:41.414+0000 7f36b39b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:06:41.419 INFO:teuthology.orchestra.run.trial119.stderr:2026-02-24T16:06:41.414+0000 7f947d4cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:06:41.419 INFO:teuthology.orchestra.run.trial119.stderr:2026-02-24T16:06:41.414+0000 7fcb60b0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:06:41.420 INFO:teuthology.orchestra.run.trial119.stderr:2026-02-24T16:06:41.418+0000 7f36b39b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:06:41.420 INFO:teuthology.orchestra.run.trial119.stderr:2026-02-24T16:06:41.418+0000 7fcb60b0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:06:41.421 INFO:teuthology.orchestra.run.trial119.stderr:2026-02-24T16:06:41.419+0000 7f947d4cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:06:41.434 INFO:teuthology.orchestra.run.trial119.stdout:42949672968 2026-02-24T16:06:41.435 DEBUG:teuthology.orchestra.run.trial119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-24T16:06:41.436 INFO:teuthology.orchestra.run.trial119.stdout:38654705673 2026-02-24T16:06:41.437 DEBUG:teuthology.orchestra.run.trial119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-24T16:06:41.438 INFO:teuthology.orchestra.run.trial119.stdout:42949672968 2026-02-24T16:06:41.438 DEBUG:teuthology.orchestra.run.trial119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-24T16:06:41.555 INFO:teuthology.orchestra.run.trial119.stderr:2026-02-24T16:06:41.551+0000 7fe8e725a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:06:41.556 INFO:teuthology.orchestra.run.trial119.stderr:2026-02-24T16:06:41.551+0000 7fbad5fa4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:06:41.557 INFO:teuthology.orchestra.run.trial119.stderr:2026-02-24T16:06:41.555+0000 7fe8e725a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:06:41.557 INFO:teuthology.orchestra.run.trial119.stderr:2026-02-24T16:06:41.555+0000 7fbad5fa4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:06:41.558 INFO:teuthology.orchestra.run.trial119.stderr:2026-02-24T16:06:41.554+0000 7fd6e555b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:06:41.567 INFO:teuthology.orchestra.run.trial119.stderr:2026-02-24T16:06:41.565+0000 7fd6e555b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:06:41.667 INFO:teuthology.orchestra.run.trial119.stdout:38654705669 2026-02-24T16:06:41.671 INFO:teuthology.orchestra.run.trial119.stdout:42949672964 2026-02-24T16:06:41.677 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705673 got 38654705669 for osd.0 2026-02-24T16:06:41.681 INFO:teuthology.orchestra.run.trial119.stdout:42949672964 2026-02-24T16:06:41.682 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672968 got 42949672964 for osd.1 2026-02-24T16:06:41.692 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672968 got 42949672964 for osd.2 2026-02-24T16:06:42.678 DEBUG:teuthology.orchestra.run.trial119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-24T16:06:42.683 DEBUG:teuthology.orchestra.run.trial119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-24T16:06:42.693 DEBUG:teuthology.orchestra.run.trial119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-24T16:06:42.756 INFO:teuthology.orchestra.run.trial119.stderr:2026-02-24T16:06:42.752+0000 7fe437c8b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:06:42.758 INFO:teuthology.orchestra.run.trial119.stderr:2026-02-24T16:06:42.756+0000 7fe437c8b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:06:42.800 INFO:teuthology.orchestra.run.trial119.stderr:2026-02-24T16:06:42.796+0000 7f490ea8b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:06:42.801 INFO:teuthology.orchestra.run.trial119.stderr:2026-02-24T16:06:42.796+0000 7fc05aecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:06:42.802 INFO:teuthology.orchestra.run.trial119.stderr:2026-02-24T16:06:42.800+0000 7f490ea8b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:06:42.802 INFO:teuthology.orchestra.run.trial119.stderr:2026-02-24T16:06:42.800+0000 7fc05aecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:06:42.870 INFO:teuthology.orchestra.run.trial119.stdout:38654705673 2026-02-24T16:06:42.881 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705673 got 38654705673 for osd.0 2026-02-24T16:06:42.882 DEBUG:teuthology.parallel:result is None 2026-02-24T16:06:42.913 INFO:teuthology.orchestra.run.trial119.stdout:42949672968 2026-02-24T16:06:42.917 INFO:teuthology.orchestra.run.trial119.stdout:42949672968 2026-02-24T16:06:42.924 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672968 got 42949672968 for osd.2 2026-02-24T16:06:42.924 DEBUG:teuthology.parallel:result is None 2026-02-24T16:06:42.928 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672968 got 42949672968 for osd.1 2026-02-24T16:06:42.928 DEBUG:teuthology.parallel:result is None 2026-02-24T16:06:42.928 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-24T16:06:42.929 DEBUG:teuthology.orchestra.run.trial119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-24T16:06:43.043 INFO:teuthology.orchestra.run.trial119.stderr:2026-02-24T16:06:43.039+0000 7fbfd225a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:06:43.045 INFO:teuthology.orchestra.run.trial119.stderr:2026-02-24T16:06:43.043+0000 7fbfd225a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:06:43.152 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T16:06:43.152 INFO:teuthology.orchestra.run.trial119.stderr:dumped all 2026-02-24T16:06:43.161 INFO:teuthology.orchestra.run.trial119.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2026-02-24T16:06:41.478454+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156499968,"kb_used":2492,"kb_used_data":2492,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156497476,"statfs":{"total":160255967232,"available":160253415424,"internally_reserved":0,"allocated":2551808,"data_stored":2551808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.015071"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-24T16:06:40.487211+0000","last_change":"2026-02-24T16:06:40.491240+0000","last_active":"2026-02-24T16:06:40.487211+0000","last_peered":"2026-02-24T16:06:40.487211+0000","last_clean":"2026-02-24T16:06:40.487211+0000","last_became_active":"2026-02-24T16:06:38.509587+0000","last_became_peered":"2026-02-24T16:06:38.509587+0000","last_unstale":"2026-02-24T16:06:40.487211+0000","last_undegraded":"2026-02-24T16:06:40.487211+0000","last_fullsized":"2026-02-24T16:06:40.487211+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T16:06:37.460632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T16:06:37.460632+0000","last_clean_scrub_stamp":"2026-02-24T16:06:37.460632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-24T16:06:40.488818+0000","last_change":"2026-02-24T16:06:40.493819+0000","last_active":"2026-02-24T16:06:40.488818+0000","last_peered":"2026-02-24T16:06:40.488818+0000","last_clean":"2026-02-24T16:06:40.488818+0000","last_became_active":"2026-02-24T16:06:38.507753+0000","last_became_peered":"2026-02-24T16:06:38.507753+0000","last_unstale":"2026-02-24T16:06:40.488818+0000","last_undegraded":"2026-02-24T16:06:40.488818+0000","last_fullsized":"2026-02-24T16:06:40.488818+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T16:06:37.460632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T16:06:37.460632+0000","last_clean_scrub_stamp":"2026-02-24T16:06:37.460632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-24T16:06:40.488938+0000","last_change":"2026-02-24T16:06:40.493510+0000","last_active":"2026-02-24T16:06:40.488938+0000","last_peered":"2026-02-24T16:06:40.488938+0000","last_clean":"2026-02-24T16:06:40.488938+0000","last_became_active":"2026-02-24T16:06:38.512986+0000","last_became_peered":"2026-02-24T16:06:38.512986+0000","last_unstale":"2026-02-24T16:06:40.488938+0000","last_undegraded":"2026-02-24T16:06:40.488938+0000","last_fullsized":"2026-02-24T16:06:40.488938+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T16:06:37.460632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T16:06:37.460632+0000","last_clean_scrub_stamp":"2026-02-24T16:06:37.460632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-24T16:06:40.489050+0000","last_change":"2026-02-24T16:06:40.493620+0000","last_active":"2026-02-24T16:06:40.489050+0000","last_peered":"2026-02-24T16:06:40.489050+0000","last_clean":"2026-02-24T16:06:40.489050+0000","last_became_active":"2026-02-24T16:06:38.514064+0000","last_became_peered":"2026-02-24T16:06:38.514064+0000","last_unstale":"2026-02-24T16:06:40.489050+0000","last_undegraded":"2026-02-24T16:06:40.489050+0000","last_fullsized":"2026-02-24T16:06:40.489050+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T16:06:37.460632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T16:06:37.460632+0000","last_clean_scrub_stamp":"2026-02-24T16:06:37.460632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'1","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-24T16:06:40.489678+0000","last_change":"2026-02-24T16:06:40.489678+0000","last_active":"2026-02-24T16:06:40.489678+0000","last_peered":"2026-02-24T16:06:40.489678+0000","last_clean":"2026-02-24T16:06:40.489678+0000","last_became_active":"2026-02-24T16:06:38.516049+0000","last_became_peered":"2026-02-24T16:06:38.516049+0000","last_unstale":"2026-02-24T16:06:40.489678+0000","last_undegraded":"2026-02-24T16:06:40.489678+0000","last_fullsized":"2026-02-24T16:06:40.489678+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T16:06:37.460632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T16:06:37.460632+0000","last_clean_scrub_stamp":"2026-02-24T16:06:37.460632+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-24T16:06:40.486870+0000","last_change":"2026-02-24T16:06:40.489450+0000","last_active":"2026-02-24T16:06:40.486870+0000","last_peered":"2026-02-24T16:06:40.486870+0000","last_clean":"2026-02-24T16:06:40.486870+0000","last_became_active":"2026-02-24T16:06:38.522954+0000","last_became_peered":"2026-02-24T16:06:38.522954+0000","last_unstale":"2026-02-24T16:06:40.486870+0000","last_undegraded":"2026-02-24T16:06:40.486870+0000","last_fullsized":"2026-02-24T16:06:40.486870+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T16:06:37.460632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T16:06:37.460632+0000","last_clean_scrub_stamp":"2026-02-24T16:06:37.460632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-24T16:06:40.484899+0000","last_change":"2026-02-24T16:06:40.487305+0000","last_active":"2026-02-24T16:06:40.484899+0000","last_peered":"2026-02-24T16:06:40.484899+0000","last_clean":"2026-02-24T16:06:40.484899+0000","last_became_active":"2026-02-24T16:06:38.512368+0000","last_became_peered":"2026-02-24T16:06:38.512368+0000","last_unstale":"2026-02-24T16:06:40.484899+0000","last_undegraded":"2026-02-24T16:06:40.484899+0000","last_fullsized":"2026-02-24T16:06:40.484899+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T16:06:37.460632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T16:06:37.460632+0000","last_clean_scrub_stamp":"2026-02-24T16:06:37.460632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":16,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-24T16:06:40.489159+0000","last_change":"2026-02-24T16:06:40.492153+0000","last_active":"2026-02-24T16:06:40.489159+0000","last_peered":"2026-02-24T16:06:40.489159+0000","last_clean":"2026-02-24T16:06:40.489159+0000","last_became_active":"2026-02-24T16:06:38.508953+0000","last_became_peered":"2026-02-24T16:06:38.508953+0000","last_unstale":"2026-02-24T16:06:40.489159+0000","last_undegraded":"2026-02-24T16:06:40.489159+0000","last_fullsized":"2026-02-24T16:06:40.489159+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T16:06:37.460632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T16:06:37.460632+0000","last_clean_scrub_stamp":"2026-02-24T16:06:37.460632+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":90,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-24T16:06:40.487052+0000","last_change":"2026-02-24T16:06:35.487810+0000","last_active":"2026-02-24T16:06:40.487052+0000","last_peered":"2026-02-24T16:06:40.487052+0000","last_clean":"2026-02-24T16:06:40.487052+0000","last_became_active":"2026-02-24T16:06:35.480818+0000","last_became_peered":"2026-02-24T16:06:35.480818+0000","last_unstale":"2026-02-24T16:06:40.487052+0000","last_undegraded":"2026-02-24T16:06:40.487052+0000","last_fullsized":"2026-02-24T16:06:40.487052+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T16:06:34.449097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T16:06:34.449097+0000","last_clean_scrub_stamp":"2026-02-24T16:06:34.449097+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160255967232,"available":160253366272,"internally_reserved":0,"allocated":2600960,"data_stored":2600960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53418655744,"available":53417480192,"internally_reserved":0,"allocated":1175552,"data_stored":1175552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949672968,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":308,"kb_used_data":308,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52166348,"statfs":{"total":53418655744,"available":53418340352,"internally_reserved":0,"allocated":315392,"data_stored":315392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":10,"seq":42949672968,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1116,"kb_used_data":1116,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165540,"statfs":{"total":53418655744,"available":53417512960,"internally_reserved":0,"allocated":1142784,"data_stored":1142784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705673,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1068,"kb_used_data":1068,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165588,"statfs":{"total":53418655744,"available":53417562112,"internally_reserved":0,"allocated":1093632,"data_stored":1093632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53418655744,"available":53417480192,"internally_reserved":0,"allocated":1175552,"data_stored":1175552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53418655744,"available":53417553920,"internally_reserved":0,"allocated":1101824,"data_stored":1101824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53418655744,"available":53417480192,"internally_reserved":0,"allocated":1175552,"data_stored":1175552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53418655744,"available":53418332160,"internally_reserved":0,"allocated":323584,"data_stored":323584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-24T16:06:43.162 DEBUG:teuthology.orchestra.run.trial119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-24T16:06:43.239 INFO:teuthology.orchestra.run.trial119.stderr:2026-02-24T16:06:43.235+0000 7f287df5c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:06:43.240 INFO:teuthology.orchestra.run.trial119.stderr:2026-02-24T16:06:43.238+0000 7f287df5c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:06:43.346 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T16:06:43.347 INFO:teuthology.orchestra.run.trial119.stderr:dumped all 2026-02-24T16:06:43.356 INFO:teuthology.orchestra.run.trial119.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2026-02-24T16:06:41.478454+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156499968,"kb_used":2492,"kb_used_data":2492,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156497476,"statfs":{"total":160255967232,"available":160253415424,"internally_reserved":0,"allocated":2551808,"data_stored":2551808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.015071"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-24T16:06:40.487211+0000","last_change":"2026-02-24T16:06:40.491240+0000","last_active":"2026-02-24T16:06:40.487211+0000","last_peered":"2026-02-24T16:06:40.487211+0000","last_clean":"2026-02-24T16:06:40.487211+0000","last_became_active":"2026-02-24T16:06:38.509587+0000","last_became_peered":"2026-02-24T16:06:38.509587+0000","last_unstale":"2026-02-24T16:06:40.487211+0000","last_undegraded":"2026-02-24T16:06:40.487211+0000","last_fullsized":"2026-02-24T16:06:40.487211+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T16:06:37.460632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T16:06:37.460632+0000","last_clean_scrub_stamp":"2026-02-24T16:06:37.460632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-24T16:06:40.488818+0000","last_change":"2026-02-24T16:06:40.493819+0000","last_active":"2026-02-24T16:06:40.488818+0000","last_peered":"2026-02-24T16:06:40.488818+0000","last_clean":"2026-02-24T16:06:40.488818+0000","last_became_active":"2026-02-24T16:06:38.507753+0000","last_became_peered":"2026-02-24T16:06:38.507753+0000","last_unstale":"2026-02-24T16:06:40.488818+0000","last_undegraded":"2026-02-24T16:06:40.488818+0000","last_fullsized":"2026-02-24T16:06:40.488818+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T16:06:37.460632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T16:06:37.460632+0000","last_clean_scrub_stamp":"2026-02-24T16:06:37.460632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-24T16:06:40.488938+0000","last_change":"2026-02-24T16:06:40.493510+0000","last_active":"2026-02-24T16:06:40.488938+0000","last_peered":"2026-02-24T16:06:40.488938+0000","last_clean":"2026-02-24T16:06:40.488938+0000","last_became_active":"2026-02-24T16:06:38.512986+0000","last_became_peered":"2026-02-24T16:06:38.512986+0000","last_unstale":"2026-02-24T16:06:40.488938+0000","last_undegraded":"2026-02-24T16:06:40.488938+0000","last_fullsized":"2026-02-24T16:06:40.488938+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T16:06:37.460632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T16:06:37.460632+0000","last_clean_scrub_stamp":"2026-02-24T16:06:37.460632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-24T16:06:40.489050+0000","last_change":"2026-02-24T16:06:40.493620+0000","last_active":"2026-02-24T16:06:40.489050+0000","last_peered":"2026-02-24T16:06:40.489050+0000","last_clean":"2026-02-24T16:06:40.489050+0000","last_became_active":"2026-02-24T16:06:38.514064+0000","last_became_peered":"2026-02-24T16:06:38.514064+0000","last_unstale":"2026-02-24T16:06:40.489050+0000","last_undegraded":"2026-02-24T16:06:40.489050+0000","last_fullsized":"2026-02-24T16:06:40.489050+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T16:06:37.460632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T16:06:37.460632+0000","last_clean_scrub_stamp":"2026-02-24T16:06:37.460632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'1","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-24T16:06:40.489678+0000","last_change":"2026-02-24T16:06:40.489678+0000","last_active":"2026-02-24T16:06:40.489678+0000","last_peered":"2026-02-24T16:06:40.489678+0000","last_clean":"2026-02-24T16:06:40.489678+0000","last_became_active":"2026-02-24T16:06:38.516049+0000","last_became_peered":"2026-02-24T16:06:38.516049+0000","last_unstale":"2026-02-24T16:06:40.489678+0000","last_undegraded":"2026-02-24T16:06:40.489678+0000","last_fullsized":"2026-02-24T16:06:40.489678+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T16:06:37.460632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T16:06:37.460632+0000","last_clean_scrub_stamp":"2026-02-24T16:06:37.460632+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-24T16:06:40.486870+0000","last_change":"2026-02-24T16:06:40.489450+0000","last_active":"2026-02-24T16:06:40.486870+0000","last_peered":"2026-02-24T16:06:40.486870+0000","last_clean":"2026-02-24T16:06:40.486870+0000","last_became_active":"2026-02-24T16:06:38.522954+0000","last_became_peered":"2026-02-24T16:06:38.522954+0000","last_unstale":"2026-02-24T16:06:40.486870+0000","last_undegraded":"2026-02-24T16:06:40.486870+0000","last_fullsized":"2026-02-24T16:06:40.486870+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T16:06:37.460632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T16:06:37.460632+0000","last_clean_scrub_stamp":"2026-02-24T16:06:37.460632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-24T16:06:40.484899+0000","last_change":"2026-02-24T16:06:40.487305+0000","last_active":"2026-02-24T16:06:40.484899+0000","last_peered":"2026-02-24T16:06:40.484899+0000","last_clean":"2026-02-24T16:06:40.484899+0000","last_became_active":"2026-02-24T16:06:38.512368+0000","last_became_peered":"2026-02-24T16:06:38.512368+0000","last_unstale":"2026-02-24T16:06:40.484899+0000","last_undegraded":"2026-02-24T16:06:40.484899+0000","last_fullsized":"2026-02-24T16:06:40.484899+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T16:06:37.460632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T16:06:37.460632+0000","last_clean_scrub_stamp":"2026-02-24T16:06:37.460632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":16,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-24T16:06:40.489159+0000","last_change":"2026-02-24T16:06:40.492153+0000","last_active":"2026-02-24T16:06:40.489159+0000","last_peered":"2026-02-24T16:06:40.489159+0000","last_clean":"2026-02-24T16:06:40.489159+0000","last_became_active":"2026-02-24T16:06:38.508953+0000","last_became_peered":"2026-02-24T16:06:38.508953+0000","last_unstale":"2026-02-24T16:06:40.489159+0000","last_undegraded":"2026-02-24T16:06:40.489159+0000","last_fullsized":"2026-02-24T16:06:40.489159+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T16:06:37.460632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T16:06:37.460632+0000","last_clean_scrub_stamp":"2026-02-24T16:06:37.460632+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":90,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-24T16:06:40.487052+0000","last_change":"2026-02-24T16:06:35.487810+0000","last_active":"2026-02-24T16:06:40.487052+0000","last_peered":"2026-02-24T16:06:40.487052+0000","last_clean":"2026-02-24T16:06:40.487052+0000","last_became_active":"2026-02-24T16:06:35.480818+0000","last_became_peered":"2026-02-24T16:06:35.480818+0000","last_unstale":"2026-02-24T16:06:40.487052+0000","last_undegraded":"2026-02-24T16:06:40.487052+0000","last_fullsized":"2026-02-24T16:06:40.487052+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-24T16:06:34.449097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-24T16:06:34.449097+0000","last_clean_scrub_stamp":"2026-02-24T16:06:34.449097+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160255967232,"available":160253366272,"internally_reserved":0,"allocated":2600960,"data_stored":2600960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53418655744,"available":53417480192,"internally_reserved":0,"allocated":1175552,"data_stored":1175552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949672968,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":308,"kb_used_data":308,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52166348,"statfs":{"total":53418655744,"available":53418340352,"internally_reserved":0,"allocated":315392,"data_stored":315392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":10,"seq":42949672968,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1116,"kb_used_data":1116,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165540,"statfs":{"total":53418655744,"available":53417512960,"internally_reserved":0,"allocated":1142784,"data_stored":1142784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705673,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1068,"kb_used_data":1068,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165588,"statfs":{"total":53418655744,"available":53417562112,"internally_reserved":0,"allocated":1093632,"data_stored":1093632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53418655744,"available":53417480192,"internally_reserved":0,"allocated":1175552,"data_stored":1175552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53418655744,"available":53417553920,"internally_reserved":0,"allocated":1101824,"data_stored":1101824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53418655744,"available":53417480192,"internally_reserved":0,"allocated":1175552,"data_stored":1175552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53418655744,"available":53418332160,"internally_reserved":0,"allocated":323584,"data_stored":323584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-24T16:06:43.357 INFO:tasks.ceph.ceph_manager.ceph:clean! 2026-02-24T16:06:43.357 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2026-02-24T16:06:43.357 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2026-02-24T16:06:43.358 DEBUG:teuthology.orchestra.run.trial119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-24T16:06:43.432 INFO:teuthology.orchestra.run.trial119.stderr:2026-02-24T16:06:43.429+0000 7f479690b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:06:43.433 INFO:teuthology.orchestra.run.trial119.stderr:2026-02-24T16:06:43.431+0000 7f479690b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:06:43.571 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T16:06:43.571 INFO:teuthology.orchestra.run.trial119.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2026-02-24T16:06:43.582 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2026-02-24T16:06:43.583 INFO:teuthology.run_tasks:Running task ssh_keys... 2026-02-24T16:06:43.653 DEBUG:teuthology.orchestra.run.trial119:> echo 'StrictHostKeyChecking no 2026-02-24T16:06:43.654 DEBUG:teuthology.orchestra.run.trial119:> ' > /home/ubuntu/.ssh/config && echo 'UserKnownHostsFile ' /dev/null >> /home/ubuntu/.ssh/config && chmod 600 /home/ubuntu/.ssh/config 2026-02-24T16:06:43.670 INFO:teuthology.task.ssh_keys:generated public key AAAAB3NzaC1yc2EAAAADAQABAAABAQC8oxBdDfTCE0FGqFlOIZiH3in3y4SXKnFNGeysYKrLIIAYwXpD4XNt4Y2nTzsOVxJDbyHY1VCrwFyaEBQ6K5c7tT38rcK7t5XKL3iQ5Yu+U44eAangh1c4lKZf2snsk2BuPbvlgJ4vZa81pDfSMHT+cxon7YXd9kQo+aRhEkFZxusBX5dH6BGwfGqZWPrNHWTYbMG70dyjsxIg8aH8YlqxdnSZWrR5ZdfeIZcvUZazJWUMlLu92TtteB5tXa33tugxaQkhT4Lkqp6mIvnQdO1ONgq3wgKE7XNQDvIL2SES2bOOvj3VUiSXNyfoYzpDLLu0W+917HFoEVuhgXKpLCmJ 2026-02-24T16:06:43.670 INFO:teuthology.task.ssh_keys:pushing keys to trial119.front.sepia.ceph.com for ubuntu 2026-02-24T16:06:43.670 DEBUG:teuthology.orchestra.run.trial119:> rm -f -- /home/ubuntu/.ssh/id_rsa 2026-02-24T16:06:43.726 DEBUG:teuthology.orchestra.run.trial119:> set -ex 2026-02-24T16:06:43.726 DEBUG:teuthology.orchestra.run.trial119:> dd of=/home/ubuntu/.ssh/id_rsa 2026-02-24T16:06:43.726 DEBUG:teuthology.orchestra.run.trial119:> chmod 0500 /home/ubuntu/.ssh/id_rsa 2026-02-24T16:06:43.783 DEBUG:teuthology.orchestra.run.trial119:> rm -f -- /home/ubuntu/.ssh/id_rsa.pub 2026-02-24T16:06:43.839 DEBUG:teuthology.orchestra.run.trial119:> set -ex 2026-02-24T16:06:43.840 DEBUG:teuthology.orchestra.run.trial119:> dd of=/home/ubuntu/.ssh/id_rsa.pub 2026-02-24T16:06:43.896 DEBUG:teuthology.orchestra.run.trial119:> cp -v -a /home/ubuntu/.ssh/authorized_keys /home/ubuntu/.ssh/authorized_keys_2026-02-24_16:06:43:896349 2026-02-24T16:06:43.952 INFO:teuthology.orchestra.run.trial119.stdout:'/home/ubuntu/.ssh/authorized_keys' -> '/home/ubuntu/.ssh/authorized_keys_2026-02-24_16:06:43:896349' 2026-02-24T16:06:43.953 DEBUG:teuthology.orchestra.run.trial119:> set -ex 2026-02-24T16:06:43.953 DEBUG:teuthology.orchestra.run.trial119:> dd of=/home/ubuntu/.ssh/authorized_keys conv=notrunc oflag=append 2026-02-24T16:06:44.009 INFO:teuthology.run_tasks:Running task cbt... 2026-02-24T16:06:44.034 INFO:tasks.cbt:cbt configuration is {'cluster': {'user': 'ubuntu', 'head': 'ubuntu@trial119.front.sepia.ceph.com', 'osds': ['ubuntu@trial119.front.sepia.ceph.com'], 'mons': ['ubuntu@trial119.front.sepia.ceph.com'], 'clients': ['ubuntu@trial119.front.sepia.ceph.com'], 'rgws': {'ubuntu@trial119.front.sepia.ceph.com': None}, 'osds_per_node': 3, 'rebuild_every_test': False, 'use_existing': True, 'is_teuthology': True, 'iterations': 1, 'tmp_dir': '/tmp/cbt', 'pool_profiles': {'replicated': {'pg_size': 256, 'pgp_size': 256, 'replication': 'replicated'}}, 'pid_dir': '/var/run/ceph'}, 'benchmarks': {'radosbench': {'concurrent_ops': 4, 'concurrent_procs': 2, 'op_size': [4194304], 'pool_monitoring_list': ['collectl'], 'pool_profile': 'replicated', 'run_monitoring_list': ['collectl'], 'time': 60, 'write_only': False}}, 'client_endpoints': None, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}} 2026-02-24T16:06:44.035 DEBUG:teuthology.orchestra.run.trial119:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/cbt 2026-02-24T16:06:44.069 DEBUG:teuthology.orchestra.run.trial119:> set -ex 2026-02-24T16:06:44.069 DEBUG:teuthology.orchestra.run.trial119:> dd of=/home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2026-02-24T16:06:44.121 DEBUG:teuthology.orchestra.run.trial119:> git clone --depth 1 -b master https://github.com/ceph/cbt.git /home/ubuntu/cephtest/cbt 2026-02-24T16:06:44.183 INFO:teuthology.orchestra.run.trial119.stderr:Cloning into '/home/ubuntu/cephtest/cbt'... 2026-02-24T16:06:44.458 INFO:tasks.cbt:Installing collectl 2026-02-24T16:06:44.458 DEBUG:teuthology.orchestra.run.trial119:> sudo mkdir collectl && cd collectl && sudo wget https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download -O collectl.tar.gz && sudo tar -xvf collectl.tar.gz && cd collectl-4.3.1 && sudo ./INSTALL 2026-02-24T16:06:44.532 INFO:teuthology.orchestra.run.trial119.stderr:--2026-02-24 16:06:44-- https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download 2026-02-24T16:06:44.585 INFO:teuthology.orchestra.run.trial119.stderr:Resolving sourceforge.net (sourceforge.net)... 104.18.12.149, 104.18.13.149, 2606:4700::6812:c95, ... 2026-02-24T16:06:44.589 INFO:teuthology.orchestra.run.trial119.stderr:Connecting to sourceforge.net (sourceforge.net)|104.18.12.149|:443... connected. 2026-02-24T16:06:44.796 INFO:teuthology.orchestra.run.trial119.stderr:HTTP request sent, awaiting response... 302 Found 2026-02-24T16:06:44.797 INFO:teuthology.orchestra.run.trial119.stderr:Location: https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABpncyUxK5XsVVWVtdgcncAp56XWbF4iOMxgqIg9Uzv96bSZbc-D9FOYfDYCxQvVLbUSjsUVtGp1k-y0Zj6zGBoWaFt9g%3D%3D&use_mirror=gigenet&r= [following] 2026-02-24T16:06:44.797 INFO:teuthology.orchestra.run.trial119.stderr:--2026-02-24 16:06:44-- https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABpncyUxK5XsVVWVtdgcncAp56XWbF4iOMxgqIg9Uzv96bSZbc-D9FOYfDYCxQvVLbUSjsUVtGp1k-y0Zj6zGBoWaFt9g%3D%3D&use_mirror=gigenet&r= 2026-02-24T16:06:44.849 INFO:teuthology.orchestra.run.trial119.stderr:Resolving downloads.sourceforge.net (downloads.sourceforge.net)... 104.18.13.149, 104.18.12.149, 2606:4700::6812:c95, ... 2026-02-24T16:06:44.853 INFO:teuthology.orchestra.run.trial119.stderr:Connecting to downloads.sourceforge.net (downloads.sourceforge.net)|104.18.13.149|:443... connected. 2026-02-24T16:06:45.034 INFO:teuthology.orchestra.run.trial119.stderr:HTTP request sent, awaiting response... 302 Found 2026-02-24T16:06:45.034 INFO:teuthology.orchestra.run.trial119.stderr:Location: https://gigenet.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?viasf=1 [following] 2026-02-24T16:06:45.034 INFO:teuthology.orchestra.run.trial119.stderr:--2026-02-24 16:06:45-- https://gigenet.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?viasf=1 2026-02-24T16:06:45.044 INFO:teuthology.orchestra.run.trial119.stderr:Resolving gigenet.dl.sourceforge.net (gigenet.dl.sourceforge.net)... 69.65.16.141 2026-02-24T16:06:45.063 INFO:teuthology.orchestra.run.trial119.stderr:Connecting to gigenet.dl.sourceforge.net (gigenet.dl.sourceforge.net)|69.65.16.141|:443... connected. 2026-02-24T16:06:50.321 INFO:teuthology.orchestra.run.trial119.stderr:HTTP request sent, awaiting response... 200 OK 2026-02-24T16:06:50.321 INFO:teuthology.orchestra.run.trial119.stderr:Length: 620693 (606K) [application/x-gzip] 2026-02-24T16:06:50.321 INFO:teuthology.orchestra.run.trial119.stderr:Saving to: ‘collectl.tar.gz’ 2026-02-24T16:06:50.321 INFO:teuthology.orchestra.run.trial119.stderr: 2026-02-24T16:06:50.340 INFO:teuthology.orchestra.run.trial119.stderr: 0K .......... .......... .......... .......... .......... 8% 2.49M 0s 2026-02-24T16:06:50.361 INFO:teuthology.orchestra.run.trial119.stderr: 50K .......... .......... .......... .......... .......... 16% 2.54M 0s 2026-02-24T16:06:50.361 INFO:teuthology.orchestra.run.trial119.stderr: 100K .......... .......... .......... .......... .......... 24% 94.0M 0s 2026-02-24T16:06:50.361 INFO:teuthology.orchestra.run.trial119.stderr: 150K .......... .......... .......... .......... .......... 32% 109M 0s 2026-02-24T16:06:50.382 INFO:teuthology.orchestra.run.trial119.stderr: 200K .......... .......... .......... .......... .......... 41% 2.60M 0s 2026-02-24T16:06:50.382 INFO:teuthology.orchestra.run.trial119.stderr: 250K .......... .......... .......... .......... .......... 49% 116M 0s 2026-02-24T16:06:50.382 INFO:teuthology.orchestra.run.trial119.stderr: 300K .......... .......... .......... .......... .......... 57% 117M 0s 2026-02-24T16:06:50.382 INFO:teuthology.orchestra.run.trial119.stderr: 350K .......... .......... .......... .......... .......... 65% 91.1M 0s 2026-02-24T16:06:50.402 INFO:teuthology.orchestra.run.trial119.stderr: 400K .......... .......... .......... .......... .......... 74% 2.68M 0s 2026-02-24T16:06:50.402 INFO:teuthology.orchestra.run.trial119.stderr: 450K .......... .......... .......... .......... .......... 82% 116M 0s 2026-02-24T16:06:50.403 INFO:teuthology.orchestra.run.trial119.stderr: 500K .......... .......... .......... .......... .......... 90% 119M 0s 2026-02-24T16:06:50.403 INFO:teuthology.orchestra.run.trial119.stderr: 550K .......... .......... .......... .......... .......... 98% 104M 0s 2026-02-24T16:06:50.403 INFO:teuthology.orchestra.run.trial119.stderr: 600K ...... 100% 11.4T=0.08s 2026-02-24T16:06:50.403 INFO:teuthology.orchestra.run.trial119.stderr: 2026-02-24T16:06:50.403 INFO:teuthology.orchestra.run.trial119.stderr:2026-02-24 16:06:50 (7.45 MB/s) - ‘collectl.tar.gz’ saved [620693/620693] 2026-02-24T16:06:50.403 INFO:teuthology.orchestra.run.trial119.stderr: 2026-02-24T16:06:50.422 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/ 2026-02-24T16:06:50.422 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/collectl.conf 2026-02-24T16:06:50.422 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/docs/ 2026-02-24T16:06:50.422 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/docs/SlowProc.html 2026-02-24T16:06:50.423 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/docs/BuddyInfo.html 2026-02-24T16:06:50.423 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/docs/WhySummary.html 2026-02-24T16:06:50.423 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/docs/Socket.html 2026-02-24T16:06:50.423 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/docs/colmux.html 2026-02-24T16:06:50.423 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/docs/Tutorial.html 2026-02-24T16:06:50.423 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/docs/Gexpr.html 2026-02-24T16:06:50.423 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/docs/Import.html 2026-02-24T16:06:50.423 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/docs/style.css 2026-02-24T16:06:50.423 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/docs/Ganglia3.jpg 2026-02-24T16:06:50.423 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/docs/Ganglia2.jpg 2026-02-24T16:06:50.423 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/docs/FileNaming.html 2026-02-24T16:06:50.423 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/docs/TheMath.html 2026-02-24T16:06:50.423 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/docs/Environmental.html 2026-02-24T16:06:50.424 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/docs/Process.html 2026-02-24T16:06:50.424 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/docs/OutputFiles.html 2026-02-24T16:06:50.424 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/docs/HiResTime.html 2026-02-24T16:06:50.424 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/docs/OutputFormats.html 2026-02-24T16:06:50.424 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/docs/slub.jpg 2026-02-24T16:06:50.424 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/docs/Lustre.html 2026-02-24T16:06:50.424 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/docs/Documentation.html 2026-02-24T16:06:50.424 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/docs/Inodes.html 2026-02-24T16:06:50.424 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/docs/Graphite.html 2026-02-24T16:06:50.424 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/docs/InputFiles.html 2026-02-24T16:06:50.424 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/docs/Export.html 2026-02-24T16:06:50.424 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/docs/Interrupts.html 2026-02-24T16:06:50.424 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/docs/Startup.html 2026-02-24T16:06:50.425 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/docs/Performance.html 2026-02-24T16:06:50.425 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/docs/Messages.html 2026-02-24T16:06:50.425 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/docs/Logging.html 2026-02-24T16:06:50.425 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/docs/Data-detail.html 2026-02-24T16:06:50.425 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/docs/CPUs.html 2026-02-24T16:06:50.425 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/docs/Matrix.html 2026-02-24T16:06:50.425 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/docs/Data-brief.html 2026-02-24T16:06:50.425 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/docs/SlabInfo.html 2026-02-24T16:06:50.425 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/docs/Misc.html 2026-02-24T16:06:50.425 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/docs/Examples.html 2026-02-24T16:06:50.425 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/docs/OpenStack.html 2026-02-24T16:06:50.425 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/docs/Plotfiles.html 2026-02-24T16:06:50.426 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/docs/ColmuxCPU.jpg 2026-02-24T16:06:50.426 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/docs/Tutorial-Lustre.html 2026-02-24T16:06:50.426 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/docs/Data-verbose.html 2026-02-24T16:06:50.426 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/docs/Data.html 2026-02-24T16:06:50.426 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/docs/Exceptions.html 2026-02-24T16:06:50.426 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/docs/RunningAsAService.html 2026-02-24T16:06:50.426 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/docs/FAQ-collectl.html 2026-02-24T16:06:50.426 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/docs/Ganglia.jpg 2026-02-24T16:06:50.426 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/docs/OperationalModes.html 2026-02-24T16:06:50.426 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/docs/Playback.html 2026-02-24T16:06:50.426 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/docs/Hello.html 2026-02-24T16:06:50.426 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/docs/Sockets.html 2026-02-24T16:06:50.426 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/docs/Network.html 2026-02-24T16:06:50.427 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/docs/Infiniband.html 2026-02-24T16:06:50.427 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/docs/Memory.html 2026-02-24T16:06:50.427 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/docs/ColmuxLustre.jpg 2026-02-24T16:06:50.427 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/docs/Disks.html 2026-02-24T16:06:50.427 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/docs/NfsInfo.html 2026-02-24T16:06:50.427 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/service/ 2026-02-24T16:06:50.427 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/service/collectl.service 2026-02-24T16:06:50.427 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/initd/ 2026-02-24T16:06:50.427 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/initd/collectl-suse 2026-02-24T16:06:50.427 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/initd/collectl-debian 2026-02-24T16:06:50.427 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/initd/collectl-generic 2026-02-24T16:06:50.427 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/initd/collectl 2026-02-24T16:06:50.427 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/misc.ph 2026-02-24T16:06:50.427 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/gexpr.ph 2026-02-24T16:06:50.428 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/statsd.ph 2026-02-24T16:06:50.428 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/envrules.std 2026-02-24T16:06:50.428 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/hello.ph 2026-02-24T16:06:50.428 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/vmsum.ph 2026-02-24T16:06:50.428 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/proctree.ph 2026-02-24T16:06:50.428 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/README-WINDOWS 2026-02-24T16:06:50.428 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/formatit.ph 2026-02-24T16:06:50.428 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/colmux 2026-02-24T16:06:50.428 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/INSTALL 2026-02-24T16:06:50.428 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/graphite.ph 2026-02-24T16:06:50.428 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/RELEASE-collectl 2026-02-24T16:06:50.428 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/vmstat.ph 2026-02-24T16:06:50.428 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/README 2026-02-24T16:06:50.428 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/man1/ 2026-02-24T16:06:50.429 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/man1/collectl.1 2026-02-24T16:06:50.429 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/man1/colmux.1 2026-02-24T16:06:50.429 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/lexpr.ph 2026-02-24T16:06:50.429 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/client.pl 2026-02-24T16:06:50.429 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/collectl 2026-02-24T16:06:50.429 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/COPYING 2026-02-24T16:06:50.429 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/vnet.ph 2026-02-24T16:06:50.429 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/GPL 2026-02-24T16:06:50.429 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/UNINSTALL 2026-02-24T16:06:50.429 INFO:teuthology.orchestra.run.trial119.stdout:collectl-4.3.1/ARTISTIC 2026-02-24T16:06:50.464 INFO:teuthology.orchestra.run.trial119.stderr:./INSTALL: line 90: chkconfig: command not found 2026-02-24T16:06:50.466 DEBUG:teuthology.orchestra.run.trial119:> sudo yum -y install librbd-devel pdsh pdsh-rcmd-ssh perf 2026-02-24T16:06:50.819 INFO:teuthology.orchestra.run.trial119.stdout:Last metadata expiration check: 0:01:03 ago on Tue 24 Feb 2026 04:05:47 PM UTC. 2026-02-24T16:06:50.905 INFO:teuthology.orchestra.run.trial119.stdout:Dependencies resolved. 2026-02-24T16:06:50.906 INFO:teuthology.orchestra.run.trial119.stdout:================================================================================ 2026-02-24T16:06:50.906 INFO:teuthology.orchestra.run.trial119.stdout: Package Arch Version Repository Size 2026-02-24T16:06:50.906 INFO:teuthology.orchestra.run.trial119.stdout:================================================================================ 2026-02-24T16:06:50.906 INFO:teuthology.orchestra.run.trial119.stdout:Installing: 2026-02-24T16:06:50.906 INFO:teuthology.orchestra.run.trial119.stdout: librbd-devel x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 24 k 2026-02-24T16:06:50.906 INFO:teuthology.orchestra.run.trial119.stdout: pdsh x86_64 2.34-7.el9 epel 118 k 2026-02-24T16:06:50.906 INFO:teuthology.orchestra.run.trial119.stdout: pdsh-rcmd-ssh x86_64 2.34-7.el9 epel 13 k 2026-02-24T16:06:50.906 INFO:teuthology.orchestra.run.trial119.stdout: perf x86_64 5.14.0-686.el9 appstream 3.3 M 2026-02-24T16:06:50.906 INFO:teuthology.orchestra.run.trial119.stdout:Installing dependencies: 2026-02-24T16:06:50.907 INFO:teuthology.orchestra.run.trial119.stdout: libradospp-devel x86_64 2:20.3.0-5343.gd26e5c22.el9 ceph 31 k 2026-02-24T16:06:50.907 INFO:teuthology.orchestra.run.trial119.stdout: libtraceevent x86_64 1.8.4-2.el9 baseos 290 k 2026-02-24T16:06:50.907 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T16:06:50.907 INFO:teuthology.orchestra.run.trial119.stdout:Transaction Summary 2026-02-24T16:06:50.907 INFO:teuthology.orchestra.run.trial119.stdout:================================================================================ 2026-02-24T16:06:50.907 INFO:teuthology.orchestra.run.trial119.stdout:Install 6 Packages 2026-02-24T16:06:50.907 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T16:06:50.907 INFO:teuthology.orchestra.run.trial119.stdout:Total download size: 3.8 M 2026-02-24T16:06:50.907 INFO:teuthology.orchestra.run.trial119.stdout:Installed size: 14 M 2026-02-24T16:06:50.907 INFO:teuthology.orchestra.run.trial119.stdout:Downloading Packages: 2026-02-24T16:06:51.893 INFO:teuthology.orchestra.run.trial119.stdout:(1/6): libradospp-devel-20.3.0-5343.gd26e5c22.e 63 kB/s | 31 kB 00:00 2026-02-24T16:06:51.913 INFO:teuthology.orchestra.run.trial119.stdout:(2/6): libtraceevent-1.8.4-2.el9.x86_64.rpm 565 kB/s | 290 kB 00:00 2026-02-24T16:06:51.953 INFO:teuthology.orchestra.run.trial119.stdout:(3/6): pdsh-2.34-7.el9.x86_64.rpm 2.9 MB/s | 118 kB 00:00 2026-02-24T16:06:51.959 INFO:teuthology.orchestra.run.trial119.stdout:(4/6): librbd-devel-20.3.0-5343.gd26e5c22.el9.x 43 kB/s | 24 kB 00:00 2026-02-24T16:06:51.961 INFO:teuthology.orchestra.run.trial119.stdout:(5/6): pdsh-rcmd-ssh-2.34-7.el9.x86_64.rpm 1.6 MB/s | 13 kB 00:00 2026-02-24T16:06:54.588 INFO:teuthology.orchestra.run.trial119.stdout:(6/6): perf-5.14.0-686.el9.x86_64.rpm 1.2 MB/s | 3.3 MB 00:02 2026-02-24T16:06:54.589 INFO:teuthology.orchestra.run.trial119.stdout:-------------------------------------------------------------------------------- 2026-02-24T16:06:54.589 INFO:teuthology.orchestra.run.trial119.stdout:Total 1.0 MB/s | 3.8 MB 00:03 2026-02-24T16:06:54.612 INFO:teuthology.orchestra.run.trial119.stdout:Running transaction check 2026-02-24T16:06:54.622 INFO:teuthology.orchestra.run.trial119.stdout:Transaction check succeeded. 2026-02-24T16:06:54.622 INFO:teuthology.orchestra.run.trial119.stdout:Running transaction test 2026-02-24T16:06:54.679 INFO:teuthology.orchestra.run.trial119.stdout:Transaction test succeeded. 2026-02-24T16:06:54.680 INFO:teuthology.orchestra.run.trial119.stdout:Running transaction 2026-02-24T16:06:54.752 INFO:teuthology.orchestra.run.trial119.stdout: Preparing : 1/1 2026-02-24T16:06:54.758 INFO:teuthology.orchestra.run.trial119.stdout: Installing : pdsh-rcmd-ssh-2.34-7.el9.x86_64 1/6 2026-02-24T16:06:54.797 INFO:teuthology.orchestra.run.trial119.stdout: Installing : pdsh-2.34-7.el9.x86_64 2/6 2026-02-24T16:06:54.802 INFO:teuthology.orchestra.run.trial119.stdout: Installing : libtraceevent-1.8.4-2.el9.x86_64 3/6 2026-02-24T16:06:54.804 INFO:teuthology.orchestra.run.trial119.stdout: Installing : libradospp-devel-2:20.3.0-5343.gd26e5c22.el9.x86_64 4/6 2026-02-24T16:06:54.969 INFO:teuthology.orchestra.run.trial119.stdout: Installing : librbd-devel-2:20.3.0-5343.gd26e5c22.el9.x86_64 5/6 2026-02-24T16:06:54.990 INFO:teuthology.orchestra.run.trial119.stdout: Installing : perf-5.14.0-686.el9.x86_64 6/6 2026-02-24T16:06:55.101 INFO:teuthology.orchestra.run.trial119.stdout: Running scriptlet: perf-5.14.0-686.el9.x86_64 6/6 2026-02-24T16:06:55.101 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : libradospp-devel-2:20.3.0-5343.gd26e5c22.el9.x86_64 1/6 2026-02-24T16:06:55.101 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : librbd-devel-2:20.3.0-5343.gd26e5c22.el9.x86_64 2/6 2026-02-24T16:06:55.101 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : libtraceevent-1.8.4-2.el9.x86_64 3/6 2026-02-24T16:06:55.101 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : perf-5.14.0-686.el9.x86_64 4/6 2026-02-24T16:06:55.101 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : pdsh-2.34-7.el9.x86_64 5/6 2026-02-24T16:06:55.165 INFO:teuthology.orchestra.run.trial119.stdout: Verifying : pdsh-rcmd-ssh-2.34-7.el9.x86_64 6/6 2026-02-24T16:06:55.165 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T16:06:55.165 INFO:teuthology.orchestra.run.trial119.stdout:Installed: 2026-02-24T16:06:55.165 INFO:teuthology.orchestra.run.trial119.stdout: libradospp-devel-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T16:06:55.166 INFO:teuthology.orchestra.run.trial119.stdout: librbd-devel-2:20.3.0-5343.gd26e5c22.el9.x86_64 2026-02-24T16:06:55.166 INFO:teuthology.orchestra.run.trial119.stdout: libtraceevent-1.8.4-2.el9.x86_64 2026-02-24T16:06:55.166 INFO:teuthology.orchestra.run.trial119.stdout: pdsh-2.34-7.el9.x86_64 2026-02-24T16:06:55.166 INFO:teuthology.orchestra.run.trial119.stdout: pdsh-rcmd-ssh-2.34-7.el9.x86_64 2026-02-24T16:06:55.166 INFO:teuthology.orchestra.run.trial119.stdout: perf-5.14.0-686.el9.x86_64 2026-02-24T16:06:55.166 INFO:teuthology.orchestra.run.trial119.stdout: 2026-02-24T16:06:55.166 INFO:teuthology.orchestra.run.trial119.stdout:Complete! 2026-02-24T16:06:55.195 DEBUG:teuthology.orchestra.run.trial119:> python3 -m pip install -r /home/ubuntu/cephtest/cbt/requirements.txt 2026-02-24T16:06:55.333 INFO:teuthology.orchestra.run.trial119.stdout:Defaulting to user installation because normal site-packages is not writeable 2026-02-24T16:06:55.345 INFO:teuthology.orchestra.run.trial119.stdout:Requirement already satisfied: pyyaml in /usr/lib64/python3.9/site-packages (from -r /home/ubuntu/cephtest/cbt/requirements.txt (line 1)) (5.4.1) 2026-02-24T16:06:55.812 INFO:teuthology.orchestra.run.trial119.stdout:Collecting lxml 2026-02-24T16:06:55.846 INFO:teuthology.orchestra.run.trial119.stdout: Downloading lxml-6.0.2-cp39-cp39-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl (5.3 MB) 2026-02-24T16:06:56.218 INFO:teuthology.orchestra.run.trial119.stdout:Collecting matplotlib 2026-02-24T16:06:56.228 INFO:teuthology.orchestra.run.trial119.stdout: Downloading matplotlib-3.9.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (8.3 MB) 2026-02-24T16:06:56.328 INFO:teuthology.orchestra.run.trial119.stdout:Requirement already satisfied: pyparsing>=2.3.1 in /usr/lib/python3.9/site-packages (from matplotlib->-r /home/ubuntu/cephtest/cbt/requirements.txt (line 3)) (2.4.7) 2026-02-24T16:06:56.705 INFO:teuthology.orchestra.run.trial119.stdout:Collecting pillow>=8 2026-02-24T16:06:56.716 INFO:teuthology.orchestra.run.trial119.stdout: Downloading pillow-11.3.0-cp39-cp39-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (6.6 MB) 2026-02-24T16:06:57.140 INFO:teuthology.orchestra.run.trial119.stdout:Collecting fonttools>=4.22.0 2026-02-24T16:06:57.148 INFO:teuthology.orchestra.run.trial119.stdout: Downloading fonttools-4.60.2-cp39-cp39-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (4.8 MB) 2026-02-24T16:06:57.335 INFO:teuthology.orchestra.run.trial119.stdout:Collecting contourpy>=1.0.1 2026-02-24T16:06:57.339 INFO:teuthology.orchestra.run.trial119.stdout: Downloading contourpy-1.3.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (321 kB) 2026-02-24T16:06:57.343 INFO:teuthology.orchestra.run.trial119.stdout:Requirement already satisfied: numpy>=1.23 in /usr/lib64/python3.9/site-packages (from matplotlib->-r /home/ubuntu/cephtest/cbt/requirements.txt (line 3)) (1.23.5) 2026-02-24T16:06:57.476 INFO:teuthology.orchestra.run.trial119.stdout:Collecting kiwisolver>=1.3.1 2026-02-24T16:06:57.487 INFO:teuthology.orchestra.run.trial119.stdout: Downloading kiwisolver-1.4.7-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.6 MB) 2026-02-24T16:06:57.530 INFO:teuthology.orchestra.run.trial119.stdout:Collecting importlib-resources>=3.2.0 2026-02-24T16:06:57.534 INFO:teuthology.orchestra.run.trial119.stdout: Downloading importlib_resources-6.5.2-py3-none-any.whl (37 kB) 2026-02-24T16:06:57.535 INFO:teuthology.orchestra.run.trial119.stdout:Requirement already satisfied: python-dateutil>=2.7 in /usr/lib/python3.9/site-packages (from matplotlib->-r /home/ubuntu/cephtest/cbt/requirements.txt (line 3)) (2.9.0.post0) 2026-02-24T16:06:57.546 INFO:teuthology.orchestra.run.trial119.stdout:Collecting cycler>=0.10 2026-02-24T16:06:57.550 INFO:teuthology.orchestra.run.trial119.stdout: Downloading cycler-0.12.1-py3-none-any.whl (8.3 kB) 2026-02-24T16:06:57.551 INFO:teuthology.orchestra.run.trial119.stdout:Requirement already satisfied: packaging>=20.0 in /usr/lib/python3.9/site-packages (from matplotlib->-r /home/ubuntu/cephtest/cbt/requirements.txt (line 3)) (20.9) 2026-02-24T16:06:57.629 INFO:teuthology.orchestra.run.trial119.stdout:Collecting zipp>=3.1.0 2026-02-24T16:06:57.633 INFO:teuthology.orchestra.run.trial119.stdout: Downloading zipp-3.23.0-py3-none-any.whl (10 kB) 2026-02-24T16:06:57.644 INFO:teuthology.orchestra.run.trial119.stdout:Requirement already satisfied: six>=1.5 in /usr/lib/python3.9/site-packages (from python-dateutil>=2.7->matplotlib->-r /home/ubuntu/cephtest/cbt/requirements.txt (line 3)) (1.15.0) 2026-02-24T16:06:57.756 INFO:teuthology.orchestra.run.trial119.stdout:Installing collected packages: zipp, pillow, kiwisolver, importlib-resources, fonttools, cycler, contourpy, matplotlib, lxml 2026-02-24T16:06:58.886 INFO:teuthology.orchestra.run.trial119.stdout:Successfully installed contourpy-1.3.0 cycler-0.12.1 fonttools-4.60.2 importlib-resources-6.5.2 kiwisolver-1.4.7 lxml-6.0.2 matplotlib-3.9.4 pillow-11.3.0 zipp-3.23.0 2026-02-24T16:06:59.009 INFO:tasks.cbt:benchmark: radosbench 2026-02-24T16:06:59.010 DEBUG:teuthology.orchestra.run.trial119:> sudo /sbin/sysctl -q -w kernel.perf_event_paranoid=0 2026-02-24T16:06:59.037 DEBUG:teuthology.orchestra.run.trial119:> /home/ubuntu/cephtest/cbt/cbt.py -a /home/ubuntu/cephtest/archive/cbt /home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2026-02-24T16:06:59.150 INFO:teuthology.orchestra.run.trial119.stderr:16:06:59 - DEBUG - cbt - Settings.cluster: 2026-02-24T16:06:59.150 INFO:teuthology.orchestra.run.trial119.stderr: {'archive_dir': '/home/ubuntu/cephtest/archive/cbt', 2026-02-24T16:06:59.150 INFO:teuthology.orchestra.run.trial119.stderr: 'clients': ['ubuntu@trial119.front.sepia.ceph.com'], 2026-02-24T16:06:59.151 INFO:teuthology.orchestra.run.trial119.stderr: 'conf_file': '/etc/ceph/ceph.conf', 2026-02-24T16:06:59.151 INFO:teuthology.orchestra.run.trial119.stderr: 'head': 'ubuntu@trial119.front.sepia.ceph.com', 2026-02-24T16:06:59.151 INFO:teuthology.orchestra.run.trial119.stderr: 'is_teuthology': True, 2026-02-24T16:06:59.151 INFO:teuthology.orchestra.run.trial119.stderr: 'iterations': 1, 2026-02-24T16:06:59.151 INFO:teuthology.orchestra.run.trial119.stderr: 'mons': ['ubuntu@trial119.front.sepia.ceph.com'], 2026-02-24T16:06:59.151 INFO:teuthology.orchestra.run.trial119.stderr: 'osds': ['ubuntu@trial119.front.sepia.ceph.com'], 2026-02-24T16:06:59.151 INFO:teuthology.orchestra.run.trial119.stderr: 'osds_per_node': 3, 2026-02-24T16:06:59.151 INFO:teuthology.orchestra.run.trial119.stderr: 'pid_dir': '/var/run/ceph', 2026-02-24T16:06:59.151 INFO:teuthology.orchestra.run.trial119.stderr: 'pool_profiles': {'replicated': {'pg_size': 256, 2026-02-24T16:06:59.151 INFO:teuthology.orchestra.run.trial119.stderr: 'pgp_size': 256, 2026-02-24T16:06:59.151 INFO:teuthology.orchestra.run.trial119.stderr: 'replication': 'replicated'}}, 2026-02-24T16:06:59.151 INFO:teuthology.orchestra.run.trial119.stderr: 'rebuild_every_test': False, 2026-02-24T16:06:59.151 INFO:teuthology.orchestra.run.trial119.stderr: 'rgws': {'ubuntu@trial119.front.sepia.ceph.com': None}, 2026-02-24T16:06:59.152 INFO:teuthology.orchestra.run.trial119.stderr: 'tmp_dir': '/tmp/cbt', 2026-02-24T16:06:59.152 INFO:teuthology.orchestra.run.trial119.stderr: 'use_existing': True, 2026-02-24T16:06:59.152 INFO:teuthology.orchestra.run.trial119.stderr: 'user': 'ubuntu'} 2026-02-24T16:06:59.152 INFO:teuthology.orchestra.run.trial119.stderr:16:06:59 - INFO - cbt - Results dir: /home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d 2026-02-24T16:06:59.152 INFO:teuthology.orchestra.run.trial119.stderr:16:06:59 - INFO - cbt - Pausing for 60s for idle monitoring. 2026-02-24T16:06:59.152 INFO:teuthology.orchestra.run.trial119.stderr:16:06:59 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial119.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/idle_monitoring/collectl 2026-02-24T16:06:59.317 INFO:teuthology.orchestra.run.trial119.stderr:16:06:59 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial119.front.sepia.ceph.com collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/idle_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2026-02-24T16:06:59.319 INFO:teuthology.orchestra.run.trial119.stderr:16:06:59 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial119.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/idle_monitoring/perf 2026-02-24T16:06:59.476 INFO:teuthology.orchestra.run.trial119.stderr:16:06:59 - DEBUG - cbt - PerfMonitoring: remote_node 2026-02-24T16:06:59.476 INFO:teuthology.orchestra.run.trial119.stderr:16:06:59 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial119.front.sepia.ceph.com for pid in `cat /var/run/ceph/crimson-osd.*.pid`; do perf stat -p ${pid} -o /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/idle_monitoring/perf/perf_stat.${pid} & done 2026-02-24T16:07:59.538 INFO:teuthology.orchestra.run.trial119.stderr:16:07:59 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial119.front.sepia.ceph.com pkill -SIGINT -f collectl 2026-02-24T16:07:59.708 INFO:teuthology.orchestra.run.trial119.stderr:16:07:59 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial119.front.sepia.ceph.com sudo pkill -SIGINT -f perf\ 2026-02-24T16:07:59.885 INFO:teuthology.orchestra.run.trial119.stderr:16:07:59 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial119.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/idle_monitoring/perf/perf.data 2026-02-24T16:07:59.887 INFO:teuthology.orchestra.run.trial119.stderr:16:07:59 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial119.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/idle_monitoring/perf/perf_stat.* 2026-02-24T16:07:59.889 INFO:teuthology.orchestra.run.trial119.stderr:16:07:59 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial119.front.sepia.ceph.com sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/* 2026-02-24T16:08:00.061 INFO:teuthology.orchestra.run.trial119.stderr:16:08:00 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=rpdcp -f 10 -R ssh -w ubuntu@trial119.front.sepia.ceph.com -r /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/* /home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d 2026-02-24T16:08:00.219 INFO:teuthology.orchestra.run.trial119.stderr:16:08:00 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial119.front.sepia.ceph.com sudo killall -9 rados 2026-02-24T16:08:00.389 INFO:teuthology.orchestra.run.trial119.stderr:16:08:00 - INFO - cbt - Results dir: /home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d 2026-02-24T16:08:00.389 INFO:teuthology.orchestra.run.trial119.stderr:16:08:00 - INFO - cbt - Skipping existing test in /home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d. 2026-02-24T16:08:00.389 INFO:teuthology.orchestra.run.trial119.stderr:16:08:00 - INFO - cbt - Running benchmark /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004 2026-02-24T16:08:00.389 INFO:teuthology.orchestra.run.trial119.stderr:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d 2026-02-24T16:08:00.390 INFO:teuthology.orchestra.run.trial119.stderr:{'concurrent_ops': 4, 'concurrent_procs': 2, 'pool_profile': 'replicated', 'time': 60, 'write_only': False, 'op_size': 4194304, 'pool_monitoring_list': 'collectl', 'run_monitoring_list': 'collectl', 'benchmark': 'radosbench', 'iteration': 0} == iteration 0 == 2026-02-24T16:08:00.390 INFO:teuthology.orchestra.run.trial119.stderr:16:08:00 - INFO - cbt - Setting OSD Read Ahead to: 0 2026-02-24T16:08:00.390 INFO:teuthology.orchestra.run.trial119.stderr:16:08:00 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial119.front.sepia.ceph.com find /dev/disk/by-partlabel/osd-device-*data -exec readlink {} \; | cut -d"/" -f 3 | sed "s/[0-9]$//" | xargs -I{} sudo sh -c "echo 0 > /sys/block/'{}'/queue/read_ahead_kb" 2026-02-24T16:08:00.390 INFO:teuthology.orchestra.run.trial119.stderr:16:08:00 - DEBUG - cbt - Cleaning existing temporary run directory: /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004 2026-02-24T16:08:00.391 INFO:teuthology.orchestra.run.trial119.stderr:16:08:00 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial119.front.sepia.ceph.com sudo rm -rf /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004 2026-02-24T16:08:00.560 INFO:teuthology.orchestra.run.trial119.stderr:16:08:00 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial119.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/pool_monitoring/collectl 2026-02-24T16:08:00.713 INFO:teuthology.orchestra.run.trial119.stderr:16:08:00 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial119.front.sepia.ceph.com collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/pool_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2026-02-24T16:08:00.714 INFO:teuthology.orchestra.run.trial119.stderr:16:08:00 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial119.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/pool_monitoring/perf 2026-02-24T16:08:00.874 INFO:teuthology.orchestra.run.trial119.stderr:16:08:00 - DEBUG - cbt - PerfMonitoring: remote_node 2026-02-24T16:08:00.874 INFO:teuthology.orchestra.run.trial119.stderr:16:08:00 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial119.front.sepia.ceph.com for pid in `cat /var/run/ceph/crimson-osd.*.pid`; do perf stat -p ${pid} -o /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/pool_monitoring/perf/perf_stat.${pid} & done 2026-02-24T16:08:00.876 INFO:teuthology.orchestra.run.trial119.stderr:16:08:00 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial119.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool delete rados-bench-cbt rados-bench-cbt --yes-i-really-really-mean-it 2026-02-24T16:08:01.219 INFO:teuthology.orchestra.run.trial119.stderr:16:08:01 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial119.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool create rados-bench-cbt 256 256 2026-02-24T16:08:02.540 INFO:teuthology.orchestra.run.trial119.stderr:16:08:02 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial119.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool application enable rados-bench-cbt radosbench 2026-02-24T16:08:03.544 INFO:teuthology.orchestra.run.trial119.stderr:16:08:03 - INFO - cbt - Checking Health after pool creation. 2026-02-24T16:08:08.549 INFO:teuthology.orchestra.run.trial119.stderr:16:08:08 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial119.front.sepia.ceph.com /usr/bin/ceph -c /etc/ceph/ceph.conf health 2026-02-24T16:08:08.904 INFO:teuthology.orchestra.run.trial119.stderr:16:08:08 - INFO - cbt - Final Pool Health Check. 2026-02-24T16:08:13.910 INFO:teuthology.orchestra.run.trial119.stderr:16:08:13 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial119.front.sepia.ceph.com /usr/bin/ceph -c /etc/ceph/ceph.conf health 2026-02-24T16:08:14.257 INFO:teuthology.orchestra.run.trial119.stderr:16:08:14 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial119.front.sepia.ceph.com pkill -SIGINT -f collectl 2026-02-24T16:08:14.426 INFO:teuthology.orchestra.run.trial119.stderr:16:08:14 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial119.front.sepia.ceph.com sudo pkill -SIGINT -f perf\ 2026-02-24T16:08:14.606 INFO:teuthology.orchestra.run.trial119.stderr:16:08:14 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial119.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/pool_monitoring/perf/perf.data 2026-02-24T16:08:14.608 INFO:teuthology.orchestra.run.trial119.stderr:16:08:14 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial119.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/pool_monitoring/perf/perf_stat.* 2026-02-24T16:08:14.610 INFO:teuthology.orchestra.run.trial119.stderr:16:08:14 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial119.front.sepia.ceph.com sync 2026-02-24T16:08:14.776 INFO:teuthology.orchestra.run.trial119.stderr:16:08:14 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial119.front.sepia.ceph.com echo 3 | sudo tee /proc/sys/vm/drop_caches 2026-02-24T16:08:15.773 INFO:teuthology.orchestra.run.trial119.stderr:16:08:15 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial119.front.sepia.ceph.com /usr/bin/rados -c /etc/ceph/ceph.conf -v 2026-02-24T16:08:15.989 INFO:teuthology.orchestra.run.trial119.stderr:16:08:15 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial119.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/write 2026-02-24T16:08:16.147 INFO:teuthology.orchestra.run.trial119.stderr:16:08:16 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial119.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf daemon osd.0 config show > /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/write/ceph_settings.out 2026-02-24T16:08:16.427 INFO:teuthology.orchestra.run.trial119.stderr:16:08:16 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial119.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/write/collectl 2026-02-24T16:08:16.592 INFO:teuthology.orchestra.run.trial119.stderr:16:08:16 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial119.front.sepia.ceph.com collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/write/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2026-02-24T16:08:16.593 INFO:teuthology.orchestra.run.trial119.stderr:16:08:16 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial119.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/write/perf 2026-02-24T16:08:16.750 INFO:teuthology.orchestra.run.trial119.stderr:16:08:16 - DEBUG - cbt - PerfMonitoring: remote_node 2026-02-24T16:08:16.750 INFO:teuthology.orchestra.run.trial119.stderr:16:08:16 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial119.front.sepia.ceph.com for pid in `cat /var/run/ceph/crimson-osd.*.pid`; do perf stat -p ${pid} -o /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/write/perf/perf_stat.${pid} & done 2026-02-24T16:08:16.751 INFO:teuthology.orchestra.run.trial119.stderr:16:08:16 - INFO - cbt - Running radosbench write test. 2026-02-24T16:08:16.752 INFO:teuthology.orchestra.run.trial119.stderr:16:08:16 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial119.front.sepia.ceph.com /usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench -b 4194304 60 write --concurrent-ios 4 --run-name `hostname -f`-0 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/write/objecter.0.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/write/output.0 2026-02-24T16:08:16.753 INFO:teuthology.orchestra.run.trial119.stderr:16:08:16 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial119.front.sepia.ceph.com /usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench -b 4194304 60 write --concurrent-ios 4 --run-name `hostname -f`-1 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/write/objecter.1.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/write/output.1 2026-02-24T16:09:17.982 INFO:teuthology.orchestra.run.trial119.stderr:16:09:17 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial119.front.sepia.ceph.com pkill -SIGINT -f collectl 2026-02-24T16:09:18.150 INFO:teuthology.orchestra.run.trial119.stderr:16:09:18 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial119.front.sepia.ceph.com sudo pkill -SIGINT -f perf\ 2026-02-24T16:09:18.324 INFO:teuthology.orchestra.run.trial119.stderr:16:09:18 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial119.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/write/perf/perf.data 2026-02-24T16:09:18.326 INFO:teuthology.orchestra.run.trial119.stderr:16:09:18 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial119.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/write/perf/perf_stat.* 2026-02-24T16:09:18.327 INFO:teuthology.orchestra.run.trial119.stderr:16:09:18 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial119.front.sepia.ceph.com find /var/run/ceph/ceph-osd*.asok -maxdepth 1 -exec sudo /usr/bin/ceph --admin-daemon {} dump_historic_ops \; > /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/write/historic_ops.out 2026-02-24T16:09:18.684 INFO:teuthology.orchestra.run.trial119.stderr:16:09:18 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial119.front.sepia.ceph.com sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/write/* 2026-02-24T16:09:18.854 INFO:teuthology.orchestra.run.trial119.stderr:16:09:18 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=rpdcp -f 10 -R ssh -w ubuntu@trial119.front.sepia.ceph.com -r /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/write/* /home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d/write 2026-02-24T16:09:19.021 INFO:teuthology.orchestra.run.trial119.stderr:16:09:19 - INFO - cbt - Convert results to json format. 2026-02-24T16:09:19.022 INFO:teuthology.orchestra.run.trial119.stderr:16:09:19 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial119.front.sepia.ceph.com sync 2026-02-24T16:09:19.235 INFO:teuthology.orchestra.run.trial119.stderr:16:09:19 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial119.front.sepia.ceph.com echo 3 | sudo tee /proc/sys/vm/drop_caches 2026-02-24T16:09:19.550 INFO:teuthology.orchestra.run.trial119.stderr:16:09:19 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial119.front.sepia.ceph.com /usr/bin/rados -c /etc/ceph/ceph.conf -v 2026-02-24T16:09:19.765 INFO:teuthology.orchestra.run.trial119.stderr:16:09:19 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial119.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/seq 2026-02-24T16:09:19.929 INFO:teuthology.orchestra.run.trial119.stderr:16:09:19 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial119.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf daemon osd.0 config show > /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/seq/ceph_settings.out 2026-02-24T16:09:20.217 INFO:teuthology.orchestra.run.trial119.stderr:16:09:20 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial119.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/seq/collectl 2026-02-24T16:09:20.380 INFO:teuthology.orchestra.run.trial119.stderr:16:09:20 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial119.front.sepia.ceph.com collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/seq/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2026-02-24T16:09:20.381 INFO:teuthology.orchestra.run.trial119.stderr:16:09:20 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial119.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/seq/perf 2026-02-24T16:09:20.541 INFO:teuthology.orchestra.run.trial119.stderr:16:09:20 - DEBUG - cbt - PerfMonitoring: remote_node 2026-02-24T16:09:20.542 INFO:teuthology.orchestra.run.trial119.stderr:16:09:20 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial119.front.sepia.ceph.com for pid in `cat /var/run/ceph/crimson-osd.*.pid`; do perf stat -p ${pid} -o /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/seq/perf/perf_stat.${pid} & done 2026-02-24T16:09:20.543 INFO:teuthology.orchestra.run.trial119.stderr:16:09:20 - INFO - cbt - Running radosbench seq test. 2026-02-24T16:09:20.543 INFO:teuthology.orchestra.run.trial119.stderr:16:09:20 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial119.front.sepia.ceph.com /usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench 60 seq --concurrent-ios 4 --run-name `hostname -f`-0 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/seq/objecter.0.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/seq/output.0 2026-02-24T16:09:20.544 INFO:teuthology.orchestra.run.trial119.stderr:16:09:20 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial119.front.sepia.ceph.com /usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench 60 seq --concurrent-ios 4 --run-name `hostname -f`-1 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/seq/objecter.1.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/seq/output.1 2026-02-24T16:09:35.766 INFO:teuthology.orchestra.run.trial119.stderr:16:09:35 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial119.front.sepia.ceph.com pkill -SIGINT -f collectl 2026-02-24T16:09:35.932 INFO:teuthology.orchestra.run.trial119.stderr:16:09:35 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial119.front.sepia.ceph.com sudo pkill -SIGINT -f perf\ 2026-02-24T16:09:36.107 INFO:teuthology.orchestra.run.trial119.stderr:16:09:36 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial119.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/seq/perf/perf.data 2026-02-24T16:09:36.108 INFO:teuthology.orchestra.run.trial119.stderr:16:09:36 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial119.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/seq/perf/perf_stat.* 2026-02-24T16:09:36.110 INFO:teuthology.orchestra.run.trial119.stderr:16:09:36 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial119.front.sepia.ceph.com find /var/run/ceph/ceph-osd*.asok -maxdepth 1 -exec sudo /usr/bin/ceph --admin-daemon {} dump_historic_ops \; > /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/seq/historic_ops.out 2026-02-24T16:09:36.467 INFO:teuthology.orchestra.run.trial119.stderr:16:09:36 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial119.front.sepia.ceph.com sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/seq/* 2026-02-24T16:09:36.636 INFO:teuthology.orchestra.run.trial119.stderr:16:09:36 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=rpdcp -f 10 -R ssh -w ubuntu@trial119.front.sepia.ceph.com -r /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/seq/* /home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d/seq 2026-02-24T16:09:36.800 INFO:teuthology.orchestra.run.trial119.stderr:16:09:36 - INFO - cbt - Convert results to json format. 2026-02-24T16:09:36.812 DEBUG:teuthology.run_tasks:Unwinding manager cbt 2026-02-24T16:09:36.824 DEBUG:teuthology.orchestra.run.trial119:> rm --one-file-system -rf -- /home/ubuntu/cephtest/cbt 2026-02-24T16:09:36.850 INFO:tasks.cbt_performance:Collecting CBT performance data config 2026-02-24T16:09:36.850 INFO:tasks.cbt_performance:reading cbt preferences 2026-02-24T16:09:36.850 DEBUG:teuthology.orchestra.run.trial119:> find /home/ubuntu/cephtest -name 'json_output.*' 2026-02-24T16:09:36.907 INFO:teuthology.orchestra.run.trial119.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d/write/json_output.0.trial119.front.sepia.ceph.com 2026-02-24T16:09:36.907 INFO:teuthology.orchestra.run.trial119.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d/write/json_output.1.trial119.front.sepia.ceph.com 2026-02-24T16:09:36.907 INFO:teuthology.orchestra.run.trial119.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d/seq/json_output.0.trial119.front.sepia.ceph.com 2026-02-24T16:09:36.907 INFO:teuthology.orchestra.run.trial119.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d/seq/json_output.1.trial119.front.sepia.ceph.com 2026-02-24T16:09:36.908 DEBUG:teuthology.orchestra.run.trial119:> set -ex 2026-02-24T16:09:36.908 DEBUG:teuthology.orchestra.run.trial119:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d/write/json_output.0.trial119.front.sepia.ceph.com of=/dev/stdout 2026-02-24T16:09:36.963 INFO:tasks.cbt_performance:reading total cpu cycles 2026-02-24T16:09:36.963 DEBUG:teuthology.orchestra.run.trial119:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d/write -name 'perf_stat.*' 2026-02-24T16:09:37.018 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2026-02-24T16:09:37.018 INFO:tasks.cbt_performance:total cpu cycles: 0 2026-02-24T16:09:37.018 DEBUG:teuthology.orchestra.run.trial119:> set -ex 2026-02-24T16:09:37.018 DEBUG:teuthology.orchestra.run.trial119:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d/write/json_output.1.trial119.front.sepia.ceph.com of=/dev/stdout 2026-02-24T16:09:37.074 INFO:tasks.cbt_performance:reading total cpu cycles 2026-02-24T16:09:37.074 DEBUG:teuthology.orchestra.run.trial119:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d/write -name 'perf_stat.*' 2026-02-24T16:09:37.130 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2026-02-24T16:09:37.130 INFO:tasks.cbt_performance:total cpu cycles: 0 2026-02-24T16:09:37.130 DEBUG:teuthology.orchestra.run.trial119:> set -ex 2026-02-24T16:09:37.130 DEBUG:teuthology.orchestra.run.trial119:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d/seq/json_output.0.trial119.front.sepia.ceph.com of=/dev/stdout 2026-02-24T16:09:37.185 INFO:tasks.cbt_performance:reading total cpu cycles 2026-02-24T16:09:37.186 DEBUG:teuthology.orchestra.run.trial119:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d/seq -name 'perf_stat.*' 2026-02-24T16:09:37.242 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2026-02-24T16:09:37.242 INFO:tasks.cbt_performance:total cpu cycles: 0 2026-02-24T16:09:37.242 DEBUG:teuthology.orchestra.run.trial119:> set -ex 2026-02-24T16:09:37.242 DEBUG:teuthology.orchestra.run.trial119:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d/seq/json_output.1.trial119.front.sepia.ceph.com of=/dev/stdout 2026-02-24T16:09:37.297 INFO:tasks.cbt_performance:reading total cpu cycles 2026-02-24T16:09:37.297 DEBUG:teuthology.orchestra.run.trial119:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d/seq -name 'perf_stat.*' 2026-02-24T16:09:37.352 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2026-02-24T16:09:37.352 INFO:tasks.cbt_performance:total cpu cycles: 0 2026-02-24T16:09:37.355 WARNING:tasks.cbt_performance:Could not send performance data to http://mira118.front.sepia.ceph.com:4000/cbt_performance: HTTPConnectionPool(host='mira118.front.sepia.ceph.com', port=4000): Max retries exceeded with url: /cbt_performance (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -2] Name or service not known')) 2026-02-24T16:09:37.355 WARNING:tasks.cbt_performance:Continuing without performance data collection 2026-02-24T16:09:37.355 DEBUG:teuthology.run_tasks:Unwinding manager ssh_keys 2026-02-24T16:09:37.370 INFO:teuthology.task.ssh_keys:Cleaning up SSH keys 2026-02-24T16:09:37.370 INFO:teuthology.task.ssh_keys:cleaning up keys added for testing 2026-02-24T16:09:37.371 INFO:teuthology.task.ssh_keys: cleaning up keys for user ubuntu on trial119.front.sepia.ceph.com 2026-02-24T16:09:37.371 DEBUG:teuthology.orchestra.run.trial119:> rm -- /home/ubuntu/.ssh/id_rsa 2026-02-24T16:09:37.409 DEBUG:teuthology.orchestra.run.trial119:> rm -- /home/ubuntu/.ssh/id_rsa.pub 2026-02-24T16:09:37.464 DEBUG:teuthology.orchestra.run.trial119:> stat -c '"%a"' /home/ubuntu/.ssh/authorized_keys 2026-02-24T16:09:37.521 INFO:teuthology.orchestra.run.trial119.stdout:"600" 2026-02-24T16:09:37.521 DEBUG:teuthology.orchestra.run.trial119:> mv -- /home/ubuntu/.ssh/authorized_keys_2026-02-24_16:06:43:896349 /home/ubuntu/.ssh/authorized_keys 2026-02-24T16:09:37.579 DEBUG:teuthology.orchestra.run.trial119:> chmod 600 /home/ubuntu/.ssh/authorized_keys 2026-02-24T16:09:37.636 DEBUG:teuthology.orchestra.run.trial119:> rm /home/ubuntu/.ssh/config 2026-02-24T16:09:37.692 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2026-02-24T16:09:37.707 DEBUG:teuthology.orchestra.run.trial119:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2026-02-24T16:09:37.807 INFO:teuthology.orchestra.run.trial119.stderr:2026-02-24T16:09:37.802+0000 7fd07a81a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:09:37.808 INFO:teuthology.orchestra.run.trial119.stderr:2026-02-24T16:09:37.807+0000 7fd07a81a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-24T16:09:37.935 INFO:teuthology.misc:Shutting down mds daemons... 2026-02-24T16:09:37.935 INFO:teuthology.misc:Shutting down osd daemons... 2026-02-24T16:09:37.935 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2026-02-24T16:09:37.935 INFO:teuthology.orchestra.run:waiting for 300 2026-02-24T16:09:38.023 INFO:tasks.ceph.osd.0:Stopped 2026-02-24T16:09:38.023 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2026-02-24T16:09:38.023 INFO:teuthology.orchestra.run:waiting for 300 2026-02-24T16:09:38.111 INFO:tasks.ceph.osd.1:Stopped 2026-02-24T16:09:38.112 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2026-02-24T16:09:38.112 INFO:teuthology.orchestra.run:waiting for 300 2026-02-24T16:09:38.192 INFO:tasks.ceph.osd.2:Stopped 2026-02-24T16:09:38.192 INFO:teuthology.misc:Shutting down mgr daemons... 2026-02-24T16:09:38.193 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2026-02-24T16:09:38.193 INFO:teuthology.orchestra.run:waiting for 300 2026-02-24T16:09:38.230 INFO:tasks.ceph.mgr.x:Stopped 2026-02-24T16:09:38.230 INFO:teuthology.misc:Shutting down mon daemons... 2026-02-24T16:09:38.231 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2026-02-24T16:09:38.231 INFO:teuthology.orchestra.run:waiting for 300 2026-02-24T16:09:38.281 INFO:tasks.ceph.mon.a:Stopped 2026-02-24T16:09:38.281 INFO:tasks.ceph:Checking cluster log for badness... 2026-02-24T16:09:38.282 DEBUG:teuthology.orchestra.run.trial119:> sudo grep -E '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | grep -E -v '\(MDS_ALL_DOWN\)' | grep -E -v '\(MDS_UP_LESS_THAN_MAX\)' | grep -E -v '\(POOL_APP_NOT_ENABLED\)' | grep -E -v '\(PG_' | grep -E -v '\(OSD_' | grep -E -v '\(OBJECT_' | grep -E -v 'overall HEALTH' | head -n 1 2026-02-24T16:09:38.347 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@trial119.front.sepia.ceph.com 2026-02-24T16:09:38.348 DEBUG:teuthology.orchestra.run.trial119:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2026-02-24T16:09:38.503 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@trial119.front.sepia.ceph.com 2026-02-24T16:09:38.503 DEBUG:teuthology.orchestra.run.trial119:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2026-02-24T16:09:38.533 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@trial119.front.sepia.ceph.com 2026-02-24T16:09:38.533 DEBUG:teuthology.orchestra.run.trial119:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2026-02-24T16:09:38.650 INFO:tasks.ceph:Archiving mon data... 2026-02-24T16:09:38.651 DEBUG:teuthology.misc:Transferring archived files from trial119:/var/lib/ceph/mon/ceph-a to /home/teuthworker/mnt/teuthology/liucm-2026-02-24_15:28:57-crimson-rados-wip-liucm-shard-redesign-crimson-only-distro-debug-trial/68197/data/mon.a.tgz 2026-02-24T16:09:38.652 DEBUG:teuthology.orchestra.run.trial119:> mktemp 2026-02-24T16:09:38.668 INFO:teuthology.orchestra.run.trial119.stdout:/tmp/tmp.eedzR8yYEC 2026-02-24T16:09:38.669 DEBUG:teuthology.orchestra.run.trial119:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.eedzR8yYEC 2026-02-24T16:09:38.756 DEBUG:teuthology.orchestra.run.trial119:> sudo chmod 0666 /tmp/tmp.eedzR8yYEC 2026-02-24T16:09:38.839 DEBUG:teuthology.orchestra.remote:trial119:/tmp/tmp.eedzR8yYEC is 202KB 2026-02-24T16:09:38.891 DEBUG:teuthology.orchestra.run.trial119:> rm -fr /tmp/tmp.eedzR8yYEC 2026-02-24T16:09:38.907 INFO:tasks.ceph:Cleaning ceph cluster... 2026-02-24T16:09:38.907 DEBUG:teuthology.orchestra.run.trial119:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2026-02-24T16:09:39.027 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2026-02-24T16:09:39.028 INFO:tasks.ceph:Archiving crash dumps... 2026-02-24T16:09:39.030 DEBUG:teuthology.misc:Transferring archived files from trial119:/var/lib/ceph/crash to /home/teuthworker/mnt/teuthology/liucm-2026-02-24_15:28:57-crimson-rados-wip-liucm-shard-redesign-crimson-only-distro-debug-trial/68197/remote/trial119/crash 2026-02-24T16:09:39.031 DEBUG:teuthology.orchestra.run.trial119:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2026-02-24T16:09:39.092 INFO:tasks.ceph:Compressing logs... 2026-02-24T16:09:39.092 DEBUG:teuthology.orchestra.run.trial119:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2026-02-24T16:09:39.156 INFO:teuthology.orchestra.run.trial119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2026-02-24T16:09:39.169 INFO:teuthology.orchestra.run.trial119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2026-02-24T16:09:39.169 INFO:teuthology.orchestra.run.trial119.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41766.log 2026-02-24T16:09:39.169 INFO:teuthology.orchestra.run.trial119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40501.log 2026-02-24T16:09:39.170 INFO:teuthology.orchestra.run.trial119.stderr:/var/log/ceph/ceph-mon.a.log: /var/log/ceph/ceph-client.admin.41766.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41420.log 2026-02-24T16:09:39.170 INFO:teuthology.orchestra.run.trial119.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41766.log.gz 2026-02-24T16:09:39.170 INFO:teuthology.orchestra.run.trial119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2026-02-24T16:09:39.170 INFO:teuthology.orchestra.run.trial119.stderr:/var/log/ceph/ceph-client.admin.40501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41469.log 2026-02-24T16:09:39.170 INFO:teuthology.orchestra.run.trial119.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.40501.log.gz 2026-02-24T16:09:39.170 INFO:teuthology.orchestra.run.trial119.stderr:/var/log/ceph/ceph-client.admin.41420.log: 91.1% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2026-02-24T16:09:39.170 INFO:teuthology.orchestra.run.trial119.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44882.log 2026-02-24T16:09:39.170 INFO:teuthology.orchestra.run.trial119.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41420.log.gz 2026-02-24T16:09:39.170 INFO:teuthology.orchestra.run.trial119.stderr:/var/log/ceph/ceph-client.admin.41469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46160.log 2026-02-24T16:09:39.170 INFO:teuthology.orchestra.run.trial119.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41469.log.gz 2026-02-24T16:09:39.171 INFO:teuthology.orchestra.run.trial119.stderr:/var/log/ceph/ceph-client.admin.44882.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41211.log 2026-02-24T16:09:39.171 INFO:teuthology.orchestra.run.trial119.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.44882.log.gz 2026-02-24T16:09:39.171 INFO:teuthology.orchestra.run.trial119.stderr:/var/log/ceph/ceph-client.admin.46160.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41059.log 66.4% 2026-02-24T16:09:39.171 INFO:teuthology.orchestra.run.trial119.stderr: -- replaced with /var/log/ceph/ceph-client.admin.46160.log.gz 2026-02-24T16:09:39.171 INFO:teuthology.orchestra.run.trial119.stderr:/var/log/ceph/ceph-client.admin.41211.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44792.log 2026-02-24T16:09:39.171 INFO:teuthology.orchestra.run.trial119.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41211.log.gz 2026-02-24T16:09:39.171 INFO:teuthology.orchestra.run.trial119.stderr:/var/log/ceph/ceph-client.admin.41059.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41768.log 2026-02-24T16:09:39.171 INFO:teuthology.orchestra.run.trial119.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41059.log.gz 2026-02-24T16:09:39.171 INFO:teuthology.orchestra.run.trial119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42008.log 2026-02-24T16:09:39.171 INFO:teuthology.orchestra.run.trial119.stderr:/var/log/ceph/ceph-client.admin.44792.log: /var/log/ceph/ceph-client.admin.41768.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44792.log.gz 2026-02-24T16:09:39.171 INFO:teuthology.orchestra.run.trial119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44850.log 2026-02-24T16:09:39.171 INFO:teuthology.orchestra.run.trial119.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41768.log.gz 2026-02-24T16:09:39.171 INFO:teuthology.orchestra.run.trial119.stderr:/var/log/ceph/ceph-client.admin.42008.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41620.log 2026-02-24T16:09:39.172 INFO:teuthology.orchestra.run.trial119.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42008.log.gz 2026-02-24T16:09:39.172 INFO:teuthology.orchestra.run.trial119.stderr:/var/log/ceph/ceph-client.admin.44850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40951.log 2026-02-24T16:09:39.172 INFO:teuthology.orchestra.run.trial119.stderr:/var/log/ceph/ceph-client.admin.41620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41915.log 2026-02-24T16:09:39.172 INFO:teuthology.orchestra.run.trial119.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41620.log.gz 2026-02-24T16:09:39.172 INFO:teuthology.orchestra.run.trial119.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44850.log.gz 2026-02-24T16:09:39.172 INFO:teuthology.orchestra.run.trial119.stderr:/var/log/ceph/ceph-client.admin.40951.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40685.log 2026-02-24T16:09:39.172 INFO:teuthology.orchestra.run.trial119.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40951.log.gz 2026-02-24T16:09:39.172 INFO:teuthology.orchestra.run.trial119.stderr:/var/log/ceph/ceph-client.admin.41915.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44936.log 2026-02-24T16:09:39.172 INFO:teuthology.orchestra.run.trial119.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41915.log.gz 2026-02-24T16:09:39.172 INFO:teuthology.orchestra.run.trial119.stderr:/var/log/ceph/ceph-client.admin.40685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41371.log 2026-02-24T16:09:39.172 INFO:teuthology.orchestra.run.trial119.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.40685.log.gz 2026-02-24T16:09:39.172 INFO:teuthology.orchestra.run.trial119.stderr:/var/log/ceph/ceph-client.admin.44936.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40638.log 2026-02-24T16:09:39.172 INFO:teuthology.orchestra.run.trial119.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.44936.log.gz 2026-02-24T16:09:39.173 INFO:teuthology.orchestra.run.trial119.stderr:/var/log/ceph/ceph-client.admin.41371.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41141.log 2026-02-24T16:09:39.173 INFO:teuthology.orchestra.run.trial119.stderr:/var/log/ceph/ceph-client.admin.40638.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41371.log.gz 2026-02-24T16:09:39.173 INFO:teuthology.orchestra.run.trial119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41916.log 2026-02-24T16:09:39.173 INFO:teuthology.orchestra.run.trial119.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.40638.log.gz 2026-02-24T16:09:39.173 INFO:teuthology.orchestra.run.trial119.stderr:/var/log/ceph/ceph-client.admin.41141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41861.log 2026-02-24T16:09:39.173 INFO:teuthology.orchestra.run.trial119.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41141.log.gz 2026-02-24T16:09:39.173 INFO:teuthology.orchestra.run.trial119.stderr:/var/log/ceph/ceph-client.admin.41916.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41281.log 2026-02-24T16:09:39.173 INFO:teuthology.orchestra.run.trial119.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41916.log.gz 2026-02-24T16:09:39.173 INFO:teuthology.orchestra.run.trial119.stderr:/var/log/ceph/ceph-client.admin.41861.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40732.log 2026-02-24T16:09:39.173 INFO:teuthology.orchestra.run.trial119.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41861.log.gz 2026-02-24T16:09:39.173 INFO:teuthology.orchestra.run.trial119.stderr:/var/log/ceph/ceph-client.admin.41281.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2026-02-24T16:09:39.173 INFO:teuthology.orchestra.run.trial119.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41281.log.gz 2026-02-24T16:09:39.173 INFO:teuthology.orchestra.run.trial119.stderr:/var/log/ceph/ceph-client.admin.40732.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42106.log 2026-02-24T16:09:39.174 INFO:teuthology.orchestra.run.trial119.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.40732.log.gz 2026-02-24T16:09:39.174 INFO:teuthology.orchestra.run.trial119.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2026-02-24T16:09:39.174 INFO:teuthology.orchestra.run.trial119.stderr:/var/log/ceph/ceph-client.admin.42106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41306.log 2026-02-24T16:09:39.174 INFO:teuthology.orchestra.run.trial119.stderr: 87.7% -- replaced with /var/log/ceph/ceph.audit.log.gz 2026-02-24T16:09:39.174 INFO:teuthology.orchestra.run.trial119.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42106.log.gz 2026-02-24T16:09:39.174 INFO:teuthology.orchestra.run.trial119.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41619.log 2026-02-24T16:09:39.174 INFO:teuthology.orchestra.run.trial119.stderr:/var/log/ceph/ceph-client.admin.41306.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.39068.log 2026-02-24T16:09:39.174 INFO:teuthology.orchestra.run.trial119.stderr: 66.7% -- replaced with /var/log/ceph/ceph-client.admin.41306.log.gz 2026-02-24T16:09:39.174 INFO:teuthology.orchestra.run.trial119.stderr:/var/log/ceph/ceph-client.admin.41619.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42057.log 2026-02-24T16:09:39.174 INFO:teuthology.orchestra.run.trial119.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41619.log.gz 2026-02-24T16:09:39.174 INFO:teuthology.orchestra.run.trial119.stderr:/var/log/ceph/ceph.tmp-client.admin.39068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41767.log 2026-02-24T16:09:39.174 INFO:teuthology.orchestra.run.trial119.stderr: 68.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.39068.log.gz 2026-02-24T16:09:39.174 INFO:teuthology.orchestra.run.trial119.stderr:/var/log/ceph/ceph-client.admin.42057.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44736.log 2026-02-24T16:09:39.175 INFO:teuthology.orchestra.run.trial119.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42057.log.gz 2026-02-24T16:09:39.175 INFO:teuthology.orchestra.run.trial119.stderr:/var/log/ceph/ceph-client.admin.41767.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41621.log 2026-02-24T16:09:39.175 INFO:teuthology.orchestra.run.trial119.stderr:/var/log/ceph/ceph-client.admin.44736.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41767.log.gz 2026-02-24T16:09:39.175 INFO:teuthology.orchestra.run.trial119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2026-02-24T16:09:39.175 INFO:teuthology.orchestra.run.trial119.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44736.log.gz 2026-02-24T16:09:39.175 INFO:teuthology.orchestra.run.trial119.stderr:/var/log/ceph/ceph-client.admin.41621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41518.log 2026-02-24T16:09:39.175 INFO:teuthology.orchestra.run.trial119.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41621.log.gz 2026-02-24T16:09:39.175 INFO:teuthology.orchestra.run.trial119.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40454.log 2026-02-24T16:09:39.175 INFO:teuthology.orchestra.run.trial119.stderr:/var/log/ceph/ceph-client.admin.41518.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2026-02-24T16:09:39.175 INFO:teuthology.orchestra.run.trial119.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41518.log.gz 2026-02-24T16:09:39.175 INFO:teuthology.orchestra.run.trial119.stderr:/var/log/ceph/ceph-client.admin.40454.log: gzip -5 50.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.40454.log.gz -- 2026-02-24T16:09:39.176 INFO:teuthology.orchestra.run.trial119.stderr: /var/log/ceph/ceph-client.admin.46837.log 2026-02-24T16:09:39.176 INFO:teuthology.orchestra.run.trial119.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45490.log 2026-02-24T16:09:39.176 INFO:teuthology.orchestra.run.trial119.stderr:/var/log/ceph/ceph-client.admin.46837.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.46837.log.gz 2026-02-24T16:09:39.176 INFO:teuthology.orchestra.run.trial119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40359.log 2026-02-24T16:09:39.176 INFO:teuthology.orchestra.run.trial119.stderr: 85.1% -- replaced with /var/log/ceph/ceph.log.gz 2026-02-24T16:09:39.176 INFO:teuthology.orchestra.run.trial119.stderr:/var/log/ceph/ceph-client.admin.45490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46158.log 2026-02-24T16:09:39.176 INFO:teuthology.orchestra.run.trial119.stderr: 66.4% -- replaced with /var/log/ceph/ceph-client.admin.45490.log.gz 2026-02-24T16:09:39.176 INFO:teuthology.orchestra.run.trial119.stderr:/var/log/ceph/ceph-client.admin.40359.log: gzip -5 --verbose 50.0% -- -- replaced with /var/log/ceph/ceph-client.admin.40359.log.gz /var/log/ceph/ceph-client.admin.45491.log 2026-02-24T16:09:39.176 INFO:teuthology.orchestra.run.trial119.stderr: 2026-02-24T16:09:39.176 INFO:teuthology.orchestra.run.trial119.stderr:/var/log/ceph/ceph-client.admin.46158.log: 65.9% -- replaced with /var/log/ceph/ceph-client.admin.46158.log.gz 2026-02-24T16:09:39.177 INFO:teuthology.orchestra.run.trial119.stderr:/var/log/ceph/ceph-client.admin.45491.log: 66.7% -- replaced with /var/log/ceph/ceph-client.admin.45491.log.gz 2026-02-24T16:09:39.177 INFO:teuthology.orchestra.run.trial119.stderr: 92.0% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2026-02-24T16:09:41.324 INFO:teuthology.orchestra.run.trial119.stderr: 92.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2026-02-24T16:09:41.429 INFO:teuthology.orchestra.run.trial119.stderr: 92.7% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2026-02-24T16:09:41.435 INFO:teuthology.orchestra.run.trial119.stderr: 92.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2026-02-24T16:09:41.438 INFO:teuthology.orchestra.run.trial119.stderr: 2026-02-24T16:09:41.438 INFO:teuthology.orchestra.run.trial119.stderr:real 0m2.289s 2026-02-24T16:09:41.438 INFO:teuthology.orchestra.run.trial119.stderr:user 0m6.249s 2026-02-24T16:09:41.438 INFO:teuthology.orchestra.run.trial119.stderr:sys 0m0.487s 2026-02-24T16:09:41.438 INFO:tasks.ceph:Archiving logs... 2026-02-24T16:09:41.438 DEBUG:teuthology.misc:Transferring archived files from trial119:/var/log/ceph to /home/teuthworker/mnt/teuthology/liucm-2026-02-24_15:28:57-crimson-rados-wip-liucm-shard-redesign-crimson-only-distro-debug-trial/68197/remote/trial119/log 2026-02-24T16:09:41.440 DEBUG:teuthology.orchestra.run.trial119:> sudo tar c -f - -C /var/log/ceph -- . 2026-02-24T16:09:42.191 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-24T16:09:42.260 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-24T16:09:42.270 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2026-02-24T16:09:42.270 DEBUG:teuthology.orchestra.run.trial119:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-24T16:09:42.296 DEBUG:teuthology.run_tasks:Unwinding manager clock 2026-02-24T16:09:42.307 INFO:teuthology.task.clock:Checking final clock skew... 2026-02-24T16:09:42.307 DEBUG:teuthology.orchestra.run.trial119:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-24T16:09:42.351 INFO:teuthology.orchestra.run.trial119.stderr:bash: line 1: ntpq: command not found 2026-02-24T16:09:42.377 INFO:teuthology.orchestra.run.trial119.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-24T16:09:42.378 INFO:teuthology.orchestra.run.trial119.stdout:=============================================================================== 2026-02-24T16:09:42.378 INFO:teuthology.orchestra.run.trial119.stdout:^- 23.186.168.133 2 7 355 120 +224us[ +224us] +/- 55ms 2026-02-24T16:09:42.378 INFO:teuthology.orchestra.run.trial119.stdout:^- 170-187-147-56.ip.linode> 2 7 377 62 -6195us[-6195us] +/- 78ms 2026-02-24T16:09:42.378 INFO:teuthology.orchestra.run.trial119.stdout:^- owners.kjsl.com 2 8 377 126 +2301us[+2400us] +/- 39ms 2026-02-24T16:09:42.378 INFO:teuthology.orchestra.run.trial119.stdout:^* time.cloudflare.com 3 7 377 124 -135us[ -35us] +/- 6878us 2026-02-24T16:09:42.380 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2026-02-24T16:09:42.394 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2026-02-24T16:09:42.395 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2026-02-24T16:09:42.408 DEBUG:teuthology.orchestra.run.trial119:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2026-02-24T16:09:42.481 DEBUG:teuthology.orchestra.run.trial119:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-24T16:09:42.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-24T16:09:42.508 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2026-02-24T16:09:42.525 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2026-02-24T16:09:42.542 INFO:teuthology.task.internal:Duration was 1902.337853 seconds 2026-02-24T16:09:42.542 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2026-02-24T16:09:42.558 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2026-02-24T16:09:42.558 DEBUG:teuthology.orchestra.run.trial119:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-24T16:09:42.597 INFO:teuthology.orchestra.run.trial119.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-24T16:09:42.944 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2026-02-24T16:09:42.944 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial119.front.sepia.ceph.com 2026-02-24T16:09:42.945 DEBUG:teuthology.orchestra.run.trial119:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-24T16:09:43.003 INFO:teuthology.task.internal.syslog:Gathering journactl... 2026-02-24T16:09:43.004 DEBUG:teuthology.orchestra.run.trial119:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-24T16:09:43.109 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2026-02-24T16:09:43.109 DEBUG:teuthology.orchestra.run.trial119:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-24T16:09:43.133 INFO:teuthology.orchestra.run.trial119.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-24T16:09:43.134 INFO:teuthology.orchestra.run.trial119.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-24T16:09:43.134 INFO:teuthology.orchestra.run.trial119.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-24T16:09:43.134 INFO:teuthology.orchestra.run.trial119.stderr: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-02-24T16:09:43.134 INFO:teuthology.orchestra.run.trial119.stderr:/home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-24T16:09:43.144 INFO:teuthology.orchestra.run.trial119.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: 82.9% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-24T16:09:43.146 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2026-02-24T16:09:43.163 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2026-02-24T16:09:43.164 DEBUG:teuthology.orchestra.run.trial119:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-24T16:09:43.210 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2026-02-24T16:09:43.228 DEBUG:teuthology.orchestra.run.trial119:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-24T16:09:43.274 INFO:teuthology.orchestra.run.trial119.stdout:kernel.core_pattern = core 2026-02-24T16:09:43.288 DEBUG:teuthology.orchestra.run.trial119:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-24T16:09:43.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-24T16:09:43.345 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2026-02-24T16:09:43.362 INFO:teuthology.task.internal:Transferring archived files... 2026-02-24T16:09:43.364 DEBUG:teuthology.misc:Transferring archived files from trial119:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/liucm-2026-02-24_15:28:57-crimson-rados-wip-liucm-shard-redesign-crimson-only-distro-debug-trial/68197/remote/trial119 2026-02-24T16:09:43.364 DEBUG:teuthology.orchestra.run.trial119:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-24T16:09:43.461 INFO:teuthology.task.internal:Removing archive directory... 2026-02-24T16:09:43.461 DEBUG:teuthology.orchestra.run.trial119:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-24T16:09:43.481 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2026-02-24T16:09:43.498 INFO:teuthology.task.internal:Not uploading archives. 2026-02-24T16:09:43.499 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2026-02-24T16:09:43.515 INFO:teuthology.task.internal:Tidying up after the test... 2026-02-24T16:09:43.516 DEBUG:teuthology.orchestra.run.trial119:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-24T16:09:43.537 INFO:teuthology.orchestra.run.trial119.stdout: 920834 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 24 16:09 /home/ubuntu/cephtest 2026-02-24T16:09:43.539 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2026-02-24T16:09:43.557 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2026-02-24T16:09:43.595 INFO:teuthology.run:Summary data: description: crimson-rados/perf/{0-crimson_install clusters/{crimson-fixed-1} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/segmented$/{crimson_seastore_segmented} perf_ignorelist settings/optimized workloads/radosbench_4M_seq_read} duration: 1902.337852716446 flavor: debug owner: scheduled_liucm@soko04.front.sepia.ceph.com success: true 2026-02-24T16:09:43.595 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-24T16:09:43.722 INFO:teuthology.run:pass