2026-02-12T12:03:40.106 INFO:root:teuthology version: 0.0.1.dev29+gc433f1062 2026-02-12T12:03:40.123 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-12T12:03:40.203 INFO:teuthology.run:Config: archive_path: /home/teuthworker/mnt/teuthology/moagrawa-2026-02-10_06:35:43-crimson-rados-main-distro-debug-trial/42493 branch: main description: crimson-rados/perf/{0-crimson_install clusters/{crimson-fixed-1} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/rbm$/{crimson_seastore_rbm_2q} perf_ignorelist settings/optimized workloads/radosbench_4M_rand_read} email: null first_in_suite: false flavor: debug job_id: '42493' kernel: branch: distro kdb: 1 sha1: distro last_in_suite: false machine_type: trial name: moagrawa-2026-02-10_06:35:43-crimson-rados-main-distro-debug-trial no_nested_subset: false os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: global: auth client required: none auth cluster required: none auth service required: none auth supported: none debug asok: 0/0 debug auth: 0/0 debug bdev: 0/0 debug bluefs: 0/0 debug bluestore: 0/0 debug buffer: 0/0 debug civetweb: 0/0 debug client: 0/0 debug compressor: 0/0 debug context: 0/0 debug crush: 0/0 debug crypto: 0/0 debug dpdk: 0/0 debug eventtrace: 0/0 debug filer: 0/0 debug filestore: 0/0 debug finisher: 0/0 debug fuse: 0/0 debug heartbeatmap: 0/0 debug javaclient: 0/0 debug journal: 0/0 debug journaler: 0/0 debug kstore: 0/0 debug leveldb: 0/0 debug lockdep: 0/0 debug mds: 0/0 debug mds balancer: 0/0 debug mds locker: 0/0 debug mds log: 0/0 debug mds log expire: 0/0 debug mds migrator: 0/0 debug memdb: 0/0 debug mgr: 0/0 debug mgrc: 0/0 debug mon: 0/0 debug monc: 0/0 debug ms: 0/0 debug objclass: 0/0 debug objectcacher: 0/0 debug objecter: 0/0 debug optracker: 0/0 debug osd: 0/0 debug paxos: 0/0 debug perfcounter: 0/0 debug rados: 0/0 debug rbd: 0/0 debug rbd mirror: 0/0 debug rbd replay: 0/0 debug refs: 0/0 debug rgw: 0/0 debug rgw lifecycle: 0/0 debug rgw notification: 0/0 debug rgw sync: 0/0 debug rocksdb: 0/0 debug striper: 0/0 debug throttle: 0/0 debug timer: 0/0 debug tp: 0/0 enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 0/0 debug ms: 0/0 debug paxos: 0/0 osd pool default crimson: true osd: crimson cpu num: 4 crimson osd obc lru size: 10 debug filestore: 0/0 debug journal: 0/0 debug ms: 0/0 debug osd: 0/0 debug seastore: 20 debug seastore backref: 20 debug seastore cache: 20 debug seastore cleaner: 20 debug seastore device: 20 debug seastore epm: 20 debug seastore fixedkv tree: 20 debug seastore journal: 20 debug seastore lba: 20 debug seastore odata: 20 debug seastore omap: 20 debug seastore onode: 20 debug seastore t: 20 debug seastore tm: 20 osd objectstore: seastore osd shutdown pgref assert: true seastore cachepin size pershard: 64M seastore cachepin type: 2Q seastore main device type: RANDOM_BLOCK_SSD seastore max concurrent transactions: 8 crimson_compat: true flavor: debug log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) - \(PG_ - \(OSD_ - \(OBJECT_ - overall HEALTH sha1: 5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log global: osd crush chooseleaf type: 0 osd pool default pg num: 128 osd pool default pgp num: 128 osd pool default size: 2 mon: {} install: ceph: flavor: debug sha1: 5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0 workunit: branch: seastore_io_uring env: CRIMSON_COMPAT: '1' sha1: 5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0 owner: scheduled_moagrawa@soko04.front.sepia.ceph.com priority: 1000 repo: https://github.com/ceph/ceph.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 2425 sha1: 5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0 sleep_before_teardown: 0 suite: crimson-rados suite_branch: seastore_io_uring suite_path: /home/teuthworker/src/github.com_mohit84_ceph_5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0/qa suite_relpath: qa suite_repo: https://github.com/mohit84/ceph.git suite_sha1: 5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0 targets: trial110.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEVV+hvOidUwHbJclrEF8MwmS14TmMH5kZRsU4/p8C1tCb/hTSvq8krxLK6jGv1rAZDERauTUCmyY9peBmhwtyg= tasks: - install: extra_packages: deb: - ceph-osd-crimson - ceph-osd-crimson-dbg rpm: - ceph-osd-crimson - ceph-osd-crimson-debuginfo - pexec: all: - sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson - install: null - ceph: conf: osd: debug monc: 20 wait-for-scrub: false - ssh_keys: null - cbt: benchmarks: radosbench: concurrent_ops: 4 concurrent_procs: 2 op_size: - 4194304 pool_monitoring_list: - collectl pool_profile: replicated readmode: rand run_monitoring_list: - collectl time: 60 write_only: false cluster: iterations: 1 osds_per_node: 3 pool_profiles: replicated: pg_size: 256 pgp_size: 256 replication: replicated user: ubuntu monitoring_profiles: perf: args: stat -p {pid} -o {perf_dir}/perf_stat.{pid} nodes: - osds perf_cmd: perf pid_glob: crimson-osd.*.pid teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: c433f1062990a0488dc29a553589bc609a460691 timestamp: 2026-02-10_06:35:43 tube: trial user: moagrawa verbose: false worker_log: /home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.2161924 2026-02-12T12:03:40.203 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_mohit84_ceph_5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0/qa; will attempt to use it 2026-02-12T12:03:40.203 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_mohit84_ceph_5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0/qa/tasks 2026-02-12T12:03:40.203 INFO:teuthology.run_tasks:Running task internal.check_packages... 2026-02-12T12:03:40.205 INFO:teuthology.task.internal:Checking packages... 2026-02-12T12:03:40.205 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'debug' and ceph hash '5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0' 2026-02-12T12:03:40.205 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2026-02-12T12:03:40.205 INFO:teuthology.packaging:ref: None 2026-02-12T12:03:40.205 INFO:teuthology.packaging:tag: None 2026-02-12T12:03:40.205 INFO:teuthology.packaging:branch: main 2026-02-12T12:03:40.205 INFO:teuthology.packaging:sha1: 5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0 2026-02-12T12:03:40.205 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&ref=main 2026-02-12T12:03:40.370 INFO:teuthology.task.internal:Found packages for ceph version 20.3.0-5273.g2bf58db4 2026-02-12T12:03:40.371 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2026-02-12T12:03:40.383 INFO:teuthology.task.internal:no buildpackages task found 2026-02-12T12:03:40.383 INFO:teuthology.run_tasks:Running task internal.save_config... 2026-02-12T12:03:40.397 INFO:teuthology.task.internal:Saving configuration 2026-02-12T12:03:40.405 INFO:teuthology.run_tasks:Running task internal.check_lock... 2026-02-12T12:03:40.416 INFO:teuthology.task.internal.check_lock:Checking locks... 2026-02-12T12:03:40.485 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial110.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/moagrawa-2026-02-10_06:35:43-crimson-rados-main-distro-debug-trial/42493', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-12 11:59:08.341705', 'locked_by': 'scheduled_moagrawa@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEVV+hvOidUwHbJclrEF8MwmS14TmMH5kZRsU4/p8C1tCb/hTSvq8krxLK6jGv1rAZDERauTUCmyY9peBmhwtyg='} 2026-02-12T12:03:40.485 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2026-02-12T12:03:40.499 INFO:teuthology.task.internal:roles: ubuntu@trial110.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2026-02-12T12:03:40.499 INFO:teuthology.run_tasks:Running task console_log... 2026-02-12T12:03:40.571 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7922e4de27a0>, signals=[15]) 2026-02-12T12:03:40.571 INFO:teuthology.run_tasks:Running task internal.connect... 2026-02-12T12:03:40.583 INFO:teuthology.task.internal:Opening connections... 2026-02-12T12:03:40.583 DEBUG:teuthology.task.internal:connecting to ubuntu@trial110.front.sepia.ceph.com 2026-02-12T12:03:40.583 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-12T12:03:40.645 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2026-02-12T12:03:40.657 DEBUG:teuthology.orchestra.run.trial110:> uname -m 2026-02-12T12:03:40.721 INFO:teuthology.orchestra.run.trial110.stdout:x86_64 2026-02-12T12:03:40.722 DEBUG:teuthology.orchestra.run.trial110:> cat /etc/os-release 2026-02-12T12:03:40.777 INFO:teuthology.orchestra.run.trial110.stdout:NAME="CentOS Stream" 2026-02-12T12:03:40.777 INFO:teuthology.orchestra.run.trial110.stdout:VERSION="9" 2026-02-12T12:03:40.777 INFO:teuthology.orchestra.run.trial110.stdout:ID="centos" 2026-02-12T12:03:40.777 INFO:teuthology.orchestra.run.trial110.stdout:ID_LIKE="rhel fedora" 2026-02-12T12:03:40.777 INFO:teuthology.orchestra.run.trial110.stdout:VERSION_ID="9" 2026-02-12T12:03:40.777 INFO:teuthology.orchestra.run.trial110.stdout:PLATFORM_ID="platform:el9" 2026-02-12T12:03:40.777 INFO:teuthology.orchestra.run.trial110.stdout:PRETTY_NAME="CentOS Stream 9" 2026-02-12T12:03:40.777 INFO:teuthology.orchestra.run.trial110.stdout:ANSI_COLOR="0;31" 2026-02-12T12:03:40.777 INFO:teuthology.orchestra.run.trial110.stdout:LOGO="fedora-logo-icon" 2026-02-12T12:03:40.777 INFO:teuthology.orchestra.run.trial110.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2026-02-12T12:03:40.777 INFO:teuthology.orchestra.run.trial110.stdout:HOME_URL="https://centos.org/" 2026-02-12T12:03:40.777 INFO:teuthology.orchestra.run.trial110.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2026-02-12T12:03:40.777 INFO:teuthology.orchestra.run.trial110.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2026-02-12T12:03:40.777 INFO:teuthology.orchestra.run.trial110.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2026-02-12T12:03:40.777 INFO:teuthology.lock.ops:Updating trial110.front.sepia.ceph.com on lock server 2026-02-12T12:03:40.833 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2026-02-12T12:03:40.852 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2026-02-12T12:03:40.864 INFO:teuthology.task.internal:Checking for old test directory... 2026-02-12T12:03:40.864 DEBUG:teuthology.orchestra.run.trial110:> test '!' -e /home/ubuntu/cephtest 2026-02-12T12:03:40.878 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2026-02-12T12:03:40.893 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2026-02-12T12:03:40.893 DEBUG:teuthology.orchestra.run.trial110:> test -z $(ls -A /var/lib/ceph) 2026-02-12T12:03:40.935 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2026-02-12T12:03:41.003 INFO:teuthology.run_tasks:Running task kernel... 2026-02-12T12:03:41.024 INFO:teuthology.task.kernel:normalize config orig: {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'} 2026-02-12T12:03:41.024 INFO:teuthology.task.kernel:config {'mon.a': {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'}}, timeout 300 2026-02-12T12:03:41.024 DEBUG:teuthology.orchestra.run.trial110:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-12T12:03:41.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-12T12:03:41.039 DEBUG:teuthology.orchestra.run.trial110:> uname -r 2026-02-12T12:03:41.095 INFO:teuthology.orchestra.run.trial110.stdout:5.14.0-645.el9.x86_64 2026-02-12T12:03:41.095 INFO:teuthology.task.kernel:Running kernel on trial110: 5.14.0-645.el9.x86_64 2026-02-12T12:03:41.095 DEBUG:teuthology.orchestra.run.trial110:> sudo yum install -y kernel 2026-02-12T12:03:41.704 INFO:teuthology.orchestra.run.trial110.stdout:CentOS Stream 9 - BaseOS 37 MB/s | 8.9 MB 00:00 2026-02-12T12:03:44.943 INFO:teuthology.orchestra.run.trial110.stdout:CentOS Stream 9 - AppStream 9.9 MB/s | 27 MB 00:02 2026-02-12T12:03:48.482 INFO:teuthology.orchestra.run.trial110.stdout:CentOS Stream 9 - CRB 6.8 MB/s | 7.8 MB 00:01 2026-02-12T12:03:49.616 INFO:teuthology.orchestra.run.trial110.stdout:CentOS Stream 9 - Extras packages 49 kB/s | 20 kB 00:00 2026-02-12T12:03:50.341 INFO:teuthology.orchestra.run.trial110.stdout:Extra Packages for Enterprise Linux 31 MB/s | 20 MB 00:00 2026-02-12T12:03:53.587 INFO:teuthology.orchestra.run.trial110.stdout:lab-extras 2.8 MB/s | 50 kB 00:00 2026-02-12T12:03:54.642 INFO:teuthology.orchestra.run.trial110.stdout:Package kernel-5.14.0-645.el9.x86_64 is already installed. 2026-02-12T12:03:54.662 INFO:teuthology.orchestra.run.trial110.stdout:Dependencies resolved. 2026-02-12T12:03:54.664 INFO:teuthology.orchestra.run.trial110.stdout:================================================================================ 2026-02-12T12:03:54.664 INFO:teuthology.orchestra.run.trial110.stdout: Package Architecture Version Repository Size 2026-02-12T12:03:54.664 INFO:teuthology.orchestra.run.trial110.stdout:================================================================================ 2026-02-12T12:03:54.664 INFO:teuthology.orchestra.run.trial110.stdout:Installing: 2026-02-12T12:03:54.664 INFO:teuthology.orchestra.run.trial110.stdout: kernel x86_64 5.14.0-677.el9 baseos 768 k 2026-02-12T12:03:54.664 INFO:teuthology.orchestra.run.trial110.stdout:Installing dependencies: 2026-02-12T12:03:54.664 INFO:teuthology.orchestra.run.trial110.stdout: kernel-core x86_64 5.14.0-677.el9 baseos 17 M 2026-02-12T12:03:54.664 INFO:teuthology.orchestra.run.trial110.stdout: kernel-modules x86_64 5.14.0-677.el9 baseos 40 M 2026-02-12T12:03:54.664 INFO:teuthology.orchestra.run.trial110.stdout: kernel-modules-core x86_64 5.14.0-677.el9 baseos 31 M 2026-02-12T12:03:54.665 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:03:54.665 INFO:teuthology.orchestra.run.trial110.stdout:Transaction Summary 2026-02-12T12:03:54.665 INFO:teuthology.orchestra.run.trial110.stdout:================================================================================ 2026-02-12T12:03:54.665 INFO:teuthology.orchestra.run.trial110.stdout:Install 4 Packages 2026-02-12T12:03:54.665 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:03:54.665 INFO:teuthology.orchestra.run.trial110.stdout:Total download size: 88 M 2026-02-12T12:03:54.665 INFO:teuthology.orchestra.run.trial110.stdout:Installed size: 135 M 2026-02-12T12:03:54.665 INFO:teuthology.orchestra.run.trial110.stdout:Downloading Packages: 2026-02-12T12:03:55.219 INFO:teuthology.orchestra.run.trial110.stdout:[MIRROR] kernel-5.14.0-677.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-677.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-12T12:03:55.222 INFO:teuthology.orchestra.run.trial110.stdout:[MIRROR] kernel-core-5.14.0-677.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-677.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-12T12:03:55.227 INFO:teuthology.orchestra.run.trial110.stdout:[MIRROR] kernel-modules-5.14.0-677.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-677.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-12T12:03:55.834 INFO:teuthology.orchestra.run.trial110.stdout:[MIRROR] kernel-5.14.0-677.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-677.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-12T12:03:55.852 INFO:teuthology.orchestra.run.trial110.stdout:[MIRROR] kernel-modules-5.14.0-677.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-677.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-12T12:03:55.878 INFO:teuthology.orchestra.run.trial110.stdout:[MIRROR] kernel-core-5.14.0-677.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-677.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-12T12:03:56.234 INFO:teuthology.orchestra.run.trial110.stdout:(1/4): kernel-5.14.0-677.el9.x86_64.rpm 520 kB/s | 768 kB 00:01 2026-02-12T12:03:56.465 INFO:teuthology.orchestra.run.trial110.stdout:(2/4): kernel-core-5.14.0-677.el9.x86_64.rpm 10 MB/s | 17 MB 00:01 2026-02-12T12:03:56.621 INFO:teuthology.orchestra.run.trial110.stdout:[MIRROR] kernel-modules-core-5.14.0-677.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-677.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-12T12:03:56.896 INFO:teuthology.orchestra.run.trial110.stdout:(3/4): kernel-modules-5.14.0-677.el9.x86_64.rpm 19 MB/s | 40 MB 00:02 2026-02-12T12:03:57.053 INFO:teuthology.orchestra.run.trial110.stdout:[MIRROR] kernel-modules-core-5.14.0-677.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-677.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-12T12:03:57.695 INFO:teuthology.orchestra.run.trial110.stdout:(4/4): kernel-modules-core-5.14.0-677.el9.x86_6 21 MB/s | 31 MB 00:01 2026-02-12T12:03:57.696 INFO:teuthology.orchestra.run.trial110.stdout:-------------------------------------------------------------------------------- 2026-02-12T12:03:57.696 INFO:teuthology.orchestra.run.trial110.stdout:Total 29 MB/s | 88 MB 00:03 2026-02-12T12:03:57.889 INFO:teuthology.orchestra.run.trial110.stdout:Running transaction check 2026-02-12T12:03:58.093 INFO:teuthology.orchestra.run.trial110.stdout:Transaction check succeeded. 2026-02-12T12:03:58.093 INFO:teuthology.orchestra.run.trial110.stdout:Running transaction test 2026-02-12T12:03:58.438 INFO:teuthology.orchestra.run.trial110.stdout:Transaction test succeeded. 2026-02-12T12:03:58.438 INFO:teuthology.orchestra.run.trial110.stdout:Running transaction 2026-02-12T12:03:59.531 INFO:teuthology.orchestra.run.trial110.stdout: Preparing : 1/1 2026-02-12T12:03:59.817 INFO:teuthology.orchestra.run.trial110.stdout: Installing : kernel-modules-core-5.14.0-677.el9.x86_64 1/4 2026-02-12T12:03:59.836 INFO:teuthology.orchestra.run.trial110.stdout: Installing : kernel-core-5.14.0-677.el9.x86_64 2/4 2026-02-12T12:04:01.613 INFO:teuthology.orchestra.run.trial110.stdout: Running scriptlet: kernel-core-5.14.0-677.el9.x86_64 2/4 2026-02-12T12:04:01.642 INFO:teuthology.orchestra.run.trial110.stdout: Installing : kernel-modules-5.14.0-677.el9.x86_64 3/4 2026-02-12T12:04:04.668 INFO:teuthology.orchestra.run.trial110.stdout: Running scriptlet: kernel-modules-5.14.0-677.el9.x86_64 3/4 2026-02-12T12:04:04.674 INFO:teuthology.orchestra.run.trial110.stdout: Installing : kernel-5.14.0-677.el9.x86_64 4/4 2026-02-12T12:04:07.959 INFO:teuthology.orchestra.run.trial110.stdout: Running scriptlet: kernel-modules-core-5.14.0-677.el9.x86_64 4/4 2026-02-12T12:04:38.429 INFO:teuthology.orchestra.run.trial110.stdout: Running scriptlet: kernel-core-5.14.0-677.el9.x86_64 4/4 2026-02-12T12:04:38.451 INFO:teuthology.orchestra.run.trial110.stdout: Running scriptlet: kernel-modules-5.14.0-677.el9.x86_64 4/4 2026-02-12T12:04:38.517 INFO:teuthology.orchestra.run.trial110.stdout: Running scriptlet: kernel-5.14.0-677.el9.x86_64 4/4 2026-02-12T12:04:38.517 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : kernel-5.14.0-677.el9.x86_64 1/4 2026-02-12T12:04:38.517 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : kernel-core-5.14.0-677.el9.x86_64 2/4 2026-02-12T12:04:38.517 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : kernel-modules-5.14.0-677.el9.x86_64 3/4 2026-02-12T12:04:38.569 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : kernel-modules-core-5.14.0-677.el9.x86_64 4/4 2026-02-12T12:04:38.569 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:04:38.569 INFO:teuthology.orchestra.run.trial110.stdout:Installed: 2026-02-12T12:04:38.569 INFO:teuthology.orchestra.run.trial110.stdout: kernel-5.14.0-677.el9.x86_64 2026-02-12T12:04:38.569 INFO:teuthology.orchestra.run.trial110.stdout: kernel-core-5.14.0-677.el9.x86_64 2026-02-12T12:04:38.569 INFO:teuthology.orchestra.run.trial110.stdout: kernel-modules-5.14.0-677.el9.x86_64 2026-02-12T12:04:38.569 INFO:teuthology.orchestra.run.trial110.stdout: kernel-modules-core-5.14.0-677.el9.x86_64 2026-02-12T12:04:38.570 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:04:38.570 INFO:teuthology.orchestra.run.trial110.stdout:Complete! 2026-02-12T12:04:38.628 DEBUG:teuthology.orchestra.run.trial110:> rpm -q kernel | sort -rV | head -n 1 2026-02-12T12:04:38.655 INFO:teuthology.orchestra.run.trial110.stdout:kernel-5.14.0-677.el9.x86_64 2026-02-12T12:04:38.656 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-677.el9.x86_64 2026-02-12T12:04:38.656 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-645.el9.x86_64 Expected: 5.14.0-677.el9.x86_64 2026-02-12T12:04:38.656 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2026-02-12T12:04:38.656 DEBUG:teuthology.task.kernel:src is distro, skipping download 2026-02-12T12:04:38.656 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2026-02-12T12:04:38.656 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@trial110.front.sepia.ceph.com, path=None, version=distro) 2026-02-12T12:04:38.656 DEBUG:teuthology.orchestra.run.trial110:> rpm -q kernel | sort -rV | head -n 1 2026-02-12T12:04:38.717 INFO:teuthology.orchestra.run.trial110.stdout:kernel-5.14.0-677.el9.x86_64 2026-02-12T12:04:38.719 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-677.el9.x86_64 2026-02-12T12:04:38.719 DEBUG:teuthology.orchestra.run.trial110:> sudo rpm -qi grub2-tools 2026-02-12T12:04:38.787 INFO:teuthology.orchestra.run.trial110.stdout:Name : grub2-tools 2026-02-12T12:04:38.788 INFO:teuthology.orchestra.run.trial110.stdout:Epoch : 1 2026-02-12T12:04:38.788 INFO:teuthology.orchestra.run.trial110.stdout:Version : 2.06 2026-02-12T12:04:38.788 INFO:teuthology.orchestra.run.trial110.stdout:Release : 118.el9 2026-02-12T12:04:38.788 INFO:teuthology.orchestra.run.trial110.stdout:Architecture: x86_64 2026-02-12T12:04:38.788 INFO:teuthology.orchestra.run.trial110.stdout:Install Date: Thu 04 Dec 2025 11:54:21 PM UTC 2026-02-12T12:04:38.788 INFO:teuthology.orchestra.run.trial110.stdout:Group : Unspecified 2026-02-12T12:04:38.788 INFO:teuthology.orchestra.run.trial110.stdout:Size : 8393909 2026-02-12T12:04:38.788 INFO:teuthology.orchestra.run.trial110.stdout:License : GPLv3+ 2026-02-12T12:04:38.788 INFO:teuthology.orchestra.run.trial110.stdout:Signature : RSA/SHA256, Wed 05 Nov 2025 09:13:15 AM UTC, Key ID 05b555b38483c65d 2026-02-12T12:04:38.788 INFO:teuthology.orchestra.run.trial110.stdout:Source RPM : grub2-2.06-118.el9.src.rpm 2026-02-12T12:04:38.788 INFO:teuthology.orchestra.run.trial110.stdout:Build Date : Tue 04 Nov 2025 03:44:14 PM UTC 2026-02-12T12:04:38.788 INFO:teuthology.orchestra.run.trial110.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2026-02-12T12:04:38.789 INFO:teuthology.orchestra.run.trial110.stdout:Packager : builder@centos.org 2026-02-12T12:04:38.789 INFO:teuthology.orchestra.run.trial110.stdout:Vendor : CentOS 2026-02-12T12:04:38.789 INFO:teuthology.orchestra.run.trial110.stdout:URL : http://www.gnu.org/software/grub/ 2026-02-12T12:04:38.789 INFO:teuthology.orchestra.run.trial110.stdout:Summary : Support tools for GRUB. 2026-02-12T12:04:38.789 INFO:teuthology.orchestra.run.trial110.stdout:Description : 2026-02-12T12:04:38.789 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:04:38.789 INFO:teuthology.orchestra.run.trial110.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2026-02-12T12:04:38.789 INFO:teuthology.orchestra.run.trial110.stdout:customizable bootloader with modular architecture. It supports a rich 2026-02-12T12:04:38.789 INFO:teuthology.orchestra.run.trial110.stdout:variety of kernel formats, file systems, computer architectures and 2026-02-12T12:04:38.789 INFO:teuthology.orchestra.run.trial110.stdout:hardware devices. 2026-02-12T12:04:38.789 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:04:38.790 INFO:teuthology.orchestra.run.trial110.stdout:This subpackage provides tools for support of all platforms. 2026-02-12T12:04:38.791 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2026-02-12T12:04:38.791 INFO:teuthology.task.kernel:Updating grub on trial110 to boot 5.14.0-677.el9.x86_64 2026-02-12T12:04:38.791 DEBUG:teuthology.orchestra.run.trial110:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f -print -quit 2026-02-12T12:04:38.856 INFO:teuthology.orchestra.run.trial110.stdout:/boot/loader/entries/2734456005d743dca7f3dc77c35a2d0d-0-rescue.conf 2026-02-12T12:04:38.857 DEBUG:teuthology.orchestra.run.trial110:> sudo command -v grubby && echo yes || echo no 2026-02-12T12:04:38.921 INFO:teuthology.orchestra.run.trial110.stdout:/sbin/grubby 2026-02-12T12:04:38.922 INFO:teuthology.orchestra.run.trial110.stdout:yes 2026-02-12T12:04:38.923 DEBUG:teuthology.orchestra.run.trial110:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f | grep -F "5.14.0-677.el9.x86_64" | head -n 1 2026-02-12T12:04:38.987 INFO:teuthology.orchestra.run.trial110.stdout:/boot/loader/entries/b0373be789df49cdafedf423703d2566-5.14.0-677.el9.x86_64.conf 2026-02-12T12:04:38.988 DEBUG:teuthology.orchestra.run.trial110:> sudo grub2-set-default b0373be789df49cdafedf423703d2566-5.14.0-677.el9.x86_64 2026-02-12T12:04:39.060 DEBUG:teuthology.orchestra.run.trial110:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2026-02-12T12:04:39.271 INFO:teuthology.orchestra.run.trial110.stderr:Generating grub configuration file ... 2026-02-12T12:04:39.334 INFO:teuthology.orchestra.run.trial110.stderr:Found linux image: /boot/vmlinuz-5.14.0-677.el9.x86_64 2026-02-12T12:04:39.341 INFO:teuthology.orchestra.run.trial110.stderr:Found initrd image: /boot/initramfs-5.14.0-677.el9.x86_64.img 2026-02-12T12:04:39.593 INFO:teuthology.orchestra.run.trial110.stderr:Found linux image: /boot/vmlinuz-5.14.0-645.el9.x86_64 2026-02-12T12:04:39.600 INFO:teuthology.orchestra.run.trial110.stderr:Found initrd image: /boot/initramfs-5.14.0-645.el9.x86_64.img 2026-02-12T12:04:39.631 INFO:teuthology.orchestra.run.trial110.stderr:Found linux image: /boot/vmlinuz-0-rescue-b0373be789df49cdafedf423703d2566 2026-02-12T12:04:39.637 INFO:teuthology.orchestra.run.trial110.stderr:Found initrd image: /boot/initramfs-0-rescue-b0373be789df49cdafedf423703d2566.img 2026-02-12T12:04:39.661 INFO:teuthology.orchestra.run.trial110.stderr:Found linux image: /boot/vmlinuz-0-rescue-2734456005d743dca7f3dc77c35a2d0d 2026-02-12T12:04:39.668 INFO:teuthology.orchestra.run.trial110.stderr:Found initrd image: /boot/initramfs-0-rescue-2734456005d743dca7f3dc77c35a2d0d.img 2026-02-12T12:04:39.751 INFO:teuthology.orchestra.run.trial110.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2026-02-12T12:04:39.759 INFO:teuthology.orchestra.run.trial110.stderr:done 2026-02-12T12:04:39.761 DEBUG:teuthology.orchestra.run.trial110:> test -d /sys/firmware/efi 2026-02-12T12:04:39.776 DEBUG:teuthology.orchestra.run.trial110:> sudo find /boot/efi/EFI -mindepth 1 -maxdepth 1 -type d || true 2026-02-12T12:04:39.841 INFO:teuthology.orchestra.run.trial110.stdout:/boot/efi/EFI/BOOT 2026-02-12T12:04:39.841 INFO:teuthology.orchestra.run.trial110.stdout:/boot/efi/EFI/centos 2026-02-12T12:04:39.843 DEBUG:teuthology.orchestra.run.trial110:> sudo test -f /boot/efi/EFI/BOOT/grub.cfg && echo yes || echo no 2026-02-12T12:04:39.909 INFO:teuthology.orchestra.run.trial110.stdout:no 2026-02-12T12:04:39.909 DEBUG:teuthology.orchestra.run.trial110:> sudo test -f /boot/efi/EFI/centos/grub.cfg && echo yes || echo no 2026-02-12T12:04:39.973 INFO:teuthology.orchestra.run.trial110.stdout:yes 2026-02-12T12:04:39.974 DEBUG:teuthology.orchestra.run.trial110:> sudo cp -f /boot/grub2/grub.cfg /boot/efi/EFI/centos/grub.cfg 2026-02-12T12:04:40.039 DEBUG:teuthology.orchestra.run.trial110:> sudo shutdown -r now 2026-02-12T12:04:40.113 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2026-02-12T12:05:10.112 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2026-02-12T12:05:10.112 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial110.front.sepia.ceph.com' 2026-02-12T12:05:10.113 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-12T12:06:10.174 DEBUG:teuthology.orchestra.remote:timed out 2026-02-12T12:06:19.183 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial110.front.sepia.ceph.com' 2026-02-12T12:06:19.184 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-12T12:06:19.277 DEBUG:teuthology.orchestra.run.trial110:> true 2026-02-12T12:06:19.431 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial110.front.sepia.ceph.com' 2026-02-12T12:06:19.431 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-677.el9.x86_64"... 2026-02-12T12:06:19.431 DEBUG:teuthology.orchestra.run.trial110:> uname -r 2026-02-12T12:06:19.486 INFO:teuthology.orchestra.run.trial110.stdout:5.14.0-677.el9.x86_64 2026-02-12T12:06:19.487 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-677.el9.x86_64 vs 5.14.0-677.el9.x86_64 2026-02-12T12:06:19.487 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2026-02-12T12:06:19.487 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2026-02-12T12:06:20.487 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2026-02-12T12:06:20.488 DEBUG:teuthology.orchestra.run.trial110:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-12T12:06:20.517 INFO:teuthology.orchestra.run.trial110.stdout:ttyS1 2026-02-12T12:06:20.549 DEBUG:teuthology.parallel:result is None 2026-02-12T12:06:20.549 INFO:teuthology.run_tasks:Running task internal.base... 2026-02-12T12:06:20.560 INFO:teuthology.task.internal:Creating test directory... 2026-02-12T12:06:20.561 DEBUG:teuthology.orchestra.run.trial110:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-12T12:06:20.607 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2026-02-12T12:06:20.618 INFO:teuthology.run_tasks:Running task internal.archive... 2026-02-12T12:06:20.632 INFO:teuthology.task.internal:Creating archive directory... 2026-02-12T12:06:20.632 DEBUG:teuthology.orchestra.run.trial110:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-12T12:06:20.669 INFO:teuthology.run_tasks:Running task internal.coredump... 2026-02-12T12:06:20.679 INFO:teuthology.task.internal:Enabling coredump saving... 2026-02-12T12:06:20.679 DEBUG:teuthology.orchestra.run.trial110:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-12T12:06:20.727 INFO:teuthology.orchestra.run.trial110.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-12T12:06:20.736 INFO:teuthology.orchestra.run.trial110.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-12T12:06:20.737 INFO:teuthology.run_tasks:Running task internal.sudo... 2026-02-12T12:06:20.749 INFO:teuthology.task.internal:Configuring sudo... 2026-02-12T12:06:20.749 DEBUG:teuthology.orchestra.run.trial110:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-12T12:06:20.802 INFO:teuthology.run_tasks:Running task internal.syslog... 2026-02-12T12:06:20.815 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2026-02-12T12:06:20.816 DEBUG:teuthology.orchestra.run.trial110:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-12T12:06:20.858 DEBUG:teuthology.orchestra.run.trial110:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-12T12:06:21.021 DEBUG:teuthology.orchestra.run.trial110:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-12T12:06:21.047 DEBUG:teuthology.orchestra.run.trial110:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-12T12:06:21.178 DEBUG:teuthology.orchestra.run.trial110:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-12T12:06:21.201 DEBUG:teuthology.orchestra.run.trial110:> set -ex 2026-02-12T12:06:21.202 DEBUG:teuthology.orchestra.run.trial110:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-12T12:06:21.267 DEBUG:teuthology.orchestra.run.trial110:> sudo service rsyslog restart 2026-02-12T12:06:21.335 INFO:teuthology.orchestra.run.trial110.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-12T12:06:21.734 INFO:teuthology.run_tasks:Running task internal.timer... 2026-02-12T12:06:21.746 INFO:teuthology.task.internal:Starting timer... 2026-02-12T12:06:21.746 INFO:teuthology.run_tasks:Running task pcp... 2026-02-12T12:06:21.762 INFO:teuthology.run_tasks:Running task selinux... 2026-02-12T12:06:21.775 DEBUG:teuthology.orchestra.run.trial110:> sudo service auditd rotate 2026-02-12T12:06:21.803 INFO:teuthology.orchestra.run.trial110.stdout:Rotating logs: 2026-02-12T12:06:21.805 DEBUG:teuthology.task.selinux:Getting current SELinux state 2026-02-12T12:06:21.805 DEBUG:teuthology.orchestra.run.trial110:> /usr/sbin/getenforce 2026-02-12T12:06:21.862 INFO:teuthology.orchestra.run.trial110.stdout:Permissive 2026-02-12T12:06:21.862 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@trial110.front.sepia.ceph.com': 'permissive'} 2026-02-12T12:06:21.863 DEBUG:teuthology.orchestra.run.trial110:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-12T12:06:21.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-12T12:06:21.926 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2026-02-12T12:06:21.926 DEBUG:teuthology.orchestra.run.trial110:> sudo /usr/sbin/setenforce permissive 2026-02-12T12:06:21.990 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2026-02-12T12:06:22.002 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2026-02-12T12:06:22.013 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2026-02-12T12:06:22.014 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit trial110.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2026-02-12T12:07:30.766 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@trial110.front.sepia.ceph.com')] 2026-02-12T12:07:30.768 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial110.front.sepia.ceph.com' 2026-02-12T12:07:30.769 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-12T12:07:30.880 DEBUG:teuthology.orchestra.run.trial110:> true 2026-02-12T12:07:30.954 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial110.front.sepia.ceph.com' 2026-02-12T12:07:30.954 INFO:teuthology.run_tasks:Running task clock... 2026-02-12T12:07:30.967 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2026-02-12T12:07:30.967 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-12T12:07:30.968 DEBUG:teuthology.orchestra.run.trial110:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-12T12:07:31.022 INFO:teuthology.orchestra.run.trial110.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2026-02-12T12:07:31.034 INFO:teuthology.orchestra.run.trial110.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2026-02-12T12:07:31.054 INFO:teuthology.orchestra.run.trial110.stderr:sudo: ntpd: command not found 2026-02-12T12:07:31.063 INFO:teuthology.orchestra.run.trial110.stdout:506 Cannot talk to daemon 2026-02-12T12:07:31.074 INFO:teuthology.orchestra.run.trial110.stderr:Failed to start ntp.service: Unit ntp.service not found. 2026-02-12T12:07:31.085 INFO:teuthology.orchestra.run.trial110.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2026-02-12T12:07:31.129 INFO:teuthology.orchestra.run.trial110.stderr:bash: line 1: ntpq: command not found 2026-02-12T12:07:31.452 INFO:teuthology.orchestra.run.trial110.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-12T12:07:31.453 INFO:teuthology.orchestra.run.trial110.stdout:=============================================================================== 2026-02-12T12:07:31.453 INFO:teuthology.orchestra.run.trial110.stdout:^? 24-111-79-186-dynamic.mi> 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-12T12:07:31.453 INFO:teuthology.orchestra.run.trial110.stdout:^? 66.187.4.132.hostodo.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-12T12:07:31.453 INFO:teuthology.orchestra.run.trial110.stdout:^? time.cloudflare.com 3 6 1 0 -21us[ -21us] +/- 6691us 2026-02-12T12:07:31.453 INFO:teuthology.orchestra.run.trial110.stdout:^? time-he.den.codehof.net 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-12T12:07:31.454 INFO:teuthology.run_tasks:Running task install... 2026-02-12T12:07:31.476 DEBUG:teuthology.task.install:project ceph 2026-02-12T12:07:31.477 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'debug', 'sha1': '5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0'}} 2026-02-12T12:07:31.477 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'flavor': 'debug', 'sha1': '5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0'} 2026-02-12T12:07:31.477 INFO:teuthology.task.install:Using flavor: debug 2026-02-12T12:07:31.492 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-osd', 'ceph-osd-classic', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-osd', 'ceph-osd-classic', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-12T12:07:31.492 INFO:teuthology.task.install:extra packages: {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']} 2026-02-12T12:07:31.493 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'debug', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0', 'tag': None, 'wait_for_package': False} 2026-02-12T12:07:31.494 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0 2026-02-12T12:07:31.647 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-moagrawa-seastore-io-uring-again-crimson-only/5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0/centos/9/flavors/debug/ 2026-02-12T12:07:31.647 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5145.g5fdb7d8f 2026-02-12T12:07:31.960 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-moagrawa-seastore-io-uring-again-crimson-only/5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0/centos/9/flavors/debug/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-moagrawa-seastore-io-uring-again-crimson-only/5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0/centos/9/flavors/debug/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-moagrawa-seastore-io-uring-again-crimson-only/5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0/centos/9/flavors/debug/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-12T12:07:31.960 DEBUG:teuthology.orchestra.run.trial110:> set -ex 2026-02-12T12:07:31.960 DEBUG:teuthology.orchestra.run.trial110:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-12T12:07:31.985 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-osd-crimson, ceph-osd-crimson-debuginfo on remote rpm x86_64 2026-02-12T12:07:31.986 DEBUG:teuthology.orchestra.run.trial110:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-12T12:07:32.051 DEBUG:teuthology.orchestra.run.trial110:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-12T12:07:32.124 DEBUG:teuthology.orchestra.run.trial110:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-12T12:07:32.187 INFO:teuthology.orchestra.run.trial110.stdout:check_obsoletes = 1 2026-02-12T12:07:32.189 DEBUG:teuthology.orchestra.run.trial110:> sudo yum clean all 2026-02-12T12:07:32.336 INFO:teuthology.orchestra.run.trial110.stdout:41 files removed 2026-02-12T12:07:32.351 DEBUG:teuthology.orchestra.run.trial110:> sudo yum -y install ceph-radosgw 2026-02-12T12:07:33.213 INFO:teuthology.orchestra.run.trial110.stdout:ceph packages for x86_64 130 kB/s | 94 kB 00:00 2026-02-12T12:07:33.843 INFO:teuthology.orchestra.run.trial110.stdout:ceph noarch packages 40 kB/s | 24 kB 00:00 2026-02-12T12:07:34.424 INFO:teuthology.orchestra.run.trial110.stdout:ceph source packages 3.7 kB/s | 2.1 kB 00:00 2026-02-12T12:07:35.244 INFO:teuthology.orchestra.run.trial110.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.9 MB 00:00 2026-02-12T12:07:36.104 INFO:teuthology.orchestra.run.trial110.stdout:CentOS Stream 9 - AppStream 87 MB/s | 27 MB 00:00 2026-02-12T12:07:39.982 INFO:teuthology.orchestra.run.trial110.stdout:CentOS Stream 9 - CRB 5.2 MB/s | 7.8 MB 00:01 2026-02-12T12:07:40.787 INFO:teuthology.orchestra.run.trial110.stdout:CentOS Stream 9 - Extras packages 232 kB/s | 20 kB 00:00 2026-02-12T12:07:41.518 INFO:teuthology.orchestra.run.trial110.stdout:Extra Packages for Enterprise Linux 31 MB/s | 20 MB 00:00 2026-02-12T12:07:44.756 INFO:teuthology.orchestra.run.trial110.stdout:lab-extras 2.5 MB/s | 50 kB 00:00 2026-02-12T12:07:46.037 INFO:teuthology.orchestra.run.trial110.stdout:Dependencies resolved. 2026-02-12T12:07:46.039 INFO:teuthology.orchestra.run.trial110.stdout:================================================================================ 2026-02-12T12:07:46.039 INFO:teuthology.orchestra.run.trial110.stdout: Package Arch Version Repository Size 2026-02-12T12:07:46.039 INFO:teuthology.orchestra.run.trial110.stdout:================================================================================ 2026-02-12T12:07:46.039 INFO:teuthology.orchestra.run.trial110.stdout:Installing: 2026-02-12T12:07:46.039 INFO:teuthology.orchestra.run.trial110.stdout: ceph-radosgw x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 11 M 2026-02-12T12:07:46.039 INFO:teuthology.orchestra.run.trial110.stdout:Upgrading: 2026-02-12T12:07:46.039 INFO:teuthology.orchestra.run.trial110.stdout: librados2 x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 3.3 M 2026-02-12T12:07:46.039 INFO:teuthology.orchestra.run.trial110.stdout: librbd1 x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 2.7 M 2026-02-12T12:07:46.039 INFO:teuthology.orchestra.run.trial110.stdout:Installing dependencies: 2026-02-12T12:07:46.039 INFO:teuthology.orchestra.run.trial110.stdout: boost-program-options x86_64 1.75.0-13.el9 appstream 104 k 2026-02-12T12:07:46.039 INFO:teuthology.orchestra.run.trial110.stdout: ceph-base x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 5.1 M 2026-02-12T12:07:46.039 INFO:teuthology.orchestra.run.trial110.stdout: ceph-common x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 21 M 2026-02-12T12:07:46.039 INFO:teuthology.orchestra.run.trial110.stdout: ceph-selinux x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 25 k 2026-02-12T12:07:46.039 INFO:teuthology.orchestra.run.trial110.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2026-02-12T12:07:46.039 INFO:teuthology.orchestra.run.trial110.stdout: libarrow x86_64 9.0.0-15.el9 epel 4.4 M 2026-02-12T12:07:46.039 INFO:teuthology.orchestra.run.trial110.stdout: libarrow-doc noarch 9.0.0-15.el9 epel 25 k 2026-02-12T12:07:46.039 INFO:teuthology.orchestra.run.trial110.stdout: libcephfs2 x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 860 k 2026-02-12T12:07:46.039 INFO:teuthology.orchestra.run.trial110.stdout: libnbd x86_64 1.20.3-4.el9 appstream 164 k 2026-02-12T12:07:46.039 INFO:teuthology.orchestra.run.trial110.stdout: liboath x86_64 2.6.12-1.el9 epel 49 k 2026-02-12T12:07:46.039 INFO:teuthology.orchestra.run.trial110.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2026-02-12T12:07:46.039 INFO:teuthology.orchestra.run.trial110.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2026-02-12T12:07:46.039 INFO:teuthology.orchestra.run.trial110.stdout: libradosstriper1 x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 263 k 2026-02-12T12:07:46.039 INFO:teuthology.orchestra.run.trial110.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2026-02-12T12:07:46.040 INFO:teuthology.orchestra.run.trial110.stdout: librgw2 x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 5.6 M 2026-02-12T12:07:46.040 INFO:teuthology.orchestra.run.trial110.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2026-02-12T12:07:46.040 INFO:teuthology.orchestra.run.trial110.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2026-02-12T12:07:46.040 INFO:teuthology.orchestra.run.trial110.stdout: mailcap noarch 2.1.49-5.el9 baseos 33 k 2026-02-12T12:07:46.040 INFO:teuthology.orchestra.run.trial110.stdout: parquet-libs x86_64 9.0.0-15.el9 epel 838 k 2026-02-12T12:07:46.040 INFO:teuthology.orchestra.run.trial110.stdout: pciutils x86_64 3.7.0-7.el9 baseos 93 k 2026-02-12T12:07:46.040 INFO:teuthology.orchestra.run.trial110.stdout: python3-ceph-argparse x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 45 k 2026-02-12T12:07:46.040 INFO:teuthology.orchestra.run.trial110.stdout: python3-ceph-common x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 184 k 2026-02-12T12:07:46.040 INFO:teuthology.orchestra.run.trial110.stdout: python3-cephfs x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 203 k 2026-02-12T12:07:46.040 INFO:teuthology.orchestra.run.trial110.stdout: python3-rados x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 324 k 2026-02-12T12:07:46.040 INFO:teuthology.orchestra.run.trial110.stdout: python3-rbd x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 305 k 2026-02-12T12:07:46.040 INFO:teuthology.orchestra.run.trial110.stdout: python3-rgw x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 100 k 2026-02-12T12:07:46.040 INFO:teuthology.orchestra.run.trial110.stdout: qatlib x86_64 25.08.0-2.el9 appstream 240 k 2026-02-12T12:07:46.040 INFO:teuthology.orchestra.run.trial110.stdout: qatzip-libs x86_64 1.3.1-1.el9 appstream 66 k 2026-02-12T12:07:46.040 INFO:teuthology.orchestra.run.trial110.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2026-02-12T12:07:46.040 INFO:teuthology.orchestra.run.trial110.stdout: thrift x86_64 0.15.0-4.el9 epel 1.6 M 2026-02-12T12:07:46.040 INFO:teuthology.orchestra.run.trial110.stdout:Installing weak dependencies: 2026-02-12T12:07:46.040 INFO:teuthology.orchestra.run.trial110.stdout: libcephfs-daemon x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 35 k 2026-02-12T12:07:46.040 INFO:teuthology.orchestra.run.trial110.stdout: libcephfs-proxy2 x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 24 k 2026-02-12T12:07:46.040 INFO:teuthology.orchestra.run.trial110.stdout: nvme-cli x86_64 2.13-1.el9 baseos 1.1 M 2026-02-12T12:07:46.040 INFO:teuthology.orchestra.run.trial110.stdout: qatlib-service x86_64 25.08.0-2.el9 appstream 37 k 2026-02-12T12:07:46.040 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:07:46.040 INFO:teuthology.orchestra.run.trial110.stdout:Transaction Summary 2026-02-12T12:07:46.040 INFO:teuthology.orchestra.run.trial110.stdout:================================================================================ 2026-02-12T12:07:46.040 INFO:teuthology.orchestra.run.trial110.stdout:Install 35 Packages 2026-02-12T12:07:46.040 INFO:teuthology.orchestra.run.trial110.stdout:Upgrade 2 Packages 2026-02-12T12:07:46.040 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:07:46.040 INFO:teuthology.orchestra.run.trial110.stdout:Total download size: 61 M 2026-02-12T12:07:46.040 INFO:teuthology.orchestra.run.trial110.stdout:Downloading Packages: 2026-02-12T12:07:56.699 INFO:teuthology.orchestra.run.trial110.stdout:(1/37): ceph-base-20.3.0-5145.g5fdb7d8f.el9.x86 511 kB/s | 5.1 MB 00:10 2026-02-12T12:07:56.770 INFO:teuthology.orchestra.run.trial110.stdout:(2/37): ceph-selinux-20.3.0-5145.g5fdb7d8f.el9. 354 kB/s | 25 kB 00:00 2026-02-12T12:07:56.840 INFO:teuthology.orchestra.run.trial110.stdout:(3/37): libcephfs-daemon-20.3.0-5145.g5fdb7d8f. 496 kB/s | 35 kB 00:00 2026-02-12T12:07:56.908 INFO:teuthology.orchestra.run.trial110.stdout:(4/37): libcephfs-proxy2-20.3.0-5145.g5fdb7d8f. 358 kB/s | 24 kB 00:00 2026-02-12T12:07:58.804 INFO:teuthology.orchestra.run.trial110.stdout:(5/37): libcephfs2-20.3.0-5145.g5fdb7d8f.el9.x8 453 kB/s | 860 kB 00:01 2026-02-12T12:07:59.381 INFO:teuthology.orchestra.run.trial110.stdout:(6/37): libradosstriper1-20.3.0-5145.g5fdb7d8f. 455 kB/s | 263 kB 00:00 2026-02-12T12:08:06.623 INFO:teuthology.orchestra.run.trial110.stdout:(7/37): ceph-radosgw-20.3.0-5145.g5fdb7d8f.el9. 568 kB/s | 11 MB 00:20 2026-02-12T12:08:06.762 INFO:teuthology.orchestra.run.trial110.stdout:(8/37): python3-ceph-argparse-20.3.0-5145.g5fdb 326 kB/s | 45 kB 00:00 2026-02-12T12:08:07.176 INFO:teuthology.orchestra.run.trial110.stdout:(9/37): python3-ceph-common-20.3.0-5145.g5fdb7d 446 kB/s | 184 kB 00:00 2026-02-12T12:08:07.578 INFO:teuthology.orchestra.run.trial110.stdout:(10/37): python3-cephfs-20.3.0-5145.g5fdb7d8f.e 505 kB/s | 203 kB 00:00 2026-02-12T12:08:08.134 INFO:teuthology.orchestra.run.trial110.stdout:(11/37): python3-rados-20.3.0-5145.g5fdb7d8f.el 583 kB/s | 324 kB 00:00 2026-02-12T12:08:08.615 INFO:teuthology.orchestra.run.trial110.stdout:(12/37): python3-rbd-20.3.0-5145.g5fdb7d8f.el9. 633 kB/s | 305 kB 00:00 2026-02-12T12:08:08.757 INFO:teuthology.orchestra.run.trial110.stdout:(13/37): python3-rgw-20.3.0-5145.g5fdb7d8f.el9. 703 kB/s | 100 kB 00:00 2026-02-12T12:08:08.777 INFO:teuthology.orchestra.run.trial110.stdout:(14/37): mailcap-2.1.49-5.el9.noarch.rpm 1.7 MB/s | 33 kB 00:00 2026-02-12T12:08:08.804 INFO:teuthology.orchestra.run.trial110.stdout:(15/37): nvme-cli-2.13-1.el9.x86_64.rpm 40 MB/s | 1.1 MB 00:00 2026-02-12T12:08:08.813 INFO:teuthology.orchestra.run.trial110.stdout:(16/37): pciutils-3.7.0-7.el9.x86_64.rpm 11 MB/s | 93 kB 00:00 2026-02-12T12:08:08.915 INFO:teuthology.orchestra.run.trial110.stdout:(17/37): boost-program-options-1.75.0-13.el9.x8 1.0 MB/s | 104 kB 00:00 2026-02-12T12:08:08.959 INFO:teuthology.orchestra.run.trial110.stdout:(18/37): libnbd-1.20.3-4.el9.x86_64.rpm 3.7 MB/s | 164 kB 00:00 2026-02-12T12:08:08.986 INFO:teuthology.orchestra.run.trial110.stdout:(19/37): libpmemobj-1.12.1-1.el9.x86_64.rpm 5.8 MB/s | 160 kB 00:00 2026-02-12T12:08:09.008 INFO:teuthology.orchestra.run.trial110.stdout:(20/37): librabbitmq-0.11.0-7.el9.x86_64.rpm 2.0 MB/s | 45 kB 00:00 2026-02-12T12:08:09.124 INFO:teuthology.orchestra.run.trial110.stdout:(21/37): librdkafka-1.6.1-102.el9.x86_64.rpm 5.6 MB/s | 662 kB 00:00 2026-02-12T12:08:09.148 INFO:teuthology.orchestra.run.trial110.stdout:(22/37): lttng-ust-2.12.0-6.el9.x86_64.rpm 12 MB/s | 292 kB 00:00 2026-02-12T12:08:09.171 INFO:teuthology.orchestra.run.trial110.stdout:(23/37): qatlib-25.08.0-2.el9.x86_64.rpm 10 MB/s | 240 kB 00:00 2026-02-12T12:08:09.195 INFO:teuthology.orchestra.run.trial110.stdout:(24/37): qatlib-service-25.08.0-2.el9.x86_64.rp 1.5 MB/s | 37 kB 00:00 2026-02-12T12:08:09.219 INFO:teuthology.orchestra.run.trial110.stdout:(25/37): qatzip-libs-1.3.1-1.el9.x86_64.rpm 2.7 MB/s | 66 kB 00:00 2026-02-12T12:08:09.456 INFO:teuthology.orchestra.run.trial110.stdout:(26/37): gperftools-libs-2.9.1-3.el9.x86_64.rpm 1.3 MB/s | 308 kB 00:00 2026-02-12T12:08:09.617 INFO:teuthology.orchestra.run.trial110.stdout:(27/37): libarrow-9.0.0-15.el9.x86_64.rpm 27 MB/s | 4.4 MB 00:00 2026-02-12T12:08:09.648 INFO:teuthology.orchestra.run.trial110.stdout:(28/37): libarrow-doc-9.0.0-15.el9.noarch.rpm 822 kB/s | 25 kB 00:00 2026-02-12T12:08:09.678 INFO:teuthology.orchestra.run.trial110.stdout:(29/37): liboath-2.6.12-1.el9.x86_64.rpm 1.6 MB/s | 49 kB 00:00 2026-02-12T12:08:09.709 INFO:teuthology.orchestra.run.trial110.stdout:(30/37): libunwind-1.6.2-1.el9.x86_64.rpm 2.2 MB/s | 67 kB 00:00 2026-02-12T12:08:09.743 INFO:teuthology.orchestra.run.trial110.stdout:(31/37): parquet-libs-9.0.0-15.el9.x86_64.rpm 24 MB/s | 838 kB 00:00 2026-02-12T12:08:09.774 INFO:teuthology.orchestra.run.trial110.stdout:(32/37): re2-20211101-20.el9.x86_64.rpm 6.0 MB/s | 191 kB 00:00 2026-02-12T12:08:09.813 INFO:teuthology.orchestra.run.trial110.stdout:(33/37): thrift-0.15.0-4.el9.x86_64.rpm 41 MB/s | 1.6 MB 00:00 2026-02-12T12:08:11.410 INFO:teuthology.orchestra.run.trial110.stdout:(34/37): librgw2-20.3.0-5145.g5fdb7d8f.el9.x86_ 479 kB/s | 5.6 MB 00:12 2026-02-12T12:08:14.061 INFO:teuthology.orchestra.run.trial110.stdout:(35/37): librados2-20.3.0-5145.g5fdb7d8f.el9.x8 792 kB/s | 3.3 MB 00:04 2026-02-12T12:08:16.129 INFO:teuthology.orchestra.run.trial110.stdout:(36/37): librbd1-20.3.0-5145.g5fdb7d8f.el9.x86_ 583 kB/s | 2.7 MB 00:04 2026-02-12T12:08:23.353 INFO:teuthology.orchestra.run.trial110.stdout:(37/37): ceph-common-20.3.0-5145.g5fdb7d8f.el9. 578 kB/s | 21 MB 00:36 2026-02-12T12:08:23.355 INFO:teuthology.orchestra.run.trial110.stdout:-------------------------------------------------------------------------------- 2026-02-12T12:08:23.355 INFO:teuthology.orchestra.run.trial110.stdout:Total 1.6 MB/s | 61 MB 00:37 2026-02-12T12:08:23.452 INFO:teuthology.orchestra.run.trial110.stdout:Running transaction check 2026-02-12T12:08:23.471 INFO:teuthology.orchestra.run.trial110.stdout:Transaction check succeeded. 2026-02-12T12:08:23.471 INFO:teuthology.orchestra.run.trial110.stdout:Running transaction test 2026-02-12T12:08:23.718 INFO:teuthology.orchestra.run.trial110.stdout:Transaction test succeeded. 2026-02-12T12:08:23.718 INFO:teuthology.orchestra.run.trial110.stdout:Running transaction 2026-02-12T12:08:24.019 INFO:teuthology.orchestra.run.trial110.stdout: Preparing : 1/1 2026-02-12T12:08:24.161 INFO:teuthology.orchestra.run.trial110.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/39 2026-02-12T12:08:24.166 INFO:teuthology.orchestra.run.trial110.stdout: Upgrading : librados2-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2/39 2026-02-12T12:08:24.212 INFO:teuthology.orchestra.run.trial110.stdout: Running scriptlet: librados2-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2/39 2026-02-12T12:08:24.213 INFO:teuthology.orchestra.run.trial110.stdout: Installing : libcephfs2-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 3/39 2026-02-12T12:08:24.225 INFO:teuthology.orchestra.run.trial110.stdout: Running scriptlet: libcephfs2-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 3/39 2026-02-12T12:08:24.227 INFO:teuthology.orchestra.run.trial110.stdout: Installing : libcephfs-daemon-2:20.3.0-5145.g5fdb7d8f.el9.x86_6 4/39 2026-02-12T12:08:24.228 INFO:teuthology.orchestra.run.trial110.stdout: Installing : libcephfs-proxy2-2:20.3.0-5145.g5fdb7d8f.el9.x86_6 5/39 2026-02-12T12:08:24.255 INFO:teuthology.orchestra.run.trial110.stdout: Running scriptlet: libcephfs-proxy2-2:20.3.0-5145.g5fdb7d8f.el9.x86_6 5/39 2026-02-12T12:08:24.257 INFO:teuthology.orchestra.run.trial110.stdout: Installing : python3-rados-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 6/39 2026-02-12T12:08:24.265 INFO:teuthology.orchestra.run.trial110.stdout: Installing : liboath-2.6.12-1.el9.x86_64 7/39 2026-02-12T12:08:24.268 INFO:teuthology.orchestra.run.trial110.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/39 2026-02-12T12:08:24.271 INFO:teuthology.orchestra.run.trial110.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/39 2026-02-12T12:08:24.275 INFO:teuthology.orchestra.run.trial110.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/39 2026-02-12T12:08:24.286 INFO:teuthology.orchestra.run.trial110.stdout: Installing : python3-ceph-argparse-2:20.3.0-5145.g5fdb7d8f.el9. 11/39 2026-02-12T12:08:24.299 INFO:teuthology.orchestra.run.trial110.stdout: Installing : python3-cephfs-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 12/39 2026-02-12T12:08:24.299 INFO:teuthology.orchestra.run.trial110.stdout: Installing : libradosstriper1-2:20.3.0-5145.g5fdb7d8f.el9.x86_6 13/39 2026-02-12T12:08:24.323 INFO:teuthology.orchestra.run.trial110.stdout: Running scriptlet: libradosstriper1-2:20.3.0-5145.g5fdb7d8f.el9.x86_6 13/39 2026-02-12T12:08:24.327 INFO:teuthology.orchestra.run.trial110.stdout: Installing : thrift-0.15.0-4.el9.x86_64 14/39 2026-02-12T12:08:24.357 INFO:teuthology.orchestra.run.trial110.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/39 2026-02-12T12:08:24.361 INFO:teuthology.orchestra.run.trial110.stdout: Installing : libarrow-9.0.0-15.el9.x86_64 16/39 2026-02-12T12:08:24.367 INFO:teuthology.orchestra.run.trial110.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/39 2026-02-12T12:08:24.369 INFO:teuthology.orchestra.run.trial110.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/39 2026-02-12T12:08:24.374 INFO:teuthology.orchestra.run.trial110.stdout: Installing : libarrow-doc-9.0.0-15.el9.noarch 19/39 2026-02-12T12:08:24.488 INFO:teuthology.orchestra.run.trial110.stdout: Installing : libnbd-1.20.3-4.el9.x86_64 20/39 2026-02-12T12:08:24.489 INFO:teuthology.orchestra.run.trial110.stdout: Upgrading : librbd1-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 21/39 2026-02-12T12:08:24.515 INFO:teuthology.orchestra.run.trial110.stdout: Running scriptlet: librbd1-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 21/39 2026-02-12T12:08:24.519 INFO:teuthology.orchestra.run.trial110.stdout: Installing : python3-rbd-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 22/39 2026-02-12T12:08:24.525 INFO:teuthology.orchestra.run.trial110.stdout: Installing : boost-program-options-1.75.0-13.el9.x86_64 23/39 2026-02-12T12:08:24.774 INFO:teuthology.orchestra.run.trial110.stdout: Installing : parquet-libs-9.0.0-15.el9.x86_64 24/39 2026-02-12T12:08:24.775 INFO:teuthology.orchestra.run.trial110.stdout: Installing : librgw2-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 25/39 2026-02-12T12:08:24.791 INFO:teuthology.orchestra.run.trial110.stdout: Running scriptlet: librgw2-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 25/39 2026-02-12T12:08:24.795 INFO:teuthology.orchestra.run.trial110.stdout: Installing : python3-rgw-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 26/39 2026-02-12T12:08:24.799 INFO:teuthology.orchestra.run.trial110.stdout: Installing : pciutils-3.7.0-7.el9.x86_64 27/39 2026-02-12T12:08:24.801 INFO:teuthology.orchestra.run.trial110.stdout: Installing : qatlib-25.08.0-2.el9.x86_64 28/39 2026-02-12T12:08:24.802 INFO:teuthology.orchestra.run.trial110.stdout: Installing : qatlib-service-25.08.0-2.el9.x86_64 29/39 2026-02-12T12:08:24.820 INFO:teuthology.orchestra.run.trial110.stdout: Running scriptlet: qatlib-service-25.08.0-2.el9.x86_64 29/39 2026-02-12T12:08:24.932 INFO:teuthology.orchestra.run.trial110.stdout: Installing : qatzip-libs-1.3.1-1.el9.x86_64 30/39 2026-02-12T12:08:24.934 INFO:teuthology.orchestra.run.trial110.stdout: Installing : nvme-cli-2.13-1.el9.x86_64 31/39 2026-02-12T12:08:25.216 INFO:teuthology.orchestra.run.trial110.stdout: Running scriptlet: nvme-cli-2.13-1.el9.x86_64 31/39 2026-02-12T12:08:25.216 INFO:teuthology.orchestra.run.trial110.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service → /usr/lib/systemd/system/nvmefc-boot-connections.service. 2026-02-12T12:08:25.216 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:08:25.258 INFO:teuthology.orchestra.run.trial110.stdout: Installing : mailcap-2.1.49-5.el9.noarch 32/39 2026-02-12T12:08:25.261 INFO:teuthology.orchestra.run.trial110.stdout: Installing : python3-ceph-common-2:20.3.0-5145.g5fdb7d8f.el9.x8 33/39 2026-02-12T12:08:26.207 INFO:teuthology.orchestra.run.trial110.stdout: Running scriptlet: ceph-common-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 34/39 2026-02-12T12:08:26.210 INFO:teuthology.orchestra.run.trial110.stdout: Installing : ceph-common-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 34/39 2026-02-12T12:08:26.472 INFO:teuthology.orchestra.run.trial110.stdout: Running scriptlet: ceph-common-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 34/39 2026-02-12T12:08:26.475 INFO:teuthology.orchestra.run.trial110.stdout: Installing : ceph-base-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 35/39 2026-02-12T12:08:26.512 INFO:teuthology.orchestra.run.trial110.stdout: Running scriptlet: ceph-base-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 35/39 2026-02-12T12:08:26.512 INFO:teuthology.orchestra.run.trial110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2026-02-12T12:08:26.512 INFO:teuthology.orchestra.run.trial110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2026-02-12T12:08:26.512 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:08:26.515 INFO:teuthology.orchestra.run.trial110.stdout: Installing : ceph-selinux-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 36/39 2026-02-12T12:08:31.019 INFO:teuthology.orchestra.run.trial110.stdout: Running scriptlet: ceph-selinux-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 36/39 2026-02-12T12:08:31.019 INFO:teuthology.orchestra.run.trial110.stdout:skipping the directory /sys 2026-02-12T12:08:31.019 INFO:teuthology.orchestra.run.trial110.stdout:skipping the directory /proc 2026-02-12T12:08:31.019 INFO:teuthology.orchestra.run.trial110.stdout:skipping the directory /mnt 2026-02-12T12:08:31.020 INFO:teuthology.orchestra.run.trial110.stdout:skipping the directory /var/tmp 2026-02-12T12:08:31.020 INFO:teuthology.orchestra.run.trial110.stdout:skipping the directory /home 2026-02-12T12:08:31.020 INFO:teuthology.orchestra.run.trial110.stdout:skipping the directory /root 2026-02-12T12:08:31.020 INFO:teuthology.orchestra.run.trial110.stdout:skipping the directory /tmp 2026-02-12T12:08:31.020 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:08:31.512 INFO:teuthology.orchestra.run.trial110.stdout: Installing : ceph-radosgw-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 37/39 2026-02-12T12:08:31.535 INFO:teuthology.orchestra.run.trial110.stdout: Running scriptlet: ceph-radosgw-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 37/39 2026-02-12T12:08:31.536 INFO:teuthology.orchestra.run.trial110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-12T12:08:31.536 INFO:teuthology.orchestra.run.trial110.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2026-02-12T12:08:31.536 INFO:teuthology.orchestra.run.trial110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-12T12:08:31.536 INFO:teuthology.orchestra.run.trial110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-12T12:08:31.536 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:08:31.538 INFO:teuthology.orchestra.run.trial110.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 38/39 2026-02-12T12:08:31.550 INFO:teuthology.orchestra.run.trial110.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 38/39 2026-02-12T12:08:31.550 INFO:teuthology.orchestra.run.trial110.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 39/39 2026-02-12T12:08:31.603 INFO:teuthology.orchestra.run.trial110.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 39/39 2026-02-12T12:08:31.603 INFO:teuthology.orchestra.run.trial110.stdout:Creating group 'qat' with GID 992. 2026-02-12T12:08:31.603 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:08:31.879 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : ceph-base-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 1/39 2026-02-12T12:08:31.879 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : ceph-common-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2/39 2026-02-12T12:08:31.879 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : ceph-radosgw-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 3/39 2026-02-12T12:08:31.879 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : ceph-selinux-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 4/39 2026-02-12T12:08:31.880 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : libcephfs-daemon-2:20.3.0-5145.g5fdb7d8f.el9.x86_6 5/39 2026-02-12T12:08:31.880 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : libcephfs-proxy2-2:20.3.0-5145.g5fdb7d8f.el9.x86_6 6/39 2026-02-12T12:08:31.880 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : libcephfs2-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 7/39 2026-02-12T12:08:31.880 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : libradosstriper1-2:20.3.0-5145.g5fdb7d8f.el9.x86_6 8/39 2026-02-12T12:08:31.880 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : librgw2-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 9/39 2026-02-12T12:08:31.880 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : python3-ceph-argparse-2:20.3.0-5145.g5fdb7d8f.el9. 10/39 2026-02-12T12:08:31.880 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : python3-ceph-common-2:20.3.0-5145.g5fdb7d8f.el9.x8 11/39 2026-02-12T12:08:31.880 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : python3-cephfs-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 12/39 2026-02-12T12:08:31.880 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : python3-rados-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 13/39 2026-02-12T12:08:31.880 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : python3-rbd-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 14/39 2026-02-12T12:08:31.880 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : python3-rgw-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 15/39 2026-02-12T12:08:31.880 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : mailcap-2.1.49-5.el9.noarch 16/39 2026-02-12T12:08:31.881 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : nvme-cli-2.13-1.el9.x86_64 17/39 2026-02-12T12:08:31.881 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : pciutils-3.7.0-7.el9.x86_64 18/39 2026-02-12T12:08:31.881 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : boost-program-options-1.75.0-13.el9.x86_64 19/39 2026-02-12T12:08:31.881 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : libnbd-1.20.3-4.el9.x86_64 20/39 2026-02-12T12:08:31.881 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 21/39 2026-02-12T12:08:31.881 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 22/39 2026-02-12T12:08:31.881 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 23/39 2026-02-12T12:08:31.881 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 24/39 2026-02-12T12:08:31.881 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : qatlib-25.08.0-2.el9.x86_64 25/39 2026-02-12T12:08:31.881 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : qatlib-service-25.08.0-2.el9.x86_64 26/39 2026-02-12T12:08:31.881 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : qatzip-libs-1.3.1-1.el9.x86_64 27/39 2026-02-12T12:08:31.881 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 28/39 2026-02-12T12:08:31.881 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : libarrow-9.0.0-15.el9.x86_64 29/39 2026-02-12T12:08:31.882 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : libarrow-doc-9.0.0-15.el9.noarch 30/39 2026-02-12T12:08:31.882 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : liboath-2.6.12-1.el9.x86_64 31/39 2026-02-12T12:08:31.882 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 32/39 2026-02-12T12:08:31.882 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : parquet-libs-9.0.0-15.el9.x86_64 33/39 2026-02-12T12:08:31.882 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : re2-1:20211101-20.el9.x86_64 34/39 2026-02-12T12:08:31.882 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : thrift-0.15.0-4.el9.x86_64 35/39 2026-02-12T12:08:31.882 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : librados2-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 36/39 2026-02-12T12:08:31.882 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 37/39 2026-02-12T12:08:31.882 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : librbd1-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 38/39 2026-02-12T12:08:31.940 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 39/39 2026-02-12T12:08:31.940 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:08:31.940 INFO:teuthology.orchestra.run.trial110.stdout:Upgraded: 2026-02-12T12:08:31.940 INFO:teuthology.orchestra.run.trial110.stdout: librados2-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:08:31.941 INFO:teuthology.orchestra.run.trial110.stdout: librbd1-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:08:31.941 INFO:teuthology.orchestra.run.trial110.stdout:Installed: 2026-02-12T12:08:31.941 INFO:teuthology.orchestra.run.trial110.stdout: boost-program-options-1.75.0-13.el9.x86_64 2026-02-12T12:08:31.941 INFO:teuthology.orchestra.run.trial110.stdout: ceph-base-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:08:31.941 INFO:teuthology.orchestra.run.trial110.stdout: ceph-common-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:08:31.941 INFO:teuthology.orchestra.run.trial110.stdout: ceph-radosgw-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:08:31.941 INFO:teuthology.orchestra.run.trial110.stdout: ceph-selinux-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:08:31.941 INFO:teuthology.orchestra.run.trial110.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2026-02-12T12:08:31.941 INFO:teuthology.orchestra.run.trial110.stdout: libarrow-9.0.0-15.el9.x86_64 2026-02-12T12:08:31.941 INFO:teuthology.orchestra.run.trial110.stdout: libarrow-doc-9.0.0-15.el9.noarch 2026-02-12T12:08:31.941 INFO:teuthology.orchestra.run.trial110.stdout: libcephfs-daemon-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:08:31.941 INFO:teuthology.orchestra.run.trial110.stdout: libcephfs-proxy2-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:08:31.942 INFO:teuthology.orchestra.run.trial110.stdout: libcephfs2-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:08:31.942 INFO:teuthology.orchestra.run.trial110.stdout: libnbd-1.20.3-4.el9.x86_64 2026-02-12T12:08:31.942 INFO:teuthology.orchestra.run.trial110.stdout: liboath-2.6.12-1.el9.x86_64 2026-02-12T12:08:31.942 INFO:teuthology.orchestra.run.trial110.stdout: libpmemobj-1.12.1-1.el9.x86_64 2026-02-12T12:08:31.942 INFO:teuthology.orchestra.run.trial110.stdout: librabbitmq-0.11.0-7.el9.x86_64 2026-02-12T12:08:31.942 INFO:teuthology.orchestra.run.trial110.stdout: libradosstriper1-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:08:31.942 INFO:teuthology.orchestra.run.trial110.stdout: librdkafka-1.6.1-102.el9.x86_64 2026-02-12T12:08:31.942 INFO:teuthology.orchestra.run.trial110.stdout: librgw2-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:08:31.942 INFO:teuthology.orchestra.run.trial110.stdout: libunwind-1.6.2-1.el9.x86_64 2026-02-12T12:08:31.942 INFO:teuthology.orchestra.run.trial110.stdout: lttng-ust-2.12.0-6.el9.x86_64 2026-02-12T12:08:31.942 INFO:teuthology.orchestra.run.trial110.stdout: mailcap-2.1.49-5.el9.noarch 2026-02-12T12:08:31.942 INFO:teuthology.orchestra.run.trial110.stdout: nvme-cli-2.13-1.el9.x86_64 2026-02-12T12:08:31.943 INFO:teuthology.orchestra.run.trial110.stdout: parquet-libs-9.0.0-15.el9.x86_64 2026-02-12T12:08:31.943 INFO:teuthology.orchestra.run.trial110.stdout: pciutils-3.7.0-7.el9.x86_64 2026-02-12T12:08:31.943 INFO:teuthology.orchestra.run.trial110.stdout: python3-ceph-argparse-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:08:31.943 INFO:teuthology.orchestra.run.trial110.stdout: python3-ceph-common-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:08:31.943 INFO:teuthology.orchestra.run.trial110.stdout: python3-cephfs-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:08:31.943 INFO:teuthology.orchestra.run.trial110.stdout: python3-rados-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:08:31.943 INFO:teuthology.orchestra.run.trial110.stdout: python3-rbd-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:08:31.943 INFO:teuthology.orchestra.run.trial110.stdout: python3-rgw-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:08:31.943 INFO:teuthology.orchestra.run.trial110.stdout: qatlib-25.08.0-2.el9.x86_64 2026-02-12T12:08:31.943 INFO:teuthology.orchestra.run.trial110.stdout: qatlib-service-25.08.0-2.el9.x86_64 2026-02-12T12:08:31.943 INFO:teuthology.orchestra.run.trial110.stdout: qatzip-libs-1.3.1-1.el9.x86_64 2026-02-12T12:08:31.943 INFO:teuthology.orchestra.run.trial110.stdout: re2-1:20211101-20.el9.x86_64 2026-02-12T12:08:31.943 INFO:teuthology.orchestra.run.trial110.stdout: thrift-0.15.0-4.el9.x86_64 2026-02-12T12:08:31.944 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:08:31.944 INFO:teuthology.orchestra.run.trial110.stdout:Complete! 2026-02-12T12:08:31.981 DEBUG:teuthology.orchestra.run.trial110:> sudo yum -y install ceph-test 2026-02-12T12:08:32.279 INFO:teuthology.orchestra.run.trial110.stdout:Last metadata expiration check: 0:00:48 ago on Thu 12 Feb 2026 12:07:44 PM UTC. 2026-02-12T12:08:32.362 INFO:teuthology.orchestra.run.trial110.stdout:Dependencies resolved. 2026-02-12T12:08:32.363 INFO:teuthology.orchestra.run.trial110.stdout:================================================================================ 2026-02-12T12:08:32.363 INFO:teuthology.orchestra.run.trial110.stdout: Package Arch Version Repository Size 2026-02-12T12:08:32.363 INFO:teuthology.orchestra.run.trial110.stdout:================================================================================ 2026-02-12T12:08:32.363 INFO:teuthology.orchestra.run.trial110.stdout:Installing: 2026-02-12T12:08:32.364 INFO:teuthology.orchestra.run.trial110.stdout: ceph-test x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 126 M 2026-02-12T12:08:32.364 INFO:teuthology.orchestra.run.trial110.stdout:Installing dependencies: 2026-02-12T12:08:32.364 INFO:teuthology.orchestra.run.trial110.stdout: hwloc-libs x86_64 2.4.1-6.el9 baseos 2.1 M 2026-02-12T12:08:32.364 INFO:teuthology.orchestra.run.trial110.stdout: libasan8 x86_64 15.1.1-2.5.el9 appstream 523 k 2026-02-12T12:08:32.364 INFO:teuthology.orchestra.run.trial110.stdout: libatomic x86_64 11.5.0-14.el9 baseos 23 k 2026-02-12T12:08:32.364 INFO:teuthology.orchestra.run.trial110.stdout: libcephsqlite x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 153 k 2026-02-12T12:08:32.364 INFO:teuthology.orchestra.run.trial110.stdout: libubsan x86_64 11.5.0-14.el9 appstream 176 k 2026-02-12T12:08:32.364 INFO:teuthology.orchestra.run.trial110.stdout: libxslt x86_64 1.1.34-12.el9 appstream 233 k 2026-02-12T12:08:32.364 INFO:teuthology.orchestra.run.trial110.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2026-02-12T12:08:32.364 INFO:teuthology.orchestra.run.trial110.stdout: protobuf x86_64 3.14.0-17.el9 appstream 1.0 M 2026-02-12T12:08:32.364 INFO:teuthology.orchestra.run.trial110.stdout: socat x86_64 1.7.4.1-8.el9 appstream 303 k 2026-02-12T12:08:32.364 INFO:teuthology.orchestra.run.trial110.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2026-02-12T12:08:32.365 INFO:teuthology.orchestra.run.trial110.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2026-02-12T12:08:32.365 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:08:32.365 INFO:teuthology.orchestra.run.trial110.stdout:Transaction Summary 2026-02-12T12:08:32.365 INFO:teuthology.orchestra.run.trial110.stdout:================================================================================ 2026-02-12T12:08:32.365 INFO:teuthology.orchestra.run.trial110.stdout:Install 12 Packages 2026-02-12T12:08:32.365 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:08:32.366 INFO:teuthology.orchestra.run.trial110.stdout:Total download size: 131 M 2026-02-12T12:08:32.366 INFO:teuthology.orchestra.run.trial110.stdout:Installed size: 977 M 2026-02-12T12:08:32.366 INFO:teuthology.orchestra.run.trial110.stdout:Downloading Packages: 2026-02-12T12:08:33.217 INFO:teuthology.orchestra.run.trial110.stdout:(1/12): libcephsqlite-20.3.0-5145.g5fdb7d8f.el9 242 kB/s | 153 kB 00:00 2026-02-12T12:08:33.611 INFO:teuthology.orchestra.run.trial110.stdout:(2/12): libatomic-11.5.0-14.el9.x86_64.rpm 58 kB/s | 23 kB 00:00 2026-02-12T12:08:33.768 INFO:teuthology.orchestra.run.trial110.stdout:(3/12): lksctp-tools-1.0.19-2.el9.x86_64.rpm 599 kB/s | 94 kB 00:00 2026-02-12T12:08:34.427 INFO:teuthology.orchestra.run.trial110.stdout:(4/12): libasan8-15.1.1-2.5.el9.x86_64.rpm 795 kB/s | 523 kB 00:00 2026-02-12T12:08:34.680 INFO:teuthology.orchestra.run.trial110.stdout:(5/12): libubsan-11.5.0-14.el9.x86_64.rpm 696 kB/s | 176 kB 00:00 2026-02-12T12:08:35.009 INFO:teuthology.orchestra.run.trial110.stdout:(6/12): libxslt-1.1.34-12.el9.x86_64.rpm 707 kB/s | 233 kB 00:00 2026-02-12T12:08:35.382 INFO:teuthology.orchestra.run.trial110.stdout:(7/12): hwloc-libs-2.4.1-6.el9.x86_64.rpm 768 kB/s | 2.1 MB 00:02 2026-02-12T12:08:35.786 INFO:teuthology.orchestra.run.trial110.stdout:(8/12): socat-1.7.4.1-8.el9.x86_64.rpm 751 kB/s | 303 kB 00:00 2026-02-12T12:08:35.883 INFO:teuthology.orchestra.run.trial110.stdout:(9/12): xmlstarlet-1.6.1-20.el9.x86_64.rpm 655 kB/s | 64 kB 00:00 2026-02-12T12:08:35.912 INFO:teuthology.orchestra.run.trial110.stdout:(10/12): yaml-cpp-0.6.3-5.el9.x86_64.rpm 4.1 MB/s | 122 kB 00:00 2026-02-12T12:08:36.180 INFO:teuthology.orchestra.run.trial110.stdout:(11/12): protobuf-3.14.0-17.el9.x86_64.rpm 878 kB/s | 1.0 MB 00:01 2026-02-12T12:13:53.121 INFO:teuthology.orchestra.run.trial110.stdout:(12/12): ceph-test-20.3.0-5145.g5fdb7d8f.el9.x8 403 kB/s | 126 MB 05:20 2026-02-12T12:13:53.124 INFO:teuthology.orchestra.run.trial110.stdout:-------------------------------------------------------------------------------- 2026-02-12T12:13:53.124 INFO:teuthology.orchestra.run.trial110.stdout:Total 418 kB/s | 131 MB 05:20 2026-02-12T12:13:53.191 INFO:teuthology.orchestra.run.trial110.stdout:Running transaction check 2026-02-12T12:13:53.202 INFO:teuthology.orchestra.run.trial110.stdout:Transaction check succeeded. 2026-02-12T12:13:53.203 INFO:teuthology.orchestra.run.trial110.stdout:Running transaction test 2026-02-12T12:13:53.631 INFO:teuthology.orchestra.run.trial110.stdout:Transaction test succeeded. 2026-02-12T12:13:53.631 INFO:teuthology.orchestra.run.trial110.stdout:Running transaction 2026-02-12T12:13:54.087 INFO:teuthology.orchestra.run.trial110.stdout: Preparing : 1/1 2026-02-12T12:13:54.096 INFO:teuthology.orchestra.run.trial110.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/12 2026-02-12T12:13:54.106 INFO:teuthology.orchestra.run.trial110.stdout: Installing : socat-1.7.4.1-8.el9.x86_64 2/12 2026-02-12T12:13:54.111 INFO:teuthology.orchestra.run.trial110.stdout: Installing : protobuf-3.14.0-17.el9.x86_64 3/12 2026-02-12T12:13:54.115 INFO:teuthology.orchestra.run.trial110.stdout: Installing : libxslt-1.1.34-12.el9.x86_64 4/12 2026-02-12T12:13:54.117 INFO:teuthology.orchestra.run.trial110.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 5/12 2026-02-12T12:13:54.122 INFO:teuthology.orchestra.run.trial110.stdout: Installing : libubsan-11.5.0-14.el9.x86_64 6/12 2026-02-12T12:13:54.123 INFO:teuthology.orchestra.run.trial110.stdout: Installing : libasan8-15.1.1-2.5.el9.x86_64 7/12 2026-02-12T12:13:54.140 INFO:teuthology.orchestra.run.trial110.stdout: Running scriptlet: libasan8-15.1.1-2.5.el9.x86_64 7/12 2026-02-12T12:13:54.142 INFO:teuthology.orchestra.run.trial110.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 8/12 2026-02-12T12:13:54.153 INFO:teuthology.orchestra.run.trial110.stdout: Installing : libatomic-11.5.0-14.el9.x86_64 9/12 2026-02-12T12:13:54.162 INFO:teuthology.orchestra.run.trial110.stdout: Installing : hwloc-libs-2.4.1-6.el9.x86_64 10/12 2026-02-12T12:13:54.163 INFO:teuthology.orchestra.run.trial110.stdout: Installing : libcephsqlite-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 11/12 2026-02-12T12:14:01.250 INFO:teuthology.orchestra.run.trial110.stdout: Running scriptlet: libcephsqlite-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 11/12 2026-02-12T12:14:01.268 INFO:teuthology.orchestra.run.trial110.stdout: Installing : ceph-test-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 12/12 2026-02-12T12:14:02.026 INFO:teuthology.orchestra.run.trial110.stdout: Running scriptlet: ceph-test-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 12/12 2026-02-12T12:14:02.026 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : ceph-test-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 1/12 2026-02-12T12:14:02.026 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : libcephsqlite-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2/12 2026-02-12T12:14:02.026 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : hwloc-libs-2.4.1-6.el9.x86_64 3/12 2026-02-12T12:14:02.026 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : libatomic-11.5.0-14.el9.x86_64 4/12 2026-02-12T12:14:02.026 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 5/12 2026-02-12T12:14:02.026 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : libasan8-15.1.1-2.5.el9.x86_64 6/12 2026-02-12T12:14:02.026 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : libubsan-11.5.0-14.el9.x86_64 7/12 2026-02-12T12:14:02.027 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : libxslt-1.1.34-12.el9.x86_64 8/12 2026-02-12T12:14:02.027 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : protobuf-3.14.0-17.el9.x86_64 9/12 2026-02-12T12:14:02.027 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : socat-1.7.4.1-8.el9.x86_64 10/12 2026-02-12T12:14:02.027 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 11/12 2026-02-12T12:14:02.079 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 12/12 2026-02-12T12:14:02.079 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:14:02.079 INFO:teuthology.orchestra.run.trial110.stdout:Installed: 2026-02-12T12:14:02.079 INFO:teuthology.orchestra.run.trial110.stdout: ceph-test-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:14:02.079 INFO:teuthology.orchestra.run.trial110.stdout: hwloc-libs-2.4.1-6.el9.x86_64 2026-02-12T12:14:02.079 INFO:teuthology.orchestra.run.trial110.stdout: libasan8-15.1.1-2.5.el9.x86_64 2026-02-12T12:14:02.079 INFO:teuthology.orchestra.run.trial110.stdout: libatomic-11.5.0-14.el9.x86_64 2026-02-12T12:14:02.079 INFO:teuthology.orchestra.run.trial110.stdout: libcephsqlite-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:14:02.079 INFO:teuthology.orchestra.run.trial110.stdout: libubsan-11.5.0-14.el9.x86_64 2026-02-12T12:14:02.080 INFO:teuthology.orchestra.run.trial110.stdout: libxslt-1.1.34-12.el9.x86_64 2026-02-12T12:14:02.080 INFO:teuthology.orchestra.run.trial110.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2026-02-12T12:14:02.080 INFO:teuthology.orchestra.run.trial110.stdout: protobuf-3.14.0-17.el9.x86_64 2026-02-12T12:14:02.080 INFO:teuthology.orchestra.run.trial110.stdout: socat-1.7.4.1-8.el9.x86_64 2026-02-12T12:14:02.080 INFO:teuthology.orchestra.run.trial110.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2026-02-12T12:14:02.080 INFO:teuthology.orchestra.run.trial110.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2026-02-12T12:14:02.080 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:14:02.080 INFO:teuthology.orchestra.run.trial110.stdout:Complete! 2026-02-12T12:14:02.121 DEBUG:teuthology.orchestra.run.trial110:> sudo yum -y install ceph 2026-02-12T12:14:02.425 INFO:teuthology.orchestra.run.trial110.stdout:Last metadata expiration check: 0:06:18 ago on Thu 12 Feb 2026 12:07:44 PM UTC. 2026-02-12T12:14:02.511 INFO:teuthology.orchestra.run.trial110.stdout:Dependencies resolved. 2026-02-12T12:14:02.514 INFO:teuthology.orchestra.run.trial110.stdout:======================================================================================= 2026-02-12T12:14:02.514 INFO:teuthology.orchestra.run.trial110.stdout: Package Arch Version Repository Size 2026-02-12T12:14:02.514 INFO:teuthology.orchestra.run.trial110.stdout:======================================================================================= 2026-02-12T12:14:02.514 INFO:teuthology.orchestra.run.trial110.stdout:Installing: 2026-02-12T12:14:02.515 INFO:teuthology.orchestra.run.trial110.stdout: ceph x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 6.5 k 2026-02-12T12:14:02.515 INFO:teuthology.orchestra.run.trial110.stdout:Installing dependencies: 2026-02-12T12:14:02.515 INFO:teuthology.orchestra.run.trial110.stdout: abseil-cpp x86_64 20211102.0-4.el9 epel 551 k 2026-02-12T12:14:02.515 INFO:teuthology.orchestra.run.trial110.stdout: ceph-grafana-dashboards noarch 2:20.3.0-5145.g5fdb7d8f.el9 ceph-noarch 57 k 2026-02-12T12:14:02.515 INFO:teuthology.orchestra.run.trial110.stdout: ceph-mds x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 2.0 M 2026-02-12T12:14:02.515 INFO:teuthology.orchestra.run.trial110.stdout: ceph-mgr x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 937 k 2026-02-12T12:14:02.515 INFO:teuthology.orchestra.run.trial110.stdout: ceph-mgr-modules-core noarch 2:20.3.0-5145.g5fdb7d8f.el9 ceph-noarch 297 k 2026-02-12T12:14:02.515 INFO:teuthology.orchestra.run.trial110.stdout: ceph-mon x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 4.8 M 2026-02-12T12:14:02.515 INFO:teuthology.orchestra.run.trial110.stdout: ceph-osd x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 163 k 2026-02-12T12:14:02.515 INFO:teuthology.orchestra.run.trial110.stdout: ceph-osd-classic x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 15 M 2026-02-12T12:14:02.515 INFO:teuthology.orchestra.run.trial110.stdout: ceph-osd-crimson x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 87 M 2026-02-12T12:14:02.516 INFO:teuthology.orchestra.run.trial110.stdout: ceph-prometheus-alerts noarch 2:20.3.0-5145.g5fdb7d8f.el9 ceph-noarch 17 k 2026-02-12T12:14:02.516 INFO:teuthology.orchestra.run.trial110.stdout: cephadm noarch 2:20.3.0-5145.g5fdb7d8f.el9 ceph-noarch 1.0 M 2026-02-12T12:14:02.516 INFO:teuthology.orchestra.run.trial110.stdout: cryptsetup x86_64 2.8.1-2.el9 baseos 350 k 2026-02-12T12:14:02.516 INFO:teuthology.orchestra.run.trial110.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2026-02-12T12:14:02.516 INFO:teuthology.orchestra.run.trial110.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2026-02-12T12:14:02.516 INFO:teuthology.orchestra.run.trial110.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2026-02-12T12:14:02.516 INFO:teuthology.orchestra.run.trial110.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2026-02-12T12:14:02.516 INFO:teuthology.orchestra.run.trial110.stdout: ledmon-libs x86_64 1.1.0-3.el9 baseos 40 k 2026-02-12T12:14:02.516 INFO:teuthology.orchestra.run.trial110.stdout: libconfig x86_64 1.7.2-9.el9 baseos 72 k 2026-02-12T12:14:02.516 INFO:teuthology.orchestra.run.trial110.stdout: libgfortran x86_64 11.5.0-14.el9 baseos 794 k 2026-02-12T12:14:02.516 INFO:teuthology.orchestra.run.trial110.stdout: libquadmath x86_64 11.5.0-14.el9 baseos 184 k 2026-02-12T12:14:02.516 INFO:teuthology.orchestra.run.trial110.stdout: libstoragemgmt x86_64 1.10.1-1.el9 appstream 246 k 2026-02-12T12:14:02.517 INFO:teuthology.orchestra.run.trial110.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2026-02-12T12:14:02.517 INFO:teuthology.orchestra.run.trial110.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2026-02-12T12:14:02.517 INFO:teuthology.orchestra.run.trial110.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2026-02-12T12:14:02.517 INFO:teuthology.orchestra.run.trial110.stdout: openblas x86_64 0.3.29-1.el9 appstream 42 k 2026-02-12T12:14:02.517 INFO:teuthology.orchestra.run.trial110.stdout: openblas-openmp x86_64 0.3.29-1.el9 appstream 5.3 M 2026-02-12T12:14:02.517 INFO:teuthology.orchestra.run.trial110.stdout: protobuf-compiler x86_64 3.14.0-17.el9 crb 862 k 2026-02-12T12:14:02.517 INFO:teuthology.orchestra.run.trial110.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2026-02-12T12:14:02.517 INFO:teuthology.orchestra.run.trial110.stdout: python3-autocommand noarch 2.2.2-8.el9 epel 29 k 2026-02-12T12:14:02.517 INFO:teuthology.orchestra.run.trial110.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2026-02-12T12:14:02.517 INFO:teuthology.orchestra.run.trial110.stdout: python3-backports-tarfile noarch 1.2.0-1.el9 epel 60 k 2026-02-12T12:14:02.517 INFO:teuthology.orchestra.run.trial110.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2026-02-12T12:14:02.517 INFO:teuthology.orchestra.run.trial110.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2026-02-12T12:14:02.517 INFO:teuthology.orchestra.run.trial110.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2026-02-12T12:14:02.518 INFO:teuthology.orchestra.run.trial110.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2026-02-12T12:14:02.518 INFO:teuthology.orchestra.run.trial110.stdout: python3-cheroot noarch 10.0.1-4.el9 epel 173 k 2026-02-12T12:14:02.518 INFO:teuthology.orchestra.run.trial110.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2026-02-12T12:14:02.518 INFO:teuthology.orchestra.run.trial110.stdout: python3-cryptography x86_64 36.0.1-5.el9 baseos 1.2 M 2026-02-12T12:14:02.518 INFO:teuthology.orchestra.run.trial110.stdout: python3-devel x86_64 3.9.25-2.el9 appstream 244 k 2026-02-12T12:14:02.518 INFO:teuthology.orchestra.run.trial110.stdout: python3-google-auth noarch 1:2.45.0-1.el9 epel 254 k 2026-02-12T12:14:02.518 INFO:teuthology.orchestra.run.trial110.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2026-02-12T12:14:02.518 INFO:teuthology.orchestra.run.trial110.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2026-02-12T12:14:02.518 INFO:teuthology.orchestra.run.trial110.stdout: python3-isodate noarch 0.6.1-3.el9 epel 56 k 2026-02-12T12:14:02.518 INFO:teuthology.orchestra.run.trial110.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2026-02-12T12:14:02.518 INFO:teuthology.orchestra.run.trial110.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2026-02-12T12:14:02.518 INFO:teuthology.orchestra.run.trial110.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2026-02-12T12:14:02.518 INFO:teuthology.orchestra.run.trial110.stdout: python3-jaraco-context noarch 6.0.1-3.el9 epel 20 k 2026-02-12T12:14:02.519 INFO:teuthology.orchestra.run.trial110.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2026-02-12T12:14:02.519 INFO:teuthology.orchestra.run.trial110.stdout: python3-jaraco-text noarch 4.0.0-2.el9 epel 26 k 2026-02-12T12:14:02.519 INFO:teuthology.orchestra.run.trial110.stdout: python3-jinja2 noarch 2.11.3-8.el9 appstream 249 k 2026-02-12T12:14:02.519 INFO:teuthology.orchestra.run.trial110.stdout: python3-jmespath noarch 1.0.1-1.el9 appstream 48 k 2026-02-12T12:14:02.519 INFO:teuthology.orchestra.run.trial110.stdout: python3-kubernetes noarch 1:26.1.0-3.el9 epel 1.0 M 2026-02-12T12:14:02.519 INFO:teuthology.orchestra.run.trial110.stdout: python3-libstoragemgmt x86_64 1.10.1-1.el9 appstream 177 k 2026-02-12T12:14:02.519 INFO:teuthology.orchestra.run.trial110.stdout: python3-lxml x86_64 4.6.5-3.el9 appstream 1.2 M 2026-02-12T12:14:02.519 INFO:teuthology.orchestra.run.trial110.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2026-02-12T12:14:02.519 INFO:teuthology.orchestra.run.trial110.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2026-02-12T12:14:02.519 INFO:teuthology.orchestra.run.trial110.stdout: python3-msgpack x86_64 1.0.3-2.el9 epel 86 k 2026-02-12T12:14:02.519 INFO:teuthology.orchestra.run.trial110.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2026-02-12T12:14:02.519 INFO:teuthology.orchestra.run.trial110.stdout: python3-numpy x86_64 1:1.23.5-2.el9 appstream 6.1 M 2026-02-12T12:14:02.519 INFO:teuthology.orchestra.run.trial110.stdout: python3-numpy-f2py x86_64 1:1.23.5-2.el9 appstream 442 k 2026-02-12T12:14:02.520 INFO:teuthology.orchestra.run.trial110.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2026-02-12T12:14:02.520 INFO:teuthology.orchestra.run.trial110.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2026-02-12T12:14:02.520 INFO:teuthology.orchestra.run.trial110.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2026-02-12T12:14:02.520 INFO:teuthology.orchestra.run.trial110.stdout: python3-protobuf noarch 3.14.0-17.el9 appstream 267 k 2026-02-12T12:14:02.520 INFO:teuthology.orchestra.run.trial110.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2026-02-12T12:14:02.520 INFO:teuthology.orchestra.run.trial110.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2026-02-12T12:14:02.520 INFO:teuthology.orchestra.run.trial110.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2026-02-12T12:14:02.520 INFO:teuthology.orchestra.run.trial110.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2026-02-12T12:14:02.520 INFO:teuthology.orchestra.run.trial110.stdout: python3-pyparsing noarch 2.4.7-9.el9 baseos 150 k 2026-02-12T12:14:02.520 INFO:teuthology.orchestra.run.trial110.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2026-02-12T12:14:02.520 INFO:teuthology.orchestra.run.trial110.stdout: python3-requests noarch 2.25.1-10.el9 baseos 126 k 2026-02-12T12:14:02.520 INFO:teuthology.orchestra.run.trial110.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2026-02-12T12:14:02.520 INFO:teuthology.orchestra.run.trial110.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2026-02-12T12:14:02.521 INFO:teuthology.orchestra.run.trial110.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2026-02-12T12:14:02.521 INFO:teuthology.orchestra.run.trial110.stdout: python3-scipy x86_64 1.9.3-2.el9 appstream 19 M 2026-02-12T12:14:02.521 INFO:teuthology.orchestra.run.trial110.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2026-02-12T12:14:02.521 INFO:teuthology.orchestra.run.trial110.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2026-02-12T12:14:02.521 INFO:teuthology.orchestra.run.trial110.stdout: python3-typing-extensions noarch 4.15.0-1.el9 epel 86 k 2026-02-12T12:14:02.521 INFO:teuthology.orchestra.run.trial110.stdout: python3-urllib3 noarch 1.26.5-7.el9 baseos 218 k 2026-02-12T12:14:02.521 INFO:teuthology.orchestra.run.trial110.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2026-02-12T12:14:02.521 INFO:teuthology.orchestra.run.trial110.stdout: python3-xmlsec x86_64 1.3.13-1.el9 epel 48 k 2026-02-12T12:14:02.521 INFO:teuthology.orchestra.run.trial110.stdout: python3-xmltodict noarch 0.12.0-15.el9 epel 22 k 2026-02-12T12:14:02.521 INFO:teuthology.orchestra.run.trial110.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2026-02-12T12:14:02.521 INFO:teuthology.orchestra.run.trial110.stdout: unzip x86_64 6.0-59.el9 baseos 182 k 2026-02-12T12:14:02.521 INFO:teuthology.orchestra.run.trial110.stdout: xmlsec1 x86_64 1.2.29-13.el9 appstream 189 k 2026-02-12T12:14:02.521 INFO:teuthology.orchestra.run.trial110.stdout: xmlsec1-openssl x86_64 1.2.29-13.el9 appstream 90 k 2026-02-12T12:14:02.522 INFO:teuthology.orchestra.run.trial110.stdout: zip x86_64 3.0-35.el9 baseos 266 k 2026-02-12T12:14:02.522 INFO:teuthology.orchestra.run.trial110.stdout:Installing weak dependencies: 2026-02-12T12:14:02.522 INFO:teuthology.orchestra.run.trial110.stdout: ceph-mgr-cephadm noarch 2:20.3.0-5145.g5fdb7d8f.el9 ceph-noarch 183 k 2026-02-12T12:14:02.522 INFO:teuthology.orchestra.run.trial110.stdout: ceph-mgr-dashboard noarch 2:20.3.0-5145.g5fdb7d8f.el9 ceph-noarch 15 M 2026-02-12T12:14:02.522 INFO:teuthology.orchestra.run.trial110.stdout: ceph-mgr-diskprediction-local noarch 2:20.3.0-5145.g5fdb7d8f.el9 ceph-noarch 7.4 M 2026-02-12T12:14:02.522 INFO:teuthology.orchestra.run.trial110.stdout: ceph-mgr-k8sevents noarch 2:20.3.0-5145.g5fdb7d8f.el9 ceph-noarch 22 k 2026-02-12T12:14:02.522 INFO:teuthology.orchestra.run.trial110.stdout: ceph-mgr-rook noarch 2:20.3.0-5145.g5fdb7d8f.el9 ceph-noarch 50 k 2026-02-12T12:14:02.522 INFO:teuthology.orchestra.run.trial110.stdout: ceph-volume noarch 2:20.3.0-5145.g5fdb7d8f.el9 ceph-noarch 299 k 2026-02-12T12:14:02.522 INFO:teuthology.orchestra.run.trial110.stdout: python3-influxdb noarch 5.3.1-1.el9 epel 139 k 2026-02-12T12:14:02.522 INFO:teuthology.orchestra.run.trial110.stdout: python3-saml noarch 1.16.0-1.el9 epel 125 k 2026-02-12T12:14:02.522 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:14:02.523 INFO:teuthology.orchestra.run.trial110.stdout:Transaction Summary 2026-02-12T12:14:02.523 INFO:teuthology.orchestra.run.trial110.stdout:======================================================================================= 2026-02-12T12:14:02.523 INFO:teuthology.orchestra.run.trial110.stdout:Install 96 Packages 2026-02-12T12:14:02.523 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:14:02.524 INFO:teuthology.orchestra.run.trial110.stdout:Total download size: 189 M 2026-02-12T12:14:02.524 INFO:teuthology.orchestra.run.trial110.stdout:Installed size: 1.4 G 2026-02-12T12:14:02.524 INFO:teuthology.orchestra.run.trial110.stdout:Downloading Packages: 2026-02-12T12:14:03.313 INFO:teuthology.orchestra.run.trial110.stdout:(1/96): ceph-20.3.0-5145.g5fdb7d8f.el9.x86_64.r 23 kB/s | 6.5 kB 00:00 2026-02-12T12:14:05.207 INFO:teuthology.orchestra.run.trial110.stdout:(2/96): ceph-mon-20.3.0-5145.g5fdb7d8f.el9.x86_ 2.6 MB/s | 4.8 MB 00:01 2026-02-12T12:14:05.510 INFO:teuthology.orchestra.run.trial110.stdout:(3/96): ceph-osd-20.3.0-5145.g5fdb7d8f.el9.x86_ 542 kB/s | 163 kB 00:00 2026-02-12T12:14:05.770 INFO:teuthology.orchestra.run.trial110.stdout:(4/96): ceph-mgr-20.3.0-5145.g5fdb7d8f.el9.x86_ 342 kB/s | 937 kB 00:02 2026-02-12T12:14:07.604 INFO:teuthology.orchestra.run.trial110.stdout:(5/96): ceph-mds-20.3.0-5145.g5fdb7d8f.el9.x86_ 453 kB/s | 2.0 MB 00:04 2026-02-12T12:14:07.817 INFO:teuthology.orchestra.run.trial110.stdout:(6/96): ceph-grafana-dashboards-20.3.0-5145.g5f 269 kB/s | 57 kB 00:00 2026-02-12T12:14:08.297 INFO:teuthology.orchestra.run.trial110.stdout:(7/96): ceph-mgr-cephadm-20.3.0-5145.g5fdb7d8f. 382 kB/s | 183 kB 00:00 2026-02-12T12:14:37.994 INFO:teuthology.orchestra.run.trial110.stdout:(8/96): ceph-mgr-dashboard-20.3.0-5145.g5fdb7d8 515 kB/s | 15 MB 00:29 2026-02-12T12:14:48.061 INFO:teuthology.orchestra.run.trial110.stdout:(9/96): ceph-mgr-diskprediction-local-20.3.0-51 754 kB/s | 7.4 MB 00:10 2026-02-12T12:14:48.132 INFO:teuthology.orchestra.run.trial110.stdout:(10/96): ceph-mgr-k8sevents-20.3.0-5145.g5fdb7d 317 kB/s | 22 kB 00:00 2026-02-12T12:14:48.172 INFO:teuthology.orchestra.run.trial110.stdout:(11/96): ceph-osd-classic-20.3.0-5145.g5fdb7d8f 354 kB/s | 15 MB 00:42 2026-02-12T12:14:48.447 INFO:teuthology.orchestra.run.trial110.stdout:(12/96): ceph-mgr-rook-20.3.0-5145.g5fdb7d8f.el 183 kB/s | 50 kB 00:00 2026-02-12T12:14:48.515 INFO:teuthology.orchestra.run.trial110.stdout:(13/96): ceph-prometheus-alerts-20.3.0-5145.g5f 255 kB/s | 17 kB 00:00 2026-02-12T12:14:48.545 INFO:teuthology.orchestra.run.trial110.stdout:(14/96): ceph-mgr-modules-core-20.3.0-5145.g5fd 717 kB/s | 297 kB 00:00 2026-02-12T12:14:49.206 INFO:teuthology.orchestra.run.trial110.stdout:(15/96): ceph-volume-20.3.0-5145.g5fdb7d8f.el9. 433 kB/s | 299 kB 00:00 2026-02-12T12:14:50.275 INFO:teuthology.orchestra.run.trial110.stdout:(16/96): cephadm-20.3.0-5145.g5fdb7d8f.el9.noar 601 kB/s | 1.0 MB 00:01 2026-02-12T12:14:50.773 INFO:teuthology.orchestra.run.trial110.stdout:(17/96): ledmon-libs-1.1.0-3.el9.x86_64.rpm 81 kB/s | 40 kB 00:00 2026-02-12T12:14:50.802 INFO:teuthology.orchestra.run.trial110.stdout:(18/96): cryptsetup-2.8.1-2.el9.x86_64.rpm 219 kB/s | 350 kB 00:01 2026-02-12T12:14:51.076 INFO:teuthology.orchestra.run.trial110.stdout:(19/96): libconfig-1.7.2-9.el9.x86_64.rpm 238 kB/s | 72 kB 00:00 2026-02-12T12:14:51.531 INFO:teuthology.orchestra.run.trial110.stdout:(20/96): libquadmath-11.5.0-14.el9.x86_64.rpm 406 kB/s | 184 kB 00:00 2026-02-12T12:14:51.853 INFO:teuthology.orchestra.run.trial110.stdout:(21/96): libgfortran-11.5.0-14.el9.x86_64.rpm 755 kB/s | 794 kB 00:01 2026-02-12T12:14:51.986 INFO:teuthology.orchestra.run.trial110.stdout:(22/96): python3-cffi-1.14.5-5.el9.x86_64.rpm 556 kB/s | 253 kB 00:00 2026-02-12T12:14:52.214 INFO:teuthology.orchestra.run.trial110.stdout:(23/96): python3-ply-3.11-14.el9.noarch.rpm 466 kB/s | 106 kB 00:00 2026-02-12T12:14:52.519 INFO:teuthology.orchestra.run.trial110.stdout:(24/96): python3-pycparser-2.20-6.el9.noarch.rp 444 kB/s | 135 kB 00:00 2026-02-12T12:14:52.823 INFO:teuthology.orchestra.run.trial110.stdout:(25/96): python3-pyparsing-2.4.7-9.el9.noarch.r 495 kB/s | 150 kB 00:00 2026-02-12T12:14:53.054 INFO:teuthology.orchestra.run.trial110.stdout:(26/96): python3-requests-2.25.1-10.el9.noarch. 548 kB/s | 126 kB 00:00 2026-02-12T12:14:53.355 INFO:teuthology.orchestra.run.trial110.stdout:(27/96): python3-cryptography-36.0.1-5.el9.x86_ 849 kB/s | 1.2 MB 00:01 2026-02-12T12:14:53.433 INFO:teuthology.orchestra.run.trial110.stdout:(28/96): python3-urllib3-1.26.5-7.el9.noarch.rp 574 kB/s | 218 kB 00:00 2026-02-12T12:14:53.657 INFO:teuthology.orchestra.run.trial110.stdout:(29/96): unzip-6.0-59.el9.x86_64.rpm 602 kB/s | 182 kB 00:00 2026-02-12T12:14:53.683 INFO:teuthology.orchestra.run.trial110.stdout:(30/96): flexiblas-3.0.4-8.el9.x86_64.rpm 1.2 MB/s | 33 kB 00:00 2026-02-12T12:14:53.738 INFO:teuthology.orchestra.run.trial110.stdout:(31/96): flexiblas-netlib-3.0.4-8.el9.x86_64.rp 55 MB/s | 3.0 MB 00:00 2026-02-12T12:14:53.746 INFO:teuthology.orchestra.run.trial110.stdout:(32/96): flexiblas-openblas-openmp-3.0.4-8.el9. 2.2 MB/s | 18 kB 00:00 2026-02-12T12:14:53.756 INFO:teuthology.orchestra.run.trial110.stdout:(33/96): libstoragemgmt-1.10.1-1.el9.x86_64.rpm 24 MB/s | 246 kB 00:00 2026-02-12T12:14:53.766 INFO:teuthology.orchestra.run.trial110.stdout:(34/96): lua-5.4.4-4.el9.x86_64.rpm 18 MB/s | 188 kB 00:00 2026-02-12T12:14:53.775 INFO:teuthology.orchestra.run.trial110.stdout:(35/96): openblas-0.3.29-1.el9.x86_64.rpm 4.9 MB/s | 42 kB 00:00 2026-02-12T12:14:53.801 INFO:teuthology.orchestra.run.trial110.stdout:(36/96): openblas-openmp-0.3.29-1.el9.x86_64.rp 201 MB/s | 5.3 MB 00:00 2026-02-12T12:14:53.814 INFO:teuthology.orchestra.run.trial110.stdout:(37/96): zip-3.0-35.el9.x86_64.rpm 698 kB/s | 266 kB 00:00 2026-02-12T12:14:53.833 INFO:teuthology.orchestra.run.trial110.stdout:(38/96): python3-babel-2.9.1-2.el9.noarch.rpm 186 MB/s | 6.0 MB 00:00 2026-02-12T12:14:53.845 INFO:teuthology.orchestra.run.trial110.stdout:(39/96): python3-jinja2-2.11.3-8.el9.noarch.rpm 22 MB/s | 249 kB 00:00 2026-02-12T12:14:53.854 INFO:teuthology.orchestra.run.trial110.stdout:(40/96): python3-jmespath-1.0.1-1.el9.noarch.rp 5.1 MB/s | 48 kB 00:00 2026-02-12T12:14:53.861 INFO:teuthology.orchestra.run.trial110.stdout:(41/96): python3-devel-3.9.25-2.el9.x86_64.rpm 5.1 MB/s | 244 kB 00:00 2026-02-12T12:14:53.864 INFO:teuthology.orchestra.run.trial110.stdout:(42/96): python3-libstoragemgmt-1.10.1-1.el9.x8 17 MB/s | 177 kB 00:00 2026-02-12T12:14:53.873 INFO:teuthology.orchestra.run.trial110.stdout:(43/96): python3-markupsafe-1.1.1-12.el9.x86_64 3.8 MB/s | 35 kB 00:00 2026-02-12T12:14:53.898 INFO:teuthology.orchestra.run.trial110.stdout:(44/96): python3-lxml-4.6.5-3.el9.x86_64.rpm 33 MB/s | 1.2 MB 00:00 2026-02-12T12:14:53.905 INFO:teuthology.orchestra.run.trial110.stdout:(45/96): python3-numpy-1.23.5-2.el9.x86_64.rpm 189 MB/s | 6.1 MB 00:00 2026-02-12T12:14:53.910 INFO:teuthology.orchestra.run.trial110.stdout:(46/96): python3-numpy-f2py-1.23.5-2.el9.x86_64 36 MB/s | 442 kB 00:00 2026-02-12T12:14:53.915 INFO:teuthology.orchestra.run.trial110.stdout:(47/96): python3-packaging-20.9-5.el9.noarch.rp 7.7 MB/s | 77 kB 00:00 2026-02-12T12:14:53.921 INFO:teuthology.orchestra.run.trial110.stdout:(48/96): python3-protobuf-3.14.0-17.el9.noarch. 24 MB/s | 267 kB 00:00 2026-02-12T12:14:53.926 INFO:teuthology.orchestra.run.trial110.stdout:(49/96): python3-pyasn1-0.4.8-6.el9.noarch.rpm 14 MB/s | 159 kB 00:00 2026-02-12T12:14:53.932 INFO:teuthology.orchestra.run.trial110.stdout:(50/96): python3-pyasn1-modules-0.4.8-6.el9.noa 24 MB/s | 279 kB 00:00 2026-02-12T12:14:53.936 INFO:teuthology.orchestra.run.trial110.stdout:(51/96): python3-requests-oauthlib-1.3.0-12.el9 5.5 MB/s | 54 kB 00:00 2026-02-12T12:14:53.946 INFO:teuthology.orchestra.run.trial110.stdout:(52/96): python3-toml-0.10.2-6.el9.noarch.rpm 4.1 MB/s | 42 kB 00:00 2026-02-12T12:14:53.956 INFO:teuthology.orchestra.run.trial110.stdout:(53/96): xmlsec1-1.2.29-13.el9.x86_64.rpm 18 MB/s | 189 kB 00:00 2026-02-12T12:14:54.000 INFO:teuthology.orchestra.run.trial110.stdout:(54/96): xmlsec1-openssl-1.2.29-13.el9.x86_64.r 2.0 MB/s | 90 kB 00:00 2026-02-12T12:14:54.055 INFO:teuthology.orchestra.run.trial110.stdout:(55/96): python3-scipy-1.9.3-2.el9.x86_64.rpm 157 MB/s | 19 MB 00:00 2026-02-12T12:14:54.099 INFO:teuthology.orchestra.run.trial110.stdout:(56/96): lua-devel-5.4.4-4.el9.x86_64.rpm 225 kB/s | 22 kB 00:00 2026-02-12T12:14:54.186 INFO:teuthology.orchestra.run.trial110.stdout:(57/96): protobuf-compiler-3.14.0-17.el9.x86_64 6.4 MB/s | 862 kB 00:00 2026-02-12T12:14:54.355 INFO:teuthology.orchestra.run.trial110.stdout:(58/96): grpc-data-1.46.7-10.el9.noarch.rpm 115 kB/s | 19 kB 00:00 2026-02-12T12:14:54.454 INFO:teuthology.orchestra.run.trial110.stdout:(59/96): abseil-cpp-20211102.0-4.el9.x86_64.rpm 1.5 MB/s | 551 kB 00:00 2026-02-12T12:14:54.497 INFO:teuthology.orchestra.run.trial110.stdout:(60/96): luarocks-3.9.2-5.el9.noarch.rpm 1.0 MB/s | 151 kB 00:00 2026-02-12T12:14:54.523 INFO:teuthology.orchestra.run.trial110.stdout:(61/96): python3-asyncssh-2.13.2-5.el9.noarch.r 7.7 MB/s | 548 kB 00:00 2026-02-12T12:14:54.542 INFO:teuthology.orchestra.run.trial110.stdout:(62/96): python3-autocommand-2.2.2-8.el9.noarch 657 kB/s | 29 kB 00:00 2026-02-12T12:14:54.567 INFO:teuthology.orchestra.run.trial110.stdout:(63/96): python3-backports-tarfile-1.2.0-1.el9. 1.3 MB/s | 60 kB 00:00 2026-02-12T12:14:54.589 INFO:teuthology.orchestra.run.trial110.stdout:(64/96): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 922 kB/s | 43 kB 00:00 2026-02-12T12:14:54.609 INFO:teuthology.orchestra.run.trial110.stdout:(65/96): python3-cachetools-4.2.4-1.el9.noarch. 770 kB/s | 32 kB 00:00 2026-02-12T12:14:54.631 INFO:teuthology.orchestra.run.trial110.stdout:(66/96): python3-certifi-2023.05.07-4.el9.noarc 336 kB/s | 14 kB 00:00 2026-02-12T12:14:54.657 INFO:teuthology.orchestra.run.trial110.stdout:(67/96): python3-cheroot-10.0.1-4.el9.noarch.rp 3.5 MB/s | 173 kB 00:00 2026-02-12T12:14:54.709 INFO:teuthology.orchestra.run.trial110.stdout:(68/96): python3-google-auth-2.45.0-1.el9.noarc 4.8 MB/s | 254 kB 00:00 2026-02-12T12:14:54.734 INFO:teuthology.orchestra.run.trial110.stdout:(69/96): python3-cherrypy-18.6.1-2.el9.noarch.r 3.4 MB/s | 358 kB 00:00 2026-02-12T12:14:54.783 INFO:teuthology.orchestra.run.trial110.stdout:(70/96): python3-grpcio-tools-1.46.7-10.el9.x86 2.9 MB/s | 144 kB 00:00 2026-02-12T12:14:54.831 INFO:teuthology.orchestra.run.trial110.stdout:(71/96): python3-influxdb-5.3.1-1.el9.noarch.rp 2.9 MB/s | 139 kB 00:00 2026-02-12T12:14:54.836 INFO:teuthology.orchestra.run.trial110.stdout:(72/96): python3-grpcio-1.46.7-10.el9.x86_64.rp 16 MB/s | 2.0 MB 00:00 2026-02-12T12:14:54.874 INFO:teuthology.orchestra.run.trial110.stdout:(73/96): python3-isodate-0.6.1-3.el9.noarch.rpm 1.3 MB/s | 56 kB 00:00 2026-02-12T12:14:54.877 INFO:teuthology.orchestra.run.trial110.stdout:(74/96): python3-jaraco-8.2.1-3.el9.noarch.rpm 259 kB/s | 11 kB 00:00 2026-02-12T12:14:54.915 INFO:teuthology.orchestra.run.trial110.stdout:(75/96): python3-jaraco-classes-3.2.1-5.el9.noa 432 kB/s | 18 kB 00:00 2026-02-12T12:14:54.919 INFO:teuthology.orchestra.run.trial110.stdout:(76/96): python3-jaraco-collections-3.0.0-8.el9 558 kB/s | 23 kB 00:00 2026-02-12T12:14:54.956 INFO:teuthology.orchestra.run.trial110.stdout:(77/96): python3-jaraco-context-6.0.1-3.el9.noa 479 kB/s | 20 kB 00:00 2026-02-12T12:14:54.960 INFO:teuthology.orchestra.run.trial110.stdout:(78/96): python3-jaraco-functools-3.5.0-2.el9.n 468 kB/s | 19 kB 00:00 2026-02-12T12:14:54.997 INFO:teuthology.orchestra.run.trial110.stdout:(79/96): python3-jaraco-text-4.0.0-2.el9.noarch 645 kB/s | 26 kB 00:00 2026-02-12T12:14:55.016 INFO:teuthology.orchestra.run.trial110.stdout:(80/96): python3-kubernetes-26.1.0-3.el9.noarch 18 MB/s | 1.0 MB 00:00 2026-02-12T12:14:55.041 INFO:teuthology.orchestra.run.trial110.stdout:(81/96): python3-more-itertools-8.12.0-2.el9.no 1.7 MB/s | 79 kB 00:00 2026-02-12T12:14:55.059 INFO:teuthology.orchestra.run.trial110.stdout:(82/96): python3-msgpack-1.0.3-2.el9.x86_64.rpm 2.0 MB/s | 86 kB 00:00 2026-02-12T12:14:55.084 INFO:teuthology.orchestra.run.trial110.stdout:(83/96): python3-natsort-7.1.1-5.el9.noarch.rpm 1.3 MB/s | 58 kB 00:00 2026-02-12T12:14:55.100 INFO:teuthology.orchestra.run.trial110.stdout:(84/96): python3-portend-3.1.0-2.el9.noarch.rpm 397 kB/s | 16 kB 00:00 2026-02-12T12:14:55.129 INFO:teuthology.orchestra.run.trial110.stdout:(85/96): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.0 MB/s | 90 kB 00:00 2026-02-12T12:14:55.142 INFO:teuthology.orchestra.run.trial110.stdout:(86/96): python3-repoze-lru-0.7-16.el9.noarch.r 743 kB/s | 31 kB 00:00 2026-02-12T12:14:55.180 INFO:teuthology.orchestra.run.trial110.stdout:(87/96): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2026-02-12T12:14:55.183 INFO:teuthology.orchestra.run.trial110.stdout:(88/96): python3-rsa-4.9-2.el9.noarch.rpm 1.4 MB/s | 59 kB 00:00 2026-02-12T12:14:55.225 INFO:teuthology.orchestra.run.trial110.stdout:(89/96): python3-tempora-5.0.0-2.el9.noarch.rpm 864 kB/s | 36 kB 00:00 2026-02-12T12:14:55.226 INFO:teuthology.orchestra.run.trial110.stdout:(90/96): python3-saml-1.16.0-1.el9.noarch.rpm 2.6 MB/s | 125 kB 00:00 2026-02-12T12:14:55.267 INFO:teuthology.orchestra.run.trial110.stdout:(91/96): python3-typing-extensions-4.15.0-1.el9 2.0 MB/s | 86 kB 00:00 2026-02-12T12:14:55.271 INFO:teuthology.orchestra.run.trial110.stdout:(92/96): python3-websocket-client-1.2.3-2.el9.n 2.0 MB/s | 90 kB 00:00 2026-02-12T12:14:55.308 INFO:teuthology.orchestra.run.trial110.stdout:(93/96): python3-xmlsec-1.3.13-1.el9.x86_64.rpm 1.1 MB/s | 48 kB 00:00 2026-02-12T12:14:55.312 INFO:teuthology.orchestra.run.trial110.stdout:(94/96): python3-xmltodict-0.12.0-15.el9.noarch 541 kB/s | 22 kB 00:00 2026-02-12T12:14:55.351 INFO:teuthology.orchestra.run.trial110.stdout:(95/96): python3-zc-lockfile-2.0-10.el9.noarch. 467 kB/s | 20 kB 00:00 2026-02-12T12:17:16.463 INFO:teuthology.orchestra.run.trial110.stdout:(96/96): ceph-osd-crimson-20.3.0-5145.g5fdb7d8f 466 kB/s | 87 MB 03:10 2026-02-12T12:17:16.466 INFO:teuthology.orchestra.run.trial110.stdout:-------------------------------------------------------------------------------- 2026-02-12T12:17:16.466 INFO:teuthology.orchestra.run.trial110.stdout:Total 998 kB/s | 189 MB 03:13 2026-02-12T12:17:16.839 INFO:teuthology.orchestra.run.trial110.stdout:Running transaction check 2026-02-12T12:17:16.878 INFO:teuthology.orchestra.run.trial110.stdout:Transaction check succeeded. 2026-02-12T12:17:16.878 INFO:teuthology.orchestra.run.trial110.stdout:Running transaction test 2026-02-12T12:17:17.569 INFO:teuthology.orchestra.run.trial110.stdout:Transaction test succeeded. 2026-02-12T12:17:17.569 INFO:teuthology.orchestra.run.trial110.stdout:Running transaction 2026-02-12T12:17:18.289 INFO:teuthology.orchestra.run.trial110.stdout: Preparing : 1/1 2026-02-12T12:17:18.291 INFO:teuthology.orchestra.run.trial110.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/96 2026-02-12T12:17:18.294 INFO:teuthology.orchestra.run.trial110.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/96 2026-02-12T12:17:18.316 INFO:teuthology.orchestra.run.trial110.stdout: Installing : xmlsec1-1.2.29-13.el9.x86_64 3/96 2026-02-12T12:17:18.349 INFO:teuthology.orchestra.run.trial110.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 4/96 2026-02-12T12:17:18.355 INFO:teuthology.orchestra.run.trial110.stdout: Installing : python3-lxml-4.6.5-3.el9.x86_64 5/96 2026-02-12T12:17:18.358 INFO:teuthology.orchestra.run.trial110.stdout: Installing : lua-5.4.4-4.el9.x86_64 6/96 2026-02-12T12:17:18.362 INFO:teuthology.orchestra.run.trial110.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 7/96 2026-02-12T12:17:18.385 INFO:teuthology.orchestra.run.trial110.stdout: Installing : unzip-6.0-59.el9.x86_64 8/96 2026-02-12T12:17:18.398 INFO:teuthology.orchestra.run.trial110.stdout: Installing : python3-urllib3-1.26.5-7.el9.noarch 9/96 2026-02-12T12:17:18.401 INFO:teuthology.orchestra.run.trial110.stdout: Installing : python3-requests-2.25.1-10.el9.noarch 10/96 2026-02-12T12:17:18.407 INFO:teuthology.orchestra.run.trial110.stdout: Installing : libquadmath-11.5.0-14.el9.x86_64 11/96 2026-02-12T12:17:18.409 INFO:teuthology.orchestra.run.trial110.stdout: Installing : libgfortran-11.5.0-14.el9.x86_64 12/96 2026-02-12T12:17:18.420 INFO:teuthology.orchestra.run.trial110.stdout: Installing : ledmon-libs-1.1.0-3.el9.x86_64 13/96 2026-02-12T12:17:18.426 INFO:teuthology.orchestra.run.trial110.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 14/96 2026-02-12T12:17:18.450 INFO:teuthology.orchestra.run.trial110.stdout: Installing : zip-3.0-35.el9.x86_64 15/96 2026-02-12T12:17:18.454 INFO:teuthology.orchestra.run.trial110.stdout: Installing : luarocks-3.9.2-5.el9.noarch 16/96 2026-02-12T12:17:18.503 INFO:teuthology.orchestra.run.trial110.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 17/96 2026-02-12T12:17:18.516 INFO:teuthology.orchestra.run.trial110.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 18/96 2026-02-12T12:17:18.519 INFO:teuthology.orchestra.run.trial110.stdout: Installing : python3-rsa-4.9-2.el9.noarch 19/96 2026-02-12T12:17:18.523 INFO:teuthology.orchestra.run.trial110.stdout: Installing : xmlsec1-openssl-1.2.29-13.el9.x86_64 20/96 2026-02-12T12:17:18.528 INFO:teuthology.orchestra.run.trial110.stdout: Installing : python3-xmlsec-1.3.13-1.el9.x86_64 21/96 2026-02-12T12:17:18.533 INFO:teuthology.orchestra.run.trial110.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 22/96 2026-02-12T12:17:18.536 INFO:teuthology.orchestra.run.trial110.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 23/96 2026-02-12T12:17:18.551 INFO:teuthology.orchestra.run.trial110.stdout: Installing : python3-xmltodict-0.12.0-15.el9.noarch 24/96 2026-02-12T12:17:18.555 INFO:teuthology.orchestra.run.trial110.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 25/96 2026-02-12T12:17:18.560 INFO:teuthology.orchestra.run.trial110.stdout: Installing : python3-typing-extensions-4.15.0-1.el9.noarch 26/96 2026-02-12T12:17:18.571 INFO:teuthology.orchestra.run.trial110.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 27/96 2026-02-12T12:17:18.580 INFO:teuthology.orchestra.run.trial110.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 28/96 2026-02-12T12:17:18.587 INFO:teuthology.orchestra.run.trial110.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 29/96 2026-02-12T12:17:18.606 INFO:teuthology.orchestra.run.trial110.stdout: Installing : python3-msgpack-1.0.3-2.el9.x86_64 30/96 2026-02-12T12:17:18.617 INFO:teuthology.orchestra.run.trial110.stdout: Installing : python3-influxdb-5.3.1-1.el9.noarch 31/96 2026-02-12T12:17:18.633 INFO:teuthology.orchestra.run.trial110.stdout: Installing : python3-isodate-0.6.1-3.el9.noarch 32/96 2026-02-12T12:17:18.637 INFO:teuthology.orchestra.run.trial110.stdout: Installing : python3-saml-1.16.0-1.el9.noarch 33/96 2026-02-12T12:17:18.645 INFO:teuthology.orchestra.run.trial110.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 34/96 2026-02-12T12:17:18.685 INFO:teuthology.orchestra.run.trial110.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 35/96 2026-02-12T12:17:19.009 INFO:teuthology.orchestra.run.trial110.stdout: Installing : python3-google-auth-1:2.45.0-1.el9.noarch 36/96 2026-02-12T12:17:19.019 INFO:teuthology.orchestra.run.trial110.stdout: Installing : python3-kubernetes-1:26.1.0-3.el9.noarch 37/96 2026-02-12T12:17:19.023 INFO:teuthology.orchestra.run.trial110.stdout: Installing : python3-backports-tarfile-1.2.0-1.el9.noarch 38/96 2026-02-12T12:17:19.029 INFO:teuthology.orchestra.run.trial110.stdout: Installing : python3-jaraco-context-6.0.1-3.el9.noarch 39/96 2026-02-12T12:17:19.031 INFO:teuthology.orchestra.run.trial110.stdout: Installing : python3-autocommand-2.2.2-8.el9.noarch 40/96 2026-02-12T12:17:19.057 INFO:teuthology.orchestra.run.trial110.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 41/96 2026-02-12T12:17:19.100 INFO:teuthology.orchestra.run.trial110.stdout: Installing : abseil-cpp-20211102.0-4.el9.x86_64 42/96 2026-02-12T12:17:19.109 INFO:teuthology.orchestra.run.trial110.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 43/96 2026-02-12T12:17:19.115 INFO:teuthology.orchestra.run.trial110.stdout: Installing : protobuf-compiler-3.14.0-17.el9.x86_64 44/96 2026-02-12T12:17:19.118 INFO:teuthology.orchestra.run.trial110.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 45/96 2026-02-12T12:17:19.124 INFO:teuthology.orchestra.run.trial110.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 46/96 2026-02-12T12:17:19.127 INFO:teuthology.orchestra.run.trial110.stdout: Installing : python3-jaraco-text-4.0.0-2.el9.noarch 47/96 2026-02-12T12:17:19.134 INFO:teuthology.orchestra.run.trial110.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 48/96 2026-02-12T12:17:19.138 INFO:teuthology.orchestra.run.trial110.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 49/96 2026-02-12T12:17:19.166 INFO:teuthology.orchestra.run.trial110.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 50/96 2026-02-12T12:17:19.175 INFO:teuthology.orchestra.run.trial110.stdout: Installing : python3-protobuf-3.14.0-17.el9.noarch 51/96 2026-02-12T12:17:19.181 INFO:teuthology.orchestra.run.trial110.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 52/96 2026-02-12T12:17:19.188 INFO:teuthology.orchestra.run.trial110.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 53/96 2026-02-12T12:17:19.223 INFO:teuthology.orchestra.run.trial110.stdout: Installing : python3-jmespath-1.0.1-1.el9.noarch 54/96 2026-02-12T12:17:19.447 INFO:teuthology.orchestra.run.trial110.stdout: Installing : python3-devel-3.9.25-2.el9.x86_64 55/96 2026-02-12T12:17:19.470 INFO:teuthology.orchestra.run.trial110.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 56/96 2026-02-12T12:17:19.474 INFO:teuthology.orchestra.run.trial110.stdout: Installing : python3-jinja2-2.11.3-8.el9.noarch 57/96 2026-02-12T12:17:19.532 INFO:teuthology.orchestra.run.trial110.stdout: Installing : openblas-0.3.29-1.el9.x86_64 58/96 2026-02-12T12:17:19.534 INFO:teuthology.orchestra.run.trial110.stdout: Installing : openblas-openmp-0.3.29-1.el9.x86_64 59/96 2026-02-12T12:17:19.555 INFO:teuthology.orchestra.run.trial110.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 60/96 2026-02-12T12:17:19.881 INFO:teuthology.orchestra.run.trial110.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 61/96 2026-02-12T12:17:19.954 INFO:teuthology.orchestra.run.trial110.stdout: Installing : python3-numpy-1:1.23.5-2.el9.x86_64 62/96 2026-02-12T12:17:20.625 INFO:teuthology.orchestra.run.trial110.stdout: Installing : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 63/96 2026-02-12T12:17:20.642 INFO:teuthology.orchestra.run.trial110.stdout: Installing : python3-scipy-1.9.3-2.el9.x86_64 64/96 2026-02-12T12:17:20.652 INFO:teuthology.orchestra.run.trial110.stdout: Installing : python3-pyparsing-2.4.7-9.el9.noarch 65/96 2026-02-12T12:17:20.658 INFO:teuthology.orchestra.run.trial110.stdout: Installing : python3-packaging-20.9-5.el9.noarch 66/96 2026-02-12T12:17:20.672 INFO:teuthology.orchestra.run.trial110.stdout: Installing : python3-ply-3.11-14.el9.noarch 67/96 2026-02-12T12:17:20.687 INFO:teuthology.orchestra.run.trial110.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 68/96 2026-02-12T12:17:20.763 INFO:teuthology.orchestra.run.trial110.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 69/96 2026-02-12T12:17:20.771 INFO:teuthology.orchestra.run.trial110.stdout: Installing : python3-cryptography-36.0.1-5.el9.x86_64 70/96 2026-02-12T12:17:20.795 INFO:teuthology.orchestra.run.trial110.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 71/96 2026-02-12T12:17:20.825 INFO:teuthology.orchestra.run.trial110.stdout: Installing : python3-cheroot-10.0.1-4.el9.noarch 72/96 2026-02-12T12:17:20.875 INFO:teuthology.orchestra.run.trial110.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 73/96 2026-02-12T12:17:20.881 INFO:teuthology.orchestra.run.trial110.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 74/96 2026-02-12T12:17:20.885 INFO:teuthology.orchestra.run.trial110.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 75/96 2026-02-12T12:17:20.886 INFO:teuthology.orchestra.run.trial110.stdout: Installing : libconfig-1.7.2-9.el9.x86_64 76/96 2026-02-12T12:17:20.903 INFO:teuthology.orchestra.run.trial110.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-12T12:17:20.903 INFO:teuthology.orchestra.run.trial110.stdout:Creating group 'libstoragemgmt' with GID 991. 2026-02-12T12:17:20.903 INFO:teuthology.orchestra.run.trial110.stdout:Creating user 'libstoragemgmt' (daemon account for libstoragemgmt) with UID 991 and GID 991. 2026-02-12T12:17:20.903 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:17:20.911 INFO:teuthology.orchestra.run.trial110.stdout: Installing : libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-12T12:17:20.935 INFO:teuthology.orchestra.run.trial110.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-12T12:17:20.935 INFO:teuthology.orchestra.run.trial110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/libstoragemgmt.service → /usr/lib/systemd/system/libstoragemgmt.service. 2026-02-12T12:17:20.935 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:17:20.968 INFO:teuthology.orchestra.run.trial110.stdout: Installing : python3-libstoragemgmt-1.10.1-1.el9.x86_64 78/96 2026-02-12T12:17:21.558 INFO:teuthology.orchestra.run.trial110.stdout: Installing : cryptsetup-2.8.1-2.el9.x86_64 79/96 2026-02-12T12:17:26.917 INFO:teuthology.orchestra.run.trial110.stdout: Installing : ceph-osd-classic-2:20.3.0-5145.g5fdb7d8f.el9.x86_6 80/96 2026-02-12T12:17:26.980 INFO:teuthology.orchestra.run.trial110.stdout: Installing : ceph-osd-crimson-2:20.3.0-5145.g5fdb7d8f.el9.x86_6 81/96 2026-02-12T12:17:26.982 INFO:teuthology.orchestra.run.trial110.stdout: Installing : ceph-volume-2:20.3.0-5145.g5fdb7d8f.el9.noarch 82/96 2026-02-12T12:17:26.995 INFO:teuthology.orchestra.run.trial110.stdout: Running scriptlet: ceph-volume-2:20.3.0-5145.g5fdb7d8f.el9.noarch 82/96 2026-02-12T12:17:26.995 INFO:teuthology.orchestra.run.trial110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-12T12:17:26.996 INFO:teuthology.orchestra.run.trial110.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2026-02-12T12:17:26.996 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:17:27.006 INFO:teuthology.orchestra.run.trial110.stdout: Installing : ceph-osd-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 83/96 2026-02-12T12:17:27.030 INFO:teuthology.orchestra.run.trial110.stdout: Running scriptlet: ceph-osd-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 83/96 2026-02-12T12:17:27.030 INFO:teuthology.orchestra.run.trial110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-12T12:17:27.030 INFO:teuthology.orchestra.run.trial110.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2026-02-12T12:17:27.030 INFO:teuthology.orchestra.run.trial110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-12T12:17:27.031 INFO:teuthology.orchestra.run.trial110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-12T12:17:27.031 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:17:27.089 INFO:teuthology.orchestra.run.trial110.stdout: Running scriptlet: cephadm-2:20.3.0-5145.g5fdb7d8f.el9.noarch 84/96 2026-02-12T12:17:27.091 INFO:teuthology.orchestra.run.trial110.stdout: Installing : cephadm-2:20.3.0-5145.g5fdb7d8f.el9.noarch 84/96 2026-02-12T12:17:27.100 INFO:teuthology.orchestra.run.trial110.stdout: Installing : ceph-prometheus-alerts-2:20.3.0-5145.g5fdb7d8f.el9 85/96 2026-02-12T12:17:27.124 INFO:teuthology.orchestra.run.trial110.stdout: Installing : ceph-grafana-dashboards-2:20.3.0-5145.g5fdb7d8f.el 86/96 2026-02-12T12:17:27.125 INFO:teuthology.orchestra.run.trial110.stdout: Installing : ceph-mgr-cephadm-2:20.3.0-5145.g5fdb7d8f.el9.noarc 87/96 2026-02-12T12:17:28.703 INFO:teuthology.orchestra.run.trial110.stdout: Running scriptlet: ceph-mgr-cephadm-2:20.3.0-5145.g5fdb7d8f.el9.noarc 87/96 2026-02-12T12:17:28.710 INFO:teuthology.orchestra.run.trial110.stdout: Installing : ceph-mgr-dashboard-2:20.3.0-5145.g5fdb7d8f.el9.noa 88/96 2026-02-12T12:17:29.165 INFO:teuthology.orchestra.run.trial110.stdout: Running scriptlet: ceph-mgr-dashboard-2:20.3.0-5145.g5fdb7d8f.el9.noa 88/96 2026-02-12T12:17:29.166 INFO:teuthology.orchestra.run.trial110.stdout: Installing : ceph-mgr-diskprediction-local-2:20.3.0-5145.g5fdb7 89/96 2026-02-12T12:17:29.177 INFO:teuthology.orchestra.run.trial110.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:20.3.0-5145.g5fdb7 89/96 2026-02-12T12:17:29.178 INFO:teuthology.orchestra.run.trial110.stdout: Installing : ceph-mgr-k8sevents-2:20.3.0-5145.g5fdb7d8f.el9.noa 90/96 2026-02-12T12:17:29.239 INFO:teuthology.orchestra.run.trial110.stdout: Running scriptlet: ceph-mgr-k8sevents-2:20.3.0-5145.g5fdb7d8f.el9.noa 90/96 2026-02-12T12:17:29.279 INFO:teuthology.orchestra.run.trial110.stdout: Installing : ceph-mgr-modules-core-2:20.3.0-5145.g5fdb7d8f.el9. 91/96 2026-02-12T12:17:29.281 INFO:teuthology.orchestra.run.trial110.stdout: Installing : ceph-mgr-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 92/96 2026-02-12T12:17:29.301 INFO:teuthology.orchestra.run.trial110.stdout: Running scriptlet: ceph-mgr-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 92/96 2026-02-12T12:17:29.301 INFO:teuthology.orchestra.run.trial110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-12T12:17:29.301 INFO:teuthology.orchestra.run.trial110.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2026-02-12T12:17:29.301 INFO:teuthology.orchestra.run.trial110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-12T12:17:29.302 INFO:teuthology.orchestra.run.trial110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-12T12:17:29.302 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:17:29.313 INFO:teuthology.orchestra.run.trial110.stdout: Installing : ceph-mgr-rook-2:20.3.0-5145.g5fdb7d8f.el9.noarch 93/96 2026-02-12T12:17:29.530 INFO:teuthology.orchestra.run.trial110.stdout: Running scriptlet: ceph-mgr-rook-2:20.3.0-5145.g5fdb7d8f.el9.noarch 93/96 2026-02-12T12:17:29.531 INFO:teuthology.orchestra.run.trial110.stdout: Installing : ceph-mon-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 94/96 2026-02-12T12:17:29.552 INFO:teuthology.orchestra.run.trial110.stdout: Running scriptlet: ceph-mon-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 94/96 2026-02-12T12:17:29.552 INFO:teuthology.orchestra.run.trial110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-12T12:17:29.552 INFO:teuthology.orchestra.run.trial110.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2026-02-12T12:17:29.552 INFO:teuthology.orchestra.run.trial110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-12T12:17:29.552 INFO:teuthology.orchestra.run.trial110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-12T12:17:29.553 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:17:29.635 INFO:teuthology.orchestra.run.trial110.stdout: Installing : ceph-mds-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 95/96 2026-02-12T12:17:29.654 INFO:teuthology.orchestra.run.trial110.stdout: Running scriptlet: ceph-mds-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 95/96 2026-02-12T12:17:29.654 INFO:teuthology.orchestra.run.trial110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-12T12:17:29.654 INFO:teuthology.orchestra.run.trial110.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2026-02-12T12:17:29.654 INFO:teuthology.orchestra.run.trial110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-12T12:17:29.655 INFO:teuthology.orchestra.run.trial110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-12T12:17:29.655 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:17:29.655 INFO:teuthology.orchestra.run.trial110.stdout: Installing : ceph-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 96/96 2026-02-12T12:17:29.661 INFO:teuthology.orchestra.run.trial110.stdout: Running scriptlet: ceph-osd-classic-2:20.3.0-5145.g5fdb7d8f.el9.x86_6 96/96 2026-02-12T12:17:29.684 INFO:teuthology.orchestra.run.trial110.stdout: Running scriptlet: ceph-osd-crimson-2:20.3.0-5145.g5fdb7d8f.el9.x86_6 96/96 2026-02-12T12:17:31.016 INFO:teuthology.orchestra.run.trial110.stdout: Running scriptlet: ceph-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 96/96 2026-02-12T12:17:31.016 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : ceph-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 1/96 2026-02-12T12:17:31.016 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : ceph-mds-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2/96 2026-02-12T12:17:31.016 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : ceph-mgr-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 3/96 2026-02-12T12:17:31.016 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : ceph-mon-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 4/96 2026-02-12T12:17:31.016 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : ceph-osd-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 5/96 2026-02-12T12:17:31.017 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : ceph-osd-classic-2:20.3.0-5145.g5fdb7d8f.el9.x86_6 6/96 2026-02-12T12:17:31.017 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : ceph-osd-crimson-2:20.3.0-5145.g5fdb7d8f.el9.x86_6 7/96 2026-02-12T12:17:31.017 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : ceph-grafana-dashboards-2:20.3.0-5145.g5fdb7d8f.el 8/96 2026-02-12T12:17:31.017 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : ceph-mgr-cephadm-2:20.3.0-5145.g5fdb7d8f.el9.noarc 9/96 2026-02-12T12:17:31.017 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : ceph-mgr-dashboard-2:20.3.0-5145.g5fdb7d8f.el9.noa 10/96 2026-02-12T12:17:31.017 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : ceph-mgr-diskprediction-local-2:20.3.0-5145.g5fdb7 11/96 2026-02-12T12:17:31.017 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : ceph-mgr-k8sevents-2:20.3.0-5145.g5fdb7d8f.el9.noa 12/96 2026-02-12T12:17:31.017 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : ceph-mgr-modules-core-2:20.3.0-5145.g5fdb7d8f.el9. 13/96 2026-02-12T12:17:31.017 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : ceph-mgr-rook-2:20.3.0-5145.g5fdb7d8f.el9.noarch 14/96 2026-02-12T12:17:31.017 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : ceph-prometheus-alerts-2:20.3.0-5145.g5fdb7d8f.el9 15/96 2026-02-12T12:17:31.017 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : ceph-volume-2:20.3.0-5145.g5fdb7d8f.el9.noarch 16/96 2026-02-12T12:17:31.018 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : cephadm-2:20.3.0-5145.g5fdb7d8f.el9.noarch 17/96 2026-02-12T12:17:31.018 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : cryptsetup-2.8.1-2.el9.x86_64 18/96 2026-02-12T12:17:31.018 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : ledmon-libs-1.1.0-3.el9.x86_64 19/96 2026-02-12T12:17:31.018 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : libconfig-1.7.2-9.el9.x86_64 20/96 2026-02-12T12:17:31.018 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : libgfortran-11.5.0-14.el9.x86_64 21/96 2026-02-12T12:17:31.018 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : libquadmath-11.5.0-14.el9.x86_64 22/96 2026-02-12T12:17:31.018 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 23/96 2026-02-12T12:17:31.018 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : python3-cryptography-36.0.1-5.el9.x86_64 24/96 2026-02-12T12:17:31.018 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : python3-ply-3.11-14.el9.noarch 25/96 2026-02-12T12:17:31.018 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 26/96 2026-02-12T12:17:31.018 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : python3-pyparsing-2.4.7-9.el9.noarch 27/96 2026-02-12T12:17:31.019 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : python3-requests-2.25.1-10.el9.noarch 28/96 2026-02-12T12:17:31.019 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : python3-urllib3-1.26.5-7.el9.noarch 29/96 2026-02-12T12:17:31.019 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : unzip-6.0-59.el9.x86_64 30/96 2026-02-12T12:17:31.019 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : zip-3.0-35.el9.x86_64 31/96 2026-02-12T12:17:31.019 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 32/96 2026-02-12T12:17:31.019 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 33/96 2026-02-12T12:17:31.019 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 34/96 2026-02-12T12:17:31.019 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : libstoragemgmt-1.10.1-1.el9.x86_64 35/96 2026-02-12T12:17:31.019 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : lua-5.4.4-4.el9.x86_64 36/96 2026-02-12T12:17:31.019 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : openblas-0.3.29-1.el9.x86_64 37/96 2026-02-12T12:17:31.019 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : openblas-openmp-0.3.29-1.el9.x86_64 38/96 2026-02-12T12:17:31.020 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 39/96 2026-02-12T12:17:31.020 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : python3-devel-3.9.25-2.el9.x86_64 40/96 2026-02-12T12:17:31.020 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : python3-jinja2-2.11.3-8.el9.noarch 41/96 2026-02-12T12:17:31.020 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : python3-jmespath-1.0.1-1.el9.noarch 42/96 2026-02-12T12:17:31.020 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : python3-libstoragemgmt-1.10.1-1.el9.x86_64 43/96 2026-02-12T12:17:31.020 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : python3-lxml-4.6.5-3.el9.x86_64 44/96 2026-02-12T12:17:31.020 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 45/96 2026-02-12T12:17:31.020 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : python3-numpy-1:1.23.5-2.el9.x86_64 46/96 2026-02-12T12:17:31.020 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 47/96 2026-02-12T12:17:31.020 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 48/96 2026-02-12T12:17:31.020 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : python3-protobuf-3.14.0-17.el9.noarch 49/96 2026-02-12T12:17:31.020 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 50/96 2026-02-12T12:17:31.020 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 51/96 2026-02-12T12:17:31.021 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 52/96 2026-02-12T12:17:31.021 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : python3-scipy-1.9.3-2.el9.x86_64 53/96 2026-02-12T12:17:31.021 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 54/96 2026-02-12T12:17:31.021 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : xmlsec1-1.2.29-13.el9.x86_64 55/96 2026-02-12T12:17:31.021 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : xmlsec1-openssl-1.2.29-13.el9.x86_64 56/96 2026-02-12T12:17:31.021 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 57/96 2026-02-12T12:17:31.021 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : protobuf-compiler-3.14.0-17.el9.x86_64 58/96 2026-02-12T12:17:31.021 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : abseil-cpp-20211102.0-4.el9.x86_64 59/96 2026-02-12T12:17:31.021 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 60/96 2026-02-12T12:17:31.021 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 61/96 2026-02-12T12:17:31.021 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 62/96 2026-02-12T12:17:31.021 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : python3-autocommand-2.2.2-8.el9.noarch 63/96 2026-02-12T12:17:31.021 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : python3-backports-tarfile-1.2.0-1.el9.noarch 64/96 2026-02-12T12:17:31.022 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 65/96 2026-02-12T12:17:31.022 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 66/96 2026-02-12T12:17:31.022 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 67/96 2026-02-12T12:17:31.022 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : python3-cheroot-10.0.1-4.el9.noarch 68/96 2026-02-12T12:17:31.022 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 69/96 2026-02-12T12:17:31.022 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : python3-google-auth-1:2.45.0-1.el9.noarch 70/96 2026-02-12T12:17:31.022 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 71/96 2026-02-12T12:17:31.022 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 72/96 2026-02-12T12:17:31.022 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : python3-influxdb-5.3.1-1.el9.noarch 73/96 2026-02-12T12:17:31.022 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : python3-isodate-0.6.1-3.el9.noarch 74/96 2026-02-12T12:17:31.022 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 75/96 2026-02-12T12:17:31.022 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 76/96 2026-02-12T12:17:31.023 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 77/96 2026-02-12T12:17:31.023 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : python3-jaraco-context-6.0.1-3.el9.noarch 78/96 2026-02-12T12:17:31.023 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 79/96 2026-02-12T12:17:31.023 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : python3-jaraco-text-4.0.0-2.el9.noarch 80/96 2026-02-12T12:17:31.023 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : python3-kubernetes-1:26.1.0-3.el9.noarch 81/96 2026-02-12T12:17:31.023 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 82/96 2026-02-12T12:17:31.023 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : python3-msgpack-1.0.3-2.el9.x86_64 83/96 2026-02-12T12:17:31.023 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 84/96 2026-02-12T12:17:31.023 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 85/96 2026-02-12T12:17:31.023 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/96 2026-02-12T12:17:31.023 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 87/96 2026-02-12T12:17:31.023 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 88/96 2026-02-12T12:17:31.023 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 89/96 2026-02-12T12:17:31.024 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : python3-saml-1.16.0-1.el9.noarch 90/96 2026-02-12T12:17:31.024 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 91/96 2026-02-12T12:17:31.024 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : python3-typing-extensions-4.15.0-1.el9.noarch 92/96 2026-02-12T12:17:31.024 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 93/96 2026-02-12T12:17:31.024 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : python3-xmlsec-1.3.13-1.el9.x86_64 94/96 2026-02-12T12:17:31.024 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : python3-xmltodict-0.12.0-15.el9.noarch 95/96 2026-02-12T12:17:31.077 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 96/96 2026-02-12T12:17:31.077 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:17:31.078 INFO:teuthology.orchestra.run.trial110.stdout:Installed: 2026-02-12T12:17:31.078 INFO:teuthology.orchestra.run.trial110.stdout: abseil-cpp-20211102.0-4.el9.x86_64 2026-02-12T12:17:31.078 INFO:teuthology.orchestra.run.trial110.stdout: ceph-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:17:31.078 INFO:teuthology.orchestra.run.trial110.stdout: ceph-grafana-dashboards-2:20.3.0-5145.g5fdb7d8f.el9.noarch 2026-02-12T12:17:31.078 INFO:teuthology.orchestra.run.trial110.stdout: ceph-mds-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:17:31.078 INFO:teuthology.orchestra.run.trial110.stdout: ceph-mgr-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:17:31.078 INFO:teuthology.orchestra.run.trial110.stdout: ceph-mgr-cephadm-2:20.3.0-5145.g5fdb7d8f.el9.noarch 2026-02-12T12:17:31.078 INFO:teuthology.orchestra.run.trial110.stdout: ceph-mgr-dashboard-2:20.3.0-5145.g5fdb7d8f.el9.noarch 2026-02-12T12:17:31.078 INFO:teuthology.orchestra.run.trial110.stdout: ceph-mgr-diskprediction-local-2:20.3.0-5145.g5fdb7d8f.el9.noarch 2026-02-12T12:17:31.078 INFO:teuthology.orchestra.run.trial110.stdout: ceph-mgr-k8sevents-2:20.3.0-5145.g5fdb7d8f.el9.noarch 2026-02-12T12:17:31.078 INFO:teuthology.orchestra.run.trial110.stdout: ceph-mgr-modules-core-2:20.3.0-5145.g5fdb7d8f.el9.noarch 2026-02-12T12:17:31.078 INFO:teuthology.orchestra.run.trial110.stdout: ceph-mgr-rook-2:20.3.0-5145.g5fdb7d8f.el9.noarch 2026-02-12T12:17:31.079 INFO:teuthology.orchestra.run.trial110.stdout: ceph-mon-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:17:31.079 INFO:teuthology.orchestra.run.trial110.stdout: ceph-osd-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:17:31.079 INFO:teuthology.orchestra.run.trial110.stdout: ceph-osd-classic-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:17:31.079 INFO:teuthology.orchestra.run.trial110.stdout: ceph-osd-crimson-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:17:31.079 INFO:teuthology.orchestra.run.trial110.stdout: ceph-prometheus-alerts-2:20.3.0-5145.g5fdb7d8f.el9.noarch 2026-02-12T12:17:31.079 INFO:teuthology.orchestra.run.trial110.stdout: ceph-volume-2:20.3.0-5145.g5fdb7d8f.el9.noarch 2026-02-12T12:17:31.079 INFO:teuthology.orchestra.run.trial110.stdout: cephadm-2:20.3.0-5145.g5fdb7d8f.el9.noarch 2026-02-12T12:17:31.079 INFO:teuthology.orchestra.run.trial110.stdout: cryptsetup-2.8.1-2.el9.x86_64 2026-02-12T12:17:31.079 INFO:teuthology.orchestra.run.trial110.stdout: flexiblas-3.0.4-8.el9.x86_64 2026-02-12T12:17:31.079 INFO:teuthology.orchestra.run.trial110.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2026-02-12T12:17:31.079 INFO:teuthology.orchestra.run.trial110.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2026-02-12T12:17:31.079 INFO:teuthology.orchestra.run.trial110.stdout: grpc-data-1.46.7-10.el9.noarch 2026-02-12T12:17:31.079 INFO:teuthology.orchestra.run.trial110.stdout: ledmon-libs-1.1.0-3.el9.x86_64 2026-02-12T12:17:31.080 INFO:teuthology.orchestra.run.trial110.stdout: libconfig-1.7.2-9.el9.x86_64 2026-02-12T12:17:31.080 INFO:teuthology.orchestra.run.trial110.stdout: libgfortran-11.5.0-14.el9.x86_64 2026-02-12T12:17:31.080 INFO:teuthology.orchestra.run.trial110.stdout: libquadmath-11.5.0-14.el9.x86_64 2026-02-12T12:17:31.080 INFO:teuthology.orchestra.run.trial110.stdout: libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-12T12:17:31.080 INFO:teuthology.orchestra.run.trial110.stdout: lua-5.4.4-4.el9.x86_64 2026-02-12T12:17:31.080 INFO:teuthology.orchestra.run.trial110.stdout: lua-devel-5.4.4-4.el9.x86_64 2026-02-12T12:17:31.080 INFO:teuthology.orchestra.run.trial110.stdout: luarocks-3.9.2-5.el9.noarch 2026-02-12T12:17:31.080 INFO:teuthology.orchestra.run.trial110.stdout: openblas-0.3.29-1.el9.x86_64 2026-02-12T12:17:31.080 INFO:teuthology.orchestra.run.trial110.stdout: openblas-openmp-0.3.29-1.el9.x86_64 2026-02-12T12:17:31.080 INFO:teuthology.orchestra.run.trial110.stdout: protobuf-compiler-3.14.0-17.el9.x86_64 2026-02-12T12:17:31.080 INFO:teuthology.orchestra.run.trial110.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2026-02-12T12:17:31.080 INFO:teuthology.orchestra.run.trial110.stdout: python3-autocommand-2.2.2-8.el9.noarch 2026-02-12T12:17:31.081 INFO:teuthology.orchestra.run.trial110.stdout: python3-babel-2.9.1-2.el9.noarch 2026-02-12T12:17:31.081 INFO:teuthology.orchestra.run.trial110.stdout: python3-backports-tarfile-1.2.0-1.el9.noarch 2026-02-12T12:17:31.081 INFO:teuthology.orchestra.run.trial110.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2026-02-12T12:17:31.081 INFO:teuthology.orchestra.run.trial110.stdout: python3-cachetools-4.2.4-1.el9.noarch 2026-02-12T12:17:31.081 INFO:teuthology.orchestra.run.trial110.stdout: python3-certifi-2023.05.07-4.el9.noarch 2026-02-12T12:17:31.081 INFO:teuthology.orchestra.run.trial110.stdout: python3-cffi-1.14.5-5.el9.x86_64 2026-02-12T12:17:31.081 INFO:teuthology.orchestra.run.trial110.stdout: python3-cheroot-10.0.1-4.el9.noarch 2026-02-12T12:17:31.081 INFO:teuthology.orchestra.run.trial110.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2026-02-12T12:17:31.081 INFO:teuthology.orchestra.run.trial110.stdout: python3-cryptography-36.0.1-5.el9.x86_64 2026-02-12T12:17:31.081 INFO:teuthology.orchestra.run.trial110.stdout: python3-devel-3.9.25-2.el9.x86_64 2026-02-12T12:17:31.081 INFO:teuthology.orchestra.run.trial110.stdout: python3-google-auth-1:2.45.0-1.el9.noarch 2026-02-12T12:17:31.081 INFO:teuthology.orchestra.run.trial110.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2026-02-12T12:17:31.081 INFO:teuthology.orchestra.run.trial110.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2026-02-12T12:17:31.082 INFO:teuthology.orchestra.run.trial110.stdout: python3-influxdb-5.3.1-1.el9.noarch 2026-02-12T12:17:31.082 INFO:teuthology.orchestra.run.trial110.stdout: python3-isodate-0.6.1-3.el9.noarch 2026-02-12T12:17:31.082 INFO:teuthology.orchestra.run.trial110.stdout: python3-jaraco-8.2.1-3.el9.noarch 2026-02-12T12:17:31.082 INFO:teuthology.orchestra.run.trial110.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2026-02-12T12:17:31.082 INFO:teuthology.orchestra.run.trial110.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2026-02-12T12:17:31.082 INFO:teuthology.orchestra.run.trial110.stdout: python3-jaraco-context-6.0.1-3.el9.noarch 2026-02-12T12:17:31.082 INFO:teuthology.orchestra.run.trial110.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2026-02-12T12:17:31.082 INFO:teuthology.orchestra.run.trial110.stdout: python3-jaraco-text-4.0.0-2.el9.noarch 2026-02-12T12:17:31.082 INFO:teuthology.orchestra.run.trial110.stdout: python3-jinja2-2.11.3-8.el9.noarch 2026-02-12T12:17:31.082 INFO:teuthology.orchestra.run.trial110.stdout: python3-jmespath-1.0.1-1.el9.noarch 2026-02-12T12:17:31.082 INFO:teuthology.orchestra.run.trial110.stdout: python3-kubernetes-1:26.1.0-3.el9.noarch 2026-02-12T12:17:31.082 INFO:teuthology.orchestra.run.trial110.stdout: python3-libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-12T12:17:31.083 INFO:teuthology.orchestra.run.trial110.stdout: python3-lxml-4.6.5-3.el9.x86_64 2026-02-12T12:17:31.083 INFO:teuthology.orchestra.run.trial110.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2026-02-12T12:17:31.083 INFO:teuthology.orchestra.run.trial110.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2026-02-12T12:17:31.083 INFO:teuthology.orchestra.run.trial110.stdout: python3-msgpack-1.0.3-2.el9.x86_64 2026-02-12T12:17:31.083 INFO:teuthology.orchestra.run.trial110.stdout: python3-natsort-7.1.1-5.el9.noarch 2026-02-12T12:17:31.083 INFO:teuthology.orchestra.run.trial110.stdout: python3-numpy-1:1.23.5-2.el9.x86_64 2026-02-12T12:17:31.083 INFO:teuthology.orchestra.run.trial110.stdout: python3-numpy-f2py-1:1.23.5-2.el9.x86_64 2026-02-12T12:17:31.083 INFO:teuthology.orchestra.run.trial110.stdout: python3-packaging-20.9-5.el9.noarch 2026-02-12T12:17:31.083 INFO:teuthology.orchestra.run.trial110.stdout: python3-ply-3.11-14.el9.noarch 2026-02-12T12:17:31.083 INFO:teuthology.orchestra.run.trial110.stdout: python3-portend-3.1.0-2.el9.noarch 2026-02-12T12:17:31.083 INFO:teuthology.orchestra.run.trial110.stdout: python3-protobuf-3.14.0-17.el9.noarch 2026-02-12T12:17:31.083 INFO:teuthology.orchestra.run.trial110.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2026-02-12T12:17:31.083 INFO:teuthology.orchestra.run.trial110.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2026-02-12T12:17:31.083 INFO:teuthology.orchestra.run.trial110.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2026-02-12T12:17:31.084 INFO:teuthology.orchestra.run.trial110.stdout: python3-pycparser-2.20-6.el9.noarch 2026-02-12T12:17:31.084 INFO:teuthology.orchestra.run.trial110.stdout: python3-pyparsing-2.4.7-9.el9.noarch 2026-02-12T12:17:31.084 INFO:teuthology.orchestra.run.trial110.stdout: python3-repoze-lru-0.7-16.el9.noarch 2026-02-12T12:17:31.084 INFO:teuthology.orchestra.run.trial110.stdout: python3-requests-2.25.1-10.el9.noarch 2026-02-12T12:17:31.084 INFO:teuthology.orchestra.run.trial110.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2026-02-12T12:17:31.084 INFO:teuthology.orchestra.run.trial110.stdout: python3-routes-2.5.1-5.el9.noarch 2026-02-12T12:17:31.084 INFO:teuthology.orchestra.run.trial110.stdout: python3-rsa-4.9-2.el9.noarch 2026-02-12T12:17:31.084 INFO:teuthology.orchestra.run.trial110.stdout: python3-saml-1.16.0-1.el9.noarch 2026-02-12T12:17:31.084 INFO:teuthology.orchestra.run.trial110.stdout: python3-scipy-1.9.3-2.el9.x86_64 2026-02-12T12:17:31.084 INFO:teuthology.orchestra.run.trial110.stdout: python3-tempora-5.0.0-2.el9.noarch 2026-02-12T12:17:31.084 INFO:teuthology.orchestra.run.trial110.stdout: python3-toml-0.10.2-6.el9.noarch 2026-02-12T12:17:31.084 INFO:teuthology.orchestra.run.trial110.stdout: python3-typing-extensions-4.15.0-1.el9.noarch 2026-02-12T12:17:31.084 INFO:teuthology.orchestra.run.trial110.stdout: python3-urllib3-1.26.5-7.el9.noarch 2026-02-12T12:17:31.085 INFO:teuthology.orchestra.run.trial110.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2026-02-12T12:17:31.085 INFO:teuthology.orchestra.run.trial110.stdout: python3-xmlsec-1.3.13-1.el9.x86_64 2026-02-12T12:17:31.085 INFO:teuthology.orchestra.run.trial110.stdout: python3-xmltodict-0.12.0-15.el9.noarch 2026-02-12T12:17:31.085 INFO:teuthology.orchestra.run.trial110.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2026-02-12T12:17:31.085 INFO:teuthology.orchestra.run.trial110.stdout: unzip-6.0-59.el9.x86_64 2026-02-12T12:17:31.085 INFO:teuthology.orchestra.run.trial110.stdout: xmlsec1-1.2.29-13.el9.x86_64 2026-02-12T12:17:31.085 INFO:teuthology.orchestra.run.trial110.stdout: xmlsec1-openssl-1.2.29-13.el9.x86_64 2026-02-12T12:17:31.085 INFO:teuthology.orchestra.run.trial110.stdout: zip-3.0-35.el9.x86_64 2026-02-12T12:17:31.085 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:17:31.085 INFO:teuthology.orchestra.run.trial110.stdout:Complete! 2026-02-12T12:17:31.140 DEBUG:teuthology.orchestra.run.trial110:> sudo yum -y install ceph-base 2026-02-12T12:17:31.440 INFO:teuthology.orchestra.run.trial110.stdout:Last metadata expiration check: 0:09:47 ago on Thu 12 Feb 2026 12:07:44 PM UTC. 2026-02-12T12:17:31.504 INFO:teuthology.orchestra.run.trial110.stdout:Package ceph-base-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:17:31.522 INFO:teuthology.orchestra.run.trial110.stdout:Dependencies resolved. 2026-02-12T12:17:31.523 INFO:teuthology.orchestra.run.trial110.stdout:Nothing to do. 2026-02-12T12:17:31.523 INFO:teuthology.orchestra.run.trial110.stdout:Complete! 2026-02-12T12:17:31.544 DEBUG:teuthology.orchestra.run.trial110:> sudo yum -y install cephadm 2026-02-12T12:17:31.845 INFO:teuthology.orchestra.run.trial110.stdout:Last metadata expiration check: 0:09:47 ago on Thu 12 Feb 2026 12:07:44 PM UTC. 2026-02-12T12:17:31.910 INFO:teuthology.orchestra.run.trial110.stdout:Package cephadm-2:20.3.0-5145.g5fdb7d8f.el9.noarch is already installed. 2026-02-12T12:17:31.928 INFO:teuthology.orchestra.run.trial110.stdout:Dependencies resolved. 2026-02-12T12:17:31.929 INFO:teuthology.orchestra.run.trial110.stdout:Nothing to do. 2026-02-12T12:17:31.929 INFO:teuthology.orchestra.run.trial110.stdout:Complete! 2026-02-12T12:17:31.950 DEBUG:teuthology.orchestra.run.trial110:> sudo yum -y install ceph-immutable-object-cache 2026-02-12T12:17:32.247 INFO:teuthology.orchestra.run.trial110.stdout:Last metadata expiration check: 0:09:48 ago on Thu 12 Feb 2026 12:07:44 PM UTC. 2026-02-12T12:17:32.332 INFO:teuthology.orchestra.run.trial110.stdout:Dependencies resolved. 2026-02-12T12:17:32.332 INFO:teuthology.orchestra.run.trial110.stdout:================================================================================ 2026-02-12T12:17:32.332 INFO:teuthology.orchestra.run.trial110.stdout: Package Arch Version Repo Size 2026-02-12T12:17:32.332 INFO:teuthology.orchestra.run.trial110.stdout:================================================================================ 2026-02-12T12:17:32.332 INFO:teuthology.orchestra.run.trial110.stdout:Installing: 2026-02-12T12:17:32.332 INFO:teuthology.orchestra.run.trial110.stdout: ceph-immutable-object-cache x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 149 k 2026-02-12T12:17:32.332 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:17:32.332 INFO:teuthology.orchestra.run.trial110.stdout:Transaction Summary 2026-02-12T12:17:32.332 INFO:teuthology.orchestra.run.trial110.stdout:================================================================================ 2026-02-12T12:17:32.333 INFO:teuthology.orchestra.run.trial110.stdout:Install 1 Package 2026-02-12T12:17:32.333 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:17:32.333 INFO:teuthology.orchestra.run.trial110.stdout:Total download size: 149 k 2026-02-12T12:17:32.333 INFO:teuthology.orchestra.run.trial110.stdout:Installed size: 455 k 2026-02-12T12:17:32.333 INFO:teuthology.orchestra.run.trial110.stdout:Downloading Packages: 2026-02-12T12:17:32.968 INFO:teuthology.orchestra.run.trial110.stdout:ceph-immutable-object-cache-20.3.0-5145.g5fdb7d 234 kB/s | 149 kB 00:00 2026-02-12T12:17:32.968 INFO:teuthology.orchestra.run.trial110.stdout:-------------------------------------------------------------------------------- 2026-02-12T12:17:32.969 INFO:teuthology.orchestra.run.trial110.stdout:Total 234 kB/s | 149 kB 00:00 2026-02-12T12:17:32.969 INFO:teuthology.orchestra.run.trial110.stdout:Running transaction check 2026-02-12T12:17:32.973 INFO:teuthology.orchestra.run.trial110.stdout:Transaction check succeeded. 2026-02-12T12:17:32.973 INFO:teuthology.orchestra.run.trial110.stdout:Running transaction test 2026-02-12T12:17:33.005 INFO:teuthology.orchestra.run.trial110.stdout:Transaction test succeeded. 2026-02-12T12:17:33.005 INFO:teuthology.orchestra.run.trial110.stdout:Running transaction 2026-02-12T12:17:33.060 INFO:teuthology.orchestra.run.trial110.stdout: Preparing : 1/1 2026-02-12T12:17:33.062 INFO:teuthology.orchestra.run.trial110.stdout: Installing : ceph-immutable-object-cache-2:20.3.0-5145.g5fdb7d8f. 1/1 2026-02-12T12:17:33.084 INFO:teuthology.orchestra.run.trial110.stdout: Running scriptlet: ceph-immutable-object-cache-2:20.3.0-5145.g5fdb7d8f. 1/1 2026-02-12T12:17:33.085 INFO:teuthology.orchestra.run.trial110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-12T12:17:33.085 INFO:teuthology.orchestra.run.trial110.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2026-02-12T12:17:33.085 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:17:33.359 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : ceph-immutable-object-cache-2:20.3.0-5145.g5fdb7d8f. 1/1 2026-02-12T12:17:33.359 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:17:33.359 INFO:teuthology.orchestra.run.trial110.stdout:Installed: 2026-02-12T12:17:33.360 INFO:teuthology.orchestra.run.trial110.stdout: ceph-immutable-object-cache-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:17:33.360 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:17:33.360 INFO:teuthology.orchestra.run.trial110.stdout:Complete! 2026-02-12T12:17:33.388 DEBUG:teuthology.orchestra.run.trial110:> sudo yum -y install ceph-mgr 2026-02-12T12:17:33.691 INFO:teuthology.orchestra.run.trial110.stdout:Last metadata expiration check: 0:09:49 ago on Thu 12 Feb 2026 12:07:44 PM UTC. 2026-02-12T12:17:33.756 INFO:teuthology.orchestra.run.trial110.stdout:Package ceph-mgr-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:17:33.774 INFO:teuthology.orchestra.run.trial110.stdout:Dependencies resolved. 2026-02-12T12:17:33.775 INFO:teuthology.orchestra.run.trial110.stdout:Nothing to do. 2026-02-12T12:17:33.775 INFO:teuthology.orchestra.run.trial110.stdout:Complete! 2026-02-12T12:17:33.796 DEBUG:teuthology.orchestra.run.trial110:> sudo yum -y install ceph-mgr-dashboard 2026-02-12T12:17:34.093 INFO:teuthology.orchestra.run.trial110.stdout:Last metadata expiration check: 0:09:50 ago on Thu 12 Feb 2026 12:07:44 PM UTC. 2026-02-12T12:17:34.158 INFO:teuthology.orchestra.run.trial110.stdout:Package ceph-mgr-dashboard-2:20.3.0-5145.g5fdb7d8f.el9.noarch is already installed. 2026-02-12T12:17:34.176 INFO:teuthology.orchestra.run.trial110.stdout:Dependencies resolved. 2026-02-12T12:17:34.177 INFO:teuthology.orchestra.run.trial110.stdout:Nothing to do. 2026-02-12T12:17:34.177 INFO:teuthology.orchestra.run.trial110.stdout:Complete! 2026-02-12T12:17:34.198 DEBUG:teuthology.orchestra.run.trial110:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-12T12:17:34.502 INFO:teuthology.orchestra.run.trial110.stdout:Last metadata expiration check: 0:09:50 ago on Thu 12 Feb 2026 12:07:44 PM UTC. 2026-02-12T12:17:34.566 INFO:teuthology.orchestra.run.trial110.stdout:Package ceph-mgr-diskprediction-local-2:20.3.0-5145.g5fdb7d8f.el9.noarch is already installed. 2026-02-12T12:17:34.584 INFO:teuthology.orchestra.run.trial110.stdout:Dependencies resolved. 2026-02-12T12:17:34.585 INFO:teuthology.orchestra.run.trial110.stdout:Nothing to do. 2026-02-12T12:17:34.585 INFO:teuthology.orchestra.run.trial110.stdout:Complete! 2026-02-12T12:17:34.607 DEBUG:teuthology.orchestra.run.trial110:> sudo yum -y install ceph-mgr-rook 2026-02-12T12:17:34.905 INFO:teuthology.orchestra.run.trial110.stdout:Last metadata expiration check: 0:09:50 ago on Thu 12 Feb 2026 12:07:44 PM UTC. 2026-02-12T12:17:34.970 INFO:teuthology.orchestra.run.trial110.stdout:Package ceph-mgr-rook-2:20.3.0-5145.g5fdb7d8f.el9.noarch is already installed. 2026-02-12T12:17:34.988 INFO:teuthology.orchestra.run.trial110.stdout:Dependencies resolved. 2026-02-12T12:17:34.989 INFO:teuthology.orchestra.run.trial110.stdout:Nothing to do. 2026-02-12T12:17:34.989 INFO:teuthology.orchestra.run.trial110.stdout:Complete! 2026-02-12T12:17:35.010 DEBUG:teuthology.orchestra.run.trial110:> sudo yum -y install ceph-mgr-cephadm 2026-02-12T12:17:35.307 INFO:teuthology.orchestra.run.trial110.stdout:Last metadata expiration check: 0:09:51 ago on Thu 12 Feb 2026 12:07:44 PM UTC. 2026-02-12T12:17:35.372 INFO:teuthology.orchestra.run.trial110.stdout:Package ceph-mgr-cephadm-2:20.3.0-5145.g5fdb7d8f.el9.noarch is already installed. 2026-02-12T12:17:35.390 INFO:teuthology.orchestra.run.trial110.stdout:Dependencies resolved. 2026-02-12T12:17:35.391 INFO:teuthology.orchestra.run.trial110.stdout:Nothing to do. 2026-02-12T12:17:35.391 INFO:teuthology.orchestra.run.trial110.stdout:Complete! 2026-02-12T12:17:35.412 DEBUG:teuthology.orchestra.run.trial110:> sudo yum -y install ceph-osd 2026-02-12T12:17:35.711 INFO:teuthology.orchestra.run.trial110.stdout:Last metadata expiration check: 0:09:51 ago on Thu 12 Feb 2026 12:07:44 PM UTC. 2026-02-12T12:17:35.776 INFO:teuthology.orchestra.run.trial110.stdout:Package ceph-osd-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:17:35.794 INFO:teuthology.orchestra.run.trial110.stdout:Dependencies resolved. 2026-02-12T12:17:35.795 INFO:teuthology.orchestra.run.trial110.stdout:Nothing to do. 2026-02-12T12:17:35.795 INFO:teuthology.orchestra.run.trial110.stdout:Complete! 2026-02-12T12:17:35.816 DEBUG:teuthology.orchestra.run.trial110:> sudo yum -y install ceph-osd-classic 2026-02-12T12:17:36.123 INFO:teuthology.orchestra.run.trial110.stdout:Last metadata expiration check: 0:09:52 ago on Thu 12 Feb 2026 12:07:44 PM UTC. 2026-02-12T12:17:36.189 INFO:teuthology.orchestra.run.trial110.stdout:Package ceph-osd-classic-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:17:36.208 INFO:teuthology.orchestra.run.trial110.stdout:Dependencies resolved. 2026-02-12T12:17:36.208 INFO:teuthology.orchestra.run.trial110.stdout:Nothing to do. 2026-02-12T12:17:36.209 INFO:teuthology.orchestra.run.trial110.stdout:Complete! 2026-02-12T12:17:36.230 DEBUG:teuthology.orchestra.run.trial110:> sudo yum -y install ceph-fuse 2026-02-12T12:17:36.532 INFO:teuthology.orchestra.run.trial110.stdout:Last metadata expiration check: 0:09:52 ago on Thu 12 Feb 2026 12:07:44 PM UTC. 2026-02-12T12:17:36.616 INFO:teuthology.orchestra.run.trial110.stdout:Dependencies resolved. 2026-02-12T12:17:36.617 INFO:teuthology.orchestra.run.trial110.stdout:================================================================================ 2026-02-12T12:17:36.617 INFO:teuthology.orchestra.run.trial110.stdout: Package Arch Version Repo Size 2026-02-12T12:17:36.617 INFO:teuthology.orchestra.run.trial110.stdout:================================================================================ 2026-02-12T12:17:36.617 INFO:teuthology.orchestra.run.trial110.stdout:Installing: 2026-02-12T12:17:36.617 INFO:teuthology.orchestra.run.trial110.stdout: ceph-fuse x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 914 k 2026-02-12T12:17:36.617 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:17:36.617 INFO:teuthology.orchestra.run.trial110.stdout:Transaction Summary 2026-02-12T12:17:36.617 INFO:teuthology.orchestra.run.trial110.stdout:================================================================================ 2026-02-12T12:17:36.618 INFO:teuthology.orchestra.run.trial110.stdout:Install 1 Package 2026-02-12T12:17:36.618 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:17:36.618 INFO:teuthology.orchestra.run.trial110.stdout:Total download size: 914 k 2026-02-12T12:17:36.618 INFO:teuthology.orchestra.run.trial110.stdout:Installed size: 2.9 M 2026-02-12T12:17:36.618 INFO:teuthology.orchestra.run.trial110.stdout:Downloading Packages: 2026-02-12T12:17:37.447 INFO:teuthology.orchestra.run.trial110.stdout:ceph-fuse-20.3.0-5145.g5fdb7d8f.el9.x86_64.rpm 1.1 MB/s | 914 kB 00:00 2026-02-12T12:17:37.448 INFO:teuthology.orchestra.run.trial110.stdout:-------------------------------------------------------------------------------- 2026-02-12T12:17:37.448 INFO:teuthology.orchestra.run.trial110.stdout:Total 1.1 MB/s | 914 kB 00:00 2026-02-12T12:17:37.448 INFO:teuthology.orchestra.run.trial110.stdout:Running transaction check 2026-02-12T12:17:37.452 INFO:teuthology.orchestra.run.trial110.stdout:Transaction check succeeded. 2026-02-12T12:17:37.453 INFO:teuthology.orchestra.run.trial110.stdout:Running transaction test 2026-02-12T12:17:37.487 INFO:teuthology.orchestra.run.trial110.stdout:Transaction test succeeded. 2026-02-12T12:17:37.487 INFO:teuthology.orchestra.run.trial110.stdout:Running transaction 2026-02-12T12:17:37.577 INFO:teuthology.orchestra.run.trial110.stdout: Preparing : 1/1 2026-02-12T12:17:37.593 INFO:teuthology.orchestra.run.trial110.stdout: Installing : ceph-fuse-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 1/1 2026-02-12T12:17:37.814 INFO:teuthology.orchestra.run.trial110.stdout: Running scriptlet: ceph-fuse-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 1/1 2026-02-12T12:17:37.871 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : ceph-fuse-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 1/1 2026-02-12T12:17:37.871 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:17:37.871 INFO:teuthology.orchestra.run.trial110.stdout:Installed: 2026-02-12T12:17:37.871 INFO:teuthology.orchestra.run.trial110.stdout: ceph-fuse-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:17:37.871 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:17:37.871 INFO:teuthology.orchestra.run.trial110.stdout:Complete! 2026-02-12T12:17:37.899 DEBUG:teuthology.orchestra.run.trial110:> sudo yum -y install ceph-volume 2026-02-12T12:17:38.199 INFO:teuthology.orchestra.run.trial110.stdout:Last metadata expiration check: 0:09:54 ago on Thu 12 Feb 2026 12:07:44 PM UTC. 2026-02-12T12:17:38.264 INFO:teuthology.orchestra.run.trial110.stdout:Package ceph-volume-2:20.3.0-5145.g5fdb7d8f.el9.noarch is already installed. 2026-02-12T12:17:38.281 INFO:teuthology.orchestra.run.trial110.stdout:Dependencies resolved. 2026-02-12T12:17:38.282 INFO:teuthology.orchestra.run.trial110.stdout:Nothing to do. 2026-02-12T12:17:38.282 INFO:teuthology.orchestra.run.trial110.stdout:Complete! 2026-02-12T12:17:38.303 DEBUG:teuthology.orchestra.run.trial110:> sudo yum -y install librados-devel 2026-02-12T12:17:38.605 INFO:teuthology.orchestra.run.trial110.stdout:Last metadata expiration check: 0:09:54 ago on Thu 12 Feb 2026 12:07:44 PM UTC. 2026-02-12T12:17:38.687 INFO:teuthology.orchestra.run.trial110.stdout:Dependencies resolved. 2026-02-12T12:17:38.688 INFO:teuthology.orchestra.run.trial110.stdout:================================================================================ 2026-02-12T12:17:38.688 INFO:teuthology.orchestra.run.trial110.stdout: Package Arch Version Repo Size 2026-02-12T12:17:38.689 INFO:teuthology.orchestra.run.trial110.stdout:================================================================================ 2026-02-12T12:17:38.689 INFO:teuthology.orchestra.run.trial110.stdout:Installing: 2026-02-12T12:17:38.689 INFO:teuthology.orchestra.run.trial110.stdout: librados-devel x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 124 k 2026-02-12T12:17:38.689 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:17:38.689 INFO:teuthology.orchestra.run.trial110.stdout:Transaction Summary 2026-02-12T12:17:38.689 INFO:teuthology.orchestra.run.trial110.stdout:================================================================================ 2026-02-12T12:17:38.689 INFO:teuthology.orchestra.run.trial110.stdout:Install 1 Package 2026-02-12T12:17:38.689 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:17:38.689 INFO:teuthology.orchestra.run.trial110.stdout:Total download size: 124 k 2026-02-12T12:17:38.689 INFO:teuthology.orchestra.run.trial110.stdout:Installed size: 434 k 2026-02-12T12:17:38.690 INFO:teuthology.orchestra.run.trial110.stdout:Downloading Packages: 2026-02-12T12:17:39.245 INFO:teuthology.orchestra.run.trial110.stdout:librados-devel-20.3.0-5145.g5fdb7d8f.el9.x86_64 222 kB/s | 124 kB 00:00 2026-02-12T12:17:39.245 INFO:teuthology.orchestra.run.trial110.stdout:-------------------------------------------------------------------------------- 2026-02-12T12:17:39.246 INFO:teuthology.orchestra.run.trial110.stdout:Total 222 kB/s | 124 kB 00:00 2026-02-12T12:17:39.246 INFO:teuthology.orchestra.run.trial110.stdout:Running transaction check 2026-02-12T12:17:39.248 INFO:teuthology.orchestra.run.trial110.stdout:Transaction check succeeded. 2026-02-12T12:17:39.249 INFO:teuthology.orchestra.run.trial110.stdout:Running transaction test 2026-02-12T12:17:39.281 INFO:teuthology.orchestra.run.trial110.stdout:Transaction test succeeded. 2026-02-12T12:17:39.281 INFO:teuthology.orchestra.run.trial110.stdout:Running transaction 2026-02-12T12:17:39.335 INFO:teuthology.orchestra.run.trial110.stdout: Preparing : 1/1 2026-02-12T12:17:39.351 INFO:teuthology.orchestra.run.trial110.stdout: Installing : librados-devel-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 1/1 2026-02-12T12:17:39.432 INFO:teuthology.orchestra.run.trial110.stdout: Running scriptlet: librados-devel-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 1/1 2026-02-12T12:17:39.489 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : librados-devel-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 1/1 2026-02-12T12:17:39.489 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:17:39.489 INFO:teuthology.orchestra.run.trial110.stdout:Installed: 2026-02-12T12:17:39.489 INFO:teuthology.orchestra.run.trial110.stdout: librados-devel-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:17:39.489 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:17:39.490 INFO:teuthology.orchestra.run.trial110.stdout:Complete! 2026-02-12T12:17:39.518 DEBUG:teuthology.orchestra.run.trial110:> sudo yum -y install libcephfs2 2026-02-12T12:17:39.819 INFO:teuthology.orchestra.run.trial110.stdout:Last metadata expiration check: 0:09:55 ago on Thu 12 Feb 2026 12:07:44 PM UTC. 2026-02-12T12:17:39.884 INFO:teuthology.orchestra.run.trial110.stdout:Package libcephfs2-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:17:39.901 INFO:teuthology.orchestra.run.trial110.stdout:Dependencies resolved. 2026-02-12T12:17:39.901 INFO:teuthology.orchestra.run.trial110.stdout:Nothing to do. 2026-02-12T12:17:39.902 INFO:teuthology.orchestra.run.trial110.stdout:Complete! 2026-02-12T12:17:39.923 DEBUG:teuthology.orchestra.run.trial110:> sudo yum -y install libcephfs-devel 2026-02-12T12:17:40.225 INFO:teuthology.orchestra.run.trial110.stdout:Last metadata expiration check: 0:09:56 ago on Thu 12 Feb 2026 12:07:44 PM UTC. 2026-02-12T12:17:40.307 INFO:teuthology.orchestra.run.trial110.stdout:Dependencies resolved. 2026-02-12T12:17:40.308 INFO:teuthology.orchestra.run.trial110.stdout:================================================================================ 2026-02-12T12:17:40.308 INFO:teuthology.orchestra.run.trial110.stdout: Package Arch Version Repo Size 2026-02-12T12:17:40.308 INFO:teuthology.orchestra.run.trial110.stdout:================================================================================ 2026-02-12T12:17:40.308 INFO:teuthology.orchestra.run.trial110.stdout:Installing: 2026-02-12T12:17:40.308 INFO:teuthology.orchestra.run.trial110.stdout: libcephfs-devel x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 37 k 2026-02-12T12:17:40.308 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:17:40.308 INFO:teuthology.orchestra.run.trial110.stdout:Transaction Summary 2026-02-12T12:17:40.308 INFO:teuthology.orchestra.run.trial110.stdout:================================================================================ 2026-02-12T12:17:40.308 INFO:teuthology.orchestra.run.trial110.stdout:Install 1 Package 2026-02-12T12:17:40.308 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:17:40.309 INFO:teuthology.orchestra.run.trial110.stdout:Total download size: 37 k 2026-02-12T12:17:40.309 INFO:teuthology.orchestra.run.trial110.stdout:Installed size: 171 k 2026-02-12T12:17:40.309 INFO:teuthology.orchestra.run.trial110.stdout:Downloading Packages: 2026-02-12T12:17:40.661 INFO:teuthology.orchestra.run.trial110.stdout:libcephfs-devel-20.3.0-5145.g5fdb7d8f.el9.x86_6 106 kB/s | 37 kB 00:00 2026-02-12T12:17:40.661 INFO:teuthology.orchestra.run.trial110.stdout:-------------------------------------------------------------------------------- 2026-02-12T12:17:40.661 INFO:teuthology.orchestra.run.trial110.stdout:Total 105 kB/s | 37 kB 00:00 2026-02-12T12:17:40.661 INFO:teuthology.orchestra.run.trial110.stdout:Running transaction check 2026-02-12T12:17:40.664 INFO:teuthology.orchestra.run.trial110.stdout:Transaction check succeeded. 2026-02-12T12:17:40.664 INFO:teuthology.orchestra.run.trial110.stdout:Running transaction test 2026-02-12T12:17:40.667 INFO:teuthology.orchestra.run.trial110.stdout:Transaction test succeeded. 2026-02-12T12:17:40.667 INFO:teuthology.orchestra.run.trial110.stdout:Running transaction 2026-02-12T12:17:40.689 INFO:teuthology.orchestra.run.trial110.stdout: Preparing : 1/1 2026-02-12T12:17:40.706 INFO:teuthology.orchestra.run.trial110.stdout: Installing : libcephfs-devel-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 1/1 2026-02-12T12:17:40.760 INFO:teuthology.orchestra.run.trial110.stdout: Running scriptlet: libcephfs-devel-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 1/1 2026-02-12T12:17:40.816 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : libcephfs-devel-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 1/1 2026-02-12T12:17:40.816 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:17:40.816 INFO:teuthology.orchestra.run.trial110.stdout:Installed: 2026-02-12T12:17:40.817 INFO:teuthology.orchestra.run.trial110.stdout: libcephfs-devel-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:17:40.817 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:17:40.817 INFO:teuthology.orchestra.run.trial110.stdout:Complete! 2026-02-12T12:17:40.842 DEBUG:teuthology.orchestra.run.trial110:> sudo yum -y install librados2 2026-02-12T12:17:41.146 INFO:teuthology.orchestra.run.trial110.stdout:Last metadata expiration check: 0:09:57 ago on Thu 12 Feb 2026 12:07:44 PM UTC. 2026-02-12T12:17:41.210 INFO:teuthology.orchestra.run.trial110.stdout:Package librados2-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:17:41.229 INFO:teuthology.orchestra.run.trial110.stdout:Dependencies resolved. 2026-02-12T12:17:41.230 INFO:teuthology.orchestra.run.trial110.stdout:Nothing to do. 2026-02-12T12:17:41.230 INFO:teuthology.orchestra.run.trial110.stdout:Complete! 2026-02-12T12:17:41.250 DEBUG:teuthology.orchestra.run.trial110:> sudo yum -y install librbd1 2026-02-12T12:17:41.550 INFO:teuthology.orchestra.run.trial110.stdout:Last metadata expiration check: 0:09:57 ago on Thu 12 Feb 2026 12:07:44 PM UTC. 2026-02-12T12:17:41.614 INFO:teuthology.orchestra.run.trial110.stdout:Package librbd1-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:17:41.632 INFO:teuthology.orchestra.run.trial110.stdout:Dependencies resolved. 2026-02-12T12:17:41.632 INFO:teuthology.orchestra.run.trial110.stdout:Nothing to do. 2026-02-12T12:17:41.632 INFO:teuthology.orchestra.run.trial110.stdout:Complete! 2026-02-12T12:17:41.654 DEBUG:teuthology.orchestra.run.trial110:> sudo yum -y install python3-rados 2026-02-12T12:17:41.952 INFO:teuthology.orchestra.run.trial110.stdout:Last metadata expiration check: 0:09:57 ago on Thu 12 Feb 2026 12:07:44 PM UTC. 2026-02-12T12:17:42.017 INFO:teuthology.orchestra.run.trial110.stdout:Package python3-rados-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:17:42.034 INFO:teuthology.orchestra.run.trial110.stdout:Dependencies resolved. 2026-02-12T12:17:42.035 INFO:teuthology.orchestra.run.trial110.stdout:Nothing to do. 2026-02-12T12:17:42.035 INFO:teuthology.orchestra.run.trial110.stdout:Complete! 2026-02-12T12:17:42.056 DEBUG:teuthology.orchestra.run.trial110:> sudo yum -y install python3-rgw 2026-02-12T12:17:42.357 INFO:teuthology.orchestra.run.trial110.stdout:Last metadata expiration check: 0:09:58 ago on Thu 12 Feb 2026 12:07:44 PM UTC. 2026-02-12T12:17:42.422 INFO:teuthology.orchestra.run.trial110.stdout:Package python3-rgw-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:17:42.440 INFO:teuthology.orchestra.run.trial110.stdout:Dependencies resolved. 2026-02-12T12:17:42.440 INFO:teuthology.orchestra.run.trial110.stdout:Nothing to do. 2026-02-12T12:17:42.441 INFO:teuthology.orchestra.run.trial110.stdout:Complete! 2026-02-12T12:17:42.461 DEBUG:teuthology.orchestra.run.trial110:> sudo yum -y install python3-cephfs 2026-02-12T12:17:42.759 INFO:teuthology.orchestra.run.trial110.stdout:Last metadata expiration check: 0:09:58 ago on Thu 12 Feb 2026 12:07:44 PM UTC. 2026-02-12T12:17:42.824 INFO:teuthology.orchestra.run.trial110.stdout:Package python3-cephfs-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:17:42.841 INFO:teuthology.orchestra.run.trial110.stdout:Dependencies resolved. 2026-02-12T12:17:42.841 INFO:teuthology.orchestra.run.trial110.stdout:Nothing to do. 2026-02-12T12:17:42.842 INFO:teuthology.orchestra.run.trial110.stdout:Complete! 2026-02-12T12:17:42.862 DEBUG:teuthology.orchestra.run.trial110:> sudo yum -y install python3-rbd 2026-02-12T12:17:43.163 INFO:teuthology.orchestra.run.trial110.stdout:Last metadata expiration check: 0:09:59 ago on Thu 12 Feb 2026 12:07:44 PM UTC. 2026-02-12T12:17:43.228 INFO:teuthology.orchestra.run.trial110.stdout:Package python3-rbd-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:17:43.245 INFO:teuthology.orchestra.run.trial110.stdout:Dependencies resolved. 2026-02-12T12:17:43.246 INFO:teuthology.orchestra.run.trial110.stdout:Nothing to do. 2026-02-12T12:17:43.246 INFO:teuthology.orchestra.run.trial110.stdout:Complete! 2026-02-12T12:17:43.267 DEBUG:teuthology.orchestra.run.trial110:> sudo yum -y install rbd-fuse 2026-02-12T12:17:43.563 INFO:teuthology.orchestra.run.trial110.stdout:Last metadata expiration check: 0:09:59 ago on Thu 12 Feb 2026 12:07:44 PM UTC. 2026-02-12T12:17:43.645 INFO:teuthology.orchestra.run.trial110.stdout:Dependencies resolved. 2026-02-12T12:17:43.646 INFO:teuthology.orchestra.run.trial110.stdout:================================================================================ 2026-02-12T12:17:43.646 INFO:teuthology.orchestra.run.trial110.stdout: Package Architecture Version Repository Size 2026-02-12T12:17:43.646 INFO:teuthology.orchestra.run.trial110.stdout:================================================================================ 2026-02-12T12:17:43.646 INFO:teuthology.orchestra.run.trial110.stdout:Installing: 2026-02-12T12:17:43.647 INFO:teuthology.orchestra.run.trial110.stdout: rbd-fuse x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 87 k 2026-02-12T12:17:43.647 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:17:43.647 INFO:teuthology.orchestra.run.trial110.stdout:Transaction Summary 2026-02-12T12:17:43.647 INFO:teuthology.orchestra.run.trial110.stdout:================================================================================ 2026-02-12T12:17:43.647 INFO:teuthology.orchestra.run.trial110.stdout:Install 1 Package 2026-02-12T12:17:43.647 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:17:43.647 INFO:teuthology.orchestra.run.trial110.stdout:Total download size: 87 k 2026-02-12T12:17:43.647 INFO:teuthology.orchestra.run.trial110.stdout:Installed size: 236 k 2026-02-12T12:17:43.647 INFO:teuthology.orchestra.run.trial110.stdout:Downloading Packages: 2026-02-12T12:17:44.152 INFO:teuthology.orchestra.run.trial110.stdout:rbd-fuse-20.3.0-5145.g5fdb7d8f.el9.x86_64.rpm 171 kB/s | 87 kB 00:00 2026-02-12T12:17:44.152 INFO:teuthology.orchestra.run.trial110.stdout:-------------------------------------------------------------------------------- 2026-02-12T12:17:44.152 INFO:teuthology.orchestra.run.trial110.stdout:Total 171 kB/s | 87 kB 00:00 2026-02-12T12:17:44.153 INFO:teuthology.orchestra.run.trial110.stdout:Running transaction check 2026-02-12T12:17:44.156 INFO:teuthology.orchestra.run.trial110.stdout:Transaction check succeeded. 2026-02-12T12:17:44.157 INFO:teuthology.orchestra.run.trial110.stdout:Running transaction test 2026-02-12T12:17:44.189 INFO:teuthology.orchestra.run.trial110.stdout:Transaction test succeeded. 2026-02-12T12:17:44.190 INFO:teuthology.orchestra.run.trial110.stdout:Running transaction 2026-02-12T12:17:44.241 INFO:teuthology.orchestra.run.trial110.stdout: Preparing : 1/1 2026-02-12T12:17:44.258 INFO:teuthology.orchestra.run.trial110.stdout: Installing : rbd-fuse-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 1/1 2026-02-12T12:17:44.335 INFO:teuthology.orchestra.run.trial110.stdout: Running scriptlet: rbd-fuse-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 1/1 2026-02-12T12:17:44.392 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : rbd-fuse-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 1/1 2026-02-12T12:17:44.392 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:17:44.392 INFO:teuthology.orchestra.run.trial110.stdout:Installed: 2026-02-12T12:17:44.392 INFO:teuthology.orchestra.run.trial110.stdout: rbd-fuse-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:17:44.392 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:17:44.392 INFO:teuthology.orchestra.run.trial110.stdout:Complete! 2026-02-12T12:17:44.419 DEBUG:teuthology.orchestra.run.trial110:> sudo yum -y install rbd-mirror 2026-02-12T12:17:44.719 INFO:teuthology.orchestra.run.trial110.stdout:Last metadata expiration check: 0:10:00 ago on Thu 12 Feb 2026 12:07:44 PM UTC. 2026-02-12T12:17:44.802 INFO:teuthology.orchestra.run.trial110.stdout:Dependencies resolved. 2026-02-12T12:17:44.803 INFO:teuthology.orchestra.run.trial110.stdout:================================================================================ 2026-02-12T12:17:44.803 INFO:teuthology.orchestra.run.trial110.stdout: Package Arch Version Repo Size 2026-02-12T12:17:44.803 INFO:teuthology.orchestra.run.trial110.stdout:================================================================================ 2026-02-12T12:17:44.803 INFO:teuthology.orchestra.run.trial110.stdout:Installing: 2026-02-12T12:17:44.803 INFO:teuthology.orchestra.run.trial110.stdout: rbd-mirror x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 2.6 M 2026-02-12T12:17:44.803 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:17:44.803 INFO:teuthology.orchestra.run.trial110.stdout:Transaction Summary 2026-02-12T12:17:44.804 INFO:teuthology.orchestra.run.trial110.stdout:================================================================================ 2026-02-12T12:17:44.804 INFO:teuthology.orchestra.run.trial110.stdout:Install 1 Package 2026-02-12T12:17:44.804 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:17:44.804 INFO:teuthology.orchestra.run.trial110.stdout:Total download size: 2.6 M 2026-02-12T12:17:44.804 INFO:teuthology.orchestra.run.trial110.stdout:Installed size: 11 M 2026-02-12T12:17:44.804 INFO:teuthology.orchestra.run.trial110.stdout:Downloading Packages: 2026-02-12T12:17:48.358 INFO:teuthology.orchestra.run.trial110.stdout:rbd-mirror-20.3.0-5145.g5fdb7d8f.el9.x86_64.rpm 747 kB/s | 2.6 MB 00:03 2026-02-12T12:17:48.358 INFO:teuthology.orchestra.run.trial110.stdout:-------------------------------------------------------------------------------- 2026-02-12T12:17:48.358 INFO:teuthology.orchestra.run.trial110.stdout:Total 747 kB/s | 2.6 MB 00:03 2026-02-12T12:17:48.359 INFO:teuthology.orchestra.run.trial110.stdout:Running transaction check 2026-02-12T12:17:48.364 INFO:teuthology.orchestra.run.trial110.stdout:Transaction check succeeded. 2026-02-12T12:17:48.364 INFO:teuthology.orchestra.run.trial110.stdout:Running transaction test 2026-02-12T12:17:48.403 INFO:teuthology.orchestra.run.trial110.stdout:Transaction test succeeded. 2026-02-12T12:17:48.403 INFO:teuthology.orchestra.run.trial110.stdout:Running transaction 2026-02-12T12:17:48.565 INFO:teuthology.orchestra.run.trial110.stdout: Preparing : 1/1 2026-02-12T12:17:48.566 INFO:teuthology.orchestra.run.trial110.stdout: Installing : rbd-mirror-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 1/1 2026-02-12T12:17:48.590 INFO:teuthology.orchestra.run.trial110.stdout: Running scriptlet: rbd-mirror-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 1/1 2026-02-12T12:17:48.590 INFO:teuthology.orchestra.run.trial110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-12T12:17:48.590 INFO:teuthology.orchestra.run.trial110.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2026-02-12T12:17:48.590 INFO:teuthology.orchestra.run.trial110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-12T12:17:48.590 INFO:teuthology.orchestra.run.trial110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-12T12:17:48.590 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:17:48.897 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : rbd-mirror-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 1/1 2026-02-12T12:17:48.898 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:17:48.898 INFO:teuthology.orchestra.run.trial110.stdout:Installed: 2026-02-12T12:17:48.898 INFO:teuthology.orchestra.run.trial110.stdout: rbd-mirror-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:17:48.898 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:17:48.898 INFO:teuthology.orchestra.run.trial110.stdout:Complete! 2026-02-12T12:17:48.926 DEBUG:teuthology.orchestra.run.trial110:> sudo yum -y install rbd-nbd 2026-02-12T12:17:49.223 INFO:teuthology.orchestra.run.trial110.stdout:Last metadata expiration check: 0:10:05 ago on Thu 12 Feb 2026 12:07:44 PM UTC. 2026-02-12T12:17:49.304 INFO:teuthology.orchestra.run.trial110.stdout:Dependencies resolved. 2026-02-12T12:17:49.305 INFO:teuthology.orchestra.run.trial110.stdout:================================================================================ 2026-02-12T12:17:49.305 INFO:teuthology.orchestra.run.trial110.stdout: Package Architecture Version Repository Size 2026-02-12T12:17:49.305 INFO:teuthology.orchestra.run.trial110.stdout:================================================================================ 2026-02-12T12:17:49.306 INFO:teuthology.orchestra.run.trial110.stdout:Installing: 2026-02-12T12:17:49.306 INFO:teuthology.orchestra.run.trial110.stdout: rbd-nbd x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 167 k 2026-02-12T12:17:49.306 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:17:49.306 INFO:teuthology.orchestra.run.trial110.stdout:Transaction Summary 2026-02-12T12:17:49.306 INFO:teuthology.orchestra.run.trial110.stdout:================================================================================ 2026-02-12T12:17:49.306 INFO:teuthology.orchestra.run.trial110.stdout:Install 1 Package 2026-02-12T12:17:49.306 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:17:49.306 INFO:teuthology.orchestra.run.trial110.stdout:Total download size: 167 k 2026-02-12T12:17:49.306 INFO:teuthology.orchestra.run.trial110.stdout:Installed size: 496 k 2026-02-12T12:17:49.306 INFO:teuthology.orchestra.run.trial110.stdout:Downloading Packages: 2026-02-12T12:17:50.167 INFO:teuthology.orchestra.run.trial110.stdout:rbd-nbd-20.3.0-5145.g5fdb7d8f.el9.x86_64.rpm 193 kB/s | 167 kB 00:00 2026-02-12T12:17:50.168 INFO:teuthology.orchestra.run.trial110.stdout:-------------------------------------------------------------------------------- 2026-02-12T12:17:50.168 INFO:teuthology.orchestra.run.trial110.stdout:Total 193 kB/s | 167 kB 00:00 2026-02-12T12:17:50.168 INFO:teuthology.orchestra.run.trial110.stdout:Running transaction check 2026-02-12T12:17:50.173 INFO:teuthology.orchestra.run.trial110.stdout:Transaction check succeeded. 2026-02-12T12:17:50.173 INFO:teuthology.orchestra.run.trial110.stdout:Running transaction test 2026-02-12T12:17:50.205 INFO:teuthology.orchestra.run.trial110.stdout:Transaction test succeeded. 2026-02-12T12:17:50.205 INFO:teuthology.orchestra.run.trial110.stdout:Running transaction 2026-02-12T12:17:50.260 INFO:teuthology.orchestra.run.trial110.stdout: Preparing : 1/1 2026-02-12T12:17:50.277 INFO:teuthology.orchestra.run.trial110.stdout: Installing : rbd-nbd-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 1/1 2026-02-12T12:17:50.355 INFO:teuthology.orchestra.run.trial110.stdout: Running scriptlet: rbd-nbd-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 1/1 2026-02-12T12:17:50.412 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : rbd-nbd-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 1/1 2026-02-12T12:17:50.412 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:17:50.412 INFO:teuthology.orchestra.run.trial110.stdout:Installed: 2026-02-12T12:17:50.412 INFO:teuthology.orchestra.run.trial110.stdout: rbd-nbd-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:17:50.412 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:17:50.412 INFO:teuthology.orchestra.run.trial110.stdout:Complete! 2026-02-12T12:17:50.441 DEBUG:teuthology.orchestra.run.trial110:> sudo yum -y install ceph-osd-crimson 2026-02-12T12:17:50.742 INFO:teuthology.orchestra.run.trial110.stdout:Last metadata expiration check: 0:10:06 ago on Thu 12 Feb 2026 12:07:44 PM UTC. 2026-02-12T12:17:50.806 INFO:teuthology.orchestra.run.trial110.stdout:Package ceph-osd-crimson-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:17:50.824 INFO:teuthology.orchestra.run.trial110.stdout:Dependencies resolved. 2026-02-12T12:17:50.825 INFO:teuthology.orchestra.run.trial110.stdout:Nothing to do. 2026-02-12T12:17:50.825 INFO:teuthology.orchestra.run.trial110.stdout:Complete! 2026-02-12T12:17:50.846 DEBUG:teuthology.orchestra.run.trial110:> sudo yum -y install ceph-osd-crimson-debuginfo 2026-02-12T12:17:51.147 INFO:teuthology.orchestra.run.trial110.stdout:Last metadata expiration check: 0:10:07 ago on Thu 12 Feb 2026 12:07:44 PM UTC. 2026-02-12T12:17:51.230 INFO:teuthology.orchestra.run.trial110.stdout:Dependencies resolved. 2026-02-12T12:17:51.231 INFO:teuthology.orchestra.run.trial110.stdout:================================================================================ 2026-02-12T12:17:51.231 INFO:teuthology.orchestra.run.trial110.stdout: Package Arch Version Repo Size 2026-02-12T12:17:51.231 INFO:teuthology.orchestra.run.trial110.stdout:================================================================================ 2026-02-12T12:17:51.231 INFO:teuthology.orchestra.run.trial110.stdout:Installing: 2026-02-12T12:17:51.231 INFO:teuthology.orchestra.run.trial110.stdout: ceph-osd-crimson-debuginfo x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 429 M 2026-02-12T12:17:51.231 INFO:teuthology.orchestra.run.trial110.stdout:Installing weak dependencies: 2026-02-12T12:17:51.231 INFO:teuthology.orchestra.run.trial110.stdout: ceph-debugsource x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 12 M 2026-02-12T12:17:51.231 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:17:51.232 INFO:teuthology.orchestra.run.trial110.stdout:Transaction Summary 2026-02-12T12:17:51.232 INFO:teuthology.orchestra.run.trial110.stdout:================================================================================ 2026-02-12T12:17:51.232 INFO:teuthology.orchestra.run.trial110.stdout:Install 2 Packages 2026-02-12T12:17:51.232 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:17:51.232 INFO:teuthology.orchestra.run.trial110.stdout:Total download size: 441 M 2026-02-12T12:17:51.232 INFO:teuthology.orchestra.run.trial110.stdout:Installed size: 2.7 G 2026-02-12T12:17:51.232 INFO:teuthology.orchestra.run.trial110.stdout:Downloading Packages: 2026-02-12T12:18:19.938 INFO:teuthology.orchestra.run.trial110.stdout:(1/2): ceph-debugsource-20.3.0-5145.g5fdb7d8f.e 429 kB/s | 12 MB 00:28 2026-02-12T12:26:57.851 INFO:teuthology.orchestra.run.trial110.stdout:(2/2): ceph-osd-crimson-debuginfo-20.3.0-5145.g 803 kB/s | 429 MB 09:06 2026-02-12T12:26:57.852 INFO:teuthology.orchestra.run.trial110.stdout:-------------------------------------------------------------------------------- 2026-02-12T12:26:57.852 INFO:teuthology.orchestra.run.trial110.stdout:Total 826 kB/s | 441 MB 09:06 2026-02-12T12:26:57.852 INFO:teuthology.orchestra.run.trial110.stdout:Running transaction check 2026-02-12T12:26:57.858 INFO:teuthology.orchestra.run.trial110.stdout:Transaction check succeeded. 2026-02-12T12:26:57.858 INFO:teuthology.orchestra.run.trial110.stdout:Running transaction test 2026-02-12T12:26:59.223 INFO:teuthology.orchestra.run.trial110.stdout:Transaction test succeeded. 2026-02-12T12:26:59.223 INFO:teuthology.orchestra.run.trial110.stdout:Running transaction 2026-02-12T12:27:02.824 INFO:teuthology.orchestra.run.trial110.stdout: Preparing : 1/1 2026-02-12T12:27:25.287 INFO:teuthology.orchestra.run.trial110.stdout: Installing : ceph-debugsource-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 1/2 2026-02-12T12:27:25.304 INFO:teuthology.orchestra.run.trial110.stdout: Installing : ceph-osd-crimson-debuginfo-2:20.3.0-5145.g5fdb7d8f.e 2/2 2026-02-12T12:27:27.164 INFO:teuthology.orchestra.run.trial110.stdout: Running scriptlet: ceph-osd-crimson-debuginfo-2:20.3.0-5145.g5fdb7d8f.e 2/2 2026-02-12T12:27:27.164 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : ceph-debugsource-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 1/2 2026-02-12T12:27:27.221 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : ceph-osd-crimson-debuginfo-2:20.3.0-5145.g5fdb7d8f.e 2/2 2026-02-12T12:27:27.221 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:27:27.221 INFO:teuthology.orchestra.run.trial110.stdout:Installed: 2026-02-12T12:27:27.221 INFO:teuthology.orchestra.run.trial110.stdout: ceph-debugsource-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:27:27.221 INFO:teuthology.orchestra.run.trial110.stdout: ceph-osd-crimson-debuginfo-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:27:27.221 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:27:27.221 INFO:teuthology.orchestra.run.trial110.stdout:Complete! 2026-02-12T12:27:27.301 DEBUG:teuthology.parallel:result is None 2026-02-12T12:27:27.302 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0 2026-02-12T12:27:27.465 DEBUG:teuthology.orchestra.run.trial110:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-12T12:27:27.487 INFO:teuthology.orchestra.run.trial110.stdout:20.3.0-5145.g5fdb7d8f.el9 2026-02-12T12:27:27.488 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5145.g5fdb7d8f.el9 2026-02-12T12:27:27.488 INFO:teuthology.task.install:The correct ceph version 20.3.0-5145.g5fdb7d8f is installed. 2026-02-12T12:27:27.490 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2026-02-12T12:27:27.491 DEBUG:teuthology.orchestra.run.trial110:> set -ex 2026-02-12T12:27:27.491 DEBUG:teuthology.orchestra.run.trial110:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-12T12:27:27.553 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2026-02-12T12:27:27.554 DEBUG:teuthology.orchestra.run.trial110:> set -ex 2026-02-12T12:27:27.554 DEBUG:teuthology.orchestra.run.trial110:> sudo dd of=/usr/bin/daemon-helper 2026-02-12T12:27:27.618 DEBUG:teuthology.orchestra.run.trial110:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-12T12:27:27.682 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2026-02-12T12:27:27.683 DEBUG:teuthology.orchestra.run.trial110:> set -ex 2026-02-12T12:27:27.683 DEBUG:teuthology.orchestra.run.trial110:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-12T12:27:27.747 DEBUG:teuthology.orchestra.run.trial110:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-12T12:27:27.812 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2026-02-12T12:27:27.812 DEBUG:teuthology.orchestra.run.trial110:> set -ex 2026-02-12T12:27:27.812 DEBUG:teuthology.orchestra.run.trial110:> sudo dd of=/usr/bin/stdin-killer 2026-02-12T12:27:27.878 DEBUG:teuthology.orchestra.run.trial110:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-12T12:27:27.943 INFO:teuthology.run_tasks:Running task pexec... 2026-02-12T12:27:27.962 INFO:teuthology.task.pexec:Executing custom commands... 2026-02-12T12:27:27.963 DEBUG:teuthology.orchestra.run.trial110:> TESTDIR=/home/ubuntu/cephtest bash -s 2026-02-12T12:27:27.985 INFO:teuthology.task.pexec:Running commands on host ubuntu@trial110.front.sepia.ceph.com 2026-02-12T12:27:27.985 INFO:teuthology.task.pexec:sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson 2026-02-12T12:27:28.036 DEBUG:teuthology.parallel:result is None 2026-02-12T12:27:28.036 INFO:teuthology.run_tasks:Running task install... 2026-02-12T12:27:28.050 DEBUG:teuthology.task.install:project ceph 2026-02-12T12:27:28.050 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'debug', 'sha1': '5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0'}} 2026-02-12T12:27:28.051 DEBUG:teuthology.task.install:config {'flavor': 'debug', 'sha1': '5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0'} 2026-02-12T12:27:28.051 INFO:teuthology.task.install:Using flavor: debug 2026-02-12T12:27:28.059 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-osd', 'ceph-osd-classic', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-osd', 'ceph-osd-classic', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-12T12:27:28.059 INFO:teuthology.task.install:extra packages: [] 2026-02-12T12:27:28.060 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'debug', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0', 'tag': None, 'wait_for_package': False} 2026-02-12T12:27:28.060 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0 2026-02-12T12:27:28.215 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-moagrawa-seastore-io-uring-again-crimson-only/5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0/centos/9/flavors/debug/ 2026-02-12T12:27:28.215 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5145.g5fdb7d8f 2026-02-12T12:27:28.596 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-moagrawa-seastore-io-uring-again-crimson-only/5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0/centos/9/flavors/debug/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-moagrawa-seastore-io-uring-again-crimson-only/5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0/centos/9/flavors/debug/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-moagrawa-seastore-io-uring-again-crimson-only/5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0/centos/9/flavors/debug/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-12T12:27:28.597 DEBUG:teuthology.orchestra.run.trial110:> set -ex 2026-02-12T12:27:28.597 DEBUG:teuthology.orchestra.run.trial110:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-12T12:27:28.623 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2026-02-12T12:27:28.623 DEBUG:teuthology.orchestra.run.trial110:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-12T12:27:28.687 DEBUG:teuthology.orchestra.run.trial110:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-12T12:27:28.753 DEBUG:teuthology.orchestra.run.trial110:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-12T12:27:28.810 INFO:teuthology.orchestra.run.trial110.stdout:check_obsoletes = 1 2026-02-12T12:27:28.820 DEBUG:teuthology.orchestra.run.trial110:> sudo yum clean all 2026-02-12T12:27:28.981 INFO:teuthology.orchestra.run.trial110.stdout:56 files removed 2026-02-12T12:27:28.996 DEBUG:teuthology.orchestra.run.trial110:> sudo yum -y install ceph-radosgw 2026-02-12T12:27:29.839 INFO:teuthology.orchestra.run.trial110.stdout:ceph packages for x86_64 135 kB/s | 94 kB 00:00 2026-02-12T12:27:30.497 INFO:teuthology.orchestra.run.trial110.stdout:ceph noarch packages 38 kB/s | 24 kB 00:00 2026-02-12T12:27:31.090 INFO:teuthology.orchestra.run.trial110.stdout:ceph source packages 3.6 kB/s | 2.1 kB 00:00 2026-02-12T12:27:32.359 INFO:teuthology.orchestra.run.trial110.stdout:CentOS Stream 9 - BaseOS 7.1 MB/s | 8.9 MB 00:01 2026-02-12T12:27:35.636 INFO:teuthology.orchestra.run.trial110.stdout:CentOS Stream 9 - AppStream 9.8 MB/s | 27 MB 00:02 2026-02-12T12:27:39.768 INFO:teuthology.orchestra.run.trial110.stdout:CentOS Stream 9 - CRB 4.5 MB/s | 7.8 MB 00:01 2026-02-12T12:27:44.331 INFO:teuthology.orchestra.run.trial110.stdout:CentOS Stream 9 - Extras packages 5.2 kB/s | 20 kB 00:03 2026-02-12T12:27:46.832 INFO:teuthology.orchestra.run.trial110.stdout:Extra Packages for Enterprise Linux 8.4 MB/s | 20 MB 00:02 2026-02-12T12:27:50.009 INFO:teuthology.orchestra.run.trial110.stdout:lab-extras 2.8 MB/s | 50 kB 00:00 2026-02-12T12:27:51.255 INFO:teuthology.orchestra.run.trial110.stdout:Package ceph-radosgw-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:27:51.273 INFO:teuthology.orchestra.run.trial110.stdout:Dependencies resolved. 2026-02-12T12:27:51.273 INFO:teuthology.orchestra.run.trial110.stdout:Nothing to do. 2026-02-12T12:27:51.274 INFO:teuthology.orchestra.run.trial110.stdout:Complete! 2026-02-12T12:27:51.298 DEBUG:teuthology.orchestra.run.trial110:> sudo yum -y install ceph-test 2026-02-12T12:27:51.593 INFO:teuthology.orchestra.run.trial110.stdout:Last metadata expiration check: 0:00:01 ago on Thu 12 Feb 2026 12:27:50 PM UTC. 2026-02-12T12:27:51.657 INFO:teuthology.orchestra.run.trial110.stdout:Package ceph-test-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:27:51.674 INFO:teuthology.orchestra.run.trial110.stdout:Dependencies resolved. 2026-02-12T12:27:51.675 INFO:teuthology.orchestra.run.trial110.stdout:Nothing to do. 2026-02-12T12:27:51.675 INFO:teuthology.orchestra.run.trial110.stdout:Complete! 2026-02-12T12:27:51.696 DEBUG:teuthology.orchestra.run.trial110:> sudo yum -y install ceph 2026-02-12T12:27:51.997 INFO:teuthology.orchestra.run.trial110.stdout:Last metadata expiration check: 0:00:01 ago on Thu 12 Feb 2026 12:27:50 PM UTC. 2026-02-12T12:27:52.061 INFO:teuthology.orchestra.run.trial110.stdout:Package ceph-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:27:52.078 INFO:teuthology.orchestra.run.trial110.stdout:Dependencies resolved. 2026-02-12T12:27:52.079 INFO:teuthology.orchestra.run.trial110.stdout:Nothing to do. 2026-02-12T12:27:52.079 INFO:teuthology.orchestra.run.trial110.stdout:Complete! 2026-02-12T12:27:52.100 DEBUG:teuthology.orchestra.run.trial110:> sudo yum -y install ceph-base 2026-02-12T12:27:52.399 INFO:teuthology.orchestra.run.trial110.stdout:Last metadata expiration check: 0:00:02 ago on Thu 12 Feb 2026 12:27:50 PM UTC. 2026-02-12T12:27:52.462 INFO:teuthology.orchestra.run.trial110.stdout:Package ceph-base-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:27:52.480 INFO:teuthology.orchestra.run.trial110.stdout:Dependencies resolved. 2026-02-12T12:27:52.481 INFO:teuthology.orchestra.run.trial110.stdout:Nothing to do. 2026-02-12T12:27:52.481 INFO:teuthology.orchestra.run.trial110.stdout:Complete! 2026-02-12T12:27:52.502 DEBUG:teuthology.orchestra.run.trial110:> sudo yum -y install cephadm 2026-02-12T12:27:52.797 INFO:teuthology.orchestra.run.trial110.stdout:Last metadata expiration check: 0:00:02 ago on Thu 12 Feb 2026 12:27:50 PM UTC. 2026-02-12T12:27:52.861 INFO:teuthology.orchestra.run.trial110.stdout:Package cephadm-2:20.3.0-5145.g5fdb7d8f.el9.noarch is already installed. 2026-02-12T12:27:52.878 INFO:teuthology.orchestra.run.trial110.stdout:Dependencies resolved. 2026-02-12T12:27:52.879 INFO:teuthology.orchestra.run.trial110.stdout:Nothing to do. 2026-02-12T12:27:52.879 INFO:teuthology.orchestra.run.trial110.stdout:Complete! 2026-02-12T12:27:52.900 DEBUG:teuthology.orchestra.run.trial110:> sudo yum -y install ceph-immutable-object-cache 2026-02-12T12:27:53.204 INFO:teuthology.orchestra.run.trial110.stdout:Last metadata expiration check: 0:00:03 ago on Thu 12 Feb 2026 12:27:50 PM UTC. 2026-02-12T12:27:53.269 INFO:teuthology.orchestra.run.trial110.stdout:Package ceph-immutable-object-cache-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:27:53.287 INFO:teuthology.orchestra.run.trial110.stdout:Dependencies resolved. 2026-02-12T12:27:53.287 INFO:teuthology.orchestra.run.trial110.stdout:Nothing to do. 2026-02-12T12:27:53.287 INFO:teuthology.orchestra.run.trial110.stdout:Complete! 2026-02-12T12:27:53.310 DEBUG:teuthology.orchestra.run.trial110:> sudo yum -y install ceph-mgr 2026-02-12T12:27:53.608 INFO:teuthology.orchestra.run.trial110.stdout:Last metadata expiration check: 0:00:03 ago on Thu 12 Feb 2026 12:27:50 PM UTC. 2026-02-12T12:27:53.672 INFO:teuthology.orchestra.run.trial110.stdout:Package ceph-mgr-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:27:53.690 INFO:teuthology.orchestra.run.trial110.stdout:Dependencies resolved. 2026-02-12T12:27:53.690 INFO:teuthology.orchestra.run.trial110.stdout:Nothing to do. 2026-02-12T12:27:53.691 INFO:teuthology.orchestra.run.trial110.stdout:Complete! 2026-02-12T12:27:53.712 DEBUG:teuthology.orchestra.run.trial110:> sudo yum -y install ceph-mgr-dashboard 2026-02-12T12:27:54.013 INFO:teuthology.orchestra.run.trial110.stdout:Last metadata expiration check: 0:00:04 ago on Thu 12 Feb 2026 12:27:50 PM UTC. 2026-02-12T12:27:54.077 INFO:teuthology.orchestra.run.trial110.stdout:Package ceph-mgr-dashboard-2:20.3.0-5145.g5fdb7d8f.el9.noarch is already installed. 2026-02-12T12:27:54.095 INFO:teuthology.orchestra.run.trial110.stdout:Dependencies resolved. 2026-02-12T12:27:54.095 INFO:teuthology.orchestra.run.trial110.stdout:Nothing to do. 2026-02-12T12:27:54.096 INFO:teuthology.orchestra.run.trial110.stdout:Complete! 2026-02-12T12:27:54.117 DEBUG:teuthology.orchestra.run.trial110:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-12T12:27:54.422 INFO:teuthology.orchestra.run.trial110.stdout:Last metadata expiration check: 0:00:04 ago on Thu 12 Feb 2026 12:27:50 PM UTC. 2026-02-12T12:27:54.487 INFO:teuthology.orchestra.run.trial110.stdout:Package ceph-mgr-diskprediction-local-2:20.3.0-5145.g5fdb7d8f.el9.noarch is already installed. 2026-02-12T12:27:54.504 INFO:teuthology.orchestra.run.trial110.stdout:Dependencies resolved. 2026-02-12T12:27:54.505 INFO:teuthology.orchestra.run.trial110.stdout:Nothing to do. 2026-02-12T12:27:54.505 INFO:teuthology.orchestra.run.trial110.stdout:Complete! 2026-02-12T12:27:54.526 DEBUG:teuthology.orchestra.run.trial110:> sudo yum -y install ceph-mgr-rook 2026-02-12T12:27:54.824 INFO:teuthology.orchestra.run.trial110.stdout:Last metadata expiration check: 0:00:04 ago on Thu 12 Feb 2026 12:27:50 PM UTC. 2026-02-12T12:27:54.889 INFO:teuthology.orchestra.run.trial110.stdout:Package ceph-mgr-rook-2:20.3.0-5145.g5fdb7d8f.el9.noarch is already installed. 2026-02-12T12:27:54.907 INFO:teuthology.orchestra.run.trial110.stdout:Dependencies resolved. 2026-02-12T12:27:54.907 INFO:teuthology.orchestra.run.trial110.stdout:Nothing to do. 2026-02-12T12:27:54.907 INFO:teuthology.orchestra.run.trial110.stdout:Complete! 2026-02-12T12:27:54.929 DEBUG:teuthology.orchestra.run.trial110:> sudo yum -y install ceph-mgr-cephadm 2026-02-12T12:27:55.229 INFO:teuthology.orchestra.run.trial110.stdout:Last metadata expiration check: 0:00:05 ago on Thu 12 Feb 2026 12:27:50 PM UTC. 2026-02-12T12:27:55.294 INFO:teuthology.orchestra.run.trial110.stdout:Package ceph-mgr-cephadm-2:20.3.0-5145.g5fdb7d8f.el9.noarch is already installed. 2026-02-12T12:27:55.312 INFO:teuthology.orchestra.run.trial110.stdout:Dependencies resolved. 2026-02-12T12:27:55.312 INFO:teuthology.orchestra.run.trial110.stdout:Nothing to do. 2026-02-12T12:27:55.313 INFO:teuthology.orchestra.run.trial110.stdout:Complete! 2026-02-12T12:27:55.334 DEBUG:teuthology.orchestra.run.trial110:> sudo yum -y install ceph-osd 2026-02-12T12:27:55.635 INFO:teuthology.orchestra.run.trial110.stdout:Last metadata expiration check: 0:00:05 ago on Thu 12 Feb 2026 12:27:50 PM UTC. 2026-02-12T12:27:55.698 INFO:teuthology.orchestra.run.trial110.stdout:Package ceph-osd-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:27:55.716 INFO:teuthology.orchestra.run.trial110.stdout:Dependencies resolved. 2026-02-12T12:27:55.717 INFO:teuthology.orchestra.run.trial110.stdout:Nothing to do. 2026-02-12T12:27:55.717 INFO:teuthology.orchestra.run.trial110.stdout:Complete! 2026-02-12T12:27:55.737 DEBUG:teuthology.orchestra.run.trial110:> sudo yum -y install ceph-osd-classic 2026-02-12T12:27:56.039 INFO:teuthology.orchestra.run.trial110.stdout:Last metadata expiration check: 0:00:06 ago on Thu 12 Feb 2026 12:27:50 PM UTC. 2026-02-12T12:27:56.103 INFO:teuthology.orchestra.run.trial110.stdout:Package ceph-osd-classic-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:27:56.121 INFO:teuthology.orchestra.run.trial110.stdout:Dependencies resolved. 2026-02-12T12:27:56.122 INFO:teuthology.orchestra.run.trial110.stdout:Nothing to do. 2026-02-12T12:27:56.122 INFO:teuthology.orchestra.run.trial110.stdout:Complete! 2026-02-12T12:27:56.143 DEBUG:teuthology.orchestra.run.trial110:> sudo yum -y install ceph-fuse 2026-02-12T12:27:56.442 INFO:teuthology.orchestra.run.trial110.stdout:Last metadata expiration check: 0:00:06 ago on Thu 12 Feb 2026 12:27:50 PM UTC. 2026-02-12T12:27:56.506 INFO:teuthology.orchestra.run.trial110.stdout:Package ceph-fuse-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:27:56.524 INFO:teuthology.orchestra.run.trial110.stdout:Dependencies resolved. 2026-02-12T12:27:56.524 INFO:teuthology.orchestra.run.trial110.stdout:Nothing to do. 2026-02-12T12:27:56.525 INFO:teuthology.orchestra.run.trial110.stdout:Complete! 2026-02-12T12:27:56.546 DEBUG:teuthology.orchestra.run.trial110:> sudo yum -y install ceph-volume 2026-02-12T12:27:56.843 INFO:teuthology.orchestra.run.trial110.stdout:Last metadata expiration check: 0:00:06 ago on Thu 12 Feb 2026 12:27:50 PM UTC. 2026-02-12T12:27:56.909 INFO:teuthology.orchestra.run.trial110.stdout:Package ceph-volume-2:20.3.0-5145.g5fdb7d8f.el9.noarch is already installed. 2026-02-12T12:27:56.926 INFO:teuthology.orchestra.run.trial110.stdout:Dependencies resolved. 2026-02-12T12:27:56.927 INFO:teuthology.orchestra.run.trial110.stdout:Nothing to do. 2026-02-12T12:27:56.927 INFO:teuthology.orchestra.run.trial110.stdout:Complete! 2026-02-12T12:27:56.949 DEBUG:teuthology.orchestra.run.trial110:> sudo yum -y install librados-devel 2026-02-12T12:27:57.247 INFO:teuthology.orchestra.run.trial110.stdout:Last metadata expiration check: 0:00:07 ago on Thu 12 Feb 2026 12:27:50 PM UTC. 2026-02-12T12:27:57.312 INFO:teuthology.orchestra.run.trial110.stdout:Package librados-devel-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:27:57.330 INFO:teuthology.orchestra.run.trial110.stdout:Dependencies resolved. 2026-02-12T12:27:57.330 INFO:teuthology.orchestra.run.trial110.stdout:Nothing to do. 2026-02-12T12:27:57.331 INFO:teuthology.orchestra.run.trial110.stdout:Complete! 2026-02-12T12:27:57.352 DEBUG:teuthology.orchestra.run.trial110:> sudo yum -y install libcephfs2 2026-02-12T12:27:57.652 INFO:teuthology.orchestra.run.trial110.stdout:Last metadata expiration check: 0:00:07 ago on Thu 12 Feb 2026 12:27:50 PM UTC. 2026-02-12T12:27:57.716 INFO:teuthology.orchestra.run.trial110.stdout:Package libcephfs2-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:27:57.734 INFO:teuthology.orchestra.run.trial110.stdout:Dependencies resolved. 2026-02-12T12:27:57.735 INFO:teuthology.orchestra.run.trial110.stdout:Nothing to do. 2026-02-12T12:27:57.735 INFO:teuthology.orchestra.run.trial110.stdout:Complete! 2026-02-12T12:27:57.757 DEBUG:teuthology.orchestra.run.trial110:> sudo yum -y install libcephfs-devel 2026-02-12T12:27:58.058 INFO:teuthology.orchestra.run.trial110.stdout:Last metadata expiration check: 0:00:08 ago on Thu 12 Feb 2026 12:27:50 PM UTC. 2026-02-12T12:27:58.123 INFO:teuthology.orchestra.run.trial110.stdout:Package libcephfs-devel-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:27:58.141 INFO:teuthology.orchestra.run.trial110.stdout:Dependencies resolved. 2026-02-12T12:27:58.141 INFO:teuthology.orchestra.run.trial110.stdout:Nothing to do. 2026-02-12T12:27:58.141 INFO:teuthology.orchestra.run.trial110.stdout:Complete! 2026-02-12T12:27:58.163 DEBUG:teuthology.orchestra.run.trial110:> sudo yum -y install librados2 2026-02-12T12:27:58.459 INFO:teuthology.orchestra.run.trial110.stdout:Last metadata expiration check: 0:00:08 ago on Thu 12 Feb 2026 12:27:50 PM UTC. 2026-02-12T12:27:58.523 INFO:teuthology.orchestra.run.trial110.stdout:Package librados2-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:27:58.540 INFO:teuthology.orchestra.run.trial110.stdout:Dependencies resolved. 2026-02-12T12:27:58.541 INFO:teuthology.orchestra.run.trial110.stdout:Nothing to do. 2026-02-12T12:27:58.541 INFO:teuthology.orchestra.run.trial110.stdout:Complete! 2026-02-12T12:27:58.562 DEBUG:teuthology.orchestra.run.trial110:> sudo yum -y install librbd1 2026-02-12T12:27:58.860 INFO:teuthology.orchestra.run.trial110.stdout:Last metadata expiration check: 0:00:08 ago on Thu 12 Feb 2026 12:27:50 PM UTC. 2026-02-12T12:27:58.924 INFO:teuthology.orchestra.run.trial110.stdout:Package librbd1-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:27:58.941 INFO:teuthology.orchestra.run.trial110.stdout:Dependencies resolved. 2026-02-12T12:27:58.942 INFO:teuthology.orchestra.run.trial110.stdout:Nothing to do. 2026-02-12T12:27:58.942 INFO:teuthology.orchestra.run.trial110.stdout:Complete! 2026-02-12T12:27:58.964 DEBUG:teuthology.orchestra.run.trial110:> sudo yum -y install python3-rados 2026-02-12T12:27:59.260 INFO:teuthology.orchestra.run.trial110.stdout:Last metadata expiration check: 0:00:09 ago on Thu 12 Feb 2026 12:27:50 PM UTC. 2026-02-12T12:27:59.325 INFO:teuthology.orchestra.run.trial110.stdout:Package python3-rados-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:27:59.342 INFO:teuthology.orchestra.run.trial110.stdout:Dependencies resolved. 2026-02-12T12:27:59.343 INFO:teuthology.orchestra.run.trial110.stdout:Nothing to do. 2026-02-12T12:27:59.343 INFO:teuthology.orchestra.run.trial110.stdout:Complete! 2026-02-12T12:27:59.365 DEBUG:teuthology.orchestra.run.trial110:> sudo yum -y install python3-rgw 2026-02-12T12:27:59.668 INFO:teuthology.orchestra.run.trial110.stdout:Last metadata expiration check: 0:00:09 ago on Thu 12 Feb 2026 12:27:50 PM UTC. 2026-02-12T12:27:59.734 INFO:teuthology.orchestra.run.trial110.stdout:Package python3-rgw-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:27:59.751 INFO:teuthology.orchestra.run.trial110.stdout:Dependencies resolved. 2026-02-12T12:27:59.752 INFO:teuthology.orchestra.run.trial110.stdout:Nothing to do. 2026-02-12T12:27:59.752 INFO:teuthology.orchestra.run.trial110.stdout:Complete! 2026-02-12T12:27:59.774 DEBUG:teuthology.orchestra.run.trial110:> sudo yum -y install python3-cephfs 2026-02-12T12:28:00.073 INFO:teuthology.orchestra.run.trial110.stdout:Last metadata expiration check: 0:00:10 ago on Thu 12 Feb 2026 12:27:50 PM UTC. 2026-02-12T12:28:00.138 INFO:teuthology.orchestra.run.trial110.stdout:Package python3-cephfs-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:28:00.155 INFO:teuthology.orchestra.run.trial110.stdout:Dependencies resolved. 2026-02-12T12:28:00.156 INFO:teuthology.orchestra.run.trial110.stdout:Nothing to do. 2026-02-12T12:28:00.156 INFO:teuthology.orchestra.run.trial110.stdout:Complete! 2026-02-12T12:28:00.178 DEBUG:teuthology.orchestra.run.trial110:> sudo yum -y install python3-rbd 2026-02-12T12:28:00.476 INFO:teuthology.orchestra.run.trial110.stdout:Last metadata expiration check: 0:00:10 ago on Thu 12 Feb 2026 12:27:50 PM UTC. 2026-02-12T12:28:00.541 INFO:teuthology.orchestra.run.trial110.stdout:Package python3-rbd-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:28:00.559 INFO:teuthology.orchestra.run.trial110.stdout:Dependencies resolved. 2026-02-12T12:28:00.559 INFO:teuthology.orchestra.run.trial110.stdout:Nothing to do. 2026-02-12T12:28:00.560 INFO:teuthology.orchestra.run.trial110.stdout:Complete! 2026-02-12T12:28:00.581 DEBUG:teuthology.orchestra.run.trial110:> sudo yum -y install rbd-fuse 2026-02-12T12:28:00.881 INFO:teuthology.orchestra.run.trial110.stdout:Last metadata expiration check: 0:00:10 ago on Thu 12 Feb 2026 12:27:50 PM UTC. 2026-02-12T12:28:00.946 INFO:teuthology.orchestra.run.trial110.stdout:Package rbd-fuse-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:28:00.963 INFO:teuthology.orchestra.run.trial110.stdout:Dependencies resolved. 2026-02-12T12:28:00.964 INFO:teuthology.orchestra.run.trial110.stdout:Nothing to do. 2026-02-12T12:28:00.964 INFO:teuthology.orchestra.run.trial110.stdout:Complete! 2026-02-12T12:28:00.986 DEBUG:teuthology.orchestra.run.trial110:> sudo yum -y install rbd-mirror 2026-02-12T12:28:01.289 INFO:teuthology.orchestra.run.trial110.stdout:Last metadata expiration check: 0:00:11 ago on Thu 12 Feb 2026 12:27:50 PM UTC. 2026-02-12T12:28:01.354 INFO:teuthology.orchestra.run.trial110.stdout:Package rbd-mirror-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:28:01.372 INFO:teuthology.orchestra.run.trial110.stdout:Dependencies resolved. 2026-02-12T12:28:01.372 INFO:teuthology.orchestra.run.trial110.stdout:Nothing to do. 2026-02-12T12:28:01.373 INFO:teuthology.orchestra.run.trial110.stdout:Complete! 2026-02-12T12:28:01.395 DEBUG:teuthology.orchestra.run.trial110:> sudo yum -y install rbd-nbd 2026-02-12T12:28:01.691 INFO:teuthology.orchestra.run.trial110.stdout:Last metadata expiration check: 0:00:11 ago on Thu 12 Feb 2026 12:27:50 PM UTC. 2026-02-12T12:28:01.756 INFO:teuthology.orchestra.run.trial110.stdout:Package rbd-nbd-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 is already installed. 2026-02-12T12:28:01.774 INFO:teuthology.orchestra.run.trial110.stdout:Dependencies resolved. 2026-02-12T12:28:01.774 INFO:teuthology.orchestra.run.trial110.stdout:Nothing to do. 2026-02-12T12:28:01.775 INFO:teuthology.orchestra.run.trial110.stdout:Complete! 2026-02-12T12:28:01.799 DEBUG:teuthology.parallel:result is None 2026-02-12T12:28:01.799 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0 2026-02-12T12:28:01.982 DEBUG:teuthology.orchestra.run.trial110:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-12T12:28:02.003 INFO:teuthology.orchestra.run.trial110.stdout:20.3.0-5145.g5fdb7d8f.el9 2026-02-12T12:28:02.003 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5145.g5fdb7d8f.el9 2026-02-12T12:28:02.004 INFO:teuthology.task.install:The correct ceph version 20.3.0-5145.g5fdb7d8f is installed. 2026-02-12T12:28:02.006 INFO:teuthology.task.install.util:Utilities already shipped, skip it... 2026-02-12T12:28:02.006 INFO:teuthology.run_tasks:Running task ceph... 2026-02-12T12:28:02.117 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2026-02-12T12:28:02.117 DEBUG:teuthology.orchestra.run.trial110:> sudo chmod 777 /var/log/ceph 2026-02-12T12:28:02.142 INFO:tasks.ceph:Disabling ceph logrotate... 2026-02-12T12:28:02.142 DEBUG:teuthology.orchestra.run.trial110:> sudo rm -f -- /etc/logrotate.d/ceph 2026-02-12T12:28:02.206 INFO:tasks.ceph:Creating extra log directories... 2026-02-12T12:28:02.206 DEBUG:teuthology.orchestra.run.trial110:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2026-02-12T12:28:02.283 INFO:tasks.ceph:Creating ceph cluster ceph... 2026-02-12T12:28:02.283 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore cachepin type': '2Q', 'seastore main device type': 'RANDOM_BLOCK_SSD', 'seastore max concurrent transactions': 8}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2026-02-12T12:28:02.283 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/mnt/teuthology/moagrawa-2026-02-10_06:35:43-crimson-rados-main-distro-debug-trial/42493', 'branch': 'main', 'description': 'crimson-rados/perf/{0-crimson_install clusters/{crimson-fixed-1} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/rbm$/{crimson_seastore_rbm_2q} perf_ignorelist settings/optimized workloads/radosbench_4M_rand_read}', 'email': None, 'first_in_suite': False, 'flavor': 'debug', 'job_id': '42493', 'kernel': {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'trial', 'name': 'moagrawa-2026-02-10_06:35:43-crimson-rados-main-distro-debug-trial', 'no_nested_subset': False, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd': {'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore cachepin type': '2Q', 'seastore main device type': 'RANDOM_BLOCK_SSD', 'seastore max concurrent transactions': 8}}, 'crimson_compat': True, 'flavor': 'debug', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH'], 'sha1': '5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'global': {'osd crush chooseleaf type': 0, 'osd pool default pg num': 128, 'osd pool default pgp num': 128, 'osd pool default size': 2}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'debug', 'sha1': '5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0'}}, 'workunit': {'branch': 'seastore_io_uring', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0'}}, 'owner': 'scheduled_moagrawa@soko04.front.sepia.ceph.com', 'priority': 1000, 'repo': 'https://github.com/ceph/ceph.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 2425, 'sha1': '5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'seastore_io_uring', 'suite_path': '/home/teuthworker/src/github.com_mohit84_ceph_5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/mohit84/ceph.git', 'suite_sha1': '5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0', 'targets': {'trial110.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEVV+hvOidUwHbJclrEF8MwmS14TmMH5kZRsU4/p8C1tCb/hTSvq8krxLK6jGv1rAZDERauTUCmyY9peBmhwtyg='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'flavor': 'debug', 'sha1': '5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0'}}, {'pexec': {'all': ['sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson']}}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore cachepin type': '2Q', 'seastore main device type': 'RANDOM_BLOCK_SSD', 'seastore max concurrent transactions': 8}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}}, 'wait-for-scrub': False, 'crimson_compat': True, 'flavor': 'debug', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH'], 'sha1': '5fdb7d8f131dcee024b0dd7d68acdff57e55c4e0', 'cluster': 'ceph'}}, {'ssh_keys': None}, {'cbt': {'benchmarks': {'radosbench': {'concurrent_ops': 4, 'concurrent_procs': 2, 'op_size': [4194304], 'pool_monitoring_list': ['collectl'], 'pool_profile': 'replicated', 'readmode': 'rand', 'run_monitoring_list': ['collectl'], 'time': 60, 'write_only': False}}, 'cluster': {'iterations': 1, 'osds_per_node': 3, 'pool_profiles': {'replicated': {'pg_size': 256, 'pgp_size': 256, 'replication': 'replicated'}}, 'user': 'ubuntu'}, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'c433f1062990a0488dc29a553589bc609a460691', 'timestamp': '2026-02-10_06:35:43', 'tube': 'trial', 'user': 'moagrawa', 'verbose': False, 'worker_log': '/home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.2161924'} 2026-02-12T12:28:02.284 DEBUG:teuthology.orchestra.run.trial110:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2026-02-12T12:28:02.328 DEBUG:teuthology.orchestra.run.trial110:> sudo install -d -m0777 -- /var/run/ceph 2026-02-12T12:28:02.392 DEBUG:teuthology.orchestra.run.trial110:> set -ex 2026-02-12T12:28:02.393 DEBUG:teuthology.orchestra.run.trial110:> dd if=/scratch_devs of=/dev/stdout 2026-02-12T12:28:02.451 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2026-02-12T12:28:02.451 DEBUG:teuthology.orchestra.run.trial110:> stat /dev/vg_nvme/lv_1 2026-02-12T12:28:02.508 INFO:teuthology.orchestra.run.trial110.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2026-02-12T12:28:02.508 INFO:teuthology.orchestra.run.trial110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-12T12:28:02.508 INFO:teuthology.orchestra.run.trial110.stdout:Device: 6h/6d Inode: 1304 Links: 1 2026-02-12T12:28:02.508 INFO:teuthology.orchestra.run.trial110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-12T12:28:02.508 INFO:teuthology.orchestra.run.trial110.stdout:Context: system_u:object_r:device_t:s0 2026-02-12T12:28:02.508 INFO:teuthology.orchestra.run.trial110.stdout:Access: 2026-02-12 12:17:48.733005916 +0000 2026-02-12T12:28:02.508 INFO:teuthology.orchestra.run.trial110.stdout:Modify: 2026-02-12 12:08:25.241328554 +0000 2026-02-12T12:28:02.508 INFO:teuthology.orchestra.run.trial110.stdout:Change: 2026-02-12 12:08:25.241328554 +0000 2026-02-12T12:28:02.508 INFO:teuthology.orchestra.run.trial110.stdout: Birth: 2026-02-12 12:08:25.241328554 +0000 2026-02-12T12:28:02.509 DEBUG:teuthology.orchestra.run.trial110:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2026-02-12T12:28:02.572 INFO:teuthology.orchestra.run.trial110.stderr:1+0 records in 2026-02-12T12:28:02.573 INFO:teuthology.orchestra.run.trial110.stderr:1+0 records out 2026-02-12T12:28:02.573 INFO:teuthology.orchestra.run.trial110.stderr:512 bytes copied, 0.000171488 s, 3.0 MB/s 2026-02-12T12:28:02.574 DEBUG:teuthology.orchestra.run.trial110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2026-02-12T12:28:02.632 DEBUG:teuthology.orchestra.run.trial110:> stat /dev/vg_nvme/lv_2 2026-02-12T12:28:02.688 INFO:teuthology.orchestra.run.trial110.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2026-02-12T12:28:02.688 INFO:teuthology.orchestra.run.trial110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-12T12:28:02.688 INFO:teuthology.orchestra.run.trial110.stdout:Device: 6h/6d Inode: 1298 Links: 1 2026-02-12T12:28:02.688 INFO:teuthology.orchestra.run.trial110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-12T12:28:02.688 INFO:teuthology.orchestra.run.trial110.stdout:Context: system_u:object_r:device_t:s0 2026-02-12T12:28:02.688 INFO:teuthology.orchestra.run.trial110.stdout:Access: 2026-02-12 12:17:48.733005916 +0000 2026-02-12T12:28:02.688 INFO:teuthology.orchestra.run.trial110.stdout:Modify: 2026-02-12 12:08:25.241328554 +0000 2026-02-12T12:28:02.689 INFO:teuthology.orchestra.run.trial110.stdout:Change: 2026-02-12 12:08:25.241328554 +0000 2026-02-12T12:28:02.689 INFO:teuthology.orchestra.run.trial110.stdout: Birth: 2026-02-12 12:08:25.241328554 +0000 2026-02-12T12:28:02.689 DEBUG:teuthology.orchestra.run.trial110:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2026-02-12T12:28:02.752 INFO:teuthology.orchestra.run.trial110.stderr:1+0 records in 2026-02-12T12:28:02.752 INFO:teuthology.orchestra.run.trial110.stderr:1+0 records out 2026-02-12T12:28:02.752 INFO:teuthology.orchestra.run.trial110.stderr:512 bytes copied, 0.000132797 s, 3.9 MB/s 2026-02-12T12:28:02.753 DEBUG:teuthology.orchestra.run.trial110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2026-02-12T12:28:02.810 DEBUG:teuthology.orchestra.run.trial110:> stat /dev/vg_nvme/lv_3 2026-02-12T12:28:02.868 INFO:teuthology.orchestra.run.trial110.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2026-02-12T12:28:02.868 INFO:teuthology.orchestra.run.trial110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-12T12:28:02.868 INFO:teuthology.orchestra.run.trial110.stdout:Device: 6h/6d Inode: 1303 Links: 1 2026-02-12T12:28:02.868 INFO:teuthology.orchestra.run.trial110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-12T12:28:02.868 INFO:teuthology.orchestra.run.trial110.stdout:Context: system_u:object_r:device_t:s0 2026-02-12T12:28:02.868 INFO:teuthology.orchestra.run.trial110.stdout:Access: 2026-02-12 12:17:48.733005916 +0000 2026-02-12T12:28:02.868 INFO:teuthology.orchestra.run.trial110.stdout:Modify: 2026-02-12 12:08:25.241328554 +0000 2026-02-12T12:28:02.869 INFO:teuthology.orchestra.run.trial110.stdout:Change: 2026-02-12 12:08:25.241328554 +0000 2026-02-12T12:28:02.869 INFO:teuthology.orchestra.run.trial110.stdout: Birth: 2026-02-12 12:08:25.241328554 +0000 2026-02-12T12:28:02.869 DEBUG:teuthology.orchestra.run.trial110:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2026-02-12T12:28:02.932 INFO:teuthology.orchestra.run.trial110.stderr:1+0 records in 2026-02-12T12:28:02.933 INFO:teuthology.orchestra.run.trial110.stderr:1+0 records out 2026-02-12T12:28:02.933 INFO:teuthology.orchestra.run.trial110.stderr:512 bytes copied, 0.000136664 s, 3.7 MB/s 2026-02-12T12:28:02.934 DEBUG:teuthology.orchestra.run.trial110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2026-02-12T12:28:02.992 DEBUG:teuthology.orchestra.run.trial110:> stat /dev/vg_nvme/lv_4 2026-02-12T12:28:03.048 INFO:teuthology.orchestra.run.trial110.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2026-02-12T12:28:03.048 INFO:teuthology.orchestra.run.trial110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-12T12:28:03.048 INFO:teuthology.orchestra.run.trial110.stdout:Device: 6h/6d Inode: 1293 Links: 1 2026-02-12T12:28:03.048 INFO:teuthology.orchestra.run.trial110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-12T12:28:03.048 INFO:teuthology.orchestra.run.trial110.stdout:Context: system_u:object_r:device_t:s0 2026-02-12T12:28:03.048 INFO:teuthology.orchestra.run.trial110.stdout:Access: 2026-02-12 12:17:48.733005916 +0000 2026-02-12T12:28:03.049 INFO:teuthology.orchestra.run.trial110.stdout:Modify: 2026-02-12 12:08:25.241328554 +0000 2026-02-12T12:28:03.049 INFO:teuthology.orchestra.run.trial110.stdout:Change: 2026-02-12 12:08:25.241328554 +0000 2026-02-12T12:28:03.049 INFO:teuthology.orchestra.run.trial110.stdout: Birth: 2026-02-12 12:08:25.241328554 +0000 2026-02-12T12:28:03.049 DEBUG:teuthology.orchestra.run.trial110:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2026-02-12T12:28:03.112 INFO:teuthology.orchestra.run.trial110.stderr:1+0 records in 2026-02-12T12:28:03.112 INFO:teuthology.orchestra.run.trial110.stderr:1+0 records out 2026-02-12T12:28:03.112 INFO:teuthology.orchestra.run.trial110.stderr:512 bytes copied, 0.000131434 s, 3.9 MB/s 2026-02-12T12:28:03.113 DEBUG:teuthology.orchestra.run.trial110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2026-02-12T12:28:03.170 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-12T12:28:03.170 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@trial110.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2026-02-12T12:28:03.170 INFO:tasks.ceph:Generating config... 2026-02-12T12:28:03.172 INFO:tasks.ceph:[osd] debug monc = 20 2026-02-12T12:28:03.173 INFO:tasks.ceph:[osd] crimson cpu num = 4 2026-02-12T12:28:03.173 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2026-02-12T12:28:03.173 INFO:tasks.ceph:[osd] debug filestore = 0/0 2026-02-12T12:28:03.173 INFO:tasks.ceph:[osd] debug journal = 0/0 2026-02-12T12:28:03.173 INFO:tasks.ceph:[osd] debug ms = 0/0 2026-02-12T12:28:03.173 INFO:tasks.ceph:[osd] debug osd = 0/0 2026-02-12T12:28:03.173 INFO:tasks.ceph:[osd] debug seastore = 20 2026-02-12T12:28:03.173 INFO:tasks.ceph:[osd] debug seastore backref = 20 2026-02-12T12:28:03.173 INFO:tasks.ceph:[osd] debug seastore cache = 20 2026-02-12T12:28:03.173 INFO:tasks.ceph:[osd] debug seastore cleaner = 20 2026-02-12T12:28:03.174 INFO:tasks.ceph:[osd] debug seastore device = 20 2026-02-12T12:28:03.174 INFO:tasks.ceph:[osd] debug seastore epm = 20 2026-02-12T12:28:03.174 INFO:tasks.ceph:[osd] debug seastore fixedkv tree = 20 2026-02-12T12:28:03.174 INFO:tasks.ceph:[osd] debug seastore journal = 20 2026-02-12T12:28:03.174 INFO:tasks.ceph:[osd] debug seastore lba = 20 2026-02-12T12:28:03.174 INFO:tasks.ceph:[osd] debug seastore odata = 20 2026-02-12T12:28:03.174 INFO:tasks.ceph:[osd] debug seastore omap = 20 2026-02-12T12:28:03.174 INFO:tasks.ceph:[osd] debug seastore onode = 20 2026-02-12T12:28:03.174 INFO:tasks.ceph:[osd] debug seastore t = 20 2026-02-12T12:28:03.174 INFO:tasks.ceph:[osd] debug seastore tm = 20 2026-02-12T12:28:03.174 INFO:tasks.ceph:[osd] osd objectstore = seastore 2026-02-12T12:28:03.174 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2026-02-12T12:28:03.174 INFO:tasks.ceph:[osd] seastore cachepin size pershard = 64M 2026-02-12T12:28:03.175 INFO:tasks.ceph:[osd] seastore cachepin type = 2Q 2026-02-12T12:28:03.175 INFO:tasks.ceph:[osd] seastore main device type = RANDOM_BLOCK_SSD 2026-02-12T12:28:03.175 INFO:tasks.ceph:[osd] seastore max concurrent transactions = 8 2026-02-12T12:28:03.175 INFO:tasks.ceph:[global] auth client required = none 2026-02-12T12:28:03.175 INFO:tasks.ceph:[global] auth cluster required = none 2026-02-12T12:28:03.175 INFO:tasks.ceph:[global] auth service required = none 2026-02-12T12:28:03.175 INFO:tasks.ceph:[global] auth supported = none 2026-02-12T12:28:03.175 INFO:tasks.ceph:[global] debug asok = 0/0 2026-02-12T12:28:03.175 INFO:tasks.ceph:[global] debug auth = 0/0 2026-02-12T12:28:03.175 INFO:tasks.ceph:[global] debug bdev = 0/0 2026-02-12T12:28:03.175 INFO:tasks.ceph:[global] debug bluefs = 0/0 2026-02-12T12:28:03.175 INFO:tasks.ceph:[global] debug bluestore = 0/0 2026-02-12T12:28:03.175 INFO:tasks.ceph:[global] debug buffer = 0/0 2026-02-12T12:28:03.176 INFO:tasks.ceph:[global] debug civetweb = 0/0 2026-02-12T12:28:03.176 INFO:tasks.ceph:[global] debug client = 0/0 2026-02-12T12:28:03.176 INFO:tasks.ceph:[global] debug compressor = 0/0 2026-02-12T12:28:03.176 INFO:tasks.ceph:[global] debug context = 0/0 2026-02-12T12:28:03.176 INFO:tasks.ceph:[global] debug crush = 0/0 2026-02-12T12:28:03.176 INFO:tasks.ceph:[global] debug crypto = 0/0 2026-02-12T12:28:03.176 INFO:tasks.ceph:[global] debug dpdk = 0/0 2026-02-12T12:28:03.176 INFO:tasks.ceph:[global] debug eventtrace = 0/0 2026-02-12T12:28:03.176 INFO:tasks.ceph:[global] debug filer = 0/0 2026-02-12T12:28:03.176 INFO:tasks.ceph:[global] debug filestore = 0/0 2026-02-12T12:28:03.176 INFO:tasks.ceph:[global] debug finisher = 0/0 2026-02-12T12:28:03.176 INFO:tasks.ceph:[global] debug fuse = 0/0 2026-02-12T12:28:03.177 INFO:tasks.ceph:[global] debug heartbeatmap = 0/0 2026-02-12T12:28:03.177 INFO:tasks.ceph:[global] debug javaclient = 0/0 2026-02-12T12:28:03.177 INFO:tasks.ceph:[global] debug journal = 0/0 2026-02-12T12:28:03.177 INFO:tasks.ceph:[global] debug journaler = 0/0 2026-02-12T12:28:03.177 INFO:tasks.ceph:[global] debug kstore = 0/0 2026-02-12T12:28:03.177 INFO:tasks.ceph:[global] debug leveldb = 0/0 2026-02-12T12:28:03.177 INFO:tasks.ceph:[global] debug lockdep = 0/0 2026-02-12T12:28:03.177 INFO:tasks.ceph:[global] debug mds = 0/0 2026-02-12T12:28:03.177 INFO:tasks.ceph:[global] debug mds balancer = 0/0 2026-02-12T12:28:03.177 INFO:tasks.ceph:[global] debug mds locker = 0/0 2026-02-12T12:28:03.177 INFO:tasks.ceph:[global] debug mds log = 0/0 2026-02-12T12:28:03.177 INFO:tasks.ceph:[global] debug mds log expire = 0/0 2026-02-12T12:28:03.177 INFO:tasks.ceph:[global] debug mds migrator = 0/0 2026-02-12T12:28:03.177 INFO:tasks.ceph:[global] debug memdb = 0/0 2026-02-12T12:28:03.178 INFO:tasks.ceph:[global] debug mgr = 0/0 2026-02-12T12:28:03.178 INFO:tasks.ceph:[global] debug mgrc = 0/0 2026-02-12T12:28:03.178 INFO:tasks.ceph:[global] debug mon = 0/0 2026-02-12T12:28:03.178 INFO:tasks.ceph:[global] debug monc = 0/0 2026-02-12T12:28:03.178 INFO:tasks.ceph:[global] debug ms = 0/0 2026-02-12T12:28:03.178 INFO:tasks.ceph:[global] debug objclass = 0/0 2026-02-12T12:28:03.178 INFO:tasks.ceph:[global] debug objectcacher = 0/0 2026-02-12T12:28:03.178 INFO:tasks.ceph:[global] debug objecter = 0/0 2026-02-12T12:28:03.178 INFO:tasks.ceph:[global] debug optracker = 0/0 2026-02-12T12:28:03.178 INFO:tasks.ceph:[global] debug osd = 0/0 2026-02-12T12:28:03.178 INFO:tasks.ceph:[global] debug paxos = 0/0 2026-02-12T12:28:03.178 INFO:tasks.ceph:[global] debug perfcounter = 0/0 2026-02-12T12:28:03.178 INFO:tasks.ceph:[global] debug rados = 0/0 2026-02-12T12:28:03.179 INFO:tasks.ceph:[global] debug rbd = 0/0 2026-02-12T12:28:03.179 INFO:tasks.ceph:[global] debug rbd mirror = 0/0 2026-02-12T12:28:03.179 INFO:tasks.ceph:[global] debug rbd replay = 0/0 2026-02-12T12:28:03.179 INFO:tasks.ceph:[global] debug refs = 0/0 2026-02-12T12:28:03.179 INFO:tasks.ceph:[global] debug rgw = 0/0 2026-02-12T12:28:03.179 INFO:tasks.ceph:[global] debug rgw lifecycle = 0/0 2026-02-12T12:28:03.179 INFO:tasks.ceph:[global] debug rgw notification = 0/0 2026-02-12T12:28:03.179 INFO:tasks.ceph:[global] debug rgw sync = 0/0 2026-02-12T12:28:03.179 INFO:tasks.ceph:[global] debug rocksdb = 0/0 2026-02-12T12:28:03.179 INFO:tasks.ceph:[global] debug striper = 0/0 2026-02-12T12:28:03.179 INFO:tasks.ceph:[global] debug throttle = 0/0 2026-02-12T12:28:03.179 INFO:tasks.ceph:[global] debug timer = 0/0 2026-02-12T12:28:03.179 INFO:tasks.ceph:[global] debug tp = 0/0 2026-02-12T12:28:03.180 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2026-02-12T12:28:03.180 INFO:tasks.ceph:[global] ms client mode = crc 2026-02-12T12:28:03.180 INFO:tasks.ceph:[global] ms cluster mode = crc 2026-02-12T12:28:03.180 INFO:tasks.ceph:[global] ms mon client mode = crc 2026-02-12T12:28:03.180 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2026-02-12T12:28:03.180 INFO:tasks.ceph:[global] ms mon service mode = crc 2026-02-12T12:28:03.180 INFO:tasks.ceph:[global] ms service mode = crc 2026-02-12T12:28:03.180 INFO:tasks.ceph:[mgr] debug mgr = 20 2026-02-12T12:28:03.180 INFO:tasks.ceph:[mgr] debug ms = 1 2026-02-12T12:28:03.180 INFO:tasks.ceph:[mon] debug mon = 0/0 2026-02-12T12:28:03.180 INFO:tasks.ceph:[mon] debug ms = 0/0 2026-02-12T12:28:03.180 INFO:tasks.ceph:[mon] debug paxos = 0/0 2026-02-12T12:28:03.180 INFO:tasks.ceph:[mon] osd pool default crimson = True 2026-02-12T12:28:03.181 INFO:tasks.ceph:Setting up mon.a... 2026-02-12T12:28:03.181 DEBUG:teuthology.orchestra.run.trial110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2026-02-12T12:28:03.249 INFO:teuthology.orchestra.run.trial110.stdout:creating /etc/ceph/ceph.keyring 2026-02-12T12:28:03.252 DEBUG:teuthology.orchestra.run.trial110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2026-02-12T12:28:03.336 DEBUG:teuthology.orchestra.run.trial110:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-12T12:28:03.400 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '10.20.193.110')] 2026-02-12T12:28:03.400 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'none', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '10.20.193.110', 'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': 'true', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore cachepin type': '2Q', 'seastore main device type': 'RANDOM_BLOCK_SSD', 'seastore max concurrent transactions': 8}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': '0/0', 'debug mon': '0/0', 'debug paxos': '0/0', 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}} 2026-02-12T12:28:03.402 DEBUG:teuthology.orchestra.run.trial110:> set -ex 2026-02-12T12:28:03.402 DEBUG:teuthology.orchestra.run.trial110:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2026-02-12T12:28:03.455 DEBUG:teuthology.orchestra.run.trial110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 10.20.193.110 --print /home/ubuntu/cephtest/ceph.monmap 2026-02-12T12:28:03.537 INFO:teuthology.orchestra.run.trial110.stderr:2026-02-12T12:28:03.536+0000 7f08440ccf40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:28:03.538 INFO:teuthology.orchestra.run.trial110.stderr:2026-02-12T12:28:03.536+0000 7f08440ccf40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:28:03.538 INFO:teuthology.orchestra.run.trial110.stderr:2026-02-12T12:28:03.536+0000 7f08440ccf40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:28:03.539 INFO:teuthology.orchestra.run.trial110.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2026-02-12T12:28:03.539 INFO:teuthology.orchestra.run.trial110.stdout:monmaptool: generated fsid cc145609-8d59-4ec8-9270-17a306f6a0f0 2026-02-12T12:28:03.539 INFO:teuthology.orchestra.run.trial110.stdout:setting min_mon_release = tentacle 2026-02-12T12:28:03.539 INFO:teuthology.orchestra.run.trial110.stdout:epoch 0 2026-02-12T12:28:03.539 INFO:teuthology.orchestra.run.trial110.stdout:fsid cc145609-8d59-4ec8-9270-17a306f6a0f0 2026-02-12T12:28:03.539 INFO:teuthology.orchestra.run.trial110.stdout:last_changed 2026-02-12T12:28:03.537253+0000 2026-02-12T12:28:03.540 INFO:teuthology.orchestra.run.trial110.stdout:created 2026-02-12T12:28:03.537253+0000 2026-02-12T12:28:03.540 INFO:teuthology.orchestra.run.trial110.stdout:min_mon_release 20 (tentacle) 2026-02-12T12:28:03.540 INFO:teuthology.orchestra.run.trial110.stdout:election_strategy: 1 2026-02-12T12:28:03.540 INFO:teuthology.orchestra.run.trial110.stdout:0: [v2:10.20.193.110:3300/0,v1:10.20.193.110:6789/0] mon.a 2026-02-12T12:28:03.540 INFO:teuthology.orchestra.run.trial110.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2026-02-12T12:28:03.541 DEBUG:teuthology.orchestra.run.trial110:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2026-02-12T12:28:03.596 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID cc145609-8d59-4ec8-9270-17a306f6a0f0... 2026-02-12T12:28:03.597 DEBUG:teuthology.orchestra.run.trial110:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2026-02-12T12:28:03.675 INFO:teuthology.orchestra.run.trial110.stdout:[global] 2026-02-12T12:28:03.675 INFO:teuthology.orchestra.run.trial110.stdout: chdir = "" 2026-02-12T12:28:03.675 INFO:teuthology.orchestra.run.trial110.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2026-02-12T12:28:03.675 INFO:teuthology.orchestra.run.trial110.stdout: auth supported = none 2026-02-12T12:28:03.675 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:28:03.676 INFO:teuthology.orchestra.run.trial110.stdout: filestore xattr use omap = true 2026-02-12T12:28:03.676 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:28:03.676 INFO:teuthology.orchestra.run.trial110.stdout: mon clock drift allowed = 1.000 2026-02-12T12:28:03.676 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:28:03.676 INFO:teuthology.orchestra.run.trial110.stdout: osd crush chooseleaf type = 0 2026-02-12T12:28:03.676 INFO:teuthology.orchestra.run.trial110.stdout: auth debug = true 2026-02-12T12:28:03.676 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:28:03.676 INFO:teuthology.orchestra.run.trial110.stdout: ms die on old message = true 2026-02-12T12:28:03.676 INFO:teuthology.orchestra.run.trial110.stdout: ms die on bug = true 2026-02-12T12:28:03.676 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:28:03.676 INFO:teuthology.orchestra.run.trial110.stdout: mon max pg per osd = 10000 # >= luminous 2026-02-12T12:28:03.677 INFO:teuthology.orchestra.run.trial110.stdout: mon pg warn max object skew = 0 2026-02-12T12:28:03.677 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:28:03.677 INFO:teuthology.orchestra.run.trial110.stdout: # disable pg_autoscaler by default for new pools 2026-02-12T12:28:03.677 INFO:teuthology.orchestra.run.trial110.stdout: osd_pool_default_pg_autoscale_mode = off 2026-02-12T12:28:03.677 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:28:03.677 INFO:teuthology.orchestra.run.trial110.stdout: osd pool default size = 2 2026-02-12T12:28:03.677 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:28:03.677 INFO:teuthology.orchestra.run.trial110.stdout: mon osd allow primary affinity = true 2026-02-12T12:28:03.677 INFO:teuthology.orchestra.run.trial110.stdout: mon osd allow pg remap = true 2026-02-12T12:28:03.677 INFO:teuthology.orchestra.run.trial110.stdout: mon warn on legacy crush tunables = false 2026-02-12T12:28:03.677 INFO:teuthology.orchestra.run.trial110.stdout: mon warn on crush straw calc version zero = false 2026-02-12T12:28:03.677 INFO:teuthology.orchestra.run.trial110.stdout: mon warn on no sortbitwise = false 2026-02-12T12:28:03.678 INFO:teuthology.orchestra.run.trial110.stdout: mon warn on osd down out interval zero = false 2026-02-12T12:28:03.678 INFO:teuthology.orchestra.run.trial110.stdout: mon warn on too few osds = false 2026-02-12T12:28:03.678 INFO:teuthology.orchestra.run.trial110.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2026-02-12T12:28:03.678 INFO:teuthology.orchestra.run.trial110.stdout: mon_warn_on_pool_no_redundancy = false 2026-02-12T12:28:03.678 INFO:teuthology.orchestra.run.trial110.stdout: mon_allow_pool_size_one = true 2026-02-12T12:28:03.678 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:28:03.678 INFO:teuthology.orchestra.run.trial110.stdout: osd pool default erasure code profile = plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2026-02-12T12:28:03.678 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:28:03.678 INFO:teuthology.orchestra.run.trial110.stdout: osd default data pool replay window = 5 2026-02-12T12:28:03.678 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:28:03.678 INFO:teuthology.orchestra.run.trial110.stdout: mon allow pool delete = true 2026-02-12T12:28:03.678 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:28:03.679 INFO:teuthology.orchestra.run.trial110.stdout: mon cluster log file level = debug 2026-02-12T12:28:03.679 INFO:teuthology.orchestra.run.trial110.stdout: debug asserts on shutdown = true 2026-02-12T12:28:03.679 INFO:teuthology.orchestra.run.trial110.stdout: mon health detail to clog = false 2026-02-12T12:28:03.679 INFO:teuthology.orchestra.run.trial110.stdout: mon host = 10.20.193.110 2026-02-12T12:28:03.679 INFO:teuthology.orchestra.run.trial110.stdout: auth client required = none 2026-02-12T12:28:03.679 INFO:teuthology.orchestra.run.trial110.stdout: auth cluster required = none 2026-02-12T12:28:03.679 INFO:teuthology.orchestra.run.trial110.stdout: auth service required = none 2026-02-12T12:28:03.679 INFO:teuthology.orchestra.run.trial110.stdout: debug asok = 0/0 2026-02-12T12:28:03.679 INFO:teuthology.orchestra.run.trial110.stdout: debug auth = 0/0 2026-02-12T12:28:03.679 INFO:teuthology.orchestra.run.trial110.stdout: debug bdev = 0/0 2026-02-12T12:28:03.679 INFO:teuthology.orchestra.run.trial110.stdout: debug bluefs = 0/0 2026-02-12T12:28:03.679 INFO:teuthology.orchestra.run.trial110.stdout: debug bluestore = 0/0 2026-02-12T12:28:03.679 INFO:teuthology.orchestra.run.trial110.stdout: debug buffer = 0/0 2026-02-12T12:28:03.680 INFO:teuthology.orchestra.run.trial110.stdout: debug civetweb = 0/0 2026-02-12T12:28:03.680 INFO:teuthology.orchestra.run.trial110.stdout: debug client = 0/0 2026-02-12T12:28:03.680 INFO:teuthology.orchestra.run.trial110.stdout: debug compressor = 0/0 2026-02-12T12:28:03.680 INFO:teuthology.orchestra.run.trial110.stdout: debug context = 0/0 2026-02-12T12:28:03.680 INFO:teuthology.orchestra.run.trial110.stdout: debug crush = 0/0 2026-02-12T12:28:03.680 INFO:teuthology.orchestra.run.trial110.stdout: debug crypto = 0/0 2026-02-12T12:28:03.680 INFO:teuthology.orchestra.run.trial110.stdout: debug dpdk = 0/0 2026-02-12T12:28:03.680 INFO:teuthology.orchestra.run.trial110.stdout: debug eventtrace = 0/0 2026-02-12T12:28:03.680 INFO:teuthology.orchestra.run.trial110.stdout: debug filer = 0/0 2026-02-12T12:28:03.680 INFO:teuthology.orchestra.run.trial110.stdout: debug filestore = 0/0 2026-02-12T12:28:03.680 INFO:teuthology.orchestra.run.trial110.stdout: debug finisher = 0/0 2026-02-12T12:28:03.680 INFO:teuthology.orchestra.run.trial110.stdout: debug fuse = 0/0 2026-02-12T12:28:03.680 INFO:teuthology.orchestra.run.trial110.stdout: debug heartbeatmap = 0/0 2026-02-12T12:28:03.681 INFO:teuthology.orchestra.run.trial110.stdout: debug javaclient = 0/0 2026-02-12T12:28:03.681 INFO:teuthology.orchestra.run.trial110.stdout: debug journal = 0/0 2026-02-12T12:28:03.681 INFO:teuthology.orchestra.run.trial110.stdout: debug journaler = 0/0 2026-02-12T12:28:03.681 INFO:teuthology.orchestra.run.trial110.stdout: debug kstore = 0/0 2026-02-12T12:28:03.681 INFO:teuthology.orchestra.run.trial110.stdout: debug leveldb = 0/0 2026-02-12T12:28:03.681 INFO:teuthology.orchestra.run.trial110.stdout: debug lockdep = 0/0 2026-02-12T12:28:03.681 INFO:teuthology.orchestra.run.trial110.stdout: debug mds = 0/0 2026-02-12T12:28:03.681 INFO:teuthology.orchestra.run.trial110.stdout: debug mds balancer = 0/0 2026-02-12T12:28:03.681 INFO:teuthology.orchestra.run.trial110.stdout: debug mds locker = 0/0 2026-02-12T12:28:03.681 INFO:teuthology.orchestra.run.trial110.stdout: debug mds log = 0/0 2026-02-12T12:28:03.681 INFO:teuthology.orchestra.run.trial110.stdout: debug mds log expire = 0/0 2026-02-12T12:28:03.681 INFO:teuthology.orchestra.run.trial110.stdout: debug mds migrator = 0/0 2026-02-12T12:28:03.681 INFO:teuthology.orchestra.run.trial110.stdout: debug memdb = 0/0 2026-02-12T12:28:03.681 INFO:teuthology.orchestra.run.trial110.stdout: debug mgr = 0/0 2026-02-12T12:28:03.682 INFO:teuthology.orchestra.run.trial110.stdout: debug mgrc = 0/0 2026-02-12T12:28:03.682 INFO:teuthology.orchestra.run.trial110.stdout: debug mon = 0/0 2026-02-12T12:28:03.682 INFO:teuthology.orchestra.run.trial110.stdout: debug monc = 0/0 2026-02-12T12:28:03.682 INFO:teuthology.orchestra.run.trial110.stdout: debug ms = 0/0 2026-02-12T12:28:03.682 INFO:teuthology.orchestra.run.trial110.stdout: debug objclass = 0/0 2026-02-12T12:28:03.682 INFO:teuthology.orchestra.run.trial110.stdout: debug objectcacher = 0/0 2026-02-12T12:28:03.682 INFO:teuthology.orchestra.run.trial110.stdout: debug objecter = 0/0 2026-02-12T12:28:03.682 INFO:teuthology.orchestra.run.trial110.stdout: debug optracker = 0/0 2026-02-12T12:28:03.682 INFO:teuthology.orchestra.run.trial110.stdout: debug osd = 0/0 2026-02-12T12:28:03.682 INFO:teuthology.orchestra.run.trial110.stdout: debug paxos = 0/0 2026-02-12T12:28:03.682 INFO:teuthology.orchestra.run.trial110.stdout: debug perfcounter = 0/0 2026-02-12T12:28:03.682 INFO:teuthology.orchestra.run.trial110.stdout: debug rados = 0/0 2026-02-12T12:28:03.682 INFO:teuthology.orchestra.run.trial110.stdout: debug rbd = 0/0 2026-02-12T12:28:03.683 INFO:teuthology.orchestra.run.trial110.stdout: debug rbd mirror = 0/0 2026-02-12T12:28:03.683 INFO:teuthology.orchestra.run.trial110.stdout: debug rbd replay = 0/0 2026-02-12T12:28:03.683 INFO:teuthology.orchestra.run.trial110.stdout: debug refs = 0/0 2026-02-12T12:28:03.683 INFO:teuthology.orchestra.run.trial110.stdout: debug rgw = 0/0 2026-02-12T12:28:03.683 INFO:teuthology.orchestra.run.trial110.stdout: debug rgw lifecycle = 0/0 2026-02-12T12:28:03.683 INFO:teuthology.orchestra.run.trial110.stdout: debug rgw notification = 0/0 2026-02-12T12:28:03.683 INFO:teuthology.orchestra.run.trial110.stdout: debug rgw sync = 0/0 2026-02-12T12:28:03.683 INFO:teuthology.orchestra.run.trial110.stdout: debug rocksdb = 0/0 2026-02-12T12:28:03.683 INFO:teuthology.orchestra.run.trial110.stdout: debug striper = 0/0 2026-02-12T12:28:03.683 INFO:teuthology.orchestra.run.trial110.stdout: debug throttle = 0/0 2026-02-12T12:28:03.683 INFO:teuthology.orchestra.run.trial110.stdout: debug timer = 0/0 2026-02-12T12:28:03.683 INFO:teuthology.orchestra.run.trial110.stdout: debug tp = 0/0 2026-02-12T12:28:03.683 INFO:teuthology.orchestra.run.trial110.stdout: enable experimental unrecoverable data corrupting features = crimson 2026-02-12T12:28:03.684 INFO:teuthology.orchestra.run.trial110.stdout: ms client mode = crc 2026-02-12T12:28:03.684 INFO:teuthology.orchestra.run.trial110.stdout: ms cluster mode = crc 2026-02-12T12:28:03.684 INFO:teuthology.orchestra.run.trial110.stdout: ms mon client mode = crc 2026-02-12T12:28:03.684 INFO:teuthology.orchestra.run.trial110.stdout: ms mon cluster mode = crc 2026-02-12T12:28:03.684 INFO:teuthology.orchestra.run.trial110.stdout: ms mon service mode = crc 2026-02-12T12:28:03.684 INFO:teuthology.orchestra.run.trial110.stdout: ms service mode = crc 2026-02-12T12:28:03.684 INFO:teuthology.orchestra.run.trial110.stdout: fsid = cc145609-8d59-4ec8-9270-17a306f6a0f0 2026-02-12T12:28:03.684 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:28:03.684 INFO:teuthology.orchestra.run.trial110.stdout:[osd] 2026-02-12T12:28:03.684 INFO:teuthology.orchestra.run.trial110.stdout: osd journal size = 100 2026-02-12T12:28:03.684 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:28:03.684 INFO:teuthology.orchestra.run.trial110.stdout: osd scrub load threshold = 5.0 2026-02-12T12:28:03.684 INFO:teuthology.orchestra.run.trial110.stdout: osd scrub max interval = 600 2026-02-12T12:28:03.685 INFO:teuthology.orchestra.run.trial110.stdout: osd mclock profile = high_recovery_ops 2026-02-12T12:28:03.685 INFO:teuthology.orchestra.run.trial110.stdout: osd mclock skip benchmark = true 2026-02-12T12:28:03.685 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:28:03.685 INFO:teuthology.orchestra.run.trial110.stdout: osd recover clone overlap = true 2026-02-12T12:28:03.685 INFO:teuthology.orchestra.run.trial110.stdout: osd recovery max chunk = 1048576 2026-02-12T12:28:03.685 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:28:03.685 INFO:teuthology.orchestra.run.trial110.stdout: osd debug shutdown = true 2026-02-12T12:28:03.685 INFO:teuthology.orchestra.run.trial110.stdout: osd debug op order = true 2026-02-12T12:28:03.685 INFO:teuthology.orchestra.run.trial110.stdout: osd debug verify stray on activate = true 2026-02-12T12:28:03.685 INFO:teuthology.orchestra.run.trial110.stdout: osd debug trim objects = true 2026-02-12T12:28:03.685 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:28:03.685 INFO:teuthology.orchestra.run.trial110.stdout: osd open classes on start = true 2026-02-12T12:28:03.685 INFO:teuthology.orchestra.run.trial110.stdout: osd debug pg log writeout = true 2026-02-12T12:28:03.686 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:28:03.686 INFO:teuthology.orchestra.run.trial110.stdout: osd deep scrub update digest min age = 30 2026-02-12T12:28:03.686 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:28:03.686 INFO:teuthology.orchestra.run.trial110.stdout: osd map max advance = 10 2026-02-12T12:28:03.686 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:28:03.686 INFO:teuthology.orchestra.run.trial110.stdout: journal zero on create = true 2026-02-12T12:28:03.686 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:28:03.686 INFO:teuthology.orchestra.run.trial110.stdout: filestore ondisk finisher threads = 3 2026-02-12T12:28:03.686 INFO:teuthology.orchestra.run.trial110.stdout: filestore apply finisher threads = 3 2026-02-12T12:28:03.686 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:28:03.686 INFO:teuthology.orchestra.run.trial110.stdout: bdev debug aio = true 2026-02-12T12:28:03.686 INFO:teuthology.orchestra.run.trial110.stdout: osd debug misdirected ops = true 2026-02-12T12:28:03.686 INFO:teuthology.orchestra.run.trial110.stdout: debug monc = 20 2026-02-12T12:28:03.687 INFO:teuthology.orchestra.run.trial110.stdout: crimson cpu num = 4 2026-02-12T12:28:03.687 INFO:teuthology.orchestra.run.trial110.stdout: crimson osd obc lru size = 10 2026-02-12T12:28:03.687 INFO:teuthology.orchestra.run.trial110.stdout: debug filestore = 0/0 2026-02-12T12:28:03.687 INFO:teuthology.orchestra.run.trial110.stdout: debug journal = 0/0 2026-02-12T12:28:03.687 INFO:teuthology.orchestra.run.trial110.stdout: debug ms = 0/0 2026-02-12T12:28:03.687 INFO:teuthology.orchestra.run.trial110.stdout: debug osd = 0/0 2026-02-12T12:28:03.687 INFO:teuthology.orchestra.run.trial110.stdout: debug seastore = 20 2026-02-12T12:28:03.687 INFO:teuthology.orchestra.run.trial110.stdout: debug seastore backref = 20 2026-02-12T12:28:03.687 INFO:teuthology.orchestra.run.trial110.stdout: debug seastore cache = 20 2026-02-12T12:28:03.687 INFO:teuthology.orchestra.run.trial110.stdout: debug seastore cleaner = 20 2026-02-12T12:28:03.687 INFO:teuthology.orchestra.run.trial110.stdout: debug seastore device = 20 2026-02-12T12:28:03.687 INFO:teuthology.orchestra.run.trial110.stdout: debug seastore epm = 20 2026-02-12T12:28:03.687 INFO:teuthology.orchestra.run.trial110.stdout: debug seastore fixedkv tree = 20 2026-02-12T12:28:03.687 INFO:teuthology.orchestra.run.trial110.stdout: debug seastore journal = 20 2026-02-12T12:28:03.688 INFO:teuthology.orchestra.run.trial110.stdout: debug seastore lba = 20 2026-02-12T12:28:03.688 INFO:teuthology.orchestra.run.trial110.stdout: debug seastore odata = 20 2026-02-12T12:28:03.688 INFO:teuthology.orchestra.run.trial110.stdout: debug seastore omap = 20 2026-02-12T12:28:03.688 INFO:teuthology.orchestra.run.trial110.stdout: debug seastore onode = 20 2026-02-12T12:28:03.688 INFO:teuthology.orchestra.run.trial110.stdout: debug seastore t = 20 2026-02-12T12:28:03.688 INFO:teuthology.orchestra.run.trial110.stdout: debug seastore tm = 20 2026-02-12T12:28:03.688 INFO:teuthology.orchestra.run.trial110.stdout: osd objectstore = seastore 2026-02-12T12:28:03.688 INFO:teuthology.orchestra.run.trial110.stdout: osd shutdown pgref assert = True 2026-02-12T12:28:03.688 INFO:teuthology.orchestra.run.trial110.stdout: seastore cachepin size pershard = 64M 2026-02-12T12:28:03.688 INFO:teuthology.orchestra.run.trial110.stdout: seastore cachepin type = 2Q 2026-02-12T12:28:03.688 INFO:teuthology.orchestra.run.trial110.stdout: seastore main device type = RANDOM_BLOCK_SSD 2026-02-12T12:28:03.688 INFO:teuthology.orchestra.run.trial110.stdout: seastore max concurrent transactions = 8 2026-02-12T12:28:03.688 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:28:03.689 INFO:teuthology.orchestra.run.trial110.stdout:[mgr] 2026-02-12T12:28:03.689 INFO:teuthology.orchestra.run.trial110.stdout: debug ms = 1 2026-02-12T12:28:03.689 INFO:teuthology.orchestra.run.trial110.stdout: debug mgr = 20 2026-02-12T12:28:03.689 INFO:teuthology.orchestra.run.trial110.stdout: debug mon = 20 2026-02-12T12:28:03.689 INFO:teuthology.orchestra.run.trial110.stdout: debug auth = 20 2026-02-12T12:28:03.689 INFO:teuthology.orchestra.run.trial110.stdout: mon reweight min pgs per osd = 4 2026-02-12T12:28:03.689 INFO:teuthology.orchestra.run.trial110.stdout: mon reweight min bytes per osd = 10 2026-02-12T12:28:03.689 INFO:teuthology.orchestra.run.trial110.stdout: mgr/telemetry/nag = false 2026-02-12T12:28:03.689 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:28:03.689 INFO:teuthology.orchestra.run.trial110.stdout:[mon] 2026-02-12T12:28:03.689 INFO:teuthology.orchestra.run.trial110.stdout: debug ms = 0/0 2026-02-12T12:28:03.689 INFO:teuthology.orchestra.run.trial110.stdout: debug mon = 0/0 2026-02-12T12:28:03.689 INFO:teuthology.orchestra.run.trial110.stdout: debug paxos = 0/0 2026-02-12T12:28:03.690 INFO:teuthology.orchestra.run.trial110.stdout: debug auth = 20 2026-02-12T12:28:03.690 INFO:teuthology.orchestra.run.trial110.stdout: mon data avail warn = 5 2026-02-12T12:28:03.690 INFO:teuthology.orchestra.run.trial110.stdout: mon mgr mkfs grace = 240 2026-02-12T12:28:03.690 INFO:teuthology.orchestra.run.trial110.stdout: mon reweight min pgs per osd = 4 2026-02-12T12:28:03.690 INFO:teuthology.orchestra.run.trial110.stdout: mon osd reporter subtree level = osd 2026-02-12T12:28:03.690 INFO:teuthology.orchestra.run.trial110.stdout: mon osd prime pg temp = true 2026-02-12T12:28:03.690 INFO:teuthology.orchestra.run.trial110.stdout: mon reweight min bytes per osd = 10 2026-02-12T12:28:03.690 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:28:03.690 INFO:teuthology.orchestra.run.trial110.stdout: # rotate auth tickets quickly to exercise renewal paths 2026-02-12T12:28:03.690 INFO:teuthology.orchestra.run.trial110.stdout: auth mon ticket ttl = 660 # 11m 2026-02-12T12:28:03.690 INFO:teuthology.orchestra.run.trial110.stdout: auth service ticket ttl = 240 # 4m 2026-02-12T12:28:03.690 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:28:03.690 INFO:teuthology.orchestra.run.trial110.stdout: # don't complain about insecure global_id in the test suite 2026-02-12T12:28:03.691 INFO:teuthology.orchestra.run.trial110.stdout: mon_warn_on_insecure_global_id_reclaim = false 2026-02-12T12:28:03.691 INFO:teuthology.orchestra.run.trial110.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2026-02-12T12:28:03.691 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:28:03.691 INFO:teuthology.orchestra.run.trial110.stdout: # 1m isn't quite enough 2026-02-12T12:28:03.691 INFO:teuthology.orchestra.run.trial110.stdout: mon_down_mkfs_grace = 2m 2026-02-12T12:28:03.691 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:28:03.691 INFO:teuthology.orchestra.run.trial110.stdout: mon_warn_on_filestore_osds = false 2026-02-12T12:28:03.691 INFO:teuthology.orchestra.run.trial110.stdout: osd pool default crimson = True 2026-02-12T12:28:03.691 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:28:03.691 INFO:teuthology.orchestra.run.trial110.stdout:[client] 2026-02-12T12:28:03.691 INFO:teuthology.orchestra.run.trial110.stdout: rgw cache enabled = true 2026-02-12T12:28:03.691 INFO:teuthology.orchestra.run.trial110.stdout: rgw enable ops log = true 2026-02-12T12:28:03.691 INFO:teuthology.orchestra.run.trial110.stdout: rgw enable usage log = true 2026-02-12T12:28:03.692 INFO:teuthology.orchestra.run.trial110.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2026-02-12T12:28:03.692 INFO:teuthology.orchestra.run.trial110.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2026-02-12T12:28:03.692 INFO:teuthology.orchestra.run.trial110.stdout:[mon.a] 2026-02-12T12:28:03.693 INFO:tasks.ceph:Creating admin key on mon.a... 2026-02-12T12:28:03.693 DEBUG:teuthology.orchestra.run.trial110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2026-02-12T12:28:03.784 INFO:tasks.ceph:Copying monmap to all nodes... 2026-02-12T12:28:03.784 DEBUG:teuthology.orchestra.run.trial110:> set -ex 2026-02-12T12:28:03.784 DEBUG:teuthology.orchestra.run.trial110:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2026-02-12T12:28:03.801 DEBUG:teuthology.orchestra.run.trial110:> set -ex 2026-02-12T12:28:03.801 DEBUG:teuthology.orchestra.run.trial110:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2026-02-12T12:28:03.856 INFO:tasks.ceph:Sending monmap to node ubuntu@trial110.front.sepia.ceph.com 2026-02-12T12:28:03.857 DEBUG:teuthology.orchestra.run.trial110:> set -ex 2026-02-12T12:28:03.857 DEBUG:teuthology.orchestra.run.trial110:> sudo dd of=/etc/ceph/ceph.keyring 2026-02-12T12:28:03.857 DEBUG:teuthology.orchestra.run.trial110:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-12T12:28:03.930 DEBUG:teuthology.orchestra.run.trial110:> set -ex 2026-02-12T12:28:03.930 DEBUG:teuthology.orchestra.run.trial110:> dd of=/home/ubuntu/cephtest/ceph.monmap 2026-02-12T12:28:03.986 INFO:tasks.ceph:Setting up mon nodes... 2026-02-12T12:28:03.986 INFO:tasks.ceph:Setting up mgr nodes... 2026-02-12T12:28:03.986 DEBUG:teuthology.orchestra.run.trial110:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2026-02-12T12:28:04.078 INFO:teuthology.orchestra.run.trial110.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2026-02-12T12:28:04.081 INFO:tasks.ceph:Setting up mds nodes... 2026-02-12T12:28:04.081 INFO:tasks.ceph_client:Setting up client nodes... 2026-02-12T12:28:04.081 DEBUG:teuthology.orchestra.run.trial110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2026-02-12T12:28:04.123 INFO:teuthology.orchestra.run.trial110.stdout:creating /etc/ceph/ceph.client.0.keyring 2026-02-12T12:28:04.135 INFO:tasks.ceph:Running mkfs on osd nodes... 2026-02-12T12:28:04.135 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@trial110.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2026-02-12T12:28:04.136 DEBUG:teuthology.orchestra.run.trial110:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2026-02-12T12:28:04.199 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-12T12:28:04.199 INFO:tasks.ceph:role: osd.0 2026-02-12T12:28:04.200 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@trial110.front.sepia.ceph.com 2026-02-12T12:28:04.200 DEBUG:teuthology.orchestra.run.trial110:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2026-02-12T12:28:04.264 INFO:teuthology.orchestra.run.trial110.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=45875200 blks 2026-02-12T12:28:04.264 INFO:teuthology.orchestra.run.trial110.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-12T12:28:04.264 INFO:teuthology.orchestra.run.trial110.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-12T12:28:04.264 INFO:teuthology.orchestra.run.trial110.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-12T12:28:04.264 INFO:teuthology.orchestra.run.trial110.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-12T12:28:04.264 INFO:teuthology.orchestra.run.trial110.stdout: = sunit=0 swidth=0 blks 2026-02-12T12:28:04.264 INFO:teuthology.orchestra.run.trial110.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-12T12:28:04.264 INFO:teuthology.orchestra.run.trial110.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-12T12:28:04.265 INFO:teuthology.orchestra.run.trial110.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-12T12:28:04.265 INFO:teuthology.orchestra.run.trial110.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-12T12:28:04.839 INFO:teuthology.orchestra.run.trial110.stdout:Discarding blocks...Done. 2026-02-12T12:28:04.841 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@trial110.front.sepia.ceph.com -o noatime 2026-02-12T12:28:04.841 DEBUG:teuthology.orchestra.run.trial110:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2026-02-12T12:28:04.914 DEBUG:teuthology.orchestra.run.trial110:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2026-02-12T12:28:04.981 DEBUG:teuthology.orchestra.run.trial110:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2026-02-12T12:28:05.045 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-12T12:28:05.046 INFO:tasks.ceph:role: osd.1 2026-02-12T12:28:05.046 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@trial110.front.sepia.ceph.com 2026-02-12T12:28:05.046 DEBUG:teuthology.orchestra.run.trial110:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2026-02-12T12:28:05.111 INFO:teuthology.orchestra.run.trial110.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=45875200 blks 2026-02-12T12:28:05.111 INFO:teuthology.orchestra.run.trial110.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-12T12:28:05.111 INFO:teuthology.orchestra.run.trial110.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-12T12:28:05.111 INFO:teuthology.orchestra.run.trial110.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-12T12:28:05.111 INFO:teuthology.orchestra.run.trial110.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-12T12:28:05.111 INFO:teuthology.orchestra.run.trial110.stdout: = sunit=0 swidth=0 blks 2026-02-12T12:28:05.112 INFO:teuthology.orchestra.run.trial110.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-12T12:28:05.112 INFO:teuthology.orchestra.run.trial110.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-12T12:28:05.112 INFO:teuthology.orchestra.run.trial110.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-12T12:28:05.112 INFO:teuthology.orchestra.run.trial110.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-12T12:28:05.673 INFO:teuthology.orchestra.run.trial110.stdout:Discarding blocks...Done. 2026-02-12T12:28:05.674 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@trial110.front.sepia.ceph.com -o noatime 2026-02-12T12:28:05.675 DEBUG:teuthology.orchestra.run.trial110:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2026-02-12T12:28:05.742 DEBUG:teuthology.orchestra.run.trial110:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2026-02-12T12:28:05.809 DEBUG:teuthology.orchestra.run.trial110:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2026-02-12T12:28:05.873 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-12T12:28:05.874 INFO:tasks.ceph:role: osd.2 2026-02-12T12:28:05.874 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@trial110.front.sepia.ceph.com 2026-02-12T12:28:05.874 DEBUG:teuthology.orchestra.run.trial110:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2026-02-12T12:28:05.938 INFO:teuthology.orchestra.run.trial110.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=45875200 blks 2026-02-12T12:28:05.938 INFO:teuthology.orchestra.run.trial110.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-12T12:28:05.938 INFO:teuthology.orchestra.run.trial110.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-12T12:28:05.938 INFO:teuthology.orchestra.run.trial110.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-12T12:28:05.938 INFO:teuthology.orchestra.run.trial110.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-12T12:28:05.938 INFO:teuthology.orchestra.run.trial110.stdout: = sunit=0 swidth=0 blks 2026-02-12T12:28:05.938 INFO:teuthology.orchestra.run.trial110.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-12T12:28:05.939 INFO:teuthology.orchestra.run.trial110.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-12T12:28:05.939 INFO:teuthology.orchestra.run.trial110.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-12T12:28:05.939 INFO:teuthology.orchestra.run.trial110.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-12T12:28:06.502 INFO:teuthology.orchestra.run.trial110.stdout:Discarding blocks...Done. 2026-02-12T12:28:06.504 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@trial110.front.sepia.ceph.com -o noatime 2026-02-12T12:28:06.505 DEBUG:teuthology.orchestra.run.trial110:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2026-02-12T12:28:06.572 DEBUG:teuthology.orchestra.run.trial110:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2026-02-12T12:28:06.639 DEBUG:teuthology.orchestra.run.trial110:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-12T12:28:06.789 INFO:teuthology.orchestra.run.trial110.stderr:INFO  osd - OSD::main: parsing early config 2026-02-12T12:28:06.846 INFO:teuthology.orchestra.run.trial110.stderr:INFO 2026-02-12 12:28:06,845 seastar - Reactor backend: linux-aio 2026-02-12T12:28:06.860 INFO:teuthology.orchestra.run.trial110.stderr:INFO 2026-02-12 12:28:06,859 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-12T12:28:06.860 INFO:teuthology.orchestra.run.trial110.stderr:INFO 2026-02-12 12:28:06,859 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-12T12:28:06.862 INFO:teuthology.orchestra.run.trial110.stderr:==21077==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-12T12:28:06.921 INFO:teuthology.orchestra.run.trial110.stderr:DEBUG 2026-02-12 12:28:06,921 [shard 0:main] seastar - reactor::drain 2026-02-12T12:28:09.062 INFO:teuthology.orchestra.run.trial110.stderr:----------------------------------------------------- 2026-02-12T12:28:09.062 INFO:teuthology.orchestra.run.trial110.stderr:Suppressions used: 2026-02-12T12:28:09.063 INFO:teuthology.orchestra.run.trial110.stderr: count bytes template 2026-02-12T12:28:09.063 INFO:teuthology.orchestra.run.trial110.stderr: 1 8 MallocExtension::Register 2026-02-12T12:28:09.063 INFO:teuthology.orchestra.run.trial110.stderr:----------------------------------------------------- 2026-02-12T12:28:09.063 INFO:teuthology.orchestra.run.trial110.stderr: 2026-02-12T12:28:09.081 INFO:teuthology.orchestra.run.trial110.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-12T12:28:09.081 INFO:teuthology.orchestra.run.trial110.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-12T12:28:09.133 INFO:teuthology.orchestra.run.trial110.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-12T12:28:09.133 INFO:teuthology.orchestra.run.trial110.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-12T12:28:09.140 INFO:teuthology.orchestra.run.trial110.stderr:INFO 2026-02-12 12:28:09,139 seastar - Reactor backend: linux-aio 2026-02-12T12:28:09.148 INFO:teuthology.orchestra.run.trial110.stderr:INFO 2026-02-12 12:28:09,147 [shard 3:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-12T12:28:09.148 INFO:teuthology.orchestra.run.trial110.stderr:INFO 2026-02-12 12:28:09,147 [shard 3:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-12T12:28:09.152 INFO:teuthology.orchestra.run.trial110.stderr:==21075==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-12T12:28:09.152 INFO:teuthology.orchestra.run.trial110.stderr:INFO 2026-02-12 12:28:09,151 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-12T12:28:09.295 INFO:teuthology.orchestra.run.trial110.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e4796c 0x9e47c55 0x9ea85f3 0x9ea8de6 0x9ea9054 0x9ea919c 0x9ea965b /lib64/libc.so.6+0x3fc2f 0xcb0ffe 0x316fa0a 0x316fd2b 0x316fe20 0x9d6554d 0x9d27585 0x9d26585 0x7bb28d7 0x7bbc3a0 0x7a3ca55 0x7a7cdbd 0x7113237 0x6f93ace 0x6f978cf 0x6f9d479 0x74115d6 0x7433c3e 0x7283c37 0x7283d2f 0x7354fe2 0x735509b 0x7355252 0x7433cbf 0x7283dac 0x7283e31 0x72afa7b 0x741c466 0x741c4eb 0x741c5c9 0x729b554 0x73a9088 0x74287fc 0x7428877 0x74288f2 0x74ec4a2 0x75da55c 0x9f024bb 0x9e56790 0x9f2ff2b 0x9f9870e 0x9fb319e 0x9fb381a 0x26ed857 0xa5c1720 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T12:28:09.295 INFO:teuthology.orchestra.run.trial110.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e4796c 0x9e47c55 0x9ea85f3 0x9ea8de6 0x9ea9054 0x9ea919c 0x9ea965b /lib64/libc.so.6+0x3fc2f 0xcb1201 0x1092db4 0xf2fdae 0x9c70283 0x9c7074d 0x9c709c0 0x9c719c0 0x9c79a43 0x9c79dbf 0x9d6290c 0x9d66fb7 0x9d27585 0x9d26585 0x7bb28d7 0x7bbc3a0 0x7a3ca55 0x7a7cdbd 0x7113237 0x6f93ace 0x6f978cf 0x6f9d479 0x74115d6 0x7433c3e 0x7283c37 0x7283d2f 0x7354fe2 0x735509b 0x7355252 0x7433cbf 0x7283dac 0x7283e31 0x72afa7b 0x741c466 0x741c4eb 0x741c5c9 0x729b554 0x73a9088 0x74287fc 0x7428877 0x74288f2 0x74ec4a2 0x75da55c 0x9f024bb 0x9e56790 0x9f2ff2b 0x9f9870e 0x9fb319e 0x9fb381a 0x26ed857 0xa5c1720 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T12:28:09.295 INFO:teuthology.orchestra.run.trial110.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e4796c 0x9e47c55 0x9ea85f3 0x9ea8de6 0x9ea9054 0x9ea919c 0x9ea965b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1585b 0x35eaa8b 0x35eac0b 0x3606a88 0x9c8294b 0x9c82a78 0x9d427fb 0x9d42990 0x9d42d22 0x9d44309 0x9d45624 0x9d45a44 0x9d622cf 0x9d66fb7 0x9d27585 0x9d26585 0x7bb28d7 0x7bbc3a0 0x7a3ca55 0x7a7cdbd 0x7113237 0x6f93ace 0x6f978cf 0x6f9d479 0x74115d6 0x7433c3e 0x7283c37 0x7283d2f 0x7354fe2 0x735509b 0x7355252 0x7433cbf 0x7283dac 0x7283e31 0x72afa7b 0x741c466 0x741c4eb 0x741c5c9 0x729b554 0x73a9088 0x74287fc 0x7428877 0x74288f2 0x74ec4a2 0x75da55c 0x9f024bb 0x9e56790 0x9f2ff2b 0x9f9870e 0x9fb319e 0x9fb381a 0x26ed857 0xa5c1720 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T12:28:09.442 INFO:teuthology.orchestra.run.trial110.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e4796c 0x9e47c55 0x9ea85f3 0x9ea8de6 0x9ea9054 0x9ea919c 0x9ea965b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xeb0a /lib64/libasan.so.8+0xec796 0xcb10b1 0x7bb2b66 0x7bbc3a0 0x7a3ca55 0x7a7cdbd 0x7113237 0x6f3791e 0x6f39c86 0x747c276 0x729b1dd 0x729b2df 0x729b3d7 0x73541ae 0x7354267 0x735441a 0x7354552 0x73545cd 0x7354652 0x73547ad 0x741bee3 0x747c2ff 0x7282672 0x729b554 0x73a9088 0x74287fc 0x7428877 0x74288f2 0x74ec4a2 0x75da55c 0x9f024bb 0x9e56790 0x9f2ff2b 0x9f9870e 0x9fb319e 0x9fb381a 0x26ed857 0xa5c1720 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T12:28:09.443 INFO:teuthology.orchestra.run.trial110.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e4796c 0x9e47c55 0x9ea85f3 0x9ea8de6 0x9ea9054 0x9ea919c 0x9ea965b /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x16b364 /lib64/libasan.so.8+0xb0581 /lib64/libasan.so.8+0xb1477 0xb6a317 0x1092c80 0xf2fdae 0x9c70450 0x9c7074d 0x9c709c0 0x9c70c76 0x9c71725 0x9c71c23 0x9c72253 0x9c723bd 0x9d5272d 0x9d65530 0x9d27585 0x9d26585 0x7bb28d7 0x7bbc3a0 0x7a3ca55 0x7a7cdbd 0x7113237 0x6f3791e 0x6f39c86 0x747c276 0x729b1dd 0x729b2df 0x729b3d7 0x73541ae 0x7354267 0x735441a 0x7354552 0x73545cd 0x7354652 0x73547ad 0x741bee3 0x747c2ff 0x7282672 0x729b554 0x73a9088 0x74287fc 0x7428877 0x74288f2 0x74ec4a2 0x75da55c 0x9f024bb 0x9e56790 0x9f2ff2b 0x9f9870e 0x9fb319e 0x9fb381a 0x26ed857 0xa5c1720 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T12:28:09.443 INFO:teuthology.orchestra.run.trial110.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e4796c 0x9e47c55 0x9ea85f3 0x9ea8de6 0x9ea9054 0x9ea919c 0x9ea965b /lib64/libc.so.6+0x3fc2f 0x1092afd 0xf2fdae 0x9c70283 0x9c7074d 0x9c709c0 0x9c70c76 0x9c71725 0x9c71c23 0x9c72253 0x9c723bd 0x9d5272d 0x9d65530 0x9d27585 0x9d26585 0x7bb28d7 0x7bbc3a0 0x7a3ca55 0x7a7cdbd 0x7113237 0x6f3791e 0x6f39c86 0x747c276 0x729b1dd 0x729b2df 0x729b3d7 0x73541ae 0x7354267 0x735441a 0x7354552 0x73545cd 0x7354652 0x73547ad 0x741bee3 0x747c2ff 0x7282672 0x729b554 0x73a9088 0x74287fc 0x7428877 0x74288f2 0x74ec4a2 0x75da55c 0x9f024bb 0x9e56790 0x9f2ff2b 0x9f9870e 0x9fb319e 0x9fb381a 0x26ed857 0xa5c1720 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T12:28:11.672 INFO:teuthology.orchestra.run.trial110.stderr:----------------------------------------------------- 2026-02-12T12:28:11.673 INFO:teuthology.orchestra.run.trial110.stderr:Suppressions used: 2026-02-12T12:28:11.673 INFO:teuthology.orchestra.run.trial110.stderr: count bytes template 2026-02-12T12:28:11.673 INFO:teuthology.orchestra.run.trial110.stderr: 1 8 MallocExtension::Register 2026-02-12T12:28:11.673 INFO:teuthology.orchestra.run.trial110.stderr:----------------------------------------------------- 2026-02-12T12:28:11.673 INFO:teuthology.orchestra.run.trial110.stderr: 2026-02-12T12:28:11.703 DEBUG:teuthology.orchestra.run.trial110:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2026-02-12T12:28:11.729 DEBUG:teuthology.orchestra.run.trial110:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-12T12:28:11.879 INFO:teuthology.orchestra.run.trial110.stderr:INFO  osd - OSD::main: parsing early config 2026-02-12T12:28:11.931 INFO:teuthology.orchestra.run.trial110.stderr:INFO 2026-02-12 12:28:11,931 seastar - Reactor backend: linux-aio 2026-02-12T12:28:11.941 INFO:teuthology.orchestra.run.trial110.stderr:INFO 2026-02-12 12:28:11,940 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-12T12:28:11.941 INFO:teuthology.orchestra.run.trial110.stderr:INFO 2026-02-12 12:28:11,940 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-12T12:28:11.943 INFO:teuthology.orchestra.run.trial110.stderr:==21147==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-12T12:28:11.976 INFO:teuthology.orchestra.run.trial110.stderr:Reactor stalled for 32 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e4796c 0x9e47c55 0x9ea85f3 0x9ea8de6 0x9ea9054 0x9ea919c 0x9ea965b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc6622 /lib64/libasan.so.8+0x151b7 0x357c0fa 0x357c462 0x357c6a8 0x357c77d 0x357d7d0 0x357df41 0x357e5b4 0x35c4899 0x35c4b94 0x35c4f31 0x35c5290 0x35c53c6 0x359249e 0x3596601 0x35968ba 0x3598069 0x35983a9 0x359865d 0x35989ef 0x3598d22 0x359907f 0x35992b7 0x35998f2 0x3599c22 0x3599f88 0x359a435 0x359a874 0x359aaa0 0x359aeaf 0x359b166 0x359b2b7 0x359b588 0x359b913 0x359bc46 0x359c139 0x359c51b 0x359c6a7 0x359cadf 0x359ce4b 0x359d1b0 0x359d4d3 0x359d5b5 0x359d843 0x359da1a 0x359dc81 0x35d1039 0x35d1789 0x35d1c75 0x35d1fef 0x35d226e 0x35d2323 0x357af42 0x357b8a5 0x357bedf 0x357af42 0x357b351 0x357b608 0x357c969 0x35bfe9d 0x34569cd 0x467a70c 0x46a4863 0x46a5d29 0x46a5dae 0x46a5eb0 0x46a5fa8 0x46a61b3 0x46a6399 0x46a6414 0x46a648f 0x46a650a 0x46a6e77 0x46a7218 0x46a7443 0x46a76aa 0x46a7975 0x46a7b93 0x46a7c4b 0x46a7d03 0x46a7dbc 0x46a7e74 0x46a7f3a 0x46a801b 0x310eb18 0x310ec1e 0x310ecd6 0x310ed8e 0x310ee46 0x3143221 2026-02-12T12:28:12.003 INFO:teuthology.orchestra.run.trial110.stderr:DEBUG 2026-02-12 12:28:12,002 [shard 0:main] seastar - reactor::drain 2026-02-12T12:28:14.149 INFO:teuthology.orchestra.run.trial110.stderr:----------------------------------------------------- 2026-02-12T12:28:14.150 INFO:teuthology.orchestra.run.trial110.stderr:Suppressions used: 2026-02-12T12:28:14.150 INFO:teuthology.orchestra.run.trial110.stderr: count bytes template 2026-02-12T12:28:14.150 INFO:teuthology.orchestra.run.trial110.stderr: 1 8 MallocExtension::Register 2026-02-12T12:28:14.150 INFO:teuthology.orchestra.run.trial110.stderr:----------------------------------------------------- 2026-02-12T12:28:14.150 INFO:teuthology.orchestra.run.trial110.stderr: 2026-02-12T12:28:14.169 INFO:teuthology.orchestra.run.trial110.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-12T12:28:14.169 INFO:teuthology.orchestra.run.trial110.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-12T12:28:14.216 INFO:teuthology.orchestra.run.trial110.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-12T12:28:14.216 INFO:teuthology.orchestra.run.trial110.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-12T12:28:14.223 INFO:teuthology.orchestra.run.trial110.stderr:INFO 2026-02-12 12:28:14,222 seastar - Reactor backend: linux-aio 2026-02-12T12:28:14.231 INFO:teuthology.orchestra.run.trial110.stderr:INFO 2026-02-12 12:28:14,230 [shard 3:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-12T12:28:14.231 INFO:teuthology.orchestra.run.trial110.stderr:INFO 2026-02-12 12:28:14,230 [shard 3:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-12T12:28:14.235 INFO:teuthology.orchestra.run.trial110.stderr:==21145==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-12T12:28:14.235 INFO:teuthology.orchestra.run.trial110.stderr:INFO 2026-02-12 12:28:14,234 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-12T12:28:14.376 INFO:teuthology.orchestra.run.trial110.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e4796c 0x9e47c55 0x9ea85f3 0x9ea8de6 0x9ea9054 0x9ea919c 0x9ea965b /lib64/libc.so.6+0x3fc2f 0x9c7ecc3 0x9d45227 0x9d453ef 0x9d45550 0x9d45a44 0x9d622cf 0x9d66fb7 0x9d27585 0x9d26585 0x7bb28d7 0x7bbc3a0 0x7a3ca55 0x7a7cdbd 0x7113237 0x6f93ace 0x6f978cf 0x6f9d479 0x74115d6 0x7433c3e 0x7283c37 0x7283d2f 0x7354fe2 0x735509b 0x7355252 0x7433cbf 0x7283dac 0x7283e31 0x72afa7b 0x741c466 0x741c4eb 0x741c5c9 0x729b554 0x73a9088 0x74287fc 0x7428877 0x74288f2 0x74ec4a2 0x75da55c 0x9f024bb 0x9e56790 0x9f2ff2b 0x9f9870e 0x9fb319e 0x9fb381a 0x26ed857 0xa5c1720 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T12:28:14.376 INFO:teuthology.orchestra.run.trial110.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e4796c 0x9e47c55 0x9ea85f3 0x9ea8de6 0x9ea9054 0x9ea919c 0x9ea965b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc597 /lib64/libasan.so.8+0x91c1 /lib64/libasan.so.8+0xee7fc 0x9d3b001 0x9d3b111 0x7bb2246 0x7bbc3a0 0x7a3ca55 0x7a7cdbd 0x7113237 0x6f93ace 0x6f978cf 0x6f9d479 0x74115d6 0x7433c3e 0x7283c37 0x7283d2f 0x7354fe2 0x735509b 0x7355252 0x7433cbf 0x7283dac 0x7283e31 0x72afa7b 0x741c466 0x741c4eb 0x741c5c9 0x729b554 0x73a9088 0x74287fc 0x7428877 0x74288f2 0x74ec4a2 0x75da55c 0x9f024bb 0x9e56790 0x9f2ff2b 0x9f9870e 0x9fb319e 0x9fb381a 0x26ed857 0xa5c1720 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T12:28:14.376 INFO:teuthology.orchestra.run.trial110.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff8ee1f7cb 0xffffffff8ee1fb70 0xffffffff8ec9525e 0xffffffff8efe6239 0xffffffff8efc97a5 0xffffffff8efd3b7b 0xffffffff8efd3fdb 0xffffffff8ec910d2 0xffffffff8f94d451 0xffffffff8fa00ba2 2026-02-12T12:28:14.376 INFO:teuthology.orchestra.run.trial110.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e4796c 0x9e47c55 0x9ea85f3 0x9ea8de6 0x9ea9054 0x9ea919c 0x9ea965b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xb0936 /lib64/libasan.so.8+0xb1477 0xb6a317 0x1092c80 0xf2fdae 0x9c70450 0x9c7074d 0x9c709c0 0x9c70c76 0x9c71725 0x9c71c23 0x9c72253 0x9c723bd 0x9d5272d 0x9d65530 0x9d27585 0x9d26585 0x7bb28d7 0x7bbc3a0 0x7a3ca55 0x7a7cdbd 0x7113237 0x6f93ace 0x6f978cf 0x6f9d479 0x74115d6 0x7433c3e 0x7283c37 0x7283d2f 0x7354fe2 0x735509b 0x7355252 0x7433cbf 0x7283dac 0x7283e31 0x72afa7b 0x741c466 0x741c4eb 0x741c5c9 0x729b554 0x73a9088 0x74287fc 0x7428877 0x74288f2 0x74ec4a2 0x75da55c 0x9f024bb 0x9e56790 0x9f2ff2b 0x9f9870e 0x9fb319e 0x9fb381a 0x26ed857 0xa5c1720 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T12:28:14.522 INFO:teuthology.orchestra.run.trial110.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e4796c 0x9e47c55 0x9ea85f3 0x9ea8de6 0x9ea9054 0x9ea919c 0x9ea965b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc597 /lib64/libasan.so.8+0x91c1 /lib64/libasan.so.8+0xee7fc 0x7485195 0x727dc36 0x72a74db 0x73cbb06 0x78a0b4d 0x7bb2209 0x7bbc3a0 0x7a3ca55 0x7a7cdbd 0x7113237 0x6f3791e 0x6f39c86 0x747c276 0x729b1dd 0x729b2df 0x729b3d7 0x73541ae 0x7354267 0x735441a 0x7354552 0x73545cd 0x7354652 0x73547ad 0x741bee3 0x747c2ff 0x7282672 0x729b554 0x73a9088 0x74287fc 0x7428877 0x74288f2 0x74ec4a2 0x75da55c 0x9f024bb 0x9e56790 0x9f2ff2b 0x9f9870e 0x9fb319e 0x9fb381a 0x26ed857 0xa5c1720 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T12:28:14.522 INFO:teuthology.orchestra.run.trial110.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff8f95fe1b 0xffffffff8ef8ac24 0xffffffff8ef8ad1b 0xffffffff8efc97bf 0xffffffff8efd3b7b 0xffffffff8efd3fdb 0xffffffff8ec910d2 0xffffffff8f94d451 0xffffffff8fa00ba2 2026-02-12T12:28:14.523 INFO:teuthology.orchestra.run.trial110.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e4796c 0x9e47c55 0x9ea85f3 0x9ea8de6 0x9ea9054 0x9ea919c 0x9ea965b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x129747 /lib64/libasan.so.8+0xc6631 /lib64/libasan.so.8+0x151b7 0x112781c 0xf2fda3 0x9c70283 0x9c7074d 0x9c709c0 0x9c726e7 0x9c72c34 0x9c72e33 0x9d58155 0x9d588b0 0x9d62b3e 0x9d66fb7 0x9d27585 0x9d26585 0x7bb28d7 0x7bbc3a0 0x7a3ca55 0x7a7cdbd 0x7113237 0x6f3791e 0x6f39c86 0x747c276 0x729b1dd 0x729b2df 0x729b3d7 0x73541ae 0x7354267 0x735441a 0x7354552 0x73545cd 0x7354652 0x73547ad 0x741bee3 0x747c2ff 0x7282672 0x729b554 0x73a9088 0x74287fc 0x7428877 0x74288f2 0x74ec4a2 0x75da55c 0x9f024bb 0x9e56790 0x9f2ff2b 0x9f9870e 0x9fb319e 0x9fb381a 0x26ed857 0xa5c1720 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T12:28:14.523 INFO:teuthology.orchestra.run.trial110.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e4796c 0x9e47c55 0x9ea85f3 0x9ea8de6 0x9ea9054 0x9ea919c 0x9ea965b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc597 /lib64/libasan.so.8+0x91c1 /lib64/libasan.so.8+0xee7fc 0x7485195 0x73cb426 0x714d9d4 0x7bb255b 0x7bbc3a0 0x7a3ca55 0x7a7cdbd 0x7113237 0x6f3791e 0x6f39c86 0x747c276 0x729b1dd 0x729b2df 0x729b3d7 0x73541ae 0x7354267 0x735441a 0x7354552 0x73545cd 0x7354652 0x73547ad 0x741bee3 0x747c2ff 0x7282672 0x729b554 0x73a9088 0x74287fc 0x7428877 0x74288f2 0x74ec4a2 0x75da55c 0x9f024bb 0x9e56790 0x9f2ff2b 0x9f9870e 0x9fb319e 0x9fb381a 0x26ed857 0xa5c1720 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T12:28:14.523 INFO:teuthology.orchestra.run.trial110.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff8ec75c69 0xffffffff8ee1f917 0xffffffff8ee1fb70 0xffffffff8ec9525e 0xffffffff8efe6239 0xffffffff8efc97a5 0xffffffff8efd3b7b 0xffffffff8efd3fdb 0xffffffff8ec910d2 0xffffffff8f94d451 0xffffffff8fa00ba2 2026-02-12T12:28:16.756 INFO:teuthology.orchestra.run.trial110.stderr:----------------------------------------------------- 2026-02-12T12:28:16.756 INFO:teuthology.orchestra.run.trial110.stderr:Suppressions used: 2026-02-12T12:28:16.756 INFO:teuthology.orchestra.run.trial110.stderr: count bytes template 2026-02-12T12:28:16.756 INFO:teuthology.orchestra.run.trial110.stderr: 1 8 MallocExtension::Register 2026-02-12T12:28:16.756 INFO:teuthology.orchestra.run.trial110.stderr:----------------------------------------------------- 2026-02-12T12:28:16.756 INFO:teuthology.orchestra.run.trial110.stderr: 2026-02-12T12:28:16.788 DEBUG:teuthology.orchestra.run.trial110:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2026-02-12T12:28:16.812 DEBUG:teuthology.orchestra.run.trial110:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-12T12:28:16.965 INFO:teuthology.orchestra.run.trial110.stderr:INFO  osd - OSD::main: parsing early config 2026-02-12T12:28:17.017 INFO:teuthology.orchestra.run.trial110.stderr:INFO 2026-02-12 12:28:17,017 seastar - Reactor backend: linux-aio 2026-02-12T12:28:17.027 INFO:teuthology.orchestra.run.trial110.stderr:INFO 2026-02-12 12:28:17,026 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-12T12:28:17.027 INFO:teuthology.orchestra.run.trial110.stderr:INFO 2026-02-12 12:28:17,026 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-12T12:28:17.029 INFO:teuthology.orchestra.run.trial110.stderr:==21211==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-12T12:28:17.088 INFO:teuthology.orchestra.run.trial110.stderr:DEBUG 2026-02-12 12:28:17,088 [shard 0:main] seastar - reactor::drain 2026-02-12T12:28:19.243 INFO:teuthology.orchestra.run.trial110.stderr:----------------------------------------------------- 2026-02-12T12:28:19.243 INFO:teuthology.orchestra.run.trial110.stderr:Suppressions used: 2026-02-12T12:28:19.243 INFO:teuthology.orchestra.run.trial110.stderr: count bytes template 2026-02-12T12:28:19.243 INFO:teuthology.orchestra.run.trial110.stderr: 1 8 MallocExtension::Register 2026-02-12T12:28:19.244 INFO:teuthology.orchestra.run.trial110.stderr:----------------------------------------------------- 2026-02-12T12:28:19.244 INFO:teuthology.orchestra.run.trial110.stderr: 2026-02-12T12:28:19.262 INFO:teuthology.orchestra.run.trial110.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-12T12:28:19.263 INFO:teuthology.orchestra.run.trial110.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-12T12:28:19.307 INFO:teuthology.orchestra.run.trial110.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-12T12:28:19.307 INFO:teuthology.orchestra.run.trial110.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-12T12:28:19.314 INFO:teuthology.orchestra.run.trial110.stderr:INFO 2026-02-12 12:28:19,313 seastar - Reactor backend: linux-aio 2026-02-12T12:28:19.326 INFO:teuthology.orchestra.run.trial110.stderr:INFO 2026-02-12 12:28:19,325 [shard 3:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-12T12:28:19.326 INFO:teuthology.orchestra.run.trial110.stderr:INFO 2026-02-12 12:28:19,325 [shard 3:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-12T12:28:19.330 INFO:teuthology.orchestra.run.trial110.stderr:==21209==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-12T12:28:19.330 INFO:teuthology.orchestra.run.trial110.stderr:INFO 2026-02-12 12:28:19,329 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-12T12:28:19.470 INFO:teuthology.orchestra.run.trial110.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e4796c 0x9e47c55 0x9ea85f3 0x9ea8de6 0x9ea9054 0x9ea919c 0x9ea965b /lib64/libc.so.6+0x3fc2f 0x1092b04 0xf2fdae 0x9c70283 0x9c7074d 0x9c709c0 0x9c726e7 0x9d58641 0x9d588b0 0x9d62b3e 0x9d66fb7 0x9d27585 0x9d26585 0x7bb28d7 0x7bbc3a0 0x7a3ca55 0x7a7cdbd 0x7113237 0x6f93ace 0x6f978cf 0x6f9d479 0x74115d6 0x7433c3e 0x7283c37 0x7283d2f 0x7354fe2 0x735509b 0x7355252 0x7433cbf 0x7283dac 0x7283e31 0x72afa7b 0x741c466 0x741c4eb 0x741c5c9 0x729b554 0x73a9088 0x74287fc 0x7428877 0x74288f2 0x74ec4a2 0x75da55c 0x9f024bb 0x9e56790 0x9f2ff2b 0x9f9870e 0x9fb319e 0x9fb381a 0x26ed857 0xa5c1720 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T12:28:19.471 INFO:teuthology.orchestra.run.trial110.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e4796c 0x9e47c55 0x9ea85f3 0x9ea8de6 0x9ea9054 0x9ea919c 0x9ea965b /lib64/libc.so.6+0x3fc2f 0x316c864 0x9c709ae 0x9c70c76 0x9c71725 0x9c71c23 0x9c72253 0x9c723bd 0x9d5272d 0x9d65530 0x9d27585 0x9d26585 0x7bb28d7 0x7bbc3a0 0x7a3ca55 0x7a7cdbd 0x7113237 0x6f93ace 0x6f978cf 0x6f9d479 0x74115d6 0x7433c3e 0x7283c37 0x7283d2f 0x7354fe2 0x735509b 0x7355252 0x7433cbf 0x7283dac 0x7283e31 0x72afa7b 0x741c466 0x741c4eb 0x741c5c9 0x729b554 0x73a9088 0x74287fc 0x7428877 0x74288f2 0x74ec4a2 0x75da55c 0x9f024bb 0x9e56790 0x9f2ff2b 0x9f9870e 0x9fb319e 0x9fb381a 0x26ed857 0xa5c1720 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T12:28:19.471 INFO:teuthology.orchestra.run.trial110.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e4796c 0x9e47c55 0x9ea85f3 0x9ea8de6 0x9ea9054 0x9ea919c 0x9ea965b /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x8cc0c /lib64/libasan.so.8+0x1006bc /lib64/libasan.so.8+0xc6619 /lib64/libasan.so.8+0x157e7 0x1092cf7 0xf2fdae 0x9c70283 0x9c7074d 0x9c709c0 0x9c70c76 0x9c71238 0x9c796df 0x9c79dbf 0x9d6290c 0x9d66fb7 0x9d27585 0x9d26585 0x7bb28d7 0x7bbc3a0 0x7a3ca55 0x7a7cdbd 0x7113237 0x6f93ace 0x6f978cf 0x6f9d479 0x74115d6 0x7433c3e 0x7283c37 0x7283d2f 0x7354fe2 0x735509b 0x7355252 0x7433cbf 0x7283dac 0x7283e31 0x72afa7b 0x741c466 0x741c4eb 0x741c5c9 0x729b554 0x73a9088 0x74287fc 0x7428877 0x74288f2 0x74ec4a2 0x75da55c 0x9f024bb 0x9e56790 0x9f2ff2b 0x9f9870e 0x9fb319e 0x9fb381a 0x26ed857 0xa5c1720 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T12:28:19.510 INFO:teuthology.orchestra.run.trial110.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e4796c 0x9e47c55 0x9ea85f3 0x9ea8de6 0x9ea9054 0x9ea919c 0x9ea965b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x129858 /lib64/libasan.so.8+0xc6631 /lib64/libasan.so.8+0x151b7 0x112781c 0xf2fda3 0x9c70283 0x9c7074d 0x9c709c0 0x9c719c0 0x9c71f2e 0x9c72253 0x9c723bd 0x9d5272d 0x9d65530 0x9d27585 0x9d26585 0x7bb28d7 0x7bbc3a0 0x7a3ca55 0x7a7cdbd 0x7113237 0x6f948ba 0x754bf4a 0x754c09e 0x9f024bb 0x9e56790 0x9f2ff2b 0x9f9870e 0x9fb319e 0x9fb381a 0x26ed857 0xa5c1720 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T12:28:19.618 INFO:teuthology.orchestra.run.trial110.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e4796c 0x9e47c55 0x9ea85f3 0x9ea8de6 0x9ea9054 0x9ea919c 0x9ea965b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1006bf /lib64/libasan.so.8+0xc6619 /lib64/libasan.so.8+0x157e7 0x1092cf7 0xf2fdae 0x9c70283 0x9c7074d 0x9c709c0 0x9c70c76 0x9c71725 0x9c71c23 0x9c72253 0x9c723bd 0x9d5272d 0x9d65530 0x9d27585 0x9d26585 0x7bb28d7 0x7bbc3a0 0x7a3ca55 0x7a7cdbd 0x7113237 0x6f3791e 0x6f39c86 0x747c276 0x729b1dd 0x729b2df 0x729b3d7 0x73541ae 0x7354267 0x735441a 0x7354552 0x73545cd 0x7354652 0x73547ad 0x741bee3 0x747c2ff 0x7282672 0x729b554 0x73a9088 0x74287fc 0x7428877 0x74288f2 0x74ec4a2 0x75da55c 0x9f024bb 0x9e56790 0x9f2ff2b 0x9f9870e 0x9fb319e 0x9fb381a 0x26ed857 0xa5c1720 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T12:28:19.618 INFO:teuthology.orchestra.run.trial110.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e4796c 0x9e47c55 0x9ea85f3 0x9ea8de6 0x9ea9054 0x9ea919c 0x9ea965b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc6611 /lib64/libasan.so.8+0x157e7 0x1092cf7 0xf2fdae 0x9c70283 0x9c7074d 0x9c709c0 0x9c719c0 0x9c79a43 0x9c79dbf 0x9d6290c 0x9d66fb7 0x9d27585 0x9d26585 0x7bb28d7 0x7bbc3a0 0x7a3ca55 0x7a7cdbd 0x7113237 0x6f3791e 0x6f39c86 0x747c276 0x729b1dd 0x729b2df 0x729b3d7 0x73541ae 0x7354267 0x735441a 0x7354552 0x73545cd 0x7354652 0x73547ad 0x741bee3 0x747c2ff 0x7282672 0x729b554 0x73a9088 0x74287fc 0x7428877 0x74288f2 0x74ec4a2 0x75da55c 0x9f024bb 0x9e56790 0x9f2ff2b 0x9f9870e 0x9fb319e 0x9fb381a 0x26ed857 0xa5c1720 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T12:28:19.618 INFO:teuthology.orchestra.run.trial110.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54ed3 0x9e4796c 0x9e47c55 0x9ea85f3 0x9ea8de6 0x9ea9054 0x9ea919c 0x9ea965b /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc597 /lib64/libasan.so.8+0x91c1 /lib64/libasan.so.8+0xee7fc 0x9b44d0b 0x9d2bd32 0x9d2be60 0x9d5e26b 0x9d5ea0e 0x9d68edb 0x78a0be9 0x7bb280e 0x7bbc3a0 0x7a3ca55 0x7a7cdbd 0x7113237 0x6f3791e 0x6f39c86 0x747c276 0x729b1dd 0x729b2df 0x729b3d7 0x73541ae 0x7354267 0x735441a 0x7354552 0x73545cd 0x7354652 0x73547ad 0x741bee3 0x747c2ff 0x7282672 0x729b554 0x73a9088 0x74287fc 0x7428877 0x74288f2 0x74ec4a2 0x75da55c 0x9f024bb 0x9e56790 0x9f2ff2b 0x9f9870e 0x9fb319e 0x9fb381a 0x26ed857 0xa5c1720 /lib64/libasan.so.8+0x2a589 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-12T12:28:19.618 INFO:teuthology.orchestra.run.trial110.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff8ec94317 0xffffffff8ee1f817 0xffffffff8ee1fb70 0xffffffff8ec9525e 0xffffffff8efe6239 0xffffffff8efc97a5 0xffffffff8efd3b7b 0xffffffff8efd3fdb 0xffffffff8ec910d2 0xffffffff8f94d451 0xffffffff8fa00ba2 2026-02-12T12:28:21.854 INFO:teuthology.orchestra.run.trial110.stderr:----------------------------------------------------- 2026-02-12T12:28:21.854 INFO:teuthology.orchestra.run.trial110.stderr:Suppressions used: 2026-02-12T12:28:21.855 INFO:teuthology.orchestra.run.trial110.stderr: count bytes template 2026-02-12T12:28:21.855 INFO:teuthology.orchestra.run.trial110.stderr: 1 8 MallocExtension::Register 2026-02-12T12:28:21.855 INFO:teuthology.orchestra.run.trial110.stderr:----------------------------------------------------- 2026-02-12T12:28:21.855 INFO:teuthology.orchestra.run.trial110.stderr: 2026-02-12T12:28:21.886 DEBUG:teuthology.orchestra.run.trial110:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2026-02-12T12:28:21.912 INFO:tasks.ceph:Reading keys from all nodes... 2026-02-12T12:28:21.912 DEBUG:teuthology.orchestra.run.trial110:> set -ex 2026-02-12T12:28:21.912 DEBUG:teuthology.orchestra.run.trial110:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2026-02-12T12:28:21.976 DEBUG:teuthology.orchestra.run.trial110:> set -ex 2026-02-12T12:28:21.977 DEBUG:teuthology.orchestra.run.trial110:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2026-02-12T12:28:22.040 DEBUG:teuthology.orchestra.run.trial110:> set -ex 2026-02-12T12:28:22.040 DEBUG:teuthology.orchestra.run.trial110:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2026-02-12T12:28:22.103 DEBUG:teuthology.orchestra.run.trial110:> set -ex 2026-02-12T12:28:22.103 DEBUG:teuthology.orchestra.run.trial110:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2026-02-12T12:28:22.167 DEBUG:teuthology.orchestra.run.trial110:> set -ex 2026-02-12T12:28:22.168 DEBUG:teuthology.orchestra.run.trial110:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2026-02-12T12:28:22.223 INFO:tasks.ceph:Adding keys to all mons... 2026-02-12T12:28:22.224 DEBUG:teuthology.orchestra.run.trial110:> sudo tee -a /etc/ceph/ceph.keyring 2026-02-12T12:28:22.285 INFO:teuthology.orchestra.run.trial110.stdout:[mgr.x] 2026-02-12T12:28:22.286 INFO:teuthology.orchestra.run.trial110.stdout: key = AQBUx41pA46gBBAAJxGnkGFoXlqENMaeBpnQBg== 2026-02-12T12:28:22.286 INFO:teuthology.orchestra.run.trial110.stdout:[osd.0] 2026-02-12T12:28:22.286 INFO:teuthology.orchestra.run.trial110.stdout: key = AQBZx41pf9QQDRAAJY7Xh7iufB65x5JHxKHEHQ== 2026-02-12T12:28:22.286 INFO:teuthology.orchestra.run.trial110.stdout:[osd.1] 2026-02-12T12:28:22.286 INFO:teuthology.orchestra.run.trial110.stdout: key = AQBex41pEUrcERAA2c0KFQ6z7csGX8wqTxBhVw== 2026-02-12T12:28:22.286 INFO:teuthology.orchestra.run.trial110.stdout:[osd.2] 2026-02-12T12:28:22.286 INFO:teuthology.orchestra.run.trial110.stdout: key = AQBjx41phqaQFxAAaxoPi6NOGUl9EudLba2C1g== 2026-02-12T12:28:22.286 INFO:teuthology.orchestra.run.trial110.stdout:[client.0] 2026-02-12T12:28:22.286 INFO:teuthology.orchestra.run.trial110.stdout: key = AQBUx41pMPNTBxAATCoVQS0zdt90wGRbfjUGLg== 2026-02-12T12:28:22.288 DEBUG:teuthology.orchestra.run.trial110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2026-02-12T12:28:22.374 DEBUG:teuthology.orchestra.run.trial110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-12T12:28:22.425 DEBUG:teuthology.orchestra.run.trial110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-12T12:28:22.472 DEBUG:teuthology.orchestra.run.trial110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-12T12:28:22.520 DEBUG:teuthology.orchestra.run.trial110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-12T12:28:22.570 INFO:tasks.ceph:Running mkfs on mon nodes... 2026-02-12T12:28:22.570 DEBUG:teuthology.orchestra.run.trial110:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2026-02-12T12:28:22.595 DEBUG:teuthology.orchestra.run.trial110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2026-02-12T12:28:22.681 INFO:teuthology.orchestra.run.trial110.stderr:2026-02-12T12:28:22.679+0000 7f21f822dc00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:28:22.681 INFO:teuthology.orchestra.run.trial110.stderr:2026-02-12T12:28:22.679+0000 7f21f822dc00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:28:22.688 DEBUG:teuthology.orchestra.run.trial110:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2026-02-12T12:28:22.713 DEBUG:teuthology.orchestra.run.trial110:> rm -- /home/ubuntu/cephtest/ceph.monmap 2026-02-12T12:28:22.768 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2026-02-12T12:28:22.769 INFO:tasks.ceph.mon.a:Restarting daemon 2026-02-12T12:28:22.769 DEBUG:teuthology.orchestra.run.trial110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2026-02-12T12:28:22.811 INFO:tasks.ceph.mon.a:Started 2026-02-12T12:28:22.811 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2026-02-12T12:28:22.811 INFO:tasks.ceph.mgr.x:Restarting daemon 2026-02-12T12:28:22.812 DEBUG:teuthology.orchestra.run.trial110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2026-02-12T12:28:22.814 INFO:tasks.ceph.mgr.x:Started 2026-02-12T12:28:22.814 DEBUG:tasks.ceph:set 0 configs 2026-02-12T12:28:22.815 DEBUG:teuthology.orchestra.run.trial110:> sudo ceph --cluster ceph config dump 2026-02-12T12:28:22.873 INFO:tasks.ceph.mon.a.trial110.stderr:2026-02-12T12:28:22.872+0000 7f00a1632c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:28:22.874 INFO:tasks.ceph.mon.a.trial110.stderr:2026-02-12T12:28:22.872+0000 7f00a1632c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:28:22.874 INFO:tasks.ceph.mon.a.trial110.stderr:2026-02-12T12:28:22.872+0000 7f00a1632c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:28:22.877 INFO:tasks.ceph.mgr.x.trial110.stderr:2026-02-12T12:28:22.875+0000 7fbf9fa36fc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:28:22.888 INFO:teuthology.orchestra.run.trial110.stderr:2026-02-12T12:28:22.885+0000 7f601985a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:28:22.890 INFO:teuthology.orchestra.run.trial110.stderr:2026-02-12T12:28:22.888+0000 7f601985a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:28:22.920 INFO:teuthology.orchestra.run.trial110.stdout:WHO MASK LEVEL OPTION VALUE RO 2026-02-12T12:28:22.928 INFO:tasks.ceph:Setting crush tunables to default 2026-02-12T12:28:22.928 DEBUG:teuthology.orchestra.run.trial110:> sudo ceph --cluster ceph osd crush tunables default 2026-02-12T12:28:23.004 INFO:teuthology.orchestra.run.trial110.stderr:2026-02-12T12:28:23.000+0000 7f7d3aa5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:28:23.005 INFO:teuthology.orchestra.run.trial110.stderr:2026-02-12T12:28:23.004+0000 7f7d3aa5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:28:23.038 INFO:teuthology.orchestra.run.trial110.stderr:adjusted tunables profile to default 2026-02-12T12:28:23.046 INFO:tasks.ceph:check_enable_crimson: True 2026-02-12T12:28:23.046 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2026-02-12T12:28:23.046 DEBUG:teuthology.orchestra.run.trial110:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2026-02-12T12:28:23.081 INFO:tasks.ceph.mgr.x.trial110.stderr:2026-02-12T12:28:23.080+0000 7fbf9fa36fc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:28:23.082 INFO:tasks.ceph.mgr.x.trial110.stderr:2026-02-12T12:28:23.080+0000 7fbf9fa36fc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:28:23.118 INFO:teuthology.orchestra.run.trial110.stderr:2026-02-12T12:28:23.115+0000 7fbebab0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:28:23.120 INFO:teuthology.orchestra.run.trial110.stderr:2026-02-12T12:28:23.118+0000 7fbebab0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:28:23.158 INFO:tasks.ceph.mon.a.trial110.stderr:2026-02-12T12:28:23.157+0000 7f00989f1640 -1 WARNING: experimental feature 'crimson' is enabled 2026-02-12T12:28:23.158 INFO:tasks.ceph.mon.a.trial110.stderr:Please be aware that this feature is experimental, untested, 2026-02-12T12:28:23.158 INFO:tasks.ceph.mon.a.trial110.stderr:unsupported, and may result in data corruption, data loss, 2026-02-12T12:28:23.158 INFO:tasks.ceph.mon.a.trial110.stderr:and/or irreparable damage to your cluster. Do not use 2026-02-12T12:28:23.158 INFO:tasks.ceph.mon.a.trial110.stderr:feature with important data. 2026-02-12T12:28:23.159 INFO:tasks.ceph.mon.a.trial110.stderr: 2026-02-12T12:28:23.928 INFO:tasks.ceph.mgr.x.trial110.stderr:/usr/lib64/python3.9/site-packages/scipy/__init__.py:73: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2026-02-12T12:28:23.929 INFO:tasks.ceph.mgr.x.trial110.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2026-02-12T12:28:23.929 INFO:tasks.ceph.mgr.x.trial110.stderr: from numpy import show_config as show_numpy_config 2026-02-12T12:28:24.048 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2026-02-12T12:28:24.048 DEBUG:teuthology.orchestra.run.trial110:> set -ex 2026-02-12T12:28:24.049 DEBUG:teuthology.orchestra.run.trial110:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2026-02-12T12:28:24.113 DEBUG:teuthology.orchestra.run.trial110:> set -ex 2026-02-12T12:28:24.114 DEBUG:teuthology.orchestra.run.trial110:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2026-02-12T12:28:24.177 DEBUG:teuthology.orchestra.run.trial110:> set -ex 2026-02-12T12:28:24.177 DEBUG:teuthology.orchestra.run.trial110:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2026-02-12T12:28:24.240 DEBUG:teuthology.orchestra.run.trial110:> sudo ceph --cluster ceph osd new d103ed9f-3542-43f8-91ff-6e926464dfec 0 2026-02-12T12:28:24.354 INFO:teuthology.orchestra.run.trial110.stderr:2026-02-12T12:28:24.350+0000 7f44e6364640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:28:24.355 INFO:teuthology.orchestra.run.trial110.stderr:2026-02-12T12:28:24.354+0000 7f44e6364640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:28:24.393 INFO:teuthology.orchestra.run.trial110.stdout:0 2026-02-12T12:28:24.402 DEBUG:teuthology.orchestra.run.trial110:> sudo ceph --cluster ceph osd new 23bf64e8-a6c2-4647-8b5e-59735720cc44 1 2026-02-12T12:28:24.476 INFO:teuthology.orchestra.run.trial110.stderr:2026-02-12T12:28:24.471+0000 7f20d4ff5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:28:24.477 INFO:teuthology.orchestra.run.trial110.stderr:2026-02-12T12:28:24.476+0000 7f20d4ff5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:28:24.514 INFO:teuthology.orchestra.run.trial110.stdout:1 2026-02-12T12:28:24.520 DEBUG:teuthology.orchestra.run.trial110:> sudo ceph --cluster ceph osd new 6d179a0c-5aac-4f28-9304-80c384f8c8e1 2 2026-02-12T12:28:24.593 INFO:teuthology.orchestra.run.trial110.stderr:2026-02-12T12:28:24.589+0000 7f1e02e1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:28:24.594 INFO:teuthology.orchestra.run.trial110.stderr:2026-02-12T12:28:24.593+0000 7f1e02e1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:28:24.631 INFO:teuthology.orchestra.run.trial110.stdout:2 2026-02-12T12:28:24.639 DEBUG:teuthology.orchestra.run.trial110:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2026-02-12T12:28:24.667 INFO:tasks.ceph.osd.0:Restarting daemon 2026-02-12T12:28:24.668 DEBUG:teuthology.orchestra.run.trial110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2026-02-12T12:28:24.709 INFO:tasks.ceph.osd.0:Started 2026-02-12T12:28:24.709 DEBUG:teuthology.orchestra.run.trial110:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2026-02-12T12:28:24.738 INFO:tasks.ceph.osd.1:Restarting daemon 2026-02-12T12:28:24.739 DEBUG:teuthology.orchestra.run.trial110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2026-02-12T12:28:24.780 INFO:tasks.ceph.osd.1:Started 2026-02-12T12:28:24.781 DEBUG:teuthology.orchestra.run.trial110:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2026-02-12T12:28:24.808 INFO:tasks.ceph.osd.2:Restarting daemon 2026-02-12T12:28:24.808 DEBUG:teuthology.orchestra.run.trial110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2026-02-12T12:28:24.850 INFO:tasks.ceph.osd.2:Started 2026-02-12T12:28:24.851 DEBUG:teuthology.orchestra.run.trial110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-12T12:28:24.927 INFO:teuthology.orchestra.run.trial110.stderr:2026-02-12T12:28:24.924+0000 7f7de068b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:28:24.928 INFO:teuthology.orchestra.run.trial110.stderr:2026-02-12T12:28:24.927+0000 7f7de068b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:28:24.964 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:28:24.964 INFO:teuthology.orchestra.run.trial110.stdout:{"epoch":6,"fsid":"cc145609-8d59-4ec8-9270-17a306f6a0f0","created":"2026-02-12T12:28:22.882886+0000","modified":"2026-02-12T12:28:24.629911+0000","last_up_change":"0.000000","last_in_change":"2026-02-12T12:28:24.629911+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"d103ed9f-3542-43f8-91ff-6e926464dfec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"23bf64e8-a6c2-4647-8b5e-59735720cc44","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d179a0c-5aac-4f28-9304-80c384f8c8e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-12T12:28:24.973 INFO:tasks.ceph.ceph_manager.ceph:[] 2026-02-12T12:28:24.974 INFO:tasks.ceph:Waiting for OSDs to come up 2026-02-12T12:28:25.275 DEBUG:teuthology.orchestra.run.trial110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-12T12:28:25.342 INFO:teuthology.misc.health.trial110.stderr:2026-02-12T12:28:25.339+0000 7f298cc51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:28:25.343 INFO:teuthology.misc.health.trial110.stderr:2026-02-12T12:28:25.342+0000 7f298cc51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:28:25.375 INFO:teuthology.misc.health.trial110.stdout: 2026-02-12T12:28:25.375 INFO:teuthology.misc.health.trial110.stdout:{"epoch":6,"fsid":"cc145609-8d59-4ec8-9270-17a306f6a0f0","created":"2026-02-12T12:28:22.882886+0000","modified":"2026-02-12T12:28:24.629911+0000","last_up_change":"0.000000","last_in_change":"2026-02-12T12:28:24.629911+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"d103ed9f-3542-43f8-91ff-6e926464dfec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"23bf64e8-a6c2-4647-8b5e-59735720cc44","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d179a0c-5aac-4f28-9304-80c384f8c8e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-12T12:28:25.382 DEBUG:teuthology.misc:0 of 3 OSDs are up 2026-02-12T12:28:26.329 INFO:tasks.ceph.mgr.x.trial110.stderr:2026-02-12T12:28:26.328+0000 7fbeba69e640 -1 mgr.server handle_report got status from non-daemon mon.a 2026-02-12T12:28:31.684 DEBUG:teuthology.orchestra.run.trial110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-12T12:28:31.757 INFO:teuthology.misc.health.trial110.stderr:2026-02-12T12:28:31.754+0000 7ffb43164640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:28:31.758 INFO:teuthology.misc.health.trial110.stderr:2026-02-12T12:28:31.757+0000 7ffb43164640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:28:31.870 INFO:teuthology.misc.health.trial110.stdout: 2026-02-12T12:28:31.870 INFO:teuthology.misc.health.trial110.stdout:{"epoch":8,"fsid":"cc145609-8d59-4ec8-9270-17a306f6a0f0","created":"2026-02-12T12:28:22.882886+0000","modified":"2026-02-12T12:28:31.259595+0000","last_up_change":"0.000000","last_in_change":"2026-02-12T12:28:24.629911+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"d103ed9f-3542-43f8-91ff-6e926464dfec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"23bf64e8-a6c2-4647-8b5e-59735720cc44","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6d179a0c-5aac-4f28-9304-80c384f8c8e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-12T12:28:31.878 DEBUG:teuthology.misc:0 of 3 OSDs are up 2026-02-12T12:28:38.180 DEBUG:teuthology.orchestra.run.trial110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-12T12:28:38.251 INFO:teuthology.misc.health.trial110.stderr:2026-02-12T12:28:38.246+0000 7fe214e5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:28:38.252 INFO:teuthology.misc.health.trial110.stderr:2026-02-12T12:28:38.249+0000 7fe214e5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:28:38.364 INFO:teuthology.misc.health.trial110.stdout: 2026-02-12T12:28:38.364 INFO:teuthology.misc.health.trial110.stdout:{"epoch":14,"fsid":"cc145609-8d59-4ec8-9270-17a306f6a0f0","created":"2026-02-12T12:28:22.882886+0000","modified":"2026-02-12T12:28:38.289005+0000","last_up_change":"2026-02-12T12:28:34.270885+0000","last_in_change":"2026-02-12T12:28:24.629911+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-12T12:28:35.337743+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d103ed9f-3542-43f8-91ff-6e926464dfec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.110:6804","nonce":3324838780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.110:6803","nonce":3324838780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.110:6805","nonce":3324838780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.110:6802","nonce":3324838780}]},"public_addr":"10.20.193.110:6804/3324838780","cluster_addr":"10.20.193.110:6803/3324838780","heartbeat_back_addr":"10.20.193.110:6805/3324838780","heartbeat_front_addr":"10.20.193.110:6802/3324838780","state":["exists","up"]},{"osd":1,"uuid":"23bf64e8-a6c2-4647-8b5e-59735720cc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.110:6806","nonce":2469450341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.110:6807","nonce":2469450341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.110:6809","nonce":2469450341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.110:6811","nonce":2469450341}]},"public_addr":"10.20.193.110:6806/2469450341","cluster_addr":"10.20.193.110:6807/2469450341","heartbeat_back_addr":"10.20.193.110:6809/2469450341","heartbeat_front_addr":"10.20.193.110:6811/2469450341","state":["exists","up"]},{"osd":2,"uuid":"6d179a0c-5aac-4f28-9304-80c384f8c8e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.110:6813","nonce":3082530054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.110:6812","nonce":3082530054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.110:6808","nonce":3082530054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.110:6810","nonce":3082530054}]},"public_addr":"10.20.193.110:6813/3082530054","cluster_addr":"10.20.193.110:6812/3082530054","heartbeat_back_addr":"10.20.193.110:6808/3082530054","heartbeat_front_addr":"10.20.193.110:6810/3082530054","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-12T12:28:38.372 DEBUG:teuthology.misc:3 of 3 OSDs are up 2026-02-12T12:28:38.373 INFO:tasks.ceph:Creating RBD pool 2026-02-12T12:28:38.373 DEBUG:teuthology.orchestra.run.trial110:> sudo ceph --cluster ceph osd pool create rbd 8 2026-02-12T12:28:38.448 INFO:teuthology.orchestra.run.trial110.stderr:2026-02-12T12:28:38.442+0000 7eff88411640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:28:38.449 INFO:teuthology.orchestra.run.trial110.stderr:2026-02-12T12:28:38.446+0000 7eff88411640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:28:39.293 INFO:teuthology.orchestra.run.trial110.stderr:pool 'rbd' created 2026-02-12T12:28:39.307 DEBUG:teuthology.orchestra.run.trial110:> rbd --cluster ceph pool init rbd 2026-02-12T12:28:39.353 INFO:teuthology.orchestra.run.trial110.stderr:2026-02-12T12:28:39.350+0000 7fb91a21a240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:28:39.353 INFO:teuthology.orchestra.run.trial110.stderr:2026-02-12T12:28:39.350+0000 7fb91a21a240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:28:39.354 INFO:teuthology.orchestra.run.trial110.stderr:2026-02-12T12:28:39.352+0000 7fb91a21a240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:28:42.331 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2026-02-12T12:28:42.332 DEBUG:teuthology.orchestra.run.trial110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2026-02-12T12:28:42.333 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2026-02-12T12:28:42.410 INFO:teuthology.orchestra.run.trial110.stderr:2026-02-12T12:28:42.406+0000 7ff343ab7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:28:42.411 INFO:teuthology.orchestra.run.trial110.stderr:2026-02-12T12:28:42.409+0000 7ff343ab7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:28:42.520 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:28:42.529 INFO:teuthology.orchestra.run.trial110.stdout:[{"version":1,"timestamp":"0.000000","name":"","changes":[]}] 2026-02-12T12:28:42.529 INFO:tasks.ceph_manager:config epoch is 1 2026-02-12T12:28:42.530 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2026-02-12T12:28:42.530 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2026-02-12T12:28:42.530 DEBUG:teuthology.orchestra.run.trial110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2026-02-12T12:28:42.608 INFO:teuthology.orchestra.run.trial110.stderr:2026-02-12T12:28:42.604+0000 7f209c50b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:28:42.609 INFO:teuthology.orchestra.run.trial110.stderr:2026-02-12T12:28:42.608+0000 7f209c50b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:28:42.757 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:28:42.765 INFO:teuthology.orchestra.run.trial110.stdout:{"epoch":5,"flags":0,"active_gid":4103,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.110:6800","nonce":3851846934},{"type":"v1","addr":"10.20.193.110:6801","nonce":3851846934}]},"active_addr":"10.20.193.110:6801/3851846934","active_change":"2026-02-12T12:28:25.323509+0000","active_mgr_features":4544132024016699391,"available":true,"standbys":[],"modules":["iostat","nfs"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_pg_upmap_activity":{"name":"update_pg_upmap_activity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Updates pg_upmap activity stats to be used in `balancer status detail`","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often the agent on each host will gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"The first TCP port the agent will try to bind to. Up to 1000 subsequent ports will be attempted if this port cannot be bound","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this option can facilitate debugging daemons that encounter runtime problems.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"How frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"Fraction of total system memory to divide among autotuned daemons. Converged systems hosting both Ceph and compute should set a lower value; systems with more than 10 GiB of memory per OSD may benefit from a slightly larger value so that the abundance of RAM can be fully utilized","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command persistent logging","long_desc":"","tags":[],"see_also":[]},"certificate_automated_rotation_enabled":{"name":"certificate_automated_rotation_enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"This flag controls whether cephadm automatically rotates certificates upon expiration.","long_desc":"","tags":[],"see_also":[]},"certificate_check_debug_mode":{"name":"certificate_check_debug_mode","type":"bool","level":"dev","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"FOR TESTING ONLY: This flag forces the certificate check instead of waiting for certificate_check_period.","long_desc":"","tags":[],"see_also":[]},"certificate_check_period":{"name":"certificate_check_period","type":"int","level":"advanced","flags":0,"default_value":"1","min":"0","max":"30","enum_allowed":[],"desc":"Specifies how often (in days) the certificate should be checked for validity.","long_desc":"","tags":[],"see_also":[]},"certificate_duration_days":{"name":"certificate_duration_days","type":"int","level":"advanced","flags":0,"default_value":"1095","min":"90","max":"3650","enum_allowed":[],"desc":"Specifies the duration of self-signed certificates generated and signed by the cephadm root CA","long_desc":"","tags":[],"see_also":[]},"certificate_renewal_threshold_days":{"name":"certificate_renewal_threshold_days","type":"int","level":"advanced","flags":0,"default_value":"30","min":"10","max":"90","enum_allowed":[],"desc":"Specifies the lead time in days to initiate certificate renewal before expiration.","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently Podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.28.1","min":"","max":"","enum_allowed":[],"desc":"Alertmanager container image","long_desc":"","tags":[],"see_also":[]},"container_image_alloy":{"name":"container_image_alloy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/alloy:latest","min":"","max":"","enum_allowed":[],"desc":"Alloy container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"Elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:12.3.1","min":"","max":"","enum_allowed":[],"desc":"Grafana container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"Haproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.9.1","min":"","max":"","enum_allowed":[],"desc":"Node exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.6","min":"","max":"","enum_allowed":[],"desc":"Nvmeof container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"Oauth2 proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v3.6.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba metrics container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"Snmp gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"quay.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"Location of dashboards to include in Grafana","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"How frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether the hardware monitoring daemon be deployed on every host?","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at the debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"Max number of daemons per service per host: guard against mistakes and runaways","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"Max number of OSDs that will be drained simultaneously when OSDs are removed","long_desc":"","tags":[],"see_also":[]},"max_parallel_osd_upgrades":{"name":"max_parallel_osd_upgrades","type":"int","level":"advanced","flags":0,"default_value":"16","min":"","max":"","enum_allowed":[],"desc":"Maximum number of OSD daemons upgraded in parallel.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"Mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default IP address for RedFish API (OOB management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"Location of alerts to include in Prometheus Alertmanager","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry URL for login purposes. This is not the default registry.","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"Cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Customized SSH config file to distribute to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many sequential SSH connection liveness check failures before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often SSH connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"stray_daemon_check_interval":{"name":"stray_daemon_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"How frequently cephadm should check for the presence of stray daemons","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use the cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest to ensure that all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROM_ALERT_CREDENTIAL_CACHE_TTL":{"name":"PROM_ALERT_CREDENTIAL_CACHE_TTL","type":"int","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_HOSTNAME_PER_DAEMON":{"name":"RGW_HOSTNAME_PER_DAEMON","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crypto_caller":{"name":"crypto_caller","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":true,"error_string":"","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"healthcheck_history_max_entries":{"name":"healthcheck_history_max_entries","type":"int","level":"advanced","flags":1,"default_value":"1000","min":"","max":"","enum_allowed":[],"desc":"Maximum number of health check history entries to keep","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"prometheus_tls_secret_name":{"name":"prometheus_tls_secret_name","type":"str","level":"advanced","flags":0,"default_value":"rook-ceph-prometheus-server-tls","min":"","max":"","enum_allowed":[],"desc":"name of tls secret in k8s for prometheus","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_cloning":{"name":"pause_cloning","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_purging":{"name":"pause_purging","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous subvolume purge threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"tentacle":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"force_disabled_modules":{},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"10.20.193.110:0","nonce":2639837741}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"10.20.193.110:0","nonce":4020329847}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"10.20.193.110:0","nonce":1419711991}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"10.20.193.110:0","nonce":4148239959}]}]} 2026-02-12T12:28:42.769 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2026-02-12T12:28:42.769 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2026-02-12T12:28:42.769 DEBUG:teuthology.orchestra.run.trial110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-12T12:28:42.848 INFO:teuthology.orchestra.run.trial110.stderr:2026-02-12T12:28:42.844+0000 7f64b850b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:28:42.849 INFO:teuthology.orchestra.run.trial110.stderr:2026-02-12T12:28:42.848+0000 7f64b850b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:28:42.959 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:28:42.959 INFO:teuthology.orchestra.run.trial110.stdout:{"epoch":18,"fsid":"cc145609-8d59-4ec8-9270-17a306f6a0f0","created":"2026-02-12T12:28:22.882886+0000","modified":"2026-02-12T12:28:42.304341+0000","last_up_change":"2026-02-12T12:28:34.270885+0000","last_in_change":"2026-02-12T12:28:24.629911+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-12T12:28:35.337743+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-12T12:28:38.561199+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":18,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d103ed9f-3542-43f8-91ff-6e926464dfec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.110:6804","nonce":3324838780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.110:6803","nonce":3324838780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.110:6805","nonce":3324838780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.110:6802","nonce":3324838780}]},"public_addr":"10.20.193.110:6804/3324838780","cluster_addr":"10.20.193.110:6803/3324838780","heartbeat_back_addr":"10.20.193.110:6805/3324838780","heartbeat_front_addr":"10.20.193.110:6802/3324838780","state":["exists","up"]},{"osd":1,"uuid":"23bf64e8-a6c2-4647-8b5e-59735720cc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.110:6806","nonce":2469450341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.110:6807","nonce":2469450341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.110:6809","nonce":2469450341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.110:6811","nonce":2469450341}]},"public_addr":"10.20.193.110:6806/2469450341","cluster_addr":"10.20.193.110:6807/2469450341","heartbeat_back_addr":"10.20.193.110:6809/2469450341","heartbeat_front_addr":"10.20.193.110:6811/2469450341","state":["exists","up"]},{"osd":2,"uuid":"6d179a0c-5aac-4f28-9304-80c384f8c8e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.110:6813","nonce":3082530054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.110:6812","nonce":3082530054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.110:6808","nonce":3082530054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.110:6810","nonce":3082530054}]},"public_addr":"10.20.193.110:6813/3082530054","cluster_addr":"10.20.193.110:6812/3082530054","heartbeat_back_addr":"10.20.193.110:6808/3082530054","heartbeat_front_addr":"10.20.193.110:6810/3082530054","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-12T12:28:42.969 INFO:tasks.ceph.ceph_manager.ceph:all up! 2026-02-12T12:28:42.969 DEBUG:teuthology.orchestra.run.trial110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-12T12:28:43.046 INFO:teuthology.orchestra.run.trial110.stderr:2026-02-12T12:28:43.043+0000 7fb794e5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:28:43.048 INFO:teuthology.orchestra.run.trial110.stderr:2026-02-12T12:28:43.046+0000 7fb794e5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:28:43.157 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:28:43.157 INFO:teuthology.orchestra.run.trial110.stdout:{"epoch":18,"fsid":"cc145609-8d59-4ec8-9270-17a306f6a0f0","created":"2026-02-12T12:28:22.882886+0000","modified":"2026-02-12T12:28:42.304341+0000","last_up_change":"2026-02-12T12:28:34.270885+0000","last_in_change":"2026-02-12T12:28:24.629911+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-12T12:28:35.337743+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-12T12:28:38.561199+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":18,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d103ed9f-3542-43f8-91ff-6e926464dfec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.110:6804","nonce":3324838780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.110:6803","nonce":3324838780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.110:6805","nonce":3324838780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.110:6802","nonce":3324838780}]},"public_addr":"10.20.193.110:6804/3324838780","cluster_addr":"10.20.193.110:6803/3324838780","heartbeat_back_addr":"10.20.193.110:6805/3324838780","heartbeat_front_addr":"10.20.193.110:6802/3324838780","state":["exists","up"]},{"osd":1,"uuid":"23bf64e8-a6c2-4647-8b5e-59735720cc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.110:6806","nonce":2469450341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.110:6807","nonce":2469450341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.110:6809","nonce":2469450341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.110:6811","nonce":2469450341}]},"public_addr":"10.20.193.110:6806/2469450341","cluster_addr":"10.20.193.110:6807/2469450341","heartbeat_back_addr":"10.20.193.110:6809/2469450341","heartbeat_front_addr":"10.20.193.110:6811/2469450341","state":["exists","up"]},{"osd":2,"uuid":"6d179a0c-5aac-4f28-9304-80c384f8c8e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.110:6813","nonce":3082530054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.110:6812","nonce":3082530054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.110:6808","nonce":3082530054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.110:6810","nonce":3082530054}]},"public_addr":"10.20.193.110:6813/3082530054","cluster_addr":"10.20.193.110:6812/3082530054","heartbeat_back_addr":"10.20.193.110:6808/3082530054","heartbeat_front_addr":"10.20.193.110:6810/3082530054","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-12T12:28:43.168 DEBUG:teuthology.orchestra.run.trial110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-12T12:28:43.169 DEBUG:teuthology.orchestra.run.trial110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-12T12:28:43.169 DEBUG:teuthology.orchestra.run.trial110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-12T12:28:43.248 INFO:teuthology.orchestra.run.trial110.stderr:2026-02-12T12:28:43.244+0000 7f6e549b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:28:43.249 INFO:teuthology.orchestra.run.trial110.stderr:2026-02-12T12:28:43.245+0000 7f8b330cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:28:43.250 INFO:teuthology.orchestra.run.trial110.stderr:2026-02-12T12:28:43.248+0000 7f6e549b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:28:43.250 INFO:teuthology.orchestra.run.trial110.stderr:2026-02-12T12:28:43.249+0000 7f8b330cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:28:43.251 INFO:teuthology.orchestra.run.trial110.stderr:2026-02-12T12:28:43.248+0000 7f7400ecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:28:43.253 INFO:teuthology.orchestra.run.trial110.stderr:2026-02-12T12:28:43.252+0000 7f7400ecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:28:43.265 INFO:teuthology.orchestra.run.trial110.stdout:38654705674 2026-02-12T12:28:43.265 DEBUG:teuthology.orchestra.run.trial110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-12T12:28:43.266 INFO:teuthology.orchestra.run.trial110.stdout:47244640264 2026-02-12T12:28:43.267 DEBUG:teuthology.orchestra.run.trial110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-12T12:28:43.268 INFO:teuthology.orchestra.run.trial110.stdout:47244640264 2026-02-12T12:28:43.268 DEBUG:teuthology.orchestra.run.trial110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-12T12:28:43.387 INFO:teuthology.orchestra.run.trial110.stderr:2026-02-12T12:28:43.383+0000 7ff2e3164640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:28:43.387 INFO:teuthology.orchestra.run.trial110.stderr:2026-02-12T12:28:43.383+0000 7f1e41f0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:28:43.388 INFO:teuthology.orchestra.run.trial110.stderr:2026-02-12T12:28:43.383+0000 7f59375f5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:28:43.388 INFO:teuthology.orchestra.run.trial110.stderr:2026-02-12T12:28:43.387+0000 7ff2e3164640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:28:43.389 INFO:teuthology.orchestra.run.trial110.stderr:2026-02-12T12:28:43.387+0000 7f1e41f0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:28:43.389 INFO:teuthology.orchestra.run.trial110.stderr:2026-02-12T12:28:43.387+0000 7f59375f5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:28:43.500 INFO:teuthology.orchestra.run.trial110.stdout:47244640260 2026-02-12T12:28:43.504 INFO:teuthology.orchestra.run.trial110.stdout:47244640260 2026-02-12T12:28:43.506 INFO:teuthology.orchestra.run.trial110.stdout:38654705668 2026-02-12T12:28:43.511 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640264 got 47244640260 for osd.1 2026-02-12T12:28:43.516 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640264 got 47244640260 for osd.2 2026-02-12T12:28:43.517 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705674 got 38654705668 for osd.0 2026-02-12T12:28:44.512 DEBUG:teuthology.orchestra.run.trial110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-12T12:28:44.517 DEBUG:teuthology.orchestra.run.trial110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-12T12:28:44.519 DEBUG:teuthology.orchestra.run.trial110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-12T12:28:44.591 INFO:teuthology.orchestra.run.trial110.stderr:2026-02-12T12:28:44.587+0000 7f009e90b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:28:44.592 INFO:teuthology.orchestra.run.trial110.stderr:2026-02-12T12:28:44.591+0000 7f009e90b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:28:44.636 INFO:teuthology.orchestra.run.trial110.stderr:2026-02-12T12:28:44.632+0000 7f21de9f5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:28:44.636 INFO:teuthology.orchestra.run.trial110.stderr:2026-02-12T12:28:44.632+0000 7f1eb1ddc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:28:44.638 INFO:teuthology.orchestra.run.trial110.stderr:2026-02-12T12:28:44.636+0000 7f21de9f5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:28:44.638 INFO:teuthology.orchestra.run.trial110.stderr:2026-02-12T12:28:44.636+0000 7f1eb1ddc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:28:44.704 INFO:teuthology.orchestra.run.trial110.stdout:47244640264 2026-02-12T12:28:44.715 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640264 got 47244640264 for osd.1 2026-02-12T12:28:44.715 DEBUG:teuthology.parallel:result is None 2026-02-12T12:28:44.750 INFO:teuthology.orchestra.run.trial110.stdout:38654705674 2026-02-12T12:28:44.751 INFO:teuthology.orchestra.run.trial110.stdout:47244640264 2026-02-12T12:28:44.761 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705674 got 38654705674 for osd.0 2026-02-12T12:28:44.762 DEBUG:teuthology.parallel:result is None 2026-02-12T12:28:44.763 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640264 got 47244640264 for osd.2 2026-02-12T12:28:44.763 DEBUG:teuthology.parallel:result is None 2026-02-12T12:28:44.763 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-12T12:28:44.763 DEBUG:teuthology.orchestra.run.trial110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T12:28:44.878 INFO:teuthology.orchestra.run.trial110.stderr:2026-02-12T12:28:44.875+0000 7f98cc41a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:28:44.879 INFO:teuthology.orchestra.run.trial110.stderr:2026-02-12T12:28:44.878+0000 7f98cc41a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:28:44.988 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:28:44.988 INFO:teuthology.orchestra.run.trial110.stderr:dumped all 2026-02-12T12:28:44.998 INFO:teuthology.orchestra.run.trial110.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2026-02-12T12:28:43.327696+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":157286352,"kb_used":62917052,"kb_used_data":62917052,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":94369300,"statfs":{"total":161061224448,"available":96634163200,"internally_reserved":0,"allocated":64427061248,"data_stored":64427061248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.034963"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-12T12:28:42.314893+0000","last_change":"2026-02-12T12:28:42.318456+0000","last_active":"2026-02-12T12:28:42.314893+0000","last_peered":"2026-02-12T12:28:42.314893+0000","last_clean":"2026-02-12T12:28:42.314893+0000","last_became_active":"2026-02-12T12:28:40.338637+0000","last_became_peered":"2026-02-12T12:28:40.338637+0000","last_unstale":"2026-02-12T12:28:42.314893+0000","last_undegraded":"2026-02-12T12:28:42.314893+0000","last_fullsized":"2026-02-12T12:28:42.314893+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T12:28:39.291899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T12:28:39.291899+0000","last_clean_scrub_stamp":"2026-02-12T12:28:39.291899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-12T12:28:42.316406+0000","last_change":"2026-02-12T12:28:42.319111+0000","last_active":"2026-02-12T12:28:42.316406+0000","last_peered":"2026-02-12T12:28:42.316406+0000","last_clean":"2026-02-12T12:28:42.316406+0000","last_became_active":"2026-02-12T12:28:40.337056+0000","last_became_peered":"2026-02-12T12:28:40.337056+0000","last_unstale":"2026-02-12T12:28:42.316406+0000","last_undegraded":"2026-02-12T12:28:42.316406+0000","last_fullsized":"2026-02-12T12:28:42.316406+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T12:28:39.291899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T12:28:39.291899+0000","last_clean_scrub_stamp":"2026-02-12T12:28:39.291899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-12T12:28:42.317382+0000","last_change":"2026-02-12T12:28:42.320068+0000","last_active":"2026-02-12T12:28:42.317382+0000","last_peered":"2026-02-12T12:28:42.317382+0000","last_clean":"2026-02-12T12:28:42.317382+0000","last_became_active":"2026-02-12T12:28:40.337910+0000","last_became_peered":"2026-02-12T12:28:40.337910+0000","last_unstale":"2026-02-12T12:28:42.317382+0000","last_undegraded":"2026-02-12T12:28:42.317382+0000","last_fullsized":"2026-02-12T12:28:42.317382+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T12:28:39.291899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T12:28:39.291899+0000","last_clean_scrub_stamp":"2026-02-12T12:28:39.291899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-12T12:28:42.316417+0000","last_change":"2026-02-12T12:28:42.318824+0000","last_active":"2026-02-12T12:28:42.316417+0000","last_peered":"2026-02-12T12:28:42.316417+0000","last_clean":"2026-02-12T12:28:42.316417+0000","last_became_active":"2026-02-12T12:28:40.341449+0000","last_became_peered":"2026-02-12T12:28:40.341449+0000","last_unstale":"2026-02-12T12:28:42.316417+0000","last_undegraded":"2026-02-12T12:28:42.316417+0000","last_fullsized":"2026-02-12T12:28:42.316417+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T12:28:39.291899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T12:28:39.291899+0000","last_clean_scrub_stamp":"2026-02-12T12:28:39.291899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"17'1","reported_seq":17,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-12T12:28:42.319094+0000","last_change":"2026-02-12T12:28:42.319094+0000","last_active":"2026-02-12T12:28:42.319094+0000","last_peered":"2026-02-12T12:28:42.319094+0000","last_clean":"2026-02-12T12:28:42.319094+0000","last_became_active":"2026-02-12T12:28:40.347700+0000","last_became_peered":"2026-02-12T12:28:40.347700+0000","last_unstale":"2026-02-12T12:28:42.319094+0000","last_undegraded":"2026-02-12T12:28:42.319094+0000","last_fullsized":"2026-02-12T12:28:42.319094+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T12:28:39.291899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T12:28:39.291899+0000","last_clean_scrub_stamp":"2026-02-12T12:28:39.291899+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-12T12:28:42.316349+0000","last_change":"2026-02-12T12:28:42.318750+0000","last_active":"2026-02-12T12:28:42.316349+0000","last_peered":"2026-02-12T12:28:42.316349+0000","last_clean":"2026-02-12T12:28:42.316349+0000","last_became_active":"2026-02-12T12:28:40.341815+0000","last_became_peered":"2026-02-12T12:28:40.341815+0000","last_unstale":"2026-02-12T12:28:42.316349+0000","last_undegraded":"2026-02-12T12:28:42.316349+0000","last_fullsized":"2026-02-12T12:28:42.316349+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T12:28:39.291899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T12:28:39.291899+0000","last_clean_scrub_stamp":"2026-02-12T12:28:39.291899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-12T12:28:42.314490+0000","last_change":"2026-02-12T12:28:42.317403+0000","last_active":"2026-02-12T12:28:42.314490+0000","last_peered":"2026-02-12T12:28:42.314490+0000","last_clean":"2026-02-12T12:28:42.314490+0000","last_became_active":"2026-02-12T12:28:40.335270+0000","last_became_peered":"2026-02-12T12:28:40.335270+0000","last_unstale":"2026-02-12T12:28:42.314490+0000","last_undegraded":"2026-02-12T12:28:42.314490+0000","last_fullsized":"2026-02-12T12:28:42.314490+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T12:28:39.291899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T12:28:39.291899+0000","last_clean_scrub_stamp":"2026-02-12T12:28:39.291899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-12T12:28:42.316415+0000","last_change":"2026-02-12T12:28:42.320157+0000","last_active":"2026-02-12T12:28:42.316415+0000","last_peered":"2026-02-12T12:28:42.316415+0000","last_clean":"2026-02-12T12:28:42.316415+0000","last_became_active":"2026-02-12T12:28:40.335737+0000","last_became_peered":"2026-02-12T12:28:40.335737+0000","last_unstale":"2026-02-12T12:28:42.316415+0000","last_undegraded":"2026-02-12T12:28:42.316415+0000","last_fullsized":"2026-02-12T12:28:42.316415+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T12:28:39.291899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T12:28:39.291899+0000","last_clean_scrub_stamp":"2026-02-12T12:28:39.291899+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":90,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-12T12:28:42.314805+0000","last_change":"2026-02-12T12:28:37.311825+0000","last_active":"2026-02-12T12:28:42.314805+0000","last_peered":"2026-02-12T12:28:42.314805+0000","last_clean":"2026-02-12T12:28:42.314805+0000","last_became_active":"2026-02-12T12:28:37.304433+0000","last_became_peered":"2026-02-12T12:28:37.304433+0000","last_unstale":"2026-02-12T12:28:42.314805+0000","last_undegraded":"2026-02-12T12:28:42.314805+0000","last_fullsized":"2026-02-12T12:28:42.314805+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T12:28:36.276882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T12:28:36.276882+0000","last_clean_scrub_stamp":"2026-02-12T12:28:36.276882+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":161061224448,"available":96634114048,"internally_reserved":0,"allocated":64427110400,"data_stored":64427110400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53687074816,"available":32211070976,"internally_reserved":0,"allocated":21476003840,"data_stored":21476003840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640264,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20971820,"kb_used_data":20971820,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456964,"statfs":{"total":53687074816,"available":32211931136,"internally_reserved":0,"allocated":21475143680,"data_stored":21475143680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640264,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972652,"kb_used_data":20972652,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456132,"statfs":{"total":53687074816,"available":32211079168,"internally_reserved":0,"allocated":21475995648,"data_stored":21475995648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972580,"kb_used_data":20972580,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456204,"statfs":{"total":53687074816,"available":32211152896,"internally_reserved":0,"allocated":21475921920,"data_stored":21475921920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53687074816,"available":32211070976,"internally_reserved":0,"allocated":21476003840,"data_stored":21476003840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53687074816,"available":32211120128,"internally_reserved":0,"allocated":21475954688,"data_stored":21475954688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53687074816,"available":32211070976,"internally_reserved":0,"allocated":21476003840,"data_stored":21476003840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53687074816,"available":32211922944,"internally_reserved":0,"allocated":21475151872,"data_stored":21475151872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T12:28:45.000 DEBUG:teuthology.orchestra.run.trial110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-12T12:28:45.077 INFO:teuthology.orchestra.run.trial110.stderr:2026-02-12T12:28:45.074+0000 7f2f5d051640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:28:45.079 INFO:teuthology.orchestra.run.trial110.stderr:2026-02-12T12:28:45.077+0000 7f2f5d051640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:28:45.187 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:28:45.187 INFO:teuthology.orchestra.run.trial110.stderr:dumped all 2026-02-12T12:28:45.196 INFO:teuthology.orchestra.run.trial110.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2026-02-12T12:28:43.327696+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":157286352,"kb_used":62917052,"kb_used_data":62917052,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":94369300,"statfs":{"total":161061224448,"available":96634163200,"internally_reserved":0,"allocated":64427061248,"data_stored":64427061248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.034963"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-12T12:28:42.314893+0000","last_change":"2026-02-12T12:28:42.318456+0000","last_active":"2026-02-12T12:28:42.314893+0000","last_peered":"2026-02-12T12:28:42.314893+0000","last_clean":"2026-02-12T12:28:42.314893+0000","last_became_active":"2026-02-12T12:28:40.338637+0000","last_became_peered":"2026-02-12T12:28:40.338637+0000","last_unstale":"2026-02-12T12:28:42.314893+0000","last_undegraded":"2026-02-12T12:28:42.314893+0000","last_fullsized":"2026-02-12T12:28:42.314893+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T12:28:39.291899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T12:28:39.291899+0000","last_clean_scrub_stamp":"2026-02-12T12:28:39.291899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-12T12:28:42.316406+0000","last_change":"2026-02-12T12:28:42.319111+0000","last_active":"2026-02-12T12:28:42.316406+0000","last_peered":"2026-02-12T12:28:42.316406+0000","last_clean":"2026-02-12T12:28:42.316406+0000","last_became_active":"2026-02-12T12:28:40.337056+0000","last_became_peered":"2026-02-12T12:28:40.337056+0000","last_unstale":"2026-02-12T12:28:42.316406+0000","last_undegraded":"2026-02-12T12:28:42.316406+0000","last_fullsized":"2026-02-12T12:28:42.316406+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T12:28:39.291899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T12:28:39.291899+0000","last_clean_scrub_stamp":"2026-02-12T12:28:39.291899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-12T12:28:42.317382+0000","last_change":"2026-02-12T12:28:42.320068+0000","last_active":"2026-02-12T12:28:42.317382+0000","last_peered":"2026-02-12T12:28:42.317382+0000","last_clean":"2026-02-12T12:28:42.317382+0000","last_became_active":"2026-02-12T12:28:40.337910+0000","last_became_peered":"2026-02-12T12:28:40.337910+0000","last_unstale":"2026-02-12T12:28:42.317382+0000","last_undegraded":"2026-02-12T12:28:42.317382+0000","last_fullsized":"2026-02-12T12:28:42.317382+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T12:28:39.291899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T12:28:39.291899+0000","last_clean_scrub_stamp":"2026-02-12T12:28:39.291899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-12T12:28:42.316417+0000","last_change":"2026-02-12T12:28:42.318824+0000","last_active":"2026-02-12T12:28:42.316417+0000","last_peered":"2026-02-12T12:28:42.316417+0000","last_clean":"2026-02-12T12:28:42.316417+0000","last_became_active":"2026-02-12T12:28:40.341449+0000","last_became_peered":"2026-02-12T12:28:40.341449+0000","last_unstale":"2026-02-12T12:28:42.316417+0000","last_undegraded":"2026-02-12T12:28:42.316417+0000","last_fullsized":"2026-02-12T12:28:42.316417+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T12:28:39.291899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T12:28:39.291899+0000","last_clean_scrub_stamp":"2026-02-12T12:28:39.291899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"17'1","reported_seq":17,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-12T12:28:42.319094+0000","last_change":"2026-02-12T12:28:42.319094+0000","last_active":"2026-02-12T12:28:42.319094+0000","last_peered":"2026-02-12T12:28:42.319094+0000","last_clean":"2026-02-12T12:28:42.319094+0000","last_became_active":"2026-02-12T12:28:40.347700+0000","last_became_peered":"2026-02-12T12:28:40.347700+0000","last_unstale":"2026-02-12T12:28:42.319094+0000","last_undegraded":"2026-02-12T12:28:42.319094+0000","last_fullsized":"2026-02-12T12:28:42.319094+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T12:28:39.291899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T12:28:39.291899+0000","last_clean_scrub_stamp":"2026-02-12T12:28:39.291899+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-12T12:28:42.316349+0000","last_change":"2026-02-12T12:28:42.318750+0000","last_active":"2026-02-12T12:28:42.316349+0000","last_peered":"2026-02-12T12:28:42.316349+0000","last_clean":"2026-02-12T12:28:42.316349+0000","last_became_active":"2026-02-12T12:28:40.341815+0000","last_became_peered":"2026-02-12T12:28:40.341815+0000","last_unstale":"2026-02-12T12:28:42.316349+0000","last_undegraded":"2026-02-12T12:28:42.316349+0000","last_fullsized":"2026-02-12T12:28:42.316349+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T12:28:39.291899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T12:28:39.291899+0000","last_clean_scrub_stamp":"2026-02-12T12:28:39.291899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-12T12:28:42.314490+0000","last_change":"2026-02-12T12:28:42.317403+0000","last_active":"2026-02-12T12:28:42.314490+0000","last_peered":"2026-02-12T12:28:42.314490+0000","last_clean":"2026-02-12T12:28:42.314490+0000","last_became_active":"2026-02-12T12:28:40.335270+0000","last_became_peered":"2026-02-12T12:28:40.335270+0000","last_unstale":"2026-02-12T12:28:42.314490+0000","last_undegraded":"2026-02-12T12:28:42.314490+0000","last_fullsized":"2026-02-12T12:28:42.314490+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T12:28:39.291899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T12:28:39.291899+0000","last_clean_scrub_stamp":"2026-02-12T12:28:39.291899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-12T12:28:42.316415+0000","last_change":"2026-02-12T12:28:42.320157+0000","last_active":"2026-02-12T12:28:42.316415+0000","last_peered":"2026-02-12T12:28:42.316415+0000","last_clean":"2026-02-12T12:28:42.316415+0000","last_became_active":"2026-02-12T12:28:40.335737+0000","last_became_peered":"2026-02-12T12:28:40.335737+0000","last_unstale":"2026-02-12T12:28:42.316415+0000","last_undegraded":"2026-02-12T12:28:42.316415+0000","last_fullsized":"2026-02-12T12:28:42.316415+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T12:28:39.291899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T12:28:39.291899+0000","last_clean_scrub_stamp":"2026-02-12T12:28:39.291899+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":90,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-12T12:28:42.314805+0000","last_change":"2026-02-12T12:28:37.311825+0000","last_active":"2026-02-12T12:28:42.314805+0000","last_peered":"2026-02-12T12:28:42.314805+0000","last_clean":"2026-02-12T12:28:42.314805+0000","last_became_active":"2026-02-12T12:28:37.304433+0000","last_became_peered":"2026-02-12T12:28:37.304433+0000","last_unstale":"2026-02-12T12:28:42.314805+0000","last_undegraded":"2026-02-12T12:28:42.314805+0000","last_fullsized":"2026-02-12T12:28:42.314805+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-12T12:28:36.276882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-12T12:28:36.276882+0000","last_clean_scrub_stamp":"2026-02-12T12:28:36.276882+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":161061224448,"available":96634114048,"internally_reserved":0,"allocated":64427110400,"data_stored":64427110400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53687074816,"available":32211070976,"internally_reserved":0,"allocated":21476003840,"data_stored":21476003840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640264,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20971820,"kb_used_data":20971820,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456964,"statfs":{"total":53687074816,"available":32211931136,"internally_reserved":0,"allocated":21475143680,"data_stored":21475143680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640264,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972652,"kb_used_data":20972652,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456132,"statfs":{"total":53687074816,"available":32211079168,"internally_reserved":0,"allocated":21475995648,"data_stored":21475995648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972580,"kb_used_data":20972580,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456204,"statfs":{"total":53687074816,"available":32211152896,"internally_reserved":0,"allocated":21475921920,"data_stored":21475921920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53687074816,"available":32211070976,"internally_reserved":0,"allocated":21476003840,"data_stored":21476003840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53687074816,"available":32211120128,"internally_reserved":0,"allocated":21475954688,"data_stored":21475954688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53687074816,"available":32211070976,"internally_reserved":0,"allocated":21476003840,"data_stored":21476003840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53687074816,"available":32211922944,"internally_reserved":0,"allocated":21475151872,"data_stored":21475151872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-12T12:28:45.198 INFO:tasks.ceph.ceph_manager.ceph:clean! 2026-02-12T12:28:45.198 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2026-02-12T12:28:45.198 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2026-02-12T12:28:45.198 DEBUG:teuthology.orchestra.run.trial110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-12T12:28:45.275 INFO:teuthology.orchestra.run.trial110.stderr:2026-02-12T12:28:45.272+0000 7f5d27846640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:28:45.276 INFO:teuthology.orchestra.run.trial110.stderr:2026-02-12T12:28:45.274+0000 7f5d27846640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:28:45.415 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:28:45.416 INFO:teuthology.orchestra.run.trial110.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2026-02-12T12:28:45.426 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2026-02-12T12:28:45.427 INFO:teuthology.run_tasks:Running task ssh_keys... 2026-02-12T12:28:45.540 DEBUG:teuthology.orchestra.run.trial110:> echo 'StrictHostKeyChecking no 2026-02-12T12:28:45.540 DEBUG:teuthology.orchestra.run.trial110:> ' > /home/ubuntu/.ssh/config && echo 'UserKnownHostsFile ' /dev/null >> /home/ubuntu/.ssh/config && chmod 600 /home/ubuntu/.ssh/config 2026-02-12T12:28:45.556 INFO:teuthology.task.ssh_keys:generated public key AAAAB3NzaC1yc2EAAAADAQABAAABAQCcHtyIvUDQQhc4raYDel446/SBzIbp1JUeRdqes59DHmRFEDWWVPDEmU4UtuRW0WINdT6s4ZiIm31zw+9+KuAK0AWtyn8yvWzKw/6fubC/8oae4ux4LU219lpZhrj7543koeOFF6D2/FBmH05v6vLGOCvIO1RqTEkaO43JG9tjAmPq68uluaWUEu1jvsto2argDJEHu7/YTtrLD+860DLMsXe/2YbWI5RDhy7W/WoMohIzvgqzkVE90gzoLiOptl5Fm8eUMAuOgZc8mJ1+RScPS1Lf3QeAISmuWk1F8lE5myApxU7a7oILf9Xson7Kcs3edzmBXK1mJoNktq/UypP7 2026-02-12T12:28:45.556 INFO:teuthology.task.ssh_keys:pushing keys to trial110.front.sepia.ceph.com for ubuntu 2026-02-12T12:28:45.556 DEBUG:teuthology.orchestra.run.trial110:> rm -f -- /home/ubuntu/.ssh/id_rsa 2026-02-12T12:28:45.611 DEBUG:teuthology.orchestra.run.trial110:> set -ex 2026-02-12T12:28:45.611 DEBUG:teuthology.orchestra.run.trial110:> dd of=/home/ubuntu/.ssh/id_rsa 2026-02-12T12:28:45.611 DEBUG:teuthology.orchestra.run.trial110:> chmod 0500 /home/ubuntu/.ssh/id_rsa 2026-02-12T12:28:45.668 DEBUG:teuthology.orchestra.run.trial110:> rm -f -- /home/ubuntu/.ssh/id_rsa.pub 2026-02-12T12:28:45.725 DEBUG:teuthology.orchestra.run.trial110:> set -ex 2026-02-12T12:28:45.726 DEBUG:teuthology.orchestra.run.trial110:> dd of=/home/ubuntu/.ssh/id_rsa.pub 2026-02-12T12:28:45.783 DEBUG:teuthology.orchestra.run.trial110:> cp -v -a /home/ubuntu/.ssh/authorized_keys /home/ubuntu/.ssh/authorized_keys_2026-02-12_12:28:45:783234 2026-02-12T12:28:45.839 INFO:teuthology.orchestra.run.trial110.stdout:'/home/ubuntu/.ssh/authorized_keys' -> '/home/ubuntu/.ssh/authorized_keys_2026-02-12_12:28:45:783234' 2026-02-12T12:28:45.840 DEBUG:teuthology.orchestra.run.trial110:> set -ex 2026-02-12T12:28:45.840 DEBUG:teuthology.orchestra.run.trial110:> dd of=/home/ubuntu/.ssh/authorized_keys conv=notrunc oflag=append 2026-02-12T12:28:45.896 INFO:teuthology.run_tasks:Running task cbt... 2026-02-12T12:28:45.920 INFO:tasks.cbt:cbt configuration is {'cluster': {'user': 'ubuntu', 'head': 'ubuntu@trial110.front.sepia.ceph.com', 'osds': ['ubuntu@trial110.front.sepia.ceph.com'], 'mons': ['ubuntu@trial110.front.sepia.ceph.com'], 'clients': ['ubuntu@trial110.front.sepia.ceph.com'], 'rgws': {'ubuntu@trial110.front.sepia.ceph.com': None}, 'osds_per_node': 3, 'rebuild_every_test': False, 'use_existing': True, 'is_teuthology': True, 'iterations': 1, 'tmp_dir': '/tmp/cbt', 'pool_profiles': {'replicated': {'pg_size': 256, 'pgp_size': 256, 'replication': 'replicated'}}, 'pid_dir': '/var/run/ceph'}, 'benchmarks': {'radosbench': {'concurrent_ops': 4, 'concurrent_procs': 2, 'op_size': [4194304], 'pool_monitoring_list': ['collectl'], 'pool_profile': 'replicated', 'readmode': 'rand', 'run_monitoring_list': ['collectl'], 'time': 60, 'write_only': False}}, 'client_endpoints': None, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}} 2026-02-12T12:28:45.921 DEBUG:teuthology.orchestra.run.trial110:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/cbt 2026-02-12T12:28:45.955 DEBUG:teuthology.orchestra.run.trial110:> set -ex 2026-02-12T12:28:45.955 DEBUG:teuthology.orchestra.run.trial110:> dd of=/home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2026-02-12T12:28:46.008 DEBUG:teuthology.orchestra.run.trial110:> git clone --depth 1 -b master https://github.com/ceph/cbt.git /home/ubuntu/cephtest/cbt 2026-02-12T12:28:46.070 INFO:teuthology.orchestra.run.trial110.stderr:Cloning into '/home/ubuntu/cephtest/cbt'... 2026-02-12T12:28:46.320 INFO:tasks.cbt:Installing collectl 2026-02-12T12:28:46.321 DEBUG:teuthology.orchestra.run.trial110:> sudo mkdir collectl && cd collectl && sudo wget https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download -O collectl.tar.gz && sudo tar -xvf collectl.tar.gz && cd collectl-4.3.1 && sudo ./INSTALL 2026-02-12T12:28:46.396 INFO:teuthology.orchestra.run.trial110.stderr:--2026-02-12 12:28:46-- https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download 2026-02-12T12:28:46.456 INFO:teuthology.orchestra.run.trial110.stderr:Resolving sourceforge.net (sourceforge.net)... 104.18.13.149, 104.18.12.149, 2606:4700::6812:c95, ... 2026-02-12T12:28:46.459 INFO:teuthology.orchestra.run.trial110.stderr:Connecting to sourceforge.net (sourceforge.net)|104.18.13.149|:443... connected. 2026-02-12T12:28:46.651 INFO:teuthology.orchestra.run.trial110.stderr:HTTP request sent, awaiting response... 302 Found 2026-02-12T12:28:46.651 INFO:teuthology.orchestra.run.trial110.stderr:Location: https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABpjcd-f6aLV4f-t3V1-U6MZ2NYP0T1u8NES_A_-4sRVR8ZRMmdlbRvhUqSttTZROXBPHOiMRJ6yQhcHDw9Gk0Bwndy9g%3D%3D&use_mirror=phoenixnap&r= [following] 2026-02-12T12:28:46.652 INFO:teuthology.orchestra.run.trial110.stderr:--2026-02-12 12:28:46-- https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABpjcd-f6aLV4f-t3V1-U6MZ2NYP0T1u8NES_A_-4sRVR8ZRMmdlbRvhUqSttTZROXBPHOiMRJ6yQhcHDw9Gk0Bwndy9g%3D%3D&use_mirror=phoenixnap&r= 2026-02-12T12:28:46.709 INFO:teuthology.orchestra.run.trial110.stderr:Resolving downloads.sourceforge.net (downloads.sourceforge.net)... 104.18.12.149, 104.18.13.149, 2606:4700::6812:d95, ... 2026-02-12T12:28:46.713 INFO:teuthology.orchestra.run.trial110.stderr:Connecting to downloads.sourceforge.net (downloads.sourceforge.net)|104.18.12.149|:443... connected. 2026-02-12T12:28:46.912 INFO:teuthology.orchestra.run.trial110.stderr:HTTP request sent, awaiting response... 302 Found 2026-02-12T12:28:46.912 INFO:teuthology.orchestra.run.trial110.stderr:Location: https://phoenixnap.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?viasf=1 [following] 2026-02-12T12:28:46.912 INFO:teuthology.orchestra.run.trial110.stderr:--2026-02-12 12:28:46-- https://phoenixnap.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?viasf=1 2026-02-12T12:28:46.921 INFO:teuthology.orchestra.run.trial110.stderr:Resolving phoenixnap.dl.sourceforge.net (phoenixnap.dl.sourceforge.net)... 184.164.141.26 2026-02-12T12:28:46.983 INFO:teuthology.orchestra.run.trial110.stderr:Connecting to phoenixnap.dl.sourceforge.net (phoenixnap.dl.sourceforge.net)|184.164.141.26|:443... connected. 2026-02-12T12:28:47.278 INFO:teuthology.orchestra.run.trial110.stderr:HTTP request sent, awaiting response... 200 OK 2026-02-12T12:28:47.279 INFO:teuthology.orchestra.run.trial110.stderr:Length: 620693 (606K) [application/x-gzip] 2026-02-12T12:28:47.279 INFO:teuthology.orchestra.run.trial110.stderr:Saving to: ‘collectl.tar.gz’ 2026-02-12T12:28:47.279 INFO:teuthology.orchestra.run.trial110.stderr: 2026-02-12T12:28:47.341 INFO:teuthology.orchestra.run.trial110.stderr: 0K .......... .......... .......... .......... .......... 8% 801K 1s 2026-02-12T12:28:47.403 INFO:teuthology.orchestra.run.trial110.stderr: 50K .......... .......... .......... .......... .......... 16% 813K 1s 2026-02-12T12:28:47.404 INFO:teuthology.orchestra.run.trial110.stderr: 100K .......... .......... .......... .......... .......... 24% 61.8M 0s 2026-02-12T12:28:47.443 INFO:teuthology.orchestra.run.trial110.stderr: 150K .......... .......... .......... .......... .......... 32% 1.21M 0s 2026-02-12T12:28:47.467 INFO:teuthology.orchestra.run.trial110.stderr: 200K .......... .......... .......... .......... .......... 41% 2.32M 0s 2026-02-12T12:28:47.468 INFO:teuthology.orchestra.run.trial110.stderr: 250K .......... .......... .......... .......... .......... 49% 150M 0s 2026-02-12T12:28:47.468 INFO:teuthology.orchestra.run.trial110.stderr: 300K .......... .......... .......... .......... .......... 57% 131M 0s 2026-02-12T12:28:47.468 INFO:teuthology.orchestra.run.trial110.stderr: 350K .......... .......... .......... .......... .......... 65% 200M 0s 2026-02-12T12:28:47.530 INFO:teuthology.orchestra.run.trial110.stderr: 400K .......... .......... .......... .......... .......... 74% 822K 0s 2026-02-12T12:28:47.530 INFO:teuthology.orchestra.run.trial110.stderr: 450K .......... .......... .......... .......... .......... 82% 196M 0s 2026-02-12T12:28:47.530 INFO:teuthology.orchestra.run.trial110.stderr: 500K .......... .......... .......... .......... .......... 90% 218M 0s 2026-02-12T12:28:47.530 INFO:teuthology.orchestra.run.trial110.stderr: 550K .......... .......... .......... .......... .......... 98% 155M 0s 2026-02-12T12:28:47.530 INFO:teuthology.orchestra.run.trial110.stderr: 600K ...... 100% 11.4T=0.2s 2026-02-12T12:28:47.530 INFO:teuthology.orchestra.run.trial110.stderr: 2026-02-12T12:28:47.530 INFO:teuthology.orchestra.run.trial110.stderr:2026-02-12 12:28:47 (2.38 MB/s) - ‘collectl.tar.gz’ saved [620693/620693] 2026-02-12T12:28:47.530 INFO:teuthology.orchestra.run.trial110.stderr: 2026-02-12T12:28:47.551 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/ 2026-02-12T12:28:47.551 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/collectl.conf 2026-02-12T12:28:47.551 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/docs/ 2026-02-12T12:28:47.551 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/docs/SlowProc.html 2026-02-12T12:28:47.551 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/docs/BuddyInfo.html 2026-02-12T12:28:47.551 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/docs/WhySummary.html 2026-02-12T12:28:47.551 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/docs/Socket.html 2026-02-12T12:28:47.551 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/docs/colmux.html 2026-02-12T12:28:47.552 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/docs/Tutorial.html 2026-02-12T12:28:47.552 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/docs/Gexpr.html 2026-02-12T12:28:47.552 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/docs/Import.html 2026-02-12T12:28:47.552 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/docs/style.css 2026-02-12T12:28:47.552 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/docs/Ganglia3.jpg 2026-02-12T12:28:47.552 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/docs/Ganglia2.jpg 2026-02-12T12:28:47.552 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/docs/FileNaming.html 2026-02-12T12:28:47.552 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/docs/TheMath.html 2026-02-12T12:28:47.552 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/docs/Environmental.html 2026-02-12T12:28:47.552 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/docs/Process.html 2026-02-12T12:28:47.552 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/docs/OutputFiles.html 2026-02-12T12:28:47.552 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/docs/HiResTime.html 2026-02-12T12:28:47.553 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/docs/OutputFormats.html 2026-02-12T12:28:47.553 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/docs/slub.jpg 2026-02-12T12:28:47.553 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/docs/Lustre.html 2026-02-12T12:28:47.553 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/docs/Documentation.html 2026-02-12T12:28:47.553 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/docs/Inodes.html 2026-02-12T12:28:47.553 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/docs/Graphite.html 2026-02-12T12:28:47.553 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/docs/InputFiles.html 2026-02-12T12:28:47.553 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/docs/Export.html 2026-02-12T12:28:47.553 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/docs/Interrupts.html 2026-02-12T12:28:47.553 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/docs/Startup.html 2026-02-12T12:28:47.553 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/docs/Performance.html 2026-02-12T12:28:47.553 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/docs/Messages.html 2026-02-12T12:28:47.553 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/docs/Logging.html 2026-02-12T12:28:47.554 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/docs/Data-detail.html 2026-02-12T12:28:47.554 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/docs/CPUs.html 2026-02-12T12:28:47.554 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/docs/Matrix.html 2026-02-12T12:28:47.554 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/docs/Data-brief.html 2026-02-12T12:28:47.554 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/docs/SlabInfo.html 2026-02-12T12:28:47.554 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/docs/Misc.html 2026-02-12T12:28:47.554 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/docs/Examples.html 2026-02-12T12:28:47.554 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/docs/OpenStack.html 2026-02-12T12:28:47.554 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/docs/Plotfiles.html 2026-02-12T12:28:47.554 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/docs/ColmuxCPU.jpg 2026-02-12T12:28:47.554 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/docs/Tutorial-Lustre.html 2026-02-12T12:28:47.554 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/docs/Data-verbose.html 2026-02-12T12:28:47.554 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/docs/Data.html 2026-02-12T12:28:47.555 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/docs/Exceptions.html 2026-02-12T12:28:47.555 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/docs/RunningAsAService.html 2026-02-12T12:28:47.555 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/docs/FAQ-collectl.html 2026-02-12T12:28:47.555 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/docs/Ganglia.jpg 2026-02-12T12:28:47.555 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/docs/OperationalModes.html 2026-02-12T12:28:47.555 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/docs/Playback.html 2026-02-12T12:28:47.555 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/docs/Hello.html 2026-02-12T12:28:47.555 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/docs/Sockets.html 2026-02-12T12:28:47.555 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/docs/Network.html 2026-02-12T12:28:47.555 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/docs/Infiniband.html 2026-02-12T12:28:47.555 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/docs/Memory.html 2026-02-12T12:28:47.555 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/docs/ColmuxLustre.jpg 2026-02-12T12:28:47.555 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/docs/Disks.html 2026-02-12T12:28:47.556 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/docs/NfsInfo.html 2026-02-12T12:28:47.556 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/service/ 2026-02-12T12:28:47.556 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/service/collectl.service 2026-02-12T12:28:47.556 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/initd/ 2026-02-12T12:28:47.556 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/initd/collectl-suse 2026-02-12T12:28:47.556 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/initd/collectl-debian 2026-02-12T12:28:47.556 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/initd/collectl-generic 2026-02-12T12:28:47.556 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/initd/collectl 2026-02-12T12:28:47.556 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/misc.ph 2026-02-12T12:28:47.556 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/gexpr.ph 2026-02-12T12:28:47.556 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/statsd.ph 2026-02-12T12:28:47.556 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/envrules.std 2026-02-12T12:28:47.556 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/hello.ph 2026-02-12T12:28:47.557 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/vmsum.ph 2026-02-12T12:28:47.557 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/proctree.ph 2026-02-12T12:28:47.557 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/README-WINDOWS 2026-02-12T12:28:47.557 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/formatit.ph 2026-02-12T12:28:47.557 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/colmux 2026-02-12T12:28:47.557 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/INSTALL 2026-02-12T12:28:47.557 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/graphite.ph 2026-02-12T12:28:47.557 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/RELEASE-collectl 2026-02-12T12:28:47.557 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/vmstat.ph 2026-02-12T12:28:47.557 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/README 2026-02-12T12:28:47.557 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/man1/ 2026-02-12T12:28:47.557 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/man1/collectl.1 2026-02-12T12:28:47.557 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/man1/colmux.1 2026-02-12T12:28:47.558 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/lexpr.ph 2026-02-12T12:28:47.558 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/client.pl 2026-02-12T12:28:47.558 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/collectl 2026-02-12T12:28:47.558 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/COPYING 2026-02-12T12:28:47.558 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/vnet.ph 2026-02-12T12:28:47.558 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/GPL 2026-02-12T12:28:47.558 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/UNINSTALL 2026-02-12T12:28:47.558 INFO:teuthology.orchestra.run.trial110.stdout:collectl-4.3.1/ARTISTIC 2026-02-12T12:28:47.593 INFO:teuthology.orchestra.run.trial110.stderr:./INSTALL: line 90: chkconfig: command not found 2026-02-12T12:28:47.595 DEBUG:teuthology.orchestra.run.trial110:> sudo yum -y install librbd-devel pdsh pdsh-rcmd-ssh perf 2026-02-12T12:28:47.941 INFO:teuthology.orchestra.run.trial110.stdout:Last metadata expiration check: 0:00:57 ago on Thu 12 Feb 2026 12:27:50 PM UTC. 2026-02-12T12:28:48.026 INFO:teuthology.orchestra.run.trial110.stdout:Dependencies resolved. 2026-02-12T12:28:48.026 INFO:teuthology.orchestra.run.trial110.stdout:================================================================================ 2026-02-12T12:28:48.026 INFO:teuthology.orchestra.run.trial110.stdout: Package Arch Version Repository Size 2026-02-12T12:28:48.026 INFO:teuthology.orchestra.run.trial110.stdout:================================================================================ 2026-02-12T12:28:48.026 INFO:teuthology.orchestra.run.trial110.stdout:Installing: 2026-02-12T12:28:48.026 INFO:teuthology.orchestra.run.trial110.stdout: librbd-devel x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 24 k 2026-02-12T12:28:48.027 INFO:teuthology.orchestra.run.trial110.stdout: pdsh x86_64 2.34-7.el9 epel 118 k 2026-02-12T12:28:48.027 INFO:teuthology.orchestra.run.trial110.stdout: pdsh-rcmd-ssh x86_64 2.34-7.el9 epel 13 k 2026-02-12T12:28:48.027 INFO:teuthology.orchestra.run.trial110.stdout: perf x86_64 5.14.0-677.el9 appstream 3.2 M 2026-02-12T12:28:48.027 INFO:teuthology.orchestra.run.trial110.stdout:Installing dependencies: 2026-02-12T12:28:48.027 INFO:teuthology.orchestra.run.trial110.stdout: bzip2 x86_64 1.0.8-10.el9 baseos 56 k 2026-02-12T12:28:48.027 INFO:teuthology.orchestra.run.trial110.stdout: libradospp-devel x86_64 2:20.3.0-5145.g5fdb7d8f.el9 ceph 31 k 2026-02-12T12:28:48.027 INFO:teuthology.orchestra.run.trial110.stdout: libtraceevent x86_64 1.8.4-2.el9 baseos 290 k 2026-02-12T12:28:48.027 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:28:48.027 INFO:teuthology.orchestra.run.trial110.stdout:Transaction Summary 2026-02-12T12:28:48.027 INFO:teuthology.orchestra.run.trial110.stdout:================================================================================ 2026-02-12T12:28:48.027 INFO:teuthology.orchestra.run.trial110.stdout:Install 7 Packages 2026-02-12T12:28:48.028 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:28:48.028 INFO:teuthology.orchestra.run.trial110.stdout:Total download size: 3.7 M 2026-02-12T12:28:48.028 INFO:teuthology.orchestra.run.trial110.stdout:Installed size: 14 M 2026-02-12T12:28:48.028 INFO:teuthology.orchestra.run.trial110.stdout:Downloading Packages: 2026-02-12T12:28:48.518 INFO:teuthology.orchestra.run.trial110.stdout:(1/7): bzip2-1.0.8-10.el9.x86_64.rpm 502 kB/s | 56 kB 00:00 2026-02-12T12:28:48.606 INFO:teuthology.orchestra.run.trial110.stdout:(2/7): libtraceevent-1.8.4-2.el9.x86_64.rpm 3.2 MB/s | 290 kB 00:00 2026-02-12T12:28:48.734 INFO:teuthology.orchestra.run.trial110.stdout:(3/7): perf-5.14.0-677.el9.x86_64.rpm 25 MB/s | 3.2 MB 00:00 2026-02-12T12:28:48.758 INFO:teuthology.orchestra.run.trial110.stdout:(4/7): librbd-devel-20.3.0-5145.g5fdb7d8f.el9.x 68 kB/s | 24 kB 00:00 2026-02-12T12:28:48.761 INFO:teuthology.orchestra.run.trial110.stdout:(5/7): libradospp-devel-20.3.0-5145.g5fdb7d8f.e 88 kB/s | 31 kB 00:00 2026-02-12T12:28:48.860 INFO:teuthology.orchestra.run.trial110.stdout:(6/7): pdsh-rcmd-ssh-2.34-7.el9.x86_64.rpm 128 kB/s | 13 kB 00:00 2026-02-12T12:28:49.023 INFO:teuthology.orchestra.run.trial110.stdout:(7/7): pdsh-2.34-7.el9.x86_64.rpm 410 kB/s | 118 kB 00:00 2026-02-12T12:28:49.024 INFO:teuthology.orchestra.run.trial110.stdout:-------------------------------------------------------------------------------- 2026-02-12T12:28:49.025 INFO:teuthology.orchestra.run.trial110.stdout:Total 3.7 MB/s | 3.7 MB 00:00 2026-02-12T12:28:49.057 INFO:teuthology.orchestra.run.trial110.stdout:Running transaction check 2026-02-12T12:28:49.067 INFO:teuthology.orchestra.run.trial110.stdout:Transaction check succeeded. 2026-02-12T12:28:49.067 INFO:teuthology.orchestra.run.trial110.stdout:Running transaction test 2026-02-12T12:28:49.124 INFO:teuthology.orchestra.run.trial110.stdout:Transaction test succeeded. 2026-02-12T12:28:49.125 INFO:teuthology.orchestra.run.trial110.stdout:Running transaction 2026-02-12T12:28:49.200 INFO:teuthology.orchestra.run.trial110.stdout: Preparing : 1/1 2026-02-12T12:28:49.206 INFO:teuthology.orchestra.run.trial110.stdout: Installing : pdsh-rcmd-ssh-2.34-7.el9.x86_64 1/7 2026-02-12T12:28:49.245 INFO:teuthology.orchestra.run.trial110.stdout: Installing : pdsh-2.34-7.el9.x86_64 2/7 2026-02-12T12:28:49.250 INFO:teuthology.orchestra.run.trial110.stdout: Installing : libtraceevent-1.8.4-2.el9.x86_64 3/7 2026-02-12T12:28:49.254 INFO:teuthology.orchestra.run.trial110.stdout: Installing : bzip2-1.0.8-10.el9.x86_64 4/7 2026-02-12T12:28:49.256 INFO:teuthology.orchestra.run.trial110.stdout: Installing : libradospp-devel-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 5/7 2026-02-12T12:28:49.417 INFO:teuthology.orchestra.run.trial110.stdout: Installing : librbd-devel-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 6/7 2026-02-12T12:28:49.438 INFO:teuthology.orchestra.run.trial110.stdout: Installing : perf-5.14.0-677.el9.x86_64 7/7 2026-02-12T12:28:49.543 INFO:teuthology.orchestra.run.trial110.stdout: Running scriptlet: perf-5.14.0-677.el9.x86_64 7/7 2026-02-12T12:28:49.543 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : libradospp-devel-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 1/7 2026-02-12T12:28:49.543 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : librbd-devel-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2/7 2026-02-12T12:28:49.543 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : bzip2-1.0.8-10.el9.x86_64 3/7 2026-02-12T12:28:49.543 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : libtraceevent-1.8.4-2.el9.x86_64 4/7 2026-02-12T12:28:49.543 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : perf-5.14.0-677.el9.x86_64 5/7 2026-02-12T12:28:49.543 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : pdsh-2.34-7.el9.x86_64 6/7 2026-02-12T12:28:49.602 INFO:teuthology.orchestra.run.trial110.stdout: Verifying : pdsh-rcmd-ssh-2.34-7.el9.x86_64 7/7 2026-02-12T12:28:49.602 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:28:49.602 INFO:teuthology.orchestra.run.trial110.stdout:Installed: 2026-02-12T12:28:49.602 INFO:teuthology.orchestra.run.trial110.stdout: bzip2-1.0.8-10.el9.x86_64 2026-02-12T12:28:49.602 INFO:teuthology.orchestra.run.trial110.stdout: libradospp-devel-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:28:49.602 INFO:teuthology.orchestra.run.trial110.stdout: librbd-devel-2:20.3.0-5145.g5fdb7d8f.el9.x86_64 2026-02-12T12:28:49.602 INFO:teuthology.orchestra.run.trial110.stdout: libtraceevent-1.8.4-2.el9.x86_64 2026-02-12T12:28:49.602 INFO:teuthology.orchestra.run.trial110.stdout: pdsh-2.34-7.el9.x86_64 2026-02-12T12:28:49.602 INFO:teuthology.orchestra.run.trial110.stdout: pdsh-rcmd-ssh-2.34-7.el9.x86_64 2026-02-12T12:28:49.602 INFO:teuthology.orchestra.run.trial110.stdout: perf-5.14.0-677.el9.x86_64 2026-02-12T12:28:49.603 INFO:teuthology.orchestra.run.trial110.stdout: 2026-02-12T12:28:49.603 INFO:teuthology.orchestra.run.trial110.stdout:Complete! 2026-02-12T12:28:49.632 DEBUG:teuthology.orchestra.run.trial110:> python3 -m pip install -r /home/ubuntu/cephtest/cbt/requirements.txt 2026-02-12T12:28:49.771 INFO:teuthology.orchestra.run.trial110.stdout:Defaulting to user installation because normal site-packages is not writeable 2026-02-12T12:28:49.781 INFO:teuthology.orchestra.run.trial110.stdout:Requirement already satisfied: pyyaml in /usr/lib64/python3.9/site-packages (from -r /home/ubuntu/cephtest/cbt/requirements.txt (line 1)) (5.4.1) 2026-02-12T12:28:49.781 INFO:teuthology.orchestra.run.trial110.stdout:Requirement already satisfied: lxml in /usr/lib64/python3.9/site-packages (from -r /home/ubuntu/cephtest/cbt/requirements.txt (line 2)) (4.6.5) 2026-02-12T12:28:50.106 INFO:teuthology.orchestra.run.trial110.stdout:Collecting matplotlib 2026-02-12T12:28:50.135 INFO:teuthology.orchestra.run.trial110.stdout: Downloading matplotlib-3.9.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (8.3 MB) 2026-02-12T12:28:50.520 INFO:teuthology.orchestra.run.trial110.stdout:Collecting fonttools>=4.22.0 2026-02-12T12:28:50.528 INFO:teuthology.orchestra.run.trial110.stdout: Downloading fonttools-4.60.2-cp39-cp39-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (4.8 MB) 2026-02-12T12:28:50.553 INFO:teuthology.orchestra.run.trial110.stdout:Requirement already satisfied: pyparsing>=2.3.1 in /usr/lib/python3.9/site-packages (from matplotlib->-r /home/ubuntu/cephtest/cbt/requirements.txt (line 3)) (2.4.7) 2026-02-12T12:28:50.553 INFO:teuthology.orchestra.run.trial110.stdout:Requirement already satisfied: packaging>=20.0 in /usr/lib/python3.9/site-packages (from matplotlib->-r /home/ubuntu/cephtest/cbt/requirements.txt (line 3)) (20.9) 2026-02-12T12:28:50.934 INFO:teuthology.orchestra.run.trial110.stdout:Collecting pillow>=8 2026-02-12T12:28:50.943 INFO:teuthology.orchestra.run.trial110.stdout: Downloading pillow-11.3.0-cp39-cp39-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (6.6 MB) 2026-02-12T12:28:50.975 INFO:teuthology.orchestra.run.trial110.stdout:Requirement already satisfied: python-dateutil>=2.7 in /usr/lib/python3.9/site-packages (from matplotlib->-r /home/ubuntu/cephtest/cbt/requirements.txt (line 3)) (2.9.0.post0) 2026-02-12T12:28:50.975 INFO:teuthology.orchestra.run.trial110.stdout:Requirement already satisfied: numpy>=1.23 in /usr/lib64/python3.9/site-packages (from matplotlib->-r /home/ubuntu/cephtest/cbt/requirements.txt (line 3)) (1.23.5) 2026-02-12T12:28:50.988 INFO:teuthology.orchestra.run.trial110.stdout:Collecting cycler>=0.10 2026-02-12T12:28:50.993 INFO:teuthology.orchestra.run.trial110.stdout: Downloading cycler-0.12.1-py3-none-any.whl (8.3 kB) 2026-02-12T12:28:51.134 INFO:teuthology.orchestra.run.trial110.stdout:Collecting contourpy>=1.0.1 2026-02-12T12:28:51.138 INFO:teuthology.orchestra.run.trial110.stdout: Downloading contourpy-1.3.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (321 kB) 2026-02-12T12:28:51.298 INFO:teuthology.orchestra.run.trial110.stdout:Collecting kiwisolver>=1.3.1 2026-02-12T12:28:51.303 INFO:teuthology.orchestra.run.trial110.stdout: Downloading kiwisolver-1.4.7-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.6 MB) 2026-02-12T12:28:51.348 INFO:teuthology.orchestra.run.trial110.stdout:Collecting importlib-resources>=3.2.0 2026-02-12T12:28:51.353 INFO:teuthology.orchestra.run.trial110.stdout: Downloading importlib_resources-6.5.2-py3-none-any.whl (37 kB) 2026-02-12T12:28:51.404 INFO:teuthology.orchestra.run.trial110.stdout:Collecting zipp>=3.1.0 2026-02-12T12:28:51.407 INFO:teuthology.orchestra.run.trial110.stdout: Downloading zipp-3.23.0-py3-none-any.whl (10 kB) 2026-02-12T12:28:51.418 INFO:teuthology.orchestra.run.trial110.stdout:Requirement already satisfied: six>=1.5 in /usr/lib/python3.9/site-packages (from python-dateutil>=2.7->matplotlib->-r /home/ubuntu/cephtest/cbt/requirements.txt (line 3)) (1.15.0) 2026-02-12T12:28:51.533 INFO:teuthology.orchestra.run.trial110.stdout:Installing collected packages: zipp, pillow, kiwisolver, importlib-resources, fonttools, cycler, contourpy, matplotlib 2026-02-12T12:28:52.598 INFO:teuthology.orchestra.run.trial110.stdout:Successfully installed contourpy-1.3.0 cycler-0.12.1 fonttools-4.60.2 importlib-resources-6.5.2 kiwisolver-1.4.7 matplotlib-3.9.4 pillow-11.3.0 zipp-3.23.0 2026-02-12T12:28:52.716 INFO:tasks.cbt:benchmark: radosbench 2026-02-12T12:28:52.716 DEBUG:teuthology.orchestra.run.trial110:> sudo /sbin/sysctl -q -w kernel.perf_event_paranoid=0 2026-02-12T12:28:52.742 DEBUG:teuthology.orchestra.run.trial110:> /home/ubuntu/cephtest/cbt/cbt.py -a /home/ubuntu/cephtest/archive/cbt /home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2026-02-12T12:28:52.854 INFO:teuthology.orchestra.run.trial110.stderr:12:28:52 - DEBUG - cbt - Settings.cluster: 2026-02-12T12:28:52.855 INFO:teuthology.orchestra.run.trial110.stderr: {'archive_dir': '/home/ubuntu/cephtest/archive/cbt', 2026-02-12T12:28:52.855 INFO:teuthology.orchestra.run.trial110.stderr: 'clients': ['ubuntu@trial110.front.sepia.ceph.com'], 2026-02-12T12:28:52.855 INFO:teuthology.orchestra.run.trial110.stderr: 'conf_file': '/etc/ceph/ceph.conf', 2026-02-12T12:28:52.855 INFO:teuthology.orchestra.run.trial110.stderr: 'head': 'ubuntu@trial110.front.sepia.ceph.com', 2026-02-12T12:28:52.855 INFO:teuthology.orchestra.run.trial110.stderr: 'is_teuthology': True, 2026-02-12T12:28:52.855 INFO:teuthology.orchestra.run.trial110.stderr: 'iterations': 1, 2026-02-12T12:28:52.855 INFO:teuthology.orchestra.run.trial110.stderr: 'mons': ['ubuntu@trial110.front.sepia.ceph.com'], 2026-02-12T12:28:52.855 INFO:teuthology.orchestra.run.trial110.stderr: 'osds': ['ubuntu@trial110.front.sepia.ceph.com'], 2026-02-12T12:28:52.855 INFO:teuthology.orchestra.run.trial110.stderr: 'osds_per_node': 3, 2026-02-12T12:28:52.855 INFO:teuthology.orchestra.run.trial110.stderr: 'pid_dir': '/var/run/ceph', 2026-02-12T12:28:52.855 INFO:teuthology.orchestra.run.trial110.stderr: 'pool_profiles': {'replicated': {'pg_size': 256, 2026-02-12T12:28:52.856 INFO:teuthology.orchestra.run.trial110.stderr: 'pgp_size': 256, 2026-02-12T12:28:52.856 INFO:teuthology.orchestra.run.trial110.stderr: 'replication': 'replicated'}}, 2026-02-12T12:28:52.856 INFO:teuthology.orchestra.run.trial110.stderr: 'rebuild_every_test': False, 2026-02-12T12:28:52.856 INFO:teuthology.orchestra.run.trial110.stderr: 'rgws': {'ubuntu@trial110.front.sepia.ceph.com': None}, 2026-02-12T12:28:52.856 INFO:teuthology.orchestra.run.trial110.stderr: 'tmp_dir': '/tmp/cbt', 2026-02-12T12:28:52.856 INFO:teuthology.orchestra.run.trial110.stderr: 'use_existing': True, 2026-02-12T12:28:52.856 INFO:teuthology.orchestra.run.trial110.stderr: 'user': 'ubuntu'} 2026-02-12T12:28:52.856 INFO:teuthology.orchestra.run.trial110.stderr:12:28:52 - INFO - cbt - Results dir: /home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c 2026-02-12T12:28:52.856 INFO:teuthology.orchestra.run.trial110.stderr:12:28:52 - INFO - cbt - Pausing for 60s for idle monitoring. 2026-02-12T12:28:52.857 INFO:teuthology.orchestra.run.trial110.stderr:12:28:52 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial110.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/idle_monitoring/collectl 2026-02-12T12:28:53.025 INFO:teuthology.orchestra.run.trial110.stderr:12:28:53 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial110.front.sepia.ceph.com collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/idle_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2026-02-12T12:28:53.027 INFO:teuthology.orchestra.run.trial110.stderr:12:28:53 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial110.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/idle_monitoring/perf 2026-02-12T12:28:53.193 INFO:teuthology.orchestra.run.trial110.stderr:12:28:53 - DEBUG - cbt - PerfMonitoring: remote_node 2026-02-12T12:28:53.194 INFO:teuthology.orchestra.run.trial110.stderr:12:28:53 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial110.front.sepia.ceph.com for pid in `cat /var/run/ceph/crimson-osd.*.pid`; do perf stat -p ${pid} -o /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/idle_monitoring/perf/perf_stat.${pid} & done 2026-02-12T12:29:53.254 INFO:teuthology.orchestra.run.trial110.stderr:12:29:53 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial110.front.sepia.ceph.com pkill -SIGINT -f collectl 2026-02-12T12:29:53.432 INFO:teuthology.orchestra.run.trial110.stderr:12:29:53 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial110.front.sepia.ceph.com sudo pkill -SIGINT -f perf\ 2026-02-12T12:29:53.621 INFO:teuthology.orchestra.run.trial110.stderr:12:29:53 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial110.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/idle_monitoring/perf/perf.data 2026-02-12T12:29:53.623 INFO:teuthology.orchestra.run.trial110.stderr:12:29:53 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial110.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/idle_monitoring/perf/perf_stat.* 2026-02-12T12:29:53.626 INFO:teuthology.orchestra.run.trial110.stderr:12:29:53 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial110.front.sepia.ceph.com sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/* 2026-02-12T12:29:53.803 INFO:teuthology.orchestra.run.trial110.stderr:12:29:53 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=rpdcp -f 10 -R ssh -w ubuntu@trial110.front.sepia.ceph.com -r /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/* /home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c 2026-02-12T12:29:53.970 INFO:teuthology.orchestra.run.trial110.stderr:12:29:53 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial110.front.sepia.ceph.com sudo killall -9 rados 2026-02-12T12:29:54.145 INFO:teuthology.orchestra.run.trial110.stderr:12:29:54 - INFO - cbt - Results dir: /home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c 2026-02-12T12:29:54.145 INFO:teuthology.orchestra.run.trial110.stderr:12:29:54 - INFO - cbt - Skipping existing test in /home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c. 2026-02-12T12:29:54.145 INFO:teuthology.orchestra.run.trial110.stderr:12:29:54 - INFO - cbt - Running benchmark /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004 2026-02-12T12:29:54.146 INFO:teuthology.orchestra.run.trial110.stderr:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c 2026-02-12T12:29:54.146 INFO:teuthology.orchestra.run.trial110.stderr:{'concurrent_ops': 4, 'concurrent_procs': 2, 'pool_profile': 'replicated', 'readmode': 'rand', 'time': 60, 'write_only': False, 'op_size': 4194304, 'pool_monitoring_list': 'collectl', 'run_monitoring_list': 'collectl', 'benchmark': 'radosbench', 'iteration': 0} == iteration 0 == 2026-02-12T12:29:54.146 INFO:teuthology.orchestra.run.trial110.stderr:12:29:54 - INFO - cbt - Setting OSD Read Ahead to: 0 2026-02-12T12:29:54.146 INFO:teuthology.orchestra.run.trial110.stderr:12:29:54 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial110.front.sepia.ceph.com find /dev/disk/by-partlabel/osd-device-*data -exec readlink {} \; | cut -d"/" -f 3 | sed "s/[0-9]$//" | xargs -I{} sudo sh -c "echo 0 > /sys/block/'{}'/queue/read_ahead_kb" 2026-02-12T12:29:54.146 INFO:teuthology.orchestra.run.trial110.stderr:12:29:54 - DEBUG - cbt - Cleaning existing temporary run directory: /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004 2026-02-12T12:29:54.147 INFO:teuthology.orchestra.run.trial110.stderr:12:29:54 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial110.front.sepia.ceph.com sudo rm -rf /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004 2026-02-12T12:29:54.325 INFO:teuthology.orchestra.run.trial110.stderr:12:29:54 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial110.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/pool_monitoring/collectl 2026-02-12T12:29:54.490 INFO:teuthology.orchestra.run.trial110.stderr:12:29:54 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial110.front.sepia.ceph.com collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/pool_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2026-02-12T12:29:54.491 INFO:teuthology.orchestra.run.trial110.stderr:12:29:54 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial110.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/pool_monitoring/perf 2026-02-12T12:29:54.657 INFO:teuthology.orchestra.run.trial110.stderr:12:29:54 - DEBUG - cbt - PerfMonitoring: remote_node 2026-02-12T12:29:54.657 INFO:teuthology.orchestra.run.trial110.stderr:12:29:54 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial110.front.sepia.ceph.com for pid in `cat /var/run/ceph/crimson-osd.*.pid`; do perf stat -p ${pid} -o /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/pool_monitoring/perf/perf_stat.${pid} & done 2026-02-12T12:29:54.659 INFO:teuthology.orchestra.run.trial110.stderr:12:29:54 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial110.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool delete rados-bench-cbt rados-bench-cbt --yes-i-really-really-mean-it 2026-02-12T12:29:55.006 INFO:teuthology.orchestra.run.trial110.stderr:12:29:55 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial110.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool create rados-bench-cbt 256 256 2026-02-12T12:29:55.435 INFO:teuthology.orchestra.run.trial110.stderr:12:29:55 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial110.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool application enable rados-bench-cbt radosbench 2026-02-12T12:29:56.439 INFO:teuthology.orchestra.run.trial110.stderr:12:29:56 - INFO - cbt - Checking Health after pool creation. 2026-02-12T12:30:01.445 INFO:teuthology.orchestra.run.trial110.stderr:12:30:01 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial110.front.sepia.ceph.com /usr/bin/ceph -c /etc/ceph/ceph.conf health 2026-02-12T12:30:01.796 INFO:teuthology.orchestra.run.trial110.stderr:12:30:01 - INFO - cbt - Final Pool Health Check. 2026-02-12T12:30:06.802 INFO:teuthology.orchestra.run.trial110.stderr:12:30:06 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial110.front.sepia.ceph.com /usr/bin/ceph -c /etc/ceph/ceph.conf health 2026-02-12T12:30:07.159 INFO:teuthology.orchestra.run.trial110.stderr:12:30:07 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial110.front.sepia.ceph.com pkill -SIGINT -f collectl 2026-02-12T12:30:07.333 INFO:teuthology.orchestra.run.trial110.stderr:12:30:07 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial110.front.sepia.ceph.com sudo pkill -SIGINT -f perf\ 2026-02-12T12:30:07.510 INFO:teuthology.orchestra.run.trial110.stderr:12:30:07 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial110.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/pool_monitoring/perf/perf.data 2026-02-12T12:30:07.512 INFO:teuthology.orchestra.run.trial110.stderr:12:30:07 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial110.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/pool_monitoring/perf/perf_stat.* 2026-02-12T12:30:07.514 INFO:teuthology.orchestra.run.trial110.stderr:12:30:07 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial110.front.sepia.ceph.com sync 2026-02-12T12:30:07.732 INFO:teuthology.orchestra.run.trial110.stderr:12:30:07 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial110.front.sepia.ceph.com echo 3 | sudo tee /proc/sys/vm/drop_caches 2026-02-12T12:30:08.724 INFO:teuthology.orchestra.run.trial110.stderr:12:30:08 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial110.front.sepia.ceph.com /usr/bin/rados -c /etc/ceph/ceph.conf -v 2026-02-12T12:30:08.933 INFO:teuthology.orchestra.run.trial110.stderr:12:30:08 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial110.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/write 2026-02-12T12:30:09.099 INFO:teuthology.orchestra.run.trial110.stderr:12:30:09 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial110.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf daemon osd.0 config show > /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/write/ceph_settings.out 2026-02-12T12:30:09.382 INFO:teuthology.orchestra.run.trial110.stderr:12:30:09 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial110.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/write/collectl 2026-02-12T12:30:09.546 INFO:teuthology.orchestra.run.trial110.stderr:12:30:09 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial110.front.sepia.ceph.com collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/write/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2026-02-12T12:30:09.547 INFO:teuthology.orchestra.run.trial110.stderr:12:30:09 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial110.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/write/perf 2026-02-12T12:30:09.710 INFO:teuthology.orchestra.run.trial110.stderr:12:30:09 - DEBUG - cbt - PerfMonitoring: remote_node 2026-02-12T12:30:09.710 INFO:teuthology.orchestra.run.trial110.stderr:12:30:09 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial110.front.sepia.ceph.com for pid in `cat /var/run/ceph/crimson-osd.*.pid`; do perf stat -p ${pid} -o /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/write/perf/perf_stat.${pid} & done 2026-02-12T12:30:09.712 INFO:teuthology.orchestra.run.trial110.stderr:12:30:09 - INFO - cbt - Running radosbench write test. 2026-02-12T12:30:09.712 INFO:teuthology.orchestra.run.trial110.stderr:12:30:09 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial110.front.sepia.ceph.com /usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench -b 4194304 60 write --concurrent-ios 4 --run-name `hostname -f`-0 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/write/objecter.0.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/write/output.0 2026-02-12T12:30:09.713 INFO:teuthology.orchestra.run.trial110.stderr:12:30:09 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial110.front.sepia.ceph.com /usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench -b 4194304 60 write --concurrent-ios 4 --run-name `hostname -f`-1 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/write/objecter.1.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/write/output.1 2026-02-12T12:31:10.940 INFO:teuthology.orchestra.run.trial110.stderr:12:31:10 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial110.front.sepia.ceph.com pkill -SIGINT -f collectl 2026-02-12T12:31:11.118 INFO:teuthology.orchestra.run.trial110.stderr:12:31:11 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial110.front.sepia.ceph.com sudo pkill -SIGINT -f perf\ 2026-02-12T12:31:11.296 INFO:teuthology.orchestra.run.trial110.stderr:12:31:11 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial110.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/write/perf/perf.data 2026-02-12T12:31:11.298 INFO:teuthology.orchestra.run.trial110.stderr:12:31:11 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial110.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/write/perf/perf_stat.* 2026-02-12T12:31:11.300 INFO:teuthology.orchestra.run.trial110.stderr:12:31:11 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial110.front.sepia.ceph.com find /var/run/ceph/ceph-osd*.asok -maxdepth 1 -exec sudo /usr/bin/ceph --admin-daemon {} dump_historic_ops \; > /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/write/historic_ops.out 2026-02-12T12:31:11.669 INFO:teuthology.orchestra.run.trial110.stderr:12:31:11 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial110.front.sepia.ceph.com sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/write/* 2026-02-12T12:31:11.848 INFO:teuthology.orchestra.run.trial110.stderr:12:31:11 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=rpdcp -f 10 -R ssh -w ubuntu@trial110.front.sepia.ceph.com -r /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/write/* /home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/write 2026-02-12T12:31:12.014 INFO:teuthology.orchestra.run.trial110.stderr:12:31:12 - INFO - cbt - Convert results to json format. 2026-02-12T12:31:12.015 INFO:teuthology.orchestra.run.trial110.stderr:12:31:12 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial110.front.sepia.ceph.com sync 2026-02-12T12:31:12.184 INFO:teuthology.orchestra.run.trial110.stderr:12:31:12 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial110.front.sepia.ceph.com echo 3 | sudo tee /proc/sys/vm/drop_caches 2026-02-12T12:31:12.494 INFO:teuthology.orchestra.run.trial110.stderr:12:31:12 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial110.front.sepia.ceph.com /usr/bin/rados -c /etc/ceph/ceph.conf -v 2026-02-12T12:31:12.703 INFO:teuthology.orchestra.run.trial110.stderr:12:31:12 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial110.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/rand 2026-02-12T12:31:12.879 INFO:teuthology.orchestra.run.trial110.stderr:12:31:12 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial110.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf daemon osd.0 config show > /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/rand/ceph_settings.out 2026-02-12T12:31:13.159 INFO:teuthology.orchestra.run.trial110.stderr:12:31:13 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial110.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/rand/collectl 2026-02-12T12:31:13.329 INFO:teuthology.orchestra.run.trial110.stderr:12:31:13 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial110.front.sepia.ceph.com collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/rand/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2026-02-12T12:31:13.331 INFO:teuthology.orchestra.run.trial110.stderr:12:31:13 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial110.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/rand/perf 2026-02-12T12:31:13.506 INFO:teuthology.orchestra.run.trial110.stderr:12:31:13 - DEBUG - cbt - PerfMonitoring: remote_node 2026-02-12T12:31:13.506 INFO:teuthology.orchestra.run.trial110.stderr:12:31:13 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial110.front.sepia.ceph.com for pid in `cat /var/run/ceph/crimson-osd.*.pid`; do perf stat -p ${pid} -o /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/rand/perf/perf_stat.${pid} & done 2026-02-12T12:31:13.507 INFO:teuthology.orchestra.run.trial110.stderr:12:31:13 - INFO - cbt - Running radosbench rand test. 2026-02-12T12:31:13.507 INFO:teuthology.orchestra.run.trial110.stderr:12:31:13 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial110.front.sepia.ceph.com /usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench 60 rand --concurrent-ios 4 --run-name `hostname -f`-0 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/rand/objecter.0.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/rand/output.0 2026-02-12T12:31:13.509 INFO:teuthology.orchestra.run.trial110.stderr:12:31:13 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial110.front.sepia.ceph.com /usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench 60 rand --concurrent-ios 4 --run-name `hostname -f`-1 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/rand/objecter.1.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/rand/output.1 2026-02-12T12:32:14.740 INFO:teuthology.orchestra.run.trial110.stderr:12:32:14 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial110.front.sepia.ceph.com pkill -SIGINT -f collectl 2026-02-12T12:32:14.926 INFO:teuthology.orchestra.run.trial110.stderr:12:32:14 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial110.front.sepia.ceph.com sudo pkill -SIGINT -f perf\ 2026-02-12T12:32:15.104 INFO:teuthology.orchestra.run.trial110.stderr:12:32:15 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial110.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/rand/perf/perf.data 2026-02-12T12:32:15.106 INFO:teuthology.orchestra.run.trial110.stderr:12:32:15 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial110.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/rand/perf/perf_stat.* 2026-02-12T12:32:15.108 INFO:teuthology.orchestra.run.trial110.stderr:12:32:15 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial110.front.sepia.ceph.com find /var/run/ceph/ceph-osd*.asok -maxdepth 1 -exec sudo /usr/bin/ceph --admin-daemon {} dump_historic_ops \; > /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/rand/historic_ops.out 2026-02-12T12:32:15.466 INFO:teuthology.orchestra.run.trial110.stderr:12:32:15 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial110.front.sepia.ceph.com sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/rand/* 2026-02-12T12:32:15.640 INFO:teuthology.orchestra.run.trial110.stderr:12:32:15 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=rpdcp -f 10 -R ssh -w ubuntu@trial110.front.sepia.ceph.com -r /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-04194304/concurrent_ops-00000004/rand/* /home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/rand 2026-02-12T12:32:15.806 INFO:teuthology.orchestra.run.trial110.stderr:12:32:15 - INFO - cbt - Convert results to json format. 2026-02-12T12:32:15.818 DEBUG:teuthology.run_tasks:Unwinding manager cbt 2026-02-12T12:32:15.833 DEBUG:teuthology.orchestra.run.trial110:> rm --one-file-system -rf -- /home/ubuntu/cephtest/cbt 2026-02-12T12:32:15.861 INFO:tasks.cbt_performance:Collecting CBT performance data config 2026-02-12T12:32:15.861 INFO:tasks.cbt_performance:reading cbt preferences 2026-02-12T12:32:15.861 DEBUG:teuthology.orchestra.run.trial110:> find /home/ubuntu/cephtest -name 'json_output.*' 2026-02-12T12:32:15.919 INFO:teuthology.orchestra.run.trial110.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/write/json_output.1.trial110.front.sepia.ceph.com 2026-02-12T12:32:15.919 INFO:teuthology.orchestra.run.trial110.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/write/json_output.0.trial110.front.sepia.ceph.com 2026-02-12T12:32:15.920 INFO:teuthology.orchestra.run.trial110.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/rand/json_output.1.trial110.front.sepia.ceph.com 2026-02-12T12:32:15.920 INFO:teuthology.orchestra.run.trial110.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/rand/json_output.0.trial110.front.sepia.ceph.com 2026-02-12T12:32:15.921 DEBUG:teuthology.orchestra.run.trial110:> set -ex 2026-02-12T12:32:15.921 DEBUG:teuthology.orchestra.run.trial110:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/write/json_output.1.trial110.front.sepia.ceph.com of=/dev/stdout 2026-02-12T12:32:15.977 INFO:tasks.cbt_performance:reading total cpu cycles 2026-02-12T12:32:15.978 DEBUG:teuthology.orchestra.run.trial110:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/write -name 'perf_stat.*' 2026-02-12T12:32:16.034 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2026-02-12T12:32:16.034 INFO:tasks.cbt_performance:total cpu cycles: 0 2026-02-12T12:32:16.034 DEBUG:teuthology.orchestra.run.trial110:> set -ex 2026-02-12T12:32:16.035 DEBUG:teuthology.orchestra.run.trial110:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/write/json_output.0.trial110.front.sepia.ceph.com of=/dev/stdout 2026-02-12T12:32:16.090 INFO:tasks.cbt_performance:reading total cpu cycles 2026-02-12T12:32:16.090 DEBUG:teuthology.orchestra.run.trial110:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/write -name 'perf_stat.*' 2026-02-12T12:32:16.146 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2026-02-12T12:32:16.146 INFO:tasks.cbt_performance:total cpu cycles: 0 2026-02-12T12:32:16.146 DEBUG:teuthology.orchestra.run.trial110:> set -ex 2026-02-12T12:32:16.146 DEBUG:teuthology.orchestra.run.trial110:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/rand/json_output.1.trial110.front.sepia.ceph.com of=/dev/stdout 2026-02-12T12:32:16.202 INFO:tasks.cbt_performance:reading total cpu cycles 2026-02-12T12:32:16.202 DEBUG:teuthology.orchestra.run.trial110:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/rand -name 'perf_stat.*' 2026-02-12T12:32:16.258 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2026-02-12T12:32:16.258 INFO:tasks.cbt_performance:total cpu cycles: 0 2026-02-12T12:32:16.259 DEBUG:teuthology.orchestra.run.trial110:> set -ex 2026-02-12T12:32:16.259 DEBUG:teuthology.orchestra.run.trial110:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/rand/json_output.0.trial110.front.sepia.ceph.com of=/dev/stdout 2026-02-12T12:32:16.315 INFO:tasks.cbt_performance:reading total cpu cycles 2026-02-12T12:32:16.316 DEBUG:teuthology.orchestra.run.trial110:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/rand -name 'perf_stat.*' 2026-02-12T12:32:16.372 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2026-02-12T12:32:16.373 INFO:tasks.cbt_performance:total cpu cycles: 0 2026-02-12T12:32:16.377 WARNING:tasks.cbt_performance:Could not send performance data to http://mira118.front.sepia.ceph.com:4000/cbt_performance: HTTPConnectionPool(host='mira118.front.sepia.ceph.com', port=4000): Max retries exceeded with url: /cbt_performance (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -2] Name or service not known')) 2026-02-12T12:32:16.377 WARNING:tasks.cbt_performance:Continuing without performance data collection 2026-02-12T12:32:16.378 DEBUG:teuthology.run_tasks:Unwinding manager ssh_keys 2026-02-12T12:32:16.393 INFO:teuthology.task.ssh_keys:Cleaning up SSH keys 2026-02-12T12:32:16.393 INFO:teuthology.task.ssh_keys:cleaning up keys added for testing 2026-02-12T12:32:16.393 INFO:teuthology.task.ssh_keys: cleaning up keys for user ubuntu on trial110.front.sepia.ceph.com 2026-02-12T12:32:16.393 DEBUG:teuthology.orchestra.run.trial110:> rm -- /home/ubuntu/.ssh/id_rsa 2026-02-12T12:32:16.429 DEBUG:teuthology.orchestra.run.trial110:> rm -- /home/ubuntu/.ssh/id_rsa.pub 2026-02-12T12:32:16.485 DEBUG:teuthology.orchestra.run.trial110:> stat -c '"%a"' /home/ubuntu/.ssh/authorized_keys 2026-02-12T12:32:16.541 INFO:teuthology.orchestra.run.trial110.stdout:"600" 2026-02-12T12:32:16.542 DEBUG:teuthology.orchestra.run.trial110:> mv -- /home/ubuntu/.ssh/authorized_keys_2026-02-12_12:28:45:783234 /home/ubuntu/.ssh/authorized_keys 2026-02-12T12:32:16.599 DEBUG:teuthology.orchestra.run.trial110:> chmod 600 /home/ubuntu/.ssh/authorized_keys 2026-02-12T12:32:16.655 DEBUG:teuthology.orchestra.run.trial110:> rm /home/ubuntu/.ssh/config 2026-02-12T12:32:16.711 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2026-02-12T12:32:16.726 DEBUG:teuthology.orchestra.run.trial110:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2026-02-12T12:32:16.826 INFO:teuthology.orchestra.run.trial110.stderr:2026-02-12T12:32:16.820+0000 7fe742e51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:32:16.827 INFO:teuthology.orchestra.run.trial110.stderr:2026-02-12T12:32:16.825+0000 7fe742e51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-12T12:32:16.949 INFO:teuthology.misc:Shutting down mds daemons... 2026-02-12T12:32:16.949 INFO:teuthology.misc:Shutting down osd daemons... 2026-02-12T12:32:16.949 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2026-02-12T12:32:16.950 INFO:teuthology.orchestra.run:waiting for 300 2026-02-12T12:32:17.023 INFO:tasks.ceph.osd.0:Stopped 2026-02-12T12:32:17.023 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2026-02-12T12:32:17.023 INFO:teuthology.orchestra.run:waiting for 300 2026-02-12T12:32:17.109 INFO:tasks.ceph.osd.1:Stopped 2026-02-12T12:32:17.109 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2026-02-12T12:32:17.109 INFO:teuthology.orchestra.run:waiting for 300 2026-02-12T12:32:17.193 INFO:tasks.ceph.osd.2:Stopped 2026-02-12T12:32:17.194 INFO:teuthology.misc:Shutting down mgr daemons... 2026-02-12T12:32:17.194 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2026-02-12T12:32:17.194 INFO:teuthology.orchestra.run:waiting for 300 2026-02-12T12:32:17.232 INFO:tasks.ceph.mgr.x:Stopped 2026-02-12T12:32:17.233 INFO:teuthology.misc:Shutting down mon daemons... 2026-02-12T12:32:17.233 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2026-02-12T12:32:17.233 INFO:teuthology.orchestra.run:waiting for 300 2026-02-12T12:32:17.282 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2026-02-12T12:32:17.282 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2026-02-12T12:32:17.282 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2026-02-12T12:32:17.282 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2026-02-12T12:32:17.283 INFO:tasks.ceph.mon.a:Stopped 2026-02-12T12:32:17.283 INFO:tasks.ceph:Checking cluster log for badness... 2026-02-12T12:32:17.283 DEBUG:teuthology.orchestra.run.trial110:> sudo grep -E '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | grep -E -v '\(MDS_ALL_DOWN\)' | grep -E -v '\(MDS_UP_LESS_THAN_MAX\)' | grep -E -v '\(POOL_APP_NOT_ENABLED\)' | grep -E -v '\(PG_' | grep -E -v '\(OSD_' | grep -E -v '\(OBJECT_' | grep -E -v 'overall HEALTH' | head -n 1 2026-02-12T12:32:17.348 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@trial110.front.sepia.ceph.com 2026-02-12T12:32:17.348 DEBUG:teuthology.orchestra.run.trial110:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2026-02-12T12:32:17.426 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@trial110.front.sepia.ceph.com 2026-02-12T12:32:17.427 DEBUG:teuthology.orchestra.run.trial110:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2026-02-12T12:32:17.500 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@trial110.front.sepia.ceph.com 2026-02-12T12:32:17.500 DEBUG:teuthology.orchestra.run.trial110:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2026-02-12T12:32:17.581 INFO:tasks.ceph:Archiving mon data... 2026-02-12T12:32:17.582 DEBUG:teuthology.misc:Transferring archived files from trial110:/var/lib/ceph/mon/ceph-a to /home/teuthworker/mnt/teuthology/moagrawa-2026-02-10_06:35:43-crimson-rados-main-distro-debug-trial/42493/data/mon.a.tgz 2026-02-12T12:32:17.582 DEBUG:teuthology.orchestra.run.trial110:> mktemp 2026-02-12T12:32:17.637 INFO:teuthology.orchestra.run.trial110.stdout:/tmp/tmp.eoP6siSpqu 2026-02-12T12:32:17.638 DEBUG:teuthology.orchestra.run.trial110:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.eoP6siSpqu 2026-02-12T12:32:17.727 DEBUG:teuthology.orchestra.run.trial110:> sudo chmod 0666 /tmp/tmp.eoP6siSpqu 2026-02-12T12:32:17.809 DEBUG:teuthology.orchestra.remote:trial110:/tmp/tmp.eoP6siSpqu is 209KB 2026-02-12T12:32:17.902 DEBUG:teuthology.orchestra.run.trial110:> rm -fr /tmp/tmp.eoP6siSpqu 2026-02-12T12:32:17.919 INFO:tasks.ceph:Cleaning ceph cluster... 2026-02-12T12:32:17.919 DEBUG:teuthology.orchestra.run.trial110:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2026-02-12T12:32:18.041 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2026-02-12T12:32:18.041 INFO:tasks.ceph:Archiving crash dumps... 2026-02-12T12:32:18.043 DEBUG:teuthology.misc:Transferring archived files from trial110:/var/lib/ceph/crash to /home/teuthworker/mnt/teuthology/moagrawa-2026-02-10_06:35:43-crimson-rados-main-distro-debug-trial/42493/remote/trial110/crash 2026-02-12T12:32:18.048 DEBUG:teuthology.orchestra.run.trial110:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2026-02-12T12:32:18.105 INFO:tasks.ceph:Compressing logs... 2026-02-12T12:32:18.105 DEBUG:teuthology.orchestra.run.trial110:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2026-02-12T12:32:18.169 INFO:teuthology.orchestra.run.trial110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2026-02-12T12:32:18.183 INFO:teuthology.orchestra.run.trial110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28216.log 2026-02-12T12:32:18.184 INFO:teuthology.orchestra.run.trial110.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23464.log 2026-02-12T12:32:18.184 INFO:teuthology.orchestra.run.trial110.stderr:/var/log/ceph/ceph-client.admin.28216.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2026-02-12T12:32:18.184 INFO:teuthology.orchestra.run.trial110.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.28216.log.gz 2026-02-12T12:32:18.184 INFO:teuthology.orchestra.run.trial110.stderr:/var/log/ceph/ceph-client.admin.23464.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22664.log 2026-02-12T12:32:18.184 INFO:teuthology.orchestra.run.trial110.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.23464.log.gz 2026-02-12T12:32:18.184 INFO:teuthology.orchestra.run.trial110.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2026-02-12T12:32:18.184 INFO:teuthology.orchestra.run.trial110.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.22664.log: -- /var/log/ceph/ceph-client.admin.22977.log 2026-02-12T12:32:18.184 INFO:teuthology.orchestra.run.trial110.stderr: 66.9% -- replaced with /var/log/ceph/ceph-client.admin.22664.log.gz 2026-02-12T12:32:18.184 INFO:teuthology.orchestra.run.trial110.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22827.log 2026-02-12T12:32:18.185 INFO:teuthology.orchestra.run.trial110.stderr:/var/log/ceph/ceph-client.admin.22977.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22978.log 2026-02-12T12:32:18.185 INFO:teuthology.orchestra.run.trial110.stderr: 91.2% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2026-02-12T12:32:18.185 INFO:teuthology.orchestra.run.trial110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22059.log 2026-02-12T12:32:18.185 INFO:teuthology.orchestra.run.trial110.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.22977.log.gz 2026-02-12T12:32:18.185 INFO:teuthology.orchestra.run.trial110.stderr:/var/log/ceph/ceph-client.admin.22827.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22569.log 2026-02-12T12:32:18.185 INFO:teuthology.orchestra.run.trial110.stderr:/var/log/ceph/ceph-client.admin.22978.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.22827.log.gz 2026-02-12T12:32:18.185 INFO:teuthology.orchestra.run.trial110.stderr:/var/log/ceph/ceph-client.admin.22059.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.22978.log.gz 2026-02-12T12:32:18.185 INFO:teuthology.orchestra.run.trial110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23415.log 2026-02-12T12:32:18.185 INFO:teuthology.orchestra.run.trial110.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.22059.log.gz 2026-02-12T12:32:18.185 INFO:teuthology.orchestra.run.trial110.stderr:gzip -5/var/log/ceph/ceph-client.admin.22569.log: --verbose -- /var/log/ceph/ceph-client.admin.22355.log 2026-02-12T12:32:18.185 INFO:teuthology.orchestra.run.trial110.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.22569.log.gz 2026-02-12T12:32:18.185 INFO:teuthology.orchestra.run.trial110.stderr:/var/log/ceph/ceph-client.admin.23415.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26248.log 2026-02-12T12:32:18.186 INFO:teuthology.orchestra.run.trial110.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.23415.log.gz 2026-02-12T12:32:18.186 INFO:teuthology.orchestra.run.trial110.stderr:/var/log/ceph/ceph-client.admin.22355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22876.log 2026-02-12T12:32:18.186 INFO:teuthology.orchestra.run.trial110.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.22355.log.gz 2026-02-12T12:32:18.186 INFO:teuthology.orchestra.run.trial110.stderr:/var/log/ceph/ceph-client.admin.26248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22325.log 2026-02-12T12:32:18.186 INFO:teuthology.orchestra.run.trial110.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.26248.log.gz 2026-02-12T12:32:18.186 INFO:teuthology.orchestra.run.trial110.stderr:/var/log/ceph/ceph-client.admin.22876.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27536.log 2026-02-12T12:32:18.186 INFO:teuthology.orchestra.run.trial110.stderr:/var/log/ceph/ceph-client.admin.22325.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.22876.log.gz 2026-02-12T12:32:18.186 INFO:teuthology.orchestra.run.trial110.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.22325.log.gz 2026-02-12T12:32:18.186 INFO:teuthology.orchestra.run.trial110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26211.log 2026-02-12T12:32:18.186 INFO:teuthology.orchestra.run.trial110.stderr:/var/log/ceph/ceph-client.admin.27536.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26097.log 2026-02-12T12:32:18.186 INFO:teuthology.orchestra.run.trial110.stderr: 65.6% -- replaced with /var/log/ceph/ceph-client.admin.27536.log.gz 2026-02-12T12:32:18.186 INFO:teuthology.orchestra.run.trial110.stderr:/var/log/ceph/ceph-client.admin.26211.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23126.log 2026-02-12T12:32:18.186 INFO:teuthology.orchestra.run.trial110.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.26211.log.gz 2026-02-12T12:32:18.187 INFO:teuthology.orchestra.run.trial110.stderr:/var/log/ceph/ceph-client.admin.26097.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22729.log 2026-02-12T12:32:18.187 INFO:teuthology.orchestra.run.trial110.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.26097.log.gz 2026-02-12T12:32:18.187 INFO:teuthology.orchestra.run.trial110.stderr:/var/log/ceph/ceph-client.admin.23126.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21828.log 2026-02-12T12:32:18.187 INFO:teuthology.orchestra.run.trial110.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.23126.log.gz 2026-02-12T12:32:18.187 INFO:teuthology.orchestra.run.trial110.stderr:/var/log/ceph/ceph-client.admin.22729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23274.log 2026-02-12T12:32:18.187 INFO:teuthology.orchestra.run.trial110.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.22729.log.gz 2026-02-12T12:32:18.187 INFO:teuthology.orchestra.run.trial110.stderr:/var/log/ceph/ceph-client.admin.21828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22979.log 2026-02-12T12:32:18.187 INFO:teuthology.orchestra.run.trial110.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.21828.log.gz 2026-02-12T12:32:18.187 INFO:teuthology.orchestra.run.trial110.stderr:/var/log/ceph/ceph-client.admin.23274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22502.log 2026-02-12T12:32:18.187 INFO:teuthology.orchestra.run.trial110.stderr:/var/log/ceph/ceph-client.admin.22979.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.23274.log.gz 2026-02-12T12:32:18.187 INFO:teuthology.orchestra.run.trial110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22639.log 2026-02-12T12:32:18.187 INFO:teuthology.orchestra.run.trial110.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.22979.log.gz 2026-02-12T12:32:18.188 INFO:teuthology.orchestra.run.trial110.stderr:/var/log/ceph/ceph-client.admin.22502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23273.log 2026-02-12T12:32:18.188 INFO:teuthology.orchestra.run.trial110.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.22502.log.gz 2026-02-12T12:32:18.188 INFO:teuthology.orchestra.run.trial110.stderr:/var/log/ceph/ceph-client.admin.22639.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23219.log 2026-02-12T12:32:18.188 INFO:teuthology.orchestra.run.trial110.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.22639.log.gz 2026-02-12T12:32:18.188 INFO:teuthology.orchestra.run.trial110.stderr:/var/log/ceph/ceph-client.admin.23273.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21875.log 2026-02-12T12:32:18.188 INFO:teuthology.orchestra.run.trial110.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.23273.log.gz 2026-02-12T12:32:18.188 INFO:teuthology.orchestra.run.trial110.stderr:/var/log/ceph/ceph-client.admin.23219.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26860.log 2026-02-12T12:32:18.188 INFO:teuthology.orchestra.run.trial110.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.23219.log.gz 2026-02-12T12:32:18.188 INFO:teuthology.orchestra.run.trial110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26302.log 2026-02-12T12:32:18.188 INFO:teuthology.orchestra.run.trial110.stderr:/var/log/ceph/ceph-client.admin.21875.log: /var/log/ceph/ceph-client.admin.26860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26858.log 2026-02-12T12:32:18.188 INFO:teuthology.orchestra.run.trial110.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.21875.log.gz 2026-02-12T12:32:18.188 INFO:teuthology.orchestra.run.trial110.stderr: 66.9% -- replaced with /var/log/ceph/ceph-client.admin.26860.log.gz 2026-02-12T12:32:18.188 INFO:teuthology.orchestra.run.trial110.stderr:/var/log/ceph/ceph-client.admin.26302.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22012.log 2026-02-12T12:32:18.188 INFO:teuthology.orchestra.run.trial110.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.26302.log.gz 2026-02-12T12:32:18.189 INFO:teuthology.orchestra.run.trial110.stderr:/var/log/ceph/ceph-client.admin.26858.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2026-02-12T12:32:18.189 INFO:teuthology.orchestra.run.trial110.stderr: 66.7% -- replaced with /var/log/ceph/ceph-client.admin.26858.log.gz 2026-02-12T12:32:18.189 INFO:teuthology.orchestra.run.trial110.stderr:/var/log/ceph/ceph-client.admin.22012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23124.log 2026-02-12T12:32:18.189 INFO:teuthology.orchestra.run.trial110.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.22012.log.gz 2026-02-12T12:32:18.189 INFO:teuthology.orchestra.run.trial110.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2026-02-12T12:32:18.189 INFO:teuthology.orchestra.run.trial110.stderr:/var/log/ceph/ceph-client.admin.23124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23366.log 2026-02-12T12:32:18.189 INFO:teuthology.orchestra.run.trial110.stderr:/var/log/ceph/ceph-osd.1.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.23124.log.gz 2026-02-12T12:32:18.189 INFO:teuthology.orchestra.run.trial110.stderr: 87.7% -- replaced with /var/log/ceph/ceph.audit.log.gz 2026-02-12T12:32:18.189 INFO:teuthology.orchestra.run.trial110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26153.log 2026-02-12T12:32:18.189 INFO:teuthology.orchestra.run.trial110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22778.log 2026-02-12T12:32:18.189 INFO:teuthology.orchestra.run.trial110.stderr:/var/log/ceph/ceph-client.admin.23366.log: /var/log/ceph/ceph-client.admin.26153.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2026-02-12T12:32:18.189 INFO:teuthology.orchestra.run.trial110.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.23366.log.gz 2026-02-12T12:32:18.190 INFO:teuthology.orchestra.run.trial110.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.26153.log.gz 2026-02-12T12:32:18.190 INFO:teuthology.orchestra.run.trial110.stderr:/var/log/ceph/ceph-client.admin.22778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22106.log 2026-02-12T12:32:18.190 INFO:teuthology.orchestra.run.trial110.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.22778.log.gz 2026-02-12T12:32:18.190 INFO:teuthology.orchestra.run.trial110.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21733.log 2026-02-12T12:32:18.190 INFO:teuthology.orchestra.run.trial110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27539.log 2026-02-12T12:32:18.190 INFO:teuthology.orchestra.run.trial110.stderr:/var/log/ceph/ceph-client.admin.22106.log: /var/log/ceph/ceph-client.admin.21733.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.22106.log.gzgzip 2026-02-12T12:32:18.190 INFO:teuthology.orchestra.run.trial110.stderr: -5 --verbose -- /var/log/ceph/ceph.log 2026-02-12T12:32:18.190 INFO:teuthology.orchestra.run.trial110.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.21733.log.gz 2026-02-12T12:32:18.190 INFO:teuthology.orchestra.run.trial110.stderr:/var/log/ceph/ceph-client.admin.27539.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.20442.log 2026-02-12T12:32:18.190 INFO:teuthology.orchestra.run.trial110.stderr: 66.7% -- replaced with /var/log/ceph/ceph-client.admin.27539.log.gz 2026-02-12T12:32:18.190 INFO:teuthology.orchestra.run.trial110.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23125.log 2026-02-12T12:32:18.190 INFO:teuthology.orchestra.run.trial110.stderr:/var/log/ceph/ceph.tmp-client.admin.20442.log: 85.5% -- replaced with /var/log/ceph/ceph.log.gz 2026-02-12T12:32:18.190 INFO:teuthology.orchestra.run.trial110.stderr: 67.7% -- replaced with /var/log/ceph/ceph.tmp-client.admin.20442.log.gz 2026-02-12T12:32:18.190 INFO:teuthology.orchestra.run.trial110.stderr:/var/log/ceph/ceph-client.admin.23125.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.23125.log.gz 2026-02-12T12:32:18.191 INFO:teuthology.orchestra.run.trial110.stderr: 92.2% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2026-02-12T12:32:20.191 INFO:teuthology.orchestra.run.trial110.stderr: 92.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2026-02-12T12:32:20.279 INFO:teuthology.orchestra.run.trial110.stderr: 92.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2026-02-12T12:32:20.449 INFO:teuthology.orchestra.run.trial110.stderr: 92.7% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2026-02-12T12:32:20.451 INFO:teuthology.orchestra.run.trial110.stderr: 2026-02-12T12:32:20.451 INFO:teuthology.orchestra.run.trial110.stderr:real 0m2.289s 2026-02-12T12:32:20.451 INFO:teuthology.orchestra.run.trial110.stderr:user 0m5.991s 2026-02-12T12:32:20.451 INFO:teuthology.orchestra.run.trial110.stderr:sys 0m0.439s 2026-02-12T12:32:20.451 INFO:tasks.ceph:Archiving logs... 2026-02-12T12:32:20.452 DEBUG:teuthology.misc:Transferring archived files from trial110:/var/log/ceph to /home/teuthworker/mnt/teuthology/moagrawa-2026-02-10_06:35:43-crimson-rados-main-distro-debug-trial/42493/remote/trial110/log 2026-02-12T12:32:20.453 DEBUG:teuthology.orchestra.run.trial110:> sudo tar c -f - -C /var/log/ceph -- . 2026-02-12T12:32:21.109 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-12T12:32:21.180 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-12T12:32:21.192 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2026-02-12T12:32:21.192 DEBUG:teuthology.orchestra.run.trial110:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-12T12:32:21.217 DEBUG:teuthology.run_tasks:Unwinding manager clock 2026-02-12T12:32:21.229 INFO:teuthology.task.clock:Checking final clock skew... 2026-02-12T12:32:21.229 DEBUG:teuthology.orchestra.run.trial110:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-12T12:32:21.272 INFO:teuthology.orchestra.run.trial110.stderr:bash: line 1: ntpq: command not found 2026-02-12T12:32:21.281 INFO:teuthology.orchestra.run.trial110.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-12T12:32:21.281 INFO:teuthology.orchestra.run.trial110.stdout:=============================================================================== 2026-02-12T12:32:21.282 INFO:teuthology.orchestra.run.trial110.stdout:^- 24-111-79-186-dynamic.mi> 2 6 177 48 +2256us[+2034us] +/- 98ms 2026-02-12T12:32:21.282 INFO:teuthology.orchestra.run.trial110.stdout:^- 66.187.4.132.hostodo.com 3 7 377 128 +1392us[+1171us] +/- 55ms 2026-02-12T12:32:21.282 INFO:teuthology.orchestra.run.trial110.stdout:^* time.cloudflare.com 3 7 377 0 -182us[ -405us] +/- 6920us 2026-02-12T12:32:21.282 INFO:teuthology.orchestra.run.trial110.stdout:^- jane.qotw.net 2 7 377 125 +1300us[+1078us] +/- 25ms 2026-02-12T12:32:21.283 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2026-02-12T12:32:21.295 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2026-02-12T12:32:21.295 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2026-02-12T12:32:21.311 DEBUG:teuthology.orchestra.run.trial110:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2026-02-12T12:32:21.372 DEBUG:teuthology.orchestra.run.trial110:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-12T12:32:21.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-12T12:32:21.399 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2026-02-12T12:32:21.416 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2026-02-12T12:32:21.432 INFO:teuthology.task.internal:Duration was 1559.686532 seconds 2026-02-12T12:32:21.433 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2026-02-12T12:32:21.451 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2026-02-12T12:32:21.451 DEBUG:teuthology.orchestra.run.trial110:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-12T12:32:21.490 INFO:teuthology.orchestra.run.trial110.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-12T12:32:21.867 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2026-02-12T12:32:21.868 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial110.front.sepia.ceph.com 2026-02-12T12:32:21.868 DEBUG:teuthology.orchestra.run.trial110:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-12T12:32:21.926 INFO:teuthology.task.internal.syslog:Gathering journactl... 2026-02-12T12:32:21.926 DEBUG:teuthology.orchestra.run.trial110:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-12T12:32:22.014 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2026-02-12T12:32:22.015 DEBUG:teuthology.orchestra.run.trial110:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-12T12:32:22.037 INFO:teuthology.orchestra.run.trial110.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-12T12:32:22.038 INFO:teuthology.orchestra.run.trial110.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-12T12:32:22.038 INFO:teuthology.orchestra.run.trial110.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-12T12:32:22.038 INFO:teuthology.orchestra.run.trial110.stderr: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-02-12T12:32:22.039 INFO:teuthology.orchestra.run.trial110.stderr:/home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-12T12:32:22.042 INFO:teuthology.orchestra.run.trial110.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: 88.8% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-12T12:32:22.044 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2026-02-12T12:32:22.061 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2026-02-12T12:32:22.061 DEBUG:teuthology.orchestra.run.trial110:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-12T12:32:22.108 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2026-02-12T12:32:22.125 DEBUG:teuthology.orchestra.run.trial110:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-12T12:32:22.171 INFO:teuthology.orchestra.run.trial110.stdout:kernel.core_pattern = core 2026-02-12T12:32:22.187 DEBUG:teuthology.orchestra.run.trial110:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-12T12:32:22.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-12T12:32:22.242 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2026-02-12T12:32:22.259 INFO:teuthology.task.internal:Transferring archived files... 2026-02-12T12:32:22.261 DEBUG:teuthology.misc:Transferring archived files from trial110:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/moagrawa-2026-02-10_06:35:43-crimson-rados-main-distro-debug-trial/42493/remote/trial110 2026-02-12T12:32:22.261 DEBUG:teuthology.orchestra.run.trial110:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-12T12:32:22.283 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2026-02-12T12:32:22.283 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2026-02-12T12:32:22.284 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2026-02-12T12:32:22.284 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a has been restored 2026-02-12T12:32:22.363 INFO:teuthology.task.internal:Removing archive directory... 2026-02-12T12:32:22.363 DEBUG:teuthology.orchestra.run.trial110:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-12T12:32:22.381 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2026-02-12T12:32:22.400 INFO:teuthology.task.internal:Not uploading archives. 2026-02-12T12:32:22.401 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2026-02-12T12:32:22.419 INFO:teuthology.task.internal:Tidying up after the test... 2026-02-12T12:32:22.419 DEBUG:teuthology.orchestra.run.trial110:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-12T12:32:22.438 INFO:teuthology.orchestra.run.trial110.stdout: 786525 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 12 12:32 /home/ubuntu/cephtest 2026-02-12T12:32:22.440 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2026-02-12T12:32:22.458 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2026-02-12T12:32:22.498 INFO:teuthology.run:Summary data: description: crimson-rados/perf/{0-crimson_install clusters/{crimson-fixed-1} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/rbm$/{crimson_seastore_rbm_2q} perf_ignorelist settings/optimized workloads/radosbench_4M_rand_read} duration: 1559.686532497406 flavor: debug owner: scheduled_moagrawa@soko04.front.sepia.ceph.com success: true 2026-02-12T12:32:22.499 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-12T12:32:22.606 INFO:teuthology.run:pass