2026-02-25T08:03:40.365 INFO:root:teuthology version: 1.2.4.dev9+gfa17720d0 2026-02-25T08:03:40.384 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-25T08:03:40.465 INFO:teuthology.run:Config: archive_path: /home/teuthworker/mnt/teuthology/matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial/69096 branch: wip-logstoremanager description: crimson-rados/perf/{0-crimson_install clusters/{crimson-fixed-1} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/rbm$/{crimson_seastore_rbm} perf_ignorelist settings/optimized workloads/radosbench_4K_rand_read} email: null first_in_suite: false flavor: debug job_id: '69096' kernel: branch: distro kdb: 1 sha1: distro last_in_suite: false machine_type: trial name: matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial no_nested_subset: false os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-logstoremanager ceph: conf: global: auth client required: none auth cluster required: none auth service required: none auth supported: none debug asok: 0/0 debug auth: 0/0 debug bdev: 0/0 debug bluefs: 0/0 debug bluestore: 0/0 debug buffer: 0/0 debug civetweb: 0/0 debug client: 0/0 debug compressor: 0/0 debug context: 0/0 debug crush: 0/0 debug crypto: 0/0 debug dpdk: 0/0 debug eventtrace: 0/0 debug filer: 0/0 debug filestore: 0/0 debug finisher: 0/0 debug fuse: 0/0 debug heartbeatmap: 0/0 debug javaclient: 0/0 debug journal: 0/0 debug journaler: 0/0 debug kstore: 0/0 debug leveldb: 0/0 debug lockdep: 0/0 debug mds: 0/0 debug mds balancer: 0/0 debug mds locker: 0/0 debug mds log: 0/0 debug mds log expire: 0/0 debug mds migrator: 0/0 debug memdb: 0/0 debug mgr: 0/0 debug mgrc: 0/0 debug mon: 0/0 debug monc: 0/0 debug ms: 0/0 debug objclass: 0/0 debug objectcacher: 0/0 debug objecter: 0/0 debug optracker: 0/0 debug osd: 0/0 debug paxos: 0/0 debug perfcounter: 0/0 debug rados: 0/0 debug rbd: 0/0 debug rbd mirror: 0/0 debug rbd replay: 0/0 debug refs: 0/0 debug rgw: 0/0 debug rgw lifecycle: 0/0 debug rgw notification: 0/0 debug rgw sync: 0/0 debug rocksdb: 0/0 debug striper: 0/0 debug throttle: 0/0 debug timer: 0/0 debug tp: 0/0 enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 0/0 debug ms: 0/0 debug paxos: 0/0 osd pool default crimson: true osd: crimson cpu num: 4 crimson osd obc lru size: 10 debug filestore: 0/0 debug journal: 0/0 debug ms: 0/0 debug osd: 0/0 debug seastore: 20 debug seastore backref: 20 debug seastore cache: 20 debug seastore cleaner: 20 debug seastore device: 20 debug seastore epm: 20 debug seastore fixedkv tree: 20 debug seastore journal: 20 debug seastore lba: 20 debug seastore odata: 20 debug seastore omap: 20 debug seastore onode: 20 debug seastore t: 20 debug seastore tm: 20 osd objectstore: seastore osd shutdown pgref assert: true seastore cachepin size pershard: 64M seastore main device type: RANDOM_BLOCK_SSD seastore max concurrent transactions: 8 crimson_compat: true flavor: debug log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) - \(PG_ - \(OSD_ - \(OBJECT_ - overall HEALTH sha1: 6aeb6975bc9a4790a399c9cc70dba9635d2a7819 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log global: osd crush chooseleaf type: 0 osd pool default pg num: 128 osd pool default pgp num: 128 osd pool default size: 2 mon: {} install: ceph: flavor: debug sha1: 6aeb6975bc9a4790a399c9cc70dba9635d2a7819 workunit: branch: main env: CRIMSON_COMPAT: '1' sha1: c32469671c6b0222f786dd36c34eb702f780a11d owner: scheduled_matan@soko04.front.sepia.ceph.com priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 869 sha1: 6aeb6975bc9a4790a399c9cc70dba9635d2a7819 sleep_before_teardown: 0 suite: crimson-rados suite_branch: main suite_path: /home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph.git suite_sha1: c32469671c6b0222f786dd36c34eb702f780a11d targets: trial190.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE1WLmp6Pwq3/iMPqif60sc3ZuonJypHy0xLtT+Qq0rcxyi9hnFxqABKTBjZnITZcBPYfOZq90iGBR+2c7Zbems= tasks: - install: extra_packages: deb: - ceph-osd-crimson - ceph-osd-crimson-dbg rpm: - ceph-osd-crimson - ceph-osd-crimson-debuginfo - pexec: all: - sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson - install: null - ceph: conf: osd: debug monc: 20 wait-for-scrub: false - ssh_keys: null - cbt: benchmarks: radosbench: concurrent_ops: 4 concurrent_procs: 2 op_size: - 4096 pool_monitoring_list: - collectl pool_profile: replicated readmode: rand run_monitoring_list: - collectl time: 60 write_only: false cluster: iterations: 1 osds_per_node: 3 pool_profiles: replicated: pg_size: 256 pgp_size: 256 replication: replicated user: ubuntu monitoring_profiles: perf: args: stat -p {pid} -o {perf_dir}/perf_stat.{pid} nodes: - osds perf_cmd: perf pid_glob: crimson-osd.*.pid teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: fa17720d0088c3ac28e473468bfc79eeaff5cd38 timestamp: 2026-02-25_07:55:35 tube: trial user: matan verbose: false worker_log: /home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.904699 2026-02-25T08:03:40.465 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa; will attempt to use it 2026-02-25T08:03:40.466 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks 2026-02-25T08:03:40.466 INFO:teuthology.run_tasks:Running task internal.check_packages... 2026-02-25T08:03:40.467 INFO:teuthology.task.internal:Checking packages... 2026-02-25T08:03:40.467 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'debug' and ceph hash '6aeb6975bc9a4790a399c9cc70dba9635d2a7819' 2026-02-25T08:03:40.467 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2026-02-25T08:03:40.467 INFO:teuthology.packaging:ref: None 2026-02-25T08:03:40.467 INFO:teuthology.packaging:tag: None 2026-02-25T08:03:40.467 INFO:teuthology.packaging:branch: wip-logstoremanager 2026-02-25T08:03:40.468 INFO:teuthology.packaging:sha1: 6aeb6975bc9a4790a399c9cc70dba9635d2a7819 2026-02-25T08:03:40.468 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&ref=wip-logstoremanager 2026-02-25T08:03:40.589 INFO:teuthology.task.internal:Found packages for ceph version 20.3.0-5338.g6aeb6975 2026-02-25T08:03:40.591 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2026-02-25T08:03:40.602 INFO:teuthology.task.internal:no buildpackages task found 2026-02-25T08:03:40.602 INFO:teuthology.run_tasks:Running task internal.save_config... 2026-02-25T08:03:40.613 INFO:teuthology.task.internal:Saving configuration 2026-02-25T08:03:40.621 INFO:teuthology.run_tasks:Running task internal.check_lock... 2026-02-25T08:03:40.633 INFO:teuthology.task.internal.check_lock:Checking locks... 2026-02-25T08:03:40.691 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial190.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial/69096', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-25 07:57:56.743000', 'locked_by': 'scheduled_matan@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE1WLmp6Pwq3/iMPqif60sc3ZuonJypHy0xLtT+Qq0rcxyi9hnFxqABKTBjZnITZcBPYfOZq90iGBR+2c7Zbems='} 2026-02-25T08:03:40.691 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2026-02-25T08:03:40.702 INFO:teuthology.task.internal:roles: ubuntu@trial190.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2026-02-25T08:03:40.702 INFO:teuthology.run_tasks:Running task console_log... 2026-02-25T08:03:40.773 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7cbc1c043100>, signals=[15]) 2026-02-25T08:03:40.774 INFO:teuthology.run_tasks:Running task internal.connect... 2026-02-25T08:03:40.785 INFO:teuthology.task.internal:Opening connections... 2026-02-25T08:03:40.785 DEBUG:teuthology.task.internal:connecting to ubuntu@trial190.front.sepia.ceph.com 2026-02-25T08:03:40.786 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-25T08:03:40.848 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2026-02-25T08:03:40.860 DEBUG:teuthology.orchestra.run.trial190:> uname -m 2026-02-25T08:03:40.924 INFO:teuthology.orchestra.run.trial190.stdout:x86_64 2026-02-25T08:03:40.925 DEBUG:teuthology.orchestra.run.trial190:> cat /etc/os-release 2026-02-25T08:03:40.980 INFO:teuthology.orchestra.run.trial190.stdout:NAME="CentOS Stream" 2026-02-25T08:03:40.980 INFO:teuthology.orchestra.run.trial190.stdout:VERSION="9" 2026-02-25T08:03:40.980 INFO:teuthology.orchestra.run.trial190.stdout:ID="centos" 2026-02-25T08:03:40.980 INFO:teuthology.orchestra.run.trial190.stdout:ID_LIKE="rhel fedora" 2026-02-25T08:03:40.980 INFO:teuthology.orchestra.run.trial190.stdout:VERSION_ID="9" 2026-02-25T08:03:40.980 INFO:teuthology.orchestra.run.trial190.stdout:PLATFORM_ID="platform:el9" 2026-02-25T08:03:40.980 INFO:teuthology.orchestra.run.trial190.stdout:PRETTY_NAME="CentOS Stream 9" 2026-02-25T08:03:40.980 INFO:teuthology.orchestra.run.trial190.stdout:ANSI_COLOR="0;31" 2026-02-25T08:03:40.980 INFO:teuthology.orchestra.run.trial190.stdout:LOGO="fedora-logo-icon" 2026-02-25T08:03:40.980 INFO:teuthology.orchestra.run.trial190.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2026-02-25T08:03:40.981 INFO:teuthology.orchestra.run.trial190.stdout:HOME_URL="https://centos.org/" 2026-02-25T08:03:40.981 INFO:teuthology.orchestra.run.trial190.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2026-02-25T08:03:40.981 INFO:teuthology.orchestra.run.trial190.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2026-02-25T08:03:40.981 INFO:teuthology.orchestra.run.trial190.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2026-02-25T08:03:40.981 INFO:teuthology.lock.ops:Updating trial190.front.sepia.ceph.com on lock server 2026-02-25T08:03:41.043 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2026-02-25T08:03:41.056 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2026-02-25T08:03:41.070 INFO:teuthology.task.internal:Checking for old test directory... 2026-02-25T08:03:41.070 DEBUG:teuthology.orchestra.run.trial190:> test '!' -e /home/ubuntu/cephtest 2026-02-25T08:03:41.085 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2026-02-25T08:03:41.096 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2026-02-25T08:03:41.096 DEBUG:teuthology.orchestra.run.trial190:> test -z $(ls -A /var/lib/ceph) 2026-02-25T08:03:41.141 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2026-02-25T08:03:41.242 INFO:teuthology.run_tasks:Running task kernel... 2026-02-25T08:03:41.262 INFO:teuthology.task.kernel:normalize config orig: {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'} 2026-02-25T08:03:41.262 INFO:teuthology.task.kernel:config {'mon.a': {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'}}, timeout 300 2026-02-25T08:03:41.262 DEBUG:teuthology.orchestra.run.trial190:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-25T08:03:41.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-25T08:03:41.277 DEBUG:teuthology.orchestra.run.trial190:> uname -r 2026-02-25T08:03:41.333 INFO:teuthology.orchestra.run.trial190.stdout:5.14.0-681.el9.x86_64 2026-02-25T08:03:41.333 INFO:teuthology.task.kernel:Running kernel on trial190: 5.14.0-681.el9.x86_64 2026-02-25T08:03:41.333 DEBUG:teuthology.orchestra.run.trial190:> sudo yum install -y kernel 2026-02-25T08:03:43.376 INFO:teuthology.orchestra.run.trial190.stdout:CentOS Stream 9 - BaseOS 5.3 MB/s | 8.9 MB 00:01 2026-02-25T08:03:47.036 INFO:teuthology.orchestra.run.trial190.stdout:CentOS Stream 9 - AppStream 8.6 MB/s | 27 MB 00:03 2026-02-25T08:03:50.442 INFO:teuthology.orchestra.run.trial190.stdout:CentOS Stream 9 - CRB 7.7 MB/s | 8.0 MB 00:01 2026-02-25T08:03:53.049 INFO:teuthology.orchestra.run.trial190.stdout:CentOS Stream 9 - Extras packages 11 kB/s | 20 kB 00:01 2026-02-25T08:03:53.784 INFO:teuthology.orchestra.run.trial190.stdout:Extra Packages for Enterprise Linux 30 MB/s | 20 MB 00:00 2026-02-25T08:03:56.951 INFO:teuthology.orchestra.run.trial190.stdout:lab-extras 2.7 MB/s | 50 kB 00:00 2026-02-25T08:03:57.964 INFO:teuthology.orchestra.run.trial190.stdout:Package kernel-5.14.0-645.el9.x86_64 is already installed. 2026-02-25T08:03:57.964 INFO:teuthology.orchestra.run.trial190.stdout:Package kernel-5.14.0-681.el9.x86_64 is already installed. 2026-02-25T08:03:57.986 INFO:teuthology.orchestra.run.trial190.stdout:Dependencies resolved. 2026-02-25T08:03:57.987 INFO:teuthology.orchestra.run.trial190.stdout:================================================================================ 2026-02-25T08:03:57.987 INFO:teuthology.orchestra.run.trial190.stdout: Package Architecture Version Repository Size 2026-02-25T08:03:57.987 INFO:teuthology.orchestra.run.trial190.stdout:================================================================================ 2026-02-25T08:03:57.988 INFO:teuthology.orchestra.run.trial190.stdout:Installing: 2026-02-25T08:03:57.988 INFO:teuthology.orchestra.run.trial190.stdout: kernel x86_64 5.14.0-686.el9 baseos 901 k 2026-02-25T08:03:57.988 INFO:teuthology.orchestra.run.trial190.stdout:Installing dependencies: 2026-02-25T08:03:57.988 INFO:teuthology.orchestra.run.trial190.stdout: kernel-core x86_64 5.14.0-686.el9 baseos 17 M 2026-02-25T08:03:57.988 INFO:teuthology.orchestra.run.trial190.stdout: kernel-modules x86_64 5.14.0-686.el9 baseos 40 M 2026-02-25T08:03:57.988 INFO:teuthology.orchestra.run.trial190.stdout: kernel-modules-core x86_64 5.14.0-686.el9 baseos 31 M 2026-02-25T08:03:57.988 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:03:57.988 INFO:teuthology.orchestra.run.trial190.stdout:Transaction Summary 2026-02-25T08:03:57.988 INFO:teuthology.orchestra.run.trial190.stdout:================================================================================ 2026-02-25T08:03:57.989 INFO:teuthology.orchestra.run.trial190.stdout:Install 4 Packages 2026-02-25T08:03:57.989 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:03:57.989 INFO:teuthology.orchestra.run.trial190.stdout:Total download size: 89 M 2026-02-25T08:03:57.989 INFO:teuthology.orchestra.run.trial190.stdout:Installed size: 135 M 2026-02-25T08:03:57.989 INFO:teuthology.orchestra.run.trial190.stdout:Downloading Packages: 2026-02-25T08:03:58.407 INFO:teuthology.orchestra.run.trial190.stdout:(1/4): kernel-5.14.0-686.el9.x86_64.rpm 5.5 MB/s | 901 kB 00:00 2026-02-25T08:03:58.582 INFO:teuthology.orchestra.run.trial190.stdout:(2/4): kernel-core-5.14.0-686.el9.x86_64.rpm 51 MB/s | 17 MB 00:00 2026-02-25T08:03:58.724 INFO:teuthology.orchestra.run.trial190.stdout:(3/4): kernel-modules-core-5.14.0-686.el9.x86_6 98 MB/s | 31 MB 00:00 2026-02-25T08:03:58.782 INFO:teuthology.orchestra.run.trial190.stdout:(4/4): kernel-modules-5.14.0-686.el9.x86_64.rpm 74 MB/s | 40 MB 00:00 2026-02-25T08:03:58.782 INFO:teuthology.orchestra.run.trial190.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:03:58.782 INFO:teuthology.orchestra.run.trial190.stdout:Total 112 MB/s | 89 MB 00:00 2026-02-25T08:03:58.982 INFO:teuthology.orchestra.run.trial190.stdout:Running transaction check 2026-02-25T08:03:59.185 INFO:teuthology.orchestra.run.trial190.stdout:Transaction check succeeded. 2026-02-25T08:03:59.185 INFO:teuthology.orchestra.run.trial190.stdout:Running transaction test 2026-02-25T08:03:59.571 INFO:teuthology.orchestra.run.trial190.stdout:Transaction test succeeded. 2026-02-25T08:03:59.571 INFO:teuthology.orchestra.run.trial190.stdout:Running transaction 2026-02-25T08:04:00.683 INFO:teuthology.orchestra.run.trial190.stdout: Preparing : 1/1 2026-02-25T08:04:00.971 INFO:teuthology.orchestra.run.trial190.stdout: Installing : kernel-modules-core-5.14.0-686.el9.x86_64 1/4 2026-02-25T08:04:00.993 INFO:teuthology.orchestra.run.trial190.stdout: Installing : kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-25T08:04:03.558 INFO:teuthology.orchestra.run.trial190.stdout: Running scriptlet: kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-25T08:04:03.591 INFO:teuthology.orchestra.run.trial190.stdout: Installing : kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-25T08:04:06.627 INFO:teuthology.orchestra.run.trial190.stdout: Running scriptlet: kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-25T08:04:06.633 INFO:teuthology.orchestra.run.trial190.stdout: Installing : kernel-5.14.0-686.el9.x86_64 4/4 2026-02-25T08:04:09.693 INFO:teuthology.orchestra.run.trial190.stdout: Running scriptlet: kernel-modules-core-5.14.0-686.el9.x86_64 4/4 2026-02-25T08:04:27.218 INFO:teuthology.orchestra.run.trial190.stdout: Running scriptlet: kernel-core-5.14.0-686.el9.x86_64 4/4 2026-02-25T08:04:27.239 INFO:teuthology.orchestra.run.trial190.stdout: Running scriptlet: kernel-modules-5.14.0-686.el9.x86_64 4/4 2026-02-25T08:04:27.426 INFO:teuthology.orchestra.run.trial190.stdout: Running scriptlet: kernel-5.14.0-686.el9.x86_64 4/4 2026-02-25T08:04:27.426 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : kernel-5.14.0-686.el9.x86_64 1/4 2026-02-25T08:04:27.426 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-25T08:04:27.426 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-25T08:04:27.484 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : kernel-modules-core-5.14.0-686.el9.x86_64 4/4 2026-02-25T08:04:27.485 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:04:27.485 INFO:teuthology.orchestra.run.trial190.stdout:Installed: 2026-02-25T08:04:27.485 INFO:teuthology.orchestra.run.trial190.stdout: kernel-5.14.0-686.el9.x86_64 2026-02-25T08:04:27.485 INFO:teuthology.orchestra.run.trial190.stdout: kernel-core-5.14.0-686.el9.x86_64 2026-02-25T08:04:27.485 INFO:teuthology.orchestra.run.trial190.stdout: kernel-modules-5.14.0-686.el9.x86_64 2026-02-25T08:04:27.485 INFO:teuthology.orchestra.run.trial190.stdout: kernel-modules-core-5.14.0-686.el9.x86_64 2026-02-25T08:04:27.485 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:04:27.486 INFO:teuthology.orchestra.run.trial190.stdout:Complete! 2026-02-25T08:04:27.549 DEBUG:teuthology.orchestra.run.trial190:> rpm -q kernel | sort -rV | head -n 1 2026-02-25T08:04:27.579 INFO:teuthology.orchestra.run.trial190.stdout:kernel-5.14.0-686.el9.x86_64 2026-02-25T08:04:27.579 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-686.el9.x86_64 2026-02-25T08:04:27.580 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-681.el9.x86_64 Expected: 5.14.0-686.el9.x86_64 2026-02-25T08:04:27.580 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2026-02-25T08:04:27.580 DEBUG:teuthology.task.kernel:src is distro, skipping download 2026-02-25T08:04:27.580 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2026-02-25T08:04:27.580 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@trial190.front.sepia.ceph.com, path=None, version=distro) 2026-02-25T08:04:27.581 DEBUG:teuthology.orchestra.run.trial190:> rpm -q kernel | sort -rV | head -n 1 2026-02-25T08:04:27.648 INFO:teuthology.orchestra.run.trial190.stdout:kernel-5.14.0-686.el9.x86_64 2026-02-25T08:04:27.648 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-686.el9.x86_64 2026-02-25T08:04:27.649 DEBUG:teuthology.orchestra.run.trial190:> sudo rpm -qi grub2-tools 2026-02-25T08:04:27.718 INFO:teuthology.orchestra.run.trial190.stdout:Name : grub2-tools 2026-02-25T08:04:27.718 INFO:teuthology.orchestra.run.trial190.stdout:Epoch : 1 2026-02-25T08:04:27.718 INFO:teuthology.orchestra.run.trial190.stdout:Version : 2.06 2026-02-25T08:04:27.718 INFO:teuthology.orchestra.run.trial190.stdout:Release : 120.el9 2026-02-25T08:04:27.718 INFO:teuthology.orchestra.run.trial190.stdout:Architecture: x86_64 2026-02-25T08:04:27.718 INFO:teuthology.orchestra.run.trial190.stdout:Install Date: Wed 18 Feb 2026 05:18:42 PM UTC 2026-02-25T08:04:27.718 INFO:teuthology.orchestra.run.trial190.stdout:Group : Unspecified 2026-02-25T08:04:27.718 INFO:teuthology.orchestra.run.trial190.stdout:Size : 9108611 2026-02-25T08:04:27.718 INFO:teuthology.orchestra.run.trial190.stdout:License : GPLv3+ 2026-02-25T08:04:27.718 INFO:teuthology.orchestra.run.trial190.stdout:Signature : RSA/SHA256, Thu 04 Dec 2025 09:42:59 AM UTC, Key ID 05b555b38483c65d 2026-02-25T08:04:27.718 INFO:teuthology.orchestra.run.trial190.stdout:Source RPM : grub2-2.06-120.el9.src.rpm 2026-02-25T08:04:27.718 INFO:teuthology.orchestra.run.trial190.stdout:Build Date : Wed 03 Dec 2025 12:10:38 PM UTC 2026-02-25T08:04:27.718 INFO:teuthology.orchestra.run.trial190.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2026-02-25T08:04:27.718 INFO:teuthology.orchestra.run.trial190.stdout:Packager : builder@centos.org 2026-02-25T08:04:27.718 INFO:teuthology.orchestra.run.trial190.stdout:Vendor : CentOS 2026-02-25T08:04:27.718 INFO:teuthology.orchestra.run.trial190.stdout:URL : http://www.gnu.org/software/grub/ 2026-02-25T08:04:27.718 INFO:teuthology.orchestra.run.trial190.stdout:Summary : Support tools for GRUB. 2026-02-25T08:04:27.718 INFO:teuthology.orchestra.run.trial190.stdout:Description : 2026-02-25T08:04:27.718 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:04:27.718 INFO:teuthology.orchestra.run.trial190.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2026-02-25T08:04:27.718 INFO:teuthology.orchestra.run.trial190.stdout:customizable bootloader with modular architecture. It supports a rich 2026-02-25T08:04:27.719 INFO:teuthology.orchestra.run.trial190.stdout:variety of kernel formats, file systems, computer architectures and 2026-02-25T08:04:27.719 INFO:teuthology.orchestra.run.trial190.stdout:hardware devices. 2026-02-25T08:04:27.719 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:04:27.719 INFO:teuthology.orchestra.run.trial190.stdout:This subpackage provides tools for support of all platforms. 2026-02-25T08:04:27.719 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2026-02-25T08:04:27.719 INFO:teuthology.task.kernel:Updating grub on trial190 to boot 5.14.0-686.el9.x86_64 2026-02-25T08:04:27.719 DEBUG:teuthology.orchestra.run.trial190:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f -print -quit 2026-02-25T08:04:27.783 INFO:teuthology.orchestra.run.trial190.stdout:/boot/loader/entries/b0373be789df49cdafedf423703d2566-5.14.0-681.el9.x86_64.conf 2026-02-25T08:04:27.784 DEBUG:teuthology.orchestra.run.trial190:> sudo command -v grubby && echo yes || echo no 2026-02-25T08:04:27.848 INFO:teuthology.orchestra.run.trial190.stdout:/sbin/grubby 2026-02-25T08:04:27.850 INFO:teuthology.orchestra.run.trial190.stdout:yes 2026-02-25T08:04:27.851 DEBUG:teuthology.orchestra.run.trial190:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f | grep -F "5.14.0-686.el9.x86_64" | head -n 1 2026-02-25T08:04:27.914 INFO:teuthology.orchestra.run.trial190.stdout:/boot/loader/entries/b0373be789df49cdafedf423703d2566-5.14.0-686.el9.x86_64.conf 2026-02-25T08:04:27.914 DEBUG:teuthology.orchestra.run.trial190:> sudo grub2-set-default b0373be789df49cdafedf423703d2566-5.14.0-686.el9.x86_64 2026-02-25T08:04:27.988 DEBUG:teuthology.orchestra.run.trial190:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2026-02-25T08:04:28.192 INFO:teuthology.orchestra.run.trial190.stderr:Generating grub configuration file ... 2026-02-25T08:04:28.259 INFO:teuthology.orchestra.run.trial190.stderr:Found linux image: /boot/vmlinuz-5.14.0-686.el9.x86_64 2026-02-25T08:04:28.267 INFO:teuthology.orchestra.run.trial190.stderr:Found initrd image: /boot/initramfs-5.14.0-686.el9.x86_64.img 2026-02-25T08:04:28.520 INFO:teuthology.orchestra.run.trial190.stderr:Found linux image: /boot/vmlinuz-5.14.0-681.el9.x86_64 2026-02-25T08:04:28.527 INFO:teuthology.orchestra.run.trial190.stderr:Found initrd image: /boot/initramfs-5.14.0-681.el9.x86_64.img 2026-02-25T08:04:28.561 INFO:teuthology.orchestra.run.trial190.stderr:Found linux image: /boot/vmlinuz-5.14.0-645.el9.x86_64 2026-02-25T08:04:28.568 INFO:teuthology.orchestra.run.trial190.stderr:Found initrd image: /boot/initramfs-5.14.0-645.el9.x86_64.img 2026-02-25T08:04:28.600 INFO:teuthology.orchestra.run.trial190.stderr:Found linux image: /boot/vmlinuz-0-rescue-b0373be789df49cdafedf423703d2566 2026-02-25T08:04:28.607 INFO:teuthology.orchestra.run.trial190.stderr:Found initrd image: /boot/initramfs-0-rescue-b0373be789df49cdafedf423703d2566.img 2026-02-25T08:04:28.630 INFO:teuthology.orchestra.run.trial190.stderr:Found linux image: /boot/vmlinuz-0-rescue-2734456005d743dca7f3dc77c35a2d0d 2026-02-25T08:04:28.636 INFO:teuthology.orchestra.run.trial190.stderr:Found initrd image: /boot/initramfs-0-rescue-2734456005d743dca7f3dc77c35a2d0d.img 2026-02-25T08:04:28.723 INFO:teuthology.orchestra.run.trial190.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2026-02-25T08:04:28.732 INFO:teuthology.orchestra.run.trial190.stderr:done 2026-02-25T08:04:28.734 DEBUG:teuthology.orchestra.run.trial190:> test -d /sys/firmware/efi 2026-02-25T08:04:28.749 DEBUG:teuthology.orchestra.run.trial190:> sudo find /boot/efi/EFI -mindepth 1 -maxdepth 1 -type d || true 2026-02-25T08:04:28.815 INFO:teuthology.orchestra.run.trial190.stdout:/boot/efi/EFI/BOOT 2026-02-25T08:04:28.815 INFO:teuthology.orchestra.run.trial190.stdout:/boot/efi/EFI/centos 2026-02-25T08:04:28.817 DEBUG:teuthology.orchestra.run.trial190:> sudo test -f /boot/efi/EFI/BOOT/grub.cfg && echo yes || echo no 2026-02-25T08:04:28.883 INFO:teuthology.orchestra.run.trial190.stdout:no 2026-02-25T08:04:28.883 DEBUG:teuthology.orchestra.run.trial190:> sudo test -f /boot/efi/EFI/centos/grub.cfg && echo yes || echo no 2026-02-25T08:04:28.950 INFO:teuthology.orchestra.run.trial190.stdout:yes 2026-02-25T08:04:28.950 DEBUG:teuthology.orchestra.run.trial190:> sudo cp -f /boot/grub2/grub.cfg /boot/efi/EFI/centos/grub.cfg 2026-02-25T08:04:29.013 INFO:teuthology.orchestra.remote:Triggering safe hard reboot via SysRq on trial190 2026-02-25T08:04:29.013 DEBUG:teuthology.orchestra.run.trial190:> sudo bash -c ' 2026-02-25T08:04:29.013 DEBUG:teuthology.orchestra.run.trial190:> echo pci > /sys/kernel/reboot/type && echo 1 > /sys/kernel/reboot/force && sync && echo s > /proc/sysrq-trigger && echo u > /proc/sysrq-trigger && sleep 2 && echo b > /proc/sysrq-trigger 2026-02-25T08:04:29.013 DEBUG:teuthology.orchestra.run.trial190:> ' 2026-02-25T08:04:59.056 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2026-02-25T08:04:59.056 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial190.front.sepia.ceph.com' 2026-02-25T08:04:59.057 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-25T08:05:13.344 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 10.20.193.190 2026-02-25T08:05:22.352 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial190.front.sepia.ceph.com' 2026-02-25T08:05:22.353 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-25T08:05:58.356 DEBUG:teuthology.orchestra.run.trial190:> true 2026-02-25T08:05:58.503 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial190.front.sepia.ceph.com' 2026-02-25T08:05:58.503 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-686.el9.x86_64"... 2026-02-25T08:05:58.503 DEBUG:teuthology.orchestra.run.trial190:> uname -r 2026-02-25T08:05:58.560 INFO:teuthology.orchestra.run.trial190.stdout:5.14.0-686.el9.x86_64 2026-02-25T08:05:58.561 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-686.el9.x86_64 vs 5.14.0-686.el9.x86_64 2026-02-25T08:05:58.561 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2026-02-25T08:05:58.561 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2026-02-25T08:05:59.562 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2026-02-25T08:05:59.562 DEBUG:teuthology.orchestra.run.trial190:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-25T08:05:59.591 INFO:teuthology.orchestra.run.trial190.stdout:ttyS1 2026-02-25T08:05:59.619 DEBUG:teuthology.parallel:result is None 2026-02-25T08:05:59.619 INFO:teuthology.run_tasks:Running task internal.base... 2026-02-25T08:05:59.634 INFO:teuthology.task.internal:Creating test directory... 2026-02-25T08:05:59.634 DEBUG:teuthology.orchestra.run.trial190:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-25T08:05:59.674 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2026-02-25T08:05:59.690 INFO:teuthology.run_tasks:Running task internal.archive... 2026-02-25T08:05:59.702 INFO:teuthology.task.internal:Creating archive directory... 2026-02-25T08:05:59.703 DEBUG:teuthology.orchestra.run.trial190:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-25T08:05:59.741 INFO:teuthology.run_tasks:Running task internal.coredump... 2026-02-25T08:05:59.751 INFO:teuthology.task.internal:Enabling coredump saving... 2026-02-25T08:05:59.751 DEBUG:teuthology.orchestra.run.trial190:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-25T08:05:59.796 INFO:teuthology.orchestra.run.trial190.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T08:05:59.804 INFO:teuthology.orchestra.run.trial190.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T08:05:59.806 INFO:teuthology.run_tasks:Running task internal.sudo... 2026-02-25T08:05:59.819 INFO:teuthology.task.internal:Configuring sudo... 2026-02-25T08:05:59.819 DEBUG:teuthology.orchestra.run.trial190:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-25T08:05:59.869 INFO:teuthology.run_tasks:Running task internal.syslog... 2026-02-25T08:05:59.885 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2026-02-25T08:05:59.885 DEBUG:teuthology.orchestra.run.trial190:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-25T08:05:59.925 DEBUG:teuthology.orchestra.run.trial190:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-25T08:06:00.045 DEBUG:teuthology.orchestra.run.trial190:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-25T08:06:00.069 DEBUG:teuthology.orchestra.run.trial190:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-25T08:06:00.185 DEBUG:teuthology.orchestra.run.trial190:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-25T08:06:00.209 DEBUG:teuthology.orchestra.run.trial190:> set -ex 2026-02-25T08:06:00.209 DEBUG:teuthology.orchestra.run.trial190:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-25T08:06:00.273 DEBUG:teuthology.orchestra.run.trial190:> sudo service rsyslog restart 2026-02-25T08:06:00.340 INFO:teuthology.orchestra.run.trial190.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-25T08:06:00.738 INFO:teuthology.run_tasks:Running task internal.timer... 2026-02-25T08:06:00.750 INFO:teuthology.task.internal:Starting timer... 2026-02-25T08:06:00.750 INFO:teuthology.run_tasks:Running task pcp... 2026-02-25T08:06:00.769 INFO:teuthology.run_tasks:Running task selinux... 2026-02-25T08:06:00.780 DEBUG:teuthology.orchestra.run.trial190:> sudo service auditd rotate 2026-02-25T08:06:00.807 INFO:teuthology.orchestra.run.trial190.stdout:Rotating logs: 2026-02-25T08:06:00.808 DEBUG:teuthology.task.selinux:Getting current SELinux state 2026-02-25T08:06:00.809 DEBUG:teuthology.orchestra.run.trial190:> /usr/sbin/getenforce 2026-02-25T08:06:00.865 INFO:teuthology.orchestra.run.trial190.stdout:Permissive 2026-02-25T08:06:00.865 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@trial190.front.sepia.ceph.com': 'permissive'} 2026-02-25T08:06:00.865 DEBUG:teuthology.orchestra.run.trial190:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-25T08:06:00.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-25T08:06:00.930 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2026-02-25T08:06:00.930 DEBUG:teuthology.orchestra.run.trial190:> sudo /usr/sbin/setenforce permissive 2026-02-25T08:06:00.993 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2026-02-25T08:06:01.007 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2026-02-25T08:06:01.014 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2026-02-25T08:06:01.014 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit trial190.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2026-02-25T08:08:26.463 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@trial190.front.sepia.ceph.com')] 2026-02-25T08:08:26.464 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial190.front.sepia.ceph.com' 2026-02-25T08:08:26.465 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-25T08:08:26.551 DEBUG:teuthology.orchestra.run.trial190:> true 2026-02-25T08:08:26.625 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial190.front.sepia.ceph.com' 2026-02-25T08:08:26.625 INFO:teuthology.run_tasks:Running task clock... 2026-02-25T08:08:26.638 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2026-02-25T08:08:26.638 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-25T08:08:26.639 DEBUG:teuthology.orchestra.run.trial190:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-25T08:08:26.692 INFO:teuthology.orchestra.run.trial190.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2026-02-25T08:08:26.704 INFO:teuthology.orchestra.run.trial190.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2026-02-25T08:08:26.723 INFO:teuthology.orchestra.run.trial190.stderr:sudo: ntpd: command not found 2026-02-25T08:08:26.733 INFO:teuthology.orchestra.run.trial190.stdout:506 Cannot talk to daemon 2026-02-25T08:08:26.744 INFO:teuthology.orchestra.run.trial190.stderr:Failed to start ntp.service: Unit ntp.service not found. 2026-02-25T08:08:26.756 INFO:teuthology.orchestra.run.trial190.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2026-02-25T08:08:26.802 INFO:teuthology.orchestra.run.trial190.stderr:bash: line 1: ntpq: command not found 2026-02-25T08:08:26.806 INFO:teuthology.orchestra.run.trial190.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-25T08:08:26.806 INFO:teuthology.orchestra.run.trial190.stdout:=============================================================================== 2026-02-25T08:08:26.806 INFO:teuthology.orchestra.run.trial190.stdout:^? nyc2.us.ntp.li 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-25T08:08:26.807 INFO:teuthology.run_tasks:Running task install... 2026-02-25T08:08:26.818 DEBUG:teuthology.task.install:project ceph 2026-02-25T08:08:26.818 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'debug', 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819'}} 2026-02-25T08:08:26.818 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'flavor': 'debug', 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819'} 2026-02-25T08:08:26.818 INFO:teuthology.task.install:Using flavor: debug 2026-02-25T08:08:26.821 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-osd', 'ceph-osd-classic', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-osd', 'ceph-osd-classic', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-25T08:08:26.821 INFO:teuthology.task.install:extra packages: {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']} 2026-02-25T08:08:26.822 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'debug', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819', 'tag': None, 'wait_for_package': False} 2026-02-25T08:08:26.822 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=6aeb6975bc9a4790a399c9cc70dba9635d2a7819 2026-02-25T08:08:26.952 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/ 2026-02-25T08:08:26.952 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5338.g6aeb6975 2026-02-25T08:08:27.299 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-25T08:08:27.300 DEBUG:teuthology.orchestra.run.trial190:> set -ex 2026-02-25T08:08:27.300 DEBUG:teuthology.orchestra.run.trial190:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-25T08:08:27.325 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-osd-crimson, ceph-osd-crimson-debuginfo on remote rpm x86_64 2026-02-25T08:08:27.326 DEBUG:teuthology.orchestra.run.trial190:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-25T08:08:27.390 DEBUG:teuthology.orchestra.run.trial190:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-25T08:08:27.463 DEBUG:teuthology.orchestra.run.trial190:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-25T08:08:27.526 INFO:teuthology.orchestra.run.trial190.stdout:check_obsoletes = 1 2026-02-25T08:08:27.527 DEBUG:teuthology.orchestra.run.trial190:> sudo yum clean all 2026-02-25T08:08:27.674 INFO:teuthology.orchestra.run.trial190.stdout:41 files removed 2026-02-25T08:08:27.688 DEBUG:teuthology.orchestra.run.trial190:> sudo yum -y install ceph-radosgw 2026-02-25T08:08:28.529 INFO:teuthology.orchestra.run.trial190.stdout:ceph packages for x86_64 135 kB/s | 94 kB 00:00 2026-02-25T08:08:29.194 INFO:teuthology.orchestra.run.trial190.stdout:ceph noarch packages 38 kB/s | 24 kB 00:00 2026-02-25T08:08:29.780 INFO:teuthology.orchestra.run.trial190.stdout:ceph source packages 3.7 kB/s | 2.1 kB 00:00 2026-02-25T08:08:39.444 INFO:teuthology.orchestra.run.trial190.stdout:CentOS Stream 9 - BaseOS 945 kB/s | 8.9 MB 00:09 2026-02-25T08:08:43.519 INFO:teuthology.orchestra.run.trial190.stdout:CentOS Stream 9 - AppStream 7.6 MB/s | 27 MB 00:03 2026-02-25T08:08:46.887 INFO:teuthology.orchestra.run.trial190.stdout:CentOS Stream 9 - CRB 8.0 MB/s | 8.0 MB 00:00 2026-02-25T08:08:48.544 INFO:teuthology.orchestra.run.trial190.stdout:CentOS Stream 9 - Extras packages 22 kB/s | 20 kB 00:00 2026-02-25T08:08:49.169 INFO:teuthology.orchestra.run.trial190.stdout:Extra Packages for Enterprise Linux 36 MB/s | 20 MB 00:00 2026-02-25T08:08:52.339 INFO:teuthology.orchestra.run.trial190.stdout:lab-extras 1.4 MB/s | 50 kB 00:00 2026-02-25T08:08:53.547 INFO:teuthology.orchestra.run.trial190.stdout:Dependencies resolved. 2026-02-25T08:08:53.549 INFO:teuthology.orchestra.run.trial190.stdout:================================================================================ 2026-02-25T08:08:53.549 INFO:teuthology.orchestra.run.trial190.stdout: Package Arch Version Repository Size 2026-02-25T08:08:53.549 INFO:teuthology.orchestra.run.trial190.stdout:================================================================================ 2026-02-25T08:08:53.549 INFO:teuthology.orchestra.run.trial190.stdout:Installing: 2026-02-25T08:08:53.549 INFO:teuthology.orchestra.run.trial190.stdout: ceph-radosgw x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 11 M 2026-02-25T08:08:53.549 INFO:teuthology.orchestra.run.trial190.stdout:Upgrading: 2026-02-25T08:08:53.549 INFO:teuthology.orchestra.run.trial190.stdout: librados2 x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 3.3 M 2026-02-25T08:08:53.549 INFO:teuthology.orchestra.run.trial190.stdout: librbd1 x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 2.7 M 2026-02-25T08:08:53.549 INFO:teuthology.orchestra.run.trial190.stdout:Installing dependencies: 2026-02-25T08:08:53.549 INFO:teuthology.orchestra.run.trial190.stdout: boost-program-options x86_64 1.75.0-13.el9 appstream 104 k 2026-02-25T08:08:53.549 INFO:teuthology.orchestra.run.trial190.stdout: ceph-base x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 5.1 M 2026-02-25T08:08:53.549 INFO:teuthology.orchestra.run.trial190.stdout: ceph-common x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 21 M 2026-02-25T08:08:53.549 INFO:teuthology.orchestra.run.trial190.stdout: ceph-selinux x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 25 k 2026-02-25T08:08:53.549 INFO:teuthology.orchestra.run.trial190.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2026-02-25T08:08:53.549 INFO:teuthology.orchestra.run.trial190.stdout: libarrow x86_64 9.0.0-15.el9 epel 4.4 M 2026-02-25T08:08:53.549 INFO:teuthology.orchestra.run.trial190.stdout: libarrow-doc noarch 9.0.0-15.el9 epel 25 k 2026-02-25T08:08:53.549 INFO:teuthology.orchestra.run.trial190.stdout: libcephfs2 x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 860 k 2026-02-25T08:08:53.549 INFO:teuthology.orchestra.run.trial190.stdout: libnbd x86_64 1.20.3-4.el9 appstream 164 k 2026-02-25T08:08:53.549 INFO:teuthology.orchestra.run.trial190.stdout: liboath x86_64 2.6.12-1.el9 epel 49 k 2026-02-25T08:08:53.549 INFO:teuthology.orchestra.run.trial190.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2026-02-25T08:08:53.549 INFO:teuthology.orchestra.run.trial190.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2026-02-25T08:08:53.549 INFO:teuthology.orchestra.run.trial190.stdout: libradosstriper1 x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 263 k 2026-02-25T08:08:53.549 INFO:teuthology.orchestra.run.trial190.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2026-02-25T08:08:53.549 INFO:teuthology.orchestra.run.trial190.stdout: librgw2 x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 5.6 M 2026-02-25T08:08:53.549 INFO:teuthology.orchestra.run.trial190.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2026-02-25T08:08:53.549 INFO:teuthology.orchestra.run.trial190.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2026-02-25T08:08:53.550 INFO:teuthology.orchestra.run.trial190.stdout: mailcap noarch 2.1.49-5.el9 baseos 33 k 2026-02-25T08:08:53.550 INFO:teuthology.orchestra.run.trial190.stdout: parquet-libs x86_64 9.0.0-15.el9 epel 838 k 2026-02-25T08:08:53.550 INFO:teuthology.orchestra.run.trial190.stdout: pciutils x86_64 3.7.0-7.el9 baseos 93 k 2026-02-25T08:08:53.550 INFO:teuthology.orchestra.run.trial190.stdout: python3-ceph-argparse x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 45 k 2026-02-25T08:08:53.550 INFO:teuthology.orchestra.run.trial190.stdout: python3-ceph-common x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 185 k 2026-02-25T08:08:53.550 INFO:teuthology.orchestra.run.trial190.stdout: python3-cephfs x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 204 k 2026-02-25T08:08:53.550 INFO:teuthology.orchestra.run.trial190.stdout: python3-rados x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 324 k 2026-02-25T08:08:53.550 INFO:teuthology.orchestra.run.trial190.stdout: python3-rbd x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 306 k 2026-02-25T08:08:53.550 INFO:teuthology.orchestra.run.trial190.stdout: python3-rgw x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 100 k 2026-02-25T08:08:53.550 INFO:teuthology.orchestra.run.trial190.stdout: qatlib x86_64 25.08.0-2.el9 appstream 240 k 2026-02-25T08:08:53.550 INFO:teuthology.orchestra.run.trial190.stdout: qatzip-libs x86_64 1.3.1-1.el9 appstream 66 k 2026-02-25T08:08:53.550 INFO:teuthology.orchestra.run.trial190.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2026-02-25T08:08:53.550 INFO:teuthology.orchestra.run.trial190.stdout: thrift x86_64 0.15.0-4.el9 epel 1.6 M 2026-02-25T08:08:53.550 INFO:teuthology.orchestra.run.trial190.stdout:Installing weak dependencies: 2026-02-25T08:08:53.550 INFO:teuthology.orchestra.run.trial190.stdout: libcephfs-daemon x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 37 k 2026-02-25T08:08:53.550 INFO:teuthology.orchestra.run.trial190.stdout: libcephfs-proxy2 x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 26 k 2026-02-25T08:08:53.550 INFO:teuthology.orchestra.run.trial190.stdout: nvme-cli x86_64 2.16-1.el9 baseos 1.2 M 2026-02-25T08:08:53.550 INFO:teuthology.orchestra.run.trial190.stdout: qatlib-service x86_64 25.08.0-2.el9 appstream 37 k 2026-02-25T08:08:53.550 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:08:53.550 INFO:teuthology.orchestra.run.trial190.stdout:Transaction Summary 2026-02-25T08:08:53.550 INFO:teuthology.orchestra.run.trial190.stdout:================================================================================ 2026-02-25T08:08:53.550 INFO:teuthology.orchestra.run.trial190.stdout:Install 35 Packages 2026-02-25T08:08:53.550 INFO:teuthology.orchestra.run.trial190.stdout:Upgrade 2 Packages 2026-02-25T08:08:53.550 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:08:53.550 INFO:teuthology.orchestra.run.trial190.stdout:Total download size: 61 M 2026-02-25T08:08:53.550 INFO:teuthology.orchestra.run.trial190.stdout:Downloading Packages: 2026-02-25T08:09:00.791 INFO:teuthology.orchestra.run.trial190.stdout:(1/37): ceph-base-20.3.0-5338.g6aeb6975.el9.x86 805 kB/s | 5.1 MB 00:06 2026-02-25T08:09:00.862 INFO:teuthology.orchestra.run.trial190.stdout:(2/37): ceph-selinux-20.3.0-5338.g6aeb6975.el9. 352 kB/s | 25 kB 00:00 2026-02-25T08:09:00.934 INFO:teuthology.orchestra.run.trial190.stdout:(3/37): libcephfs-daemon-20.3.0-5338.g6aeb6975. 514 kB/s | 37 kB 00:00 2026-02-25T08:09:01.010 INFO:teuthology.orchestra.run.trial190.stdout:(4/37): libcephfs-proxy2-20.3.0-5338.g6aeb6975. 340 kB/s | 26 kB 00:00 2026-02-25T08:09:02.784 INFO:teuthology.orchestra.run.trial190.stdout:(5/37): libcephfs2-20.3.0-5338.g6aeb6975.el9.x8 485 kB/s | 860 kB 00:01 2026-02-25T08:09:03.487 INFO:teuthology.orchestra.run.trial190.stdout:(6/37): libradosstriper1-20.3.0-5338.g6aeb6975. 374 kB/s | 263 kB 00:00 2026-02-25T08:09:19.370 INFO:teuthology.orchestra.run.trial190.stdout:(7/37): librgw2-20.3.0-5338.g6aeb6975.el9.x86_6 363 kB/s | 5.6 MB 00:15 2026-02-25T08:09:19.581 INFO:teuthology.orchestra.run.trial190.stdout:(8/37): python3-ceph-argparse-20.3.0-5338.g6aeb 214 kB/s | 45 kB 00:00 2026-02-25T08:09:20.278 INFO:teuthology.orchestra.run.trial190.stdout:(9/37): python3-ceph-common-20.3.0-5338.g6aeb69 265 kB/s | 185 kB 00:00 2026-02-25T08:09:20.575 INFO:teuthology.orchestra.run.trial190.stdout:(10/37): ceph-radosgw-20.3.0-5338.g6aeb6975.el9 436 kB/s | 11 MB 00:26 2026-02-25T08:09:21.060 INFO:teuthology.orchestra.run.trial190.stdout:(11/37): python3-cephfs-20.3.0-5338.g6aeb6975.e 260 kB/s | 204 kB 00:00 2026-02-25T08:09:21.630 INFO:teuthology.orchestra.run.trial190.stdout:(12/37): python3-rados-20.3.0-5338.g6aeb6975.el 307 kB/s | 324 kB 00:01 2026-02-25T08:09:21.965 INFO:teuthology.orchestra.run.trial190.stdout:(13/37): python3-rbd-20.3.0-5338.g6aeb6975.el9. 338 kB/s | 306 kB 00:00 2026-02-25T08:09:21.975 INFO:teuthology.orchestra.run.trial190.stdout:(14/37): mailcap-2.1.49-5.el9.noarch.rpm 3.0 MB/s | 33 kB 00:00 2026-02-25T08:09:21.996 INFO:teuthology.orchestra.run.trial190.stdout:(15/37): nvme-cli-2.16-1.el9.x86_64.rpm 58 MB/s | 1.2 MB 00:00 2026-02-25T08:09:22.001 INFO:teuthology.orchestra.run.trial190.stdout:(16/37): pciutils-3.7.0-7.el9.x86_64.rpm 18 MB/s | 93 kB 00:00 2026-02-25T08:09:22.056 INFO:teuthology.orchestra.run.trial190.stdout:(17/37): python3-rgw-20.3.0-5338.g6aeb6975.el9. 235 kB/s | 100 kB 00:00 2026-02-25T08:09:22.099 INFO:teuthology.orchestra.run.trial190.stdout:(18/37): boost-program-options-1.75.0-13.el9.x8 1.0 MB/s | 104 kB 00:00 2026-02-25T08:09:22.142 INFO:teuthology.orchestra.run.trial190.stdout:(19/37): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.6 MB/s | 160 kB 00:00 2026-02-25T08:09:22.162 INFO:teuthology.orchestra.run.trial190.stdout:(20/37): librabbitmq-0.11.0-7.el9.x86_64.rpm 2.1 MB/s | 45 kB 00:00 2026-02-25T08:09:22.174 INFO:teuthology.orchestra.run.trial190.stdout:(21/37): libnbd-1.20.3-4.el9.x86_64.rpm 1.4 MB/s | 164 kB 00:00 2026-02-25T08:09:22.218 INFO:teuthology.orchestra.run.trial190.stdout:(22/37): librdkafka-1.6.1-102.el9.x86_64.rpm 12 MB/s | 662 kB 00:00 2026-02-25T08:09:22.224 INFO:teuthology.orchestra.run.trial190.stdout:(23/37): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.8 MB/s | 292 kB 00:00 2026-02-25T08:09:22.242 INFO:teuthology.orchestra.run.trial190.stdout:(24/37): qatlib-25.08.0-2.el9.x86_64.rpm 9.9 MB/s | 240 kB 00:00 2026-02-25T08:09:22.245 INFO:teuthology.orchestra.run.trial190.stdout:(25/37): qatlib-service-25.08.0-2.el9.x86_64.rp 1.7 MB/s | 37 kB 00:00 2026-02-25T08:09:22.262 INFO:teuthology.orchestra.run.trial190.stdout:(26/37): qatzip-libs-1.3.1-1.el9.x86_64.rpm 3.2 MB/s | 66 kB 00:00 2026-02-25T08:09:22.779 INFO:teuthology.orchestra.run.trial190.stdout:(27/37): gperftools-libs-2.9.1-3.el9.x86_64.rpm 576 kB/s | 308 kB 00:00 2026-02-25T08:09:22.858 INFO:teuthology.orchestra.run.trial190.stdout:(28/37): libarrow-doc-9.0.0-15.el9.noarch.rpm 318 kB/s | 25 kB 00:00 2026-02-25T08:09:22.932 INFO:teuthology.orchestra.run.trial190.stdout:(29/37): liboath-2.6.12-1.el9.x86_64.rpm 661 kB/s | 49 kB 00:00 2026-02-25T08:09:23.011 INFO:teuthology.orchestra.run.trial190.stdout:(30/37): libunwind-1.6.2-1.el9.x86_64.rpm 854 kB/s | 67 kB 00:00 2026-02-25T08:09:23.041 INFO:teuthology.orchestra.run.trial190.stdout:(31/37): libarrow-9.0.0-15.el9.x86_64.rpm 5.7 MB/s | 4.4 MB 00:00 2026-02-25T08:09:23.174 INFO:teuthology.orchestra.run.trial190.stdout:(32/37): parquet-libs-9.0.0-15.el9.x86_64.rpm 5.0 MB/s | 838 kB 00:00 2026-02-25T08:09:23.234 INFO:teuthology.orchestra.run.trial190.stdout:(33/37): re2-20211101-20.el9.x86_64.rpm 996 kB/s | 191 kB 00:00 2026-02-25T08:09:23.326 INFO:teuthology.orchestra.run.trial190.stdout:(34/37): thrift-0.15.0-4.el9.x86_64.rpm 10 MB/s | 1.6 MB 00:00 2026-02-25T08:09:31.940 INFO:teuthology.orchestra.run.trial190.stdout:(35/37): librbd1-20.3.0-5338.g6aeb6975.el9.x86_ 320 kB/s | 2.7 MB 00:08 2026-02-25T08:09:34.453 INFO:teuthology.orchestra.run.trial190.stdout:(36/37): librados2-20.3.0-5338.g6aeb6975.el9.x8 300 kB/s | 3.3 MB 00:11 2026-02-25T08:09:37.118 INFO:teuthology.orchestra.run.trial190.stdout:(37/37): ceph-common-20.3.0-5338.g6aeb6975.el9. 498 kB/s | 21 MB 00:42 2026-02-25T08:09:37.121 INFO:teuthology.orchestra.run.trial190.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:09:37.121 INFO:teuthology.orchestra.run.trial190.stdout:Total 1.4 MB/s | 61 MB 00:43 2026-02-25T08:09:37.215 INFO:teuthology.orchestra.run.trial190.stdout:Running transaction check 2026-02-25T08:09:37.234 INFO:teuthology.orchestra.run.trial190.stdout:Transaction check succeeded. 2026-02-25T08:09:37.234 INFO:teuthology.orchestra.run.trial190.stdout:Running transaction test 2026-02-25T08:09:37.479 INFO:teuthology.orchestra.run.trial190.stdout:Transaction test succeeded. 2026-02-25T08:09:37.479 INFO:teuthology.orchestra.run.trial190.stdout:Running transaction 2026-02-25T08:09:37.780 INFO:teuthology.orchestra.run.trial190.stdout: Preparing : 1/1 2026-02-25T08:09:37.923 INFO:teuthology.orchestra.run.trial190.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/39 2026-02-25T08:09:37.927 INFO:teuthology.orchestra.run.trial190.stdout: Upgrading : librados2-2:20.3.0-5338.g6aeb6975.el9.x86_64 2/39 2026-02-25T08:09:37.976 INFO:teuthology.orchestra.run.trial190.stdout: Running scriptlet: librados2-2:20.3.0-5338.g6aeb6975.el9.x86_64 2/39 2026-02-25T08:09:37.977 INFO:teuthology.orchestra.run.trial190.stdout: Installing : libcephfs2-2:20.3.0-5338.g6aeb6975.el9.x86_64 3/39 2026-02-25T08:09:37.989 INFO:teuthology.orchestra.run.trial190.stdout: Running scriptlet: libcephfs2-2:20.3.0-5338.g6aeb6975.el9.x86_64 3/39 2026-02-25T08:09:37.991 INFO:teuthology.orchestra.run.trial190.stdout: Installing : libcephfs-daemon-2:20.3.0-5338.g6aeb6975.el9.x86_6 4/39 2026-02-25T08:09:37.991 INFO:teuthology.orchestra.run.trial190.stdout: Installing : libcephfs-proxy2-2:20.3.0-5338.g6aeb6975.el9.x86_6 5/39 2026-02-25T08:09:38.018 INFO:teuthology.orchestra.run.trial190.stdout: Running scriptlet: libcephfs-proxy2-2:20.3.0-5338.g6aeb6975.el9.x86_6 5/39 2026-02-25T08:09:38.021 INFO:teuthology.orchestra.run.trial190.stdout: Installing : python3-rados-2:20.3.0-5338.g6aeb6975.el9.x86_64 6/39 2026-02-25T08:09:38.030 INFO:teuthology.orchestra.run.trial190.stdout: Installing : liboath-2.6.12-1.el9.x86_64 7/39 2026-02-25T08:09:38.033 INFO:teuthology.orchestra.run.trial190.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/39 2026-02-25T08:09:38.036 INFO:teuthology.orchestra.run.trial190.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/39 2026-02-25T08:09:38.040 INFO:teuthology.orchestra.run.trial190.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/39 2026-02-25T08:09:38.051 INFO:teuthology.orchestra.run.trial190.stdout: Installing : python3-ceph-argparse-2:20.3.0-5338.g6aeb6975.el9. 11/39 2026-02-25T08:09:38.064 INFO:teuthology.orchestra.run.trial190.stdout: Installing : python3-cephfs-2:20.3.0-5338.g6aeb6975.el9.x86_64 12/39 2026-02-25T08:09:38.065 INFO:teuthology.orchestra.run.trial190.stdout: Installing : libradosstriper1-2:20.3.0-5338.g6aeb6975.el9.x86_6 13/39 2026-02-25T08:09:38.086 INFO:teuthology.orchestra.run.trial190.stdout: Running scriptlet: libradosstriper1-2:20.3.0-5338.g6aeb6975.el9.x86_6 13/39 2026-02-25T08:09:38.089 INFO:teuthology.orchestra.run.trial190.stdout: Installing : thrift-0.15.0-4.el9.x86_64 14/39 2026-02-25T08:09:38.119 INFO:teuthology.orchestra.run.trial190.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/39 2026-02-25T08:09:38.123 INFO:teuthology.orchestra.run.trial190.stdout: Installing : libarrow-9.0.0-15.el9.x86_64 16/39 2026-02-25T08:09:38.129 INFO:teuthology.orchestra.run.trial190.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/39 2026-02-25T08:09:38.131 INFO:teuthology.orchestra.run.trial190.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/39 2026-02-25T08:09:38.135 INFO:teuthology.orchestra.run.trial190.stdout: Installing : libarrow-doc-9.0.0-15.el9.noarch 19/39 2026-02-25T08:09:38.249 INFO:teuthology.orchestra.run.trial190.stdout: Installing : libnbd-1.20.3-4.el9.x86_64 20/39 2026-02-25T08:09:38.250 INFO:teuthology.orchestra.run.trial190.stdout: Upgrading : librbd1-2:20.3.0-5338.g6aeb6975.el9.x86_64 21/39 2026-02-25T08:09:38.278 INFO:teuthology.orchestra.run.trial190.stdout: Running scriptlet: librbd1-2:20.3.0-5338.g6aeb6975.el9.x86_64 21/39 2026-02-25T08:09:38.281 INFO:teuthology.orchestra.run.trial190.stdout: Installing : python3-rbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 22/39 2026-02-25T08:09:38.289 INFO:teuthology.orchestra.run.trial190.stdout: Installing : boost-program-options-1.75.0-13.el9.x86_64 23/39 2026-02-25T08:09:38.537 INFO:teuthology.orchestra.run.trial190.stdout: Installing : parquet-libs-9.0.0-15.el9.x86_64 24/39 2026-02-25T08:09:38.538 INFO:teuthology.orchestra.run.trial190.stdout: Installing : librgw2-2:20.3.0-5338.g6aeb6975.el9.x86_64 25/39 2026-02-25T08:09:38.554 INFO:teuthology.orchestra.run.trial190.stdout: Running scriptlet: librgw2-2:20.3.0-5338.g6aeb6975.el9.x86_64 25/39 2026-02-25T08:09:38.558 INFO:teuthology.orchestra.run.trial190.stdout: Installing : python3-rgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 26/39 2026-02-25T08:09:38.564 INFO:teuthology.orchestra.run.trial190.stdout: Installing : pciutils-3.7.0-7.el9.x86_64 27/39 2026-02-25T08:09:38.567 INFO:teuthology.orchestra.run.trial190.stdout: Installing : qatlib-25.08.0-2.el9.x86_64 28/39 2026-02-25T08:09:38.567 INFO:teuthology.orchestra.run.trial190.stdout: Installing : qatlib-service-25.08.0-2.el9.x86_64 29/39 2026-02-25T08:09:38.585 INFO:teuthology.orchestra.run.trial190.stdout: Running scriptlet: qatlib-service-25.08.0-2.el9.x86_64 29/39 2026-02-25T08:09:38.701 INFO:teuthology.orchestra.run.trial190.stdout: Installing : qatzip-libs-1.3.1-1.el9.x86_64 30/39 2026-02-25T08:09:38.704 INFO:teuthology.orchestra.run.trial190.stdout: Installing : nvme-cli-2.16-1.el9.x86_64 31/39 2026-02-25T08:09:38.983 INFO:teuthology.orchestra.run.trial190.stdout: Running scriptlet: nvme-cli-2.16-1.el9.x86_64 31/39 2026-02-25T08:09:38.983 INFO:teuthology.orchestra.run.trial190.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service → /usr/lib/systemd/system/nvmefc-boot-connections.service. 2026-02-25T08:09:38.983 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:09:39.026 INFO:teuthology.orchestra.run.trial190.stdout: Installing : mailcap-2.1.49-5.el9.noarch 32/39 2026-02-25T08:09:39.028 INFO:teuthology.orchestra.run.trial190.stdout: Installing : python3-ceph-common-2:20.3.0-5338.g6aeb6975.el9.x8 33/39 2026-02-25T08:09:39.969 INFO:teuthology.orchestra.run.trial190.stdout: Running scriptlet: ceph-common-2:20.3.0-5338.g6aeb6975.el9.x86_64 34/39 2026-02-25T08:09:39.972 INFO:teuthology.orchestra.run.trial190.stdout: Installing : ceph-common-2:20.3.0-5338.g6aeb6975.el9.x86_64 34/39 2026-02-25T08:09:40.236 INFO:teuthology.orchestra.run.trial190.stdout: Running scriptlet: ceph-common-2:20.3.0-5338.g6aeb6975.el9.x86_64 34/39 2026-02-25T08:09:40.239 INFO:teuthology.orchestra.run.trial190.stdout: Installing : ceph-base-2:20.3.0-5338.g6aeb6975.el9.x86_64 35/39 2026-02-25T08:09:40.280 INFO:teuthology.orchestra.run.trial190.stdout: Running scriptlet: ceph-base-2:20.3.0-5338.g6aeb6975.el9.x86_64 35/39 2026-02-25T08:09:40.280 INFO:teuthology.orchestra.run.trial190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2026-02-25T08:09:40.280 INFO:teuthology.orchestra.run.trial190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2026-02-25T08:09:40.280 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:09:40.284 INFO:teuthology.orchestra.run.trial190.stdout: Installing : ceph-selinux-2:20.3.0-5338.g6aeb6975.el9.x86_64 36/39 2026-02-25T08:09:44.711 INFO:teuthology.orchestra.run.trial190.stdout: Running scriptlet: ceph-selinux-2:20.3.0-5338.g6aeb6975.el9.x86_64 36/39 2026-02-25T08:09:44.712 INFO:teuthology.orchestra.run.trial190.stdout:skipping the directory /sys 2026-02-25T08:09:44.712 INFO:teuthology.orchestra.run.trial190.stdout:skipping the directory /proc 2026-02-25T08:09:44.712 INFO:teuthology.orchestra.run.trial190.stdout:skipping the directory /mnt 2026-02-25T08:09:44.712 INFO:teuthology.orchestra.run.trial190.stdout:skipping the directory /var/tmp 2026-02-25T08:09:44.712 INFO:teuthology.orchestra.run.trial190.stdout:skipping the directory /home 2026-02-25T08:09:44.712 INFO:teuthology.orchestra.run.trial190.stdout:skipping the directory /root 2026-02-25T08:09:44.712 INFO:teuthology.orchestra.run.trial190.stdout:skipping the directory /tmp 2026-02-25T08:09:44.712 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:09:45.203 INFO:teuthology.orchestra.run.trial190.stdout: Installing : ceph-radosgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 37/39 2026-02-25T08:09:45.226 INFO:teuthology.orchestra.run.trial190.stdout: Running scriptlet: ceph-radosgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 37/39 2026-02-25T08:09:45.226 INFO:teuthology.orchestra.run.trial190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T08:09:45.227 INFO:teuthology.orchestra.run.trial190.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2026-02-25T08:09:45.227 INFO:teuthology.orchestra.run.trial190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-25T08:09:45.227 INFO:teuthology.orchestra.run.trial190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-25T08:09:45.227 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:09:45.229 INFO:teuthology.orchestra.run.trial190.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 38/39 2026-02-25T08:09:45.241 INFO:teuthology.orchestra.run.trial190.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 38/39 2026-02-25T08:09:45.241 INFO:teuthology.orchestra.run.trial190.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 39/39 2026-02-25T08:09:45.295 INFO:teuthology.orchestra.run.trial190.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 39/39 2026-02-25T08:09:45.295 INFO:teuthology.orchestra.run.trial190.stdout:Creating group 'qat' with GID 992. 2026-02-25T08:09:45.295 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:09:45.588 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : ceph-base-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/39 2026-02-25T08:09:45.588 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : ceph-common-2:20.3.0-5338.g6aeb6975.el9.x86_64 2/39 2026-02-25T08:09:45.589 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : ceph-radosgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 3/39 2026-02-25T08:09:45.589 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : ceph-selinux-2:20.3.0-5338.g6aeb6975.el9.x86_64 4/39 2026-02-25T08:09:45.589 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : libcephfs-daemon-2:20.3.0-5338.g6aeb6975.el9.x86_6 5/39 2026-02-25T08:09:45.589 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : libcephfs-proxy2-2:20.3.0-5338.g6aeb6975.el9.x86_6 6/39 2026-02-25T08:09:45.589 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : libcephfs2-2:20.3.0-5338.g6aeb6975.el9.x86_64 7/39 2026-02-25T08:09:45.589 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : libradosstriper1-2:20.3.0-5338.g6aeb6975.el9.x86_6 8/39 2026-02-25T08:09:45.589 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : librgw2-2:20.3.0-5338.g6aeb6975.el9.x86_64 9/39 2026-02-25T08:09:45.589 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : python3-ceph-argparse-2:20.3.0-5338.g6aeb6975.el9. 10/39 2026-02-25T08:09:45.589 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : python3-ceph-common-2:20.3.0-5338.g6aeb6975.el9.x8 11/39 2026-02-25T08:09:45.589 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : python3-cephfs-2:20.3.0-5338.g6aeb6975.el9.x86_64 12/39 2026-02-25T08:09:45.589 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : python3-rados-2:20.3.0-5338.g6aeb6975.el9.x86_64 13/39 2026-02-25T08:09:45.590 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : python3-rbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 14/39 2026-02-25T08:09:45.590 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : python3-rgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 15/39 2026-02-25T08:09:45.590 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : mailcap-2.1.49-5.el9.noarch 16/39 2026-02-25T08:09:45.590 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : nvme-cli-2.16-1.el9.x86_64 17/39 2026-02-25T08:09:45.590 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : pciutils-3.7.0-7.el9.x86_64 18/39 2026-02-25T08:09:45.590 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : boost-program-options-1.75.0-13.el9.x86_64 19/39 2026-02-25T08:09:45.590 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : libnbd-1.20.3-4.el9.x86_64 20/39 2026-02-25T08:09:45.590 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 21/39 2026-02-25T08:09:45.590 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 22/39 2026-02-25T08:09:45.590 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 23/39 2026-02-25T08:09:45.590 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 24/39 2026-02-25T08:09:45.590 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : qatlib-25.08.0-2.el9.x86_64 25/39 2026-02-25T08:09:45.591 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : qatlib-service-25.08.0-2.el9.x86_64 26/39 2026-02-25T08:09:45.591 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : qatzip-libs-1.3.1-1.el9.x86_64 27/39 2026-02-25T08:09:45.591 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 28/39 2026-02-25T08:09:45.591 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : libarrow-9.0.0-15.el9.x86_64 29/39 2026-02-25T08:09:45.591 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : libarrow-doc-9.0.0-15.el9.noarch 30/39 2026-02-25T08:09:45.591 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : liboath-2.6.12-1.el9.x86_64 31/39 2026-02-25T08:09:45.591 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 32/39 2026-02-25T08:09:45.591 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : parquet-libs-9.0.0-15.el9.x86_64 33/39 2026-02-25T08:09:45.591 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : re2-1:20211101-20.el9.x86_64 34/39 2026-02-25T08:09:45.591 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : thrift-0.15.0-4.el9.x86_64 35/39 2026-02-25T08:09:45.591 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : librados2-2:20.3.0-5338.g6aeb6975.el9.x86_64 36/39 2026-02-25T08:09:45.591 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 37/39 2026-02-25T08:09:45.591 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : librbd1-2:20.3.0-5338.g6aeb6975.el9.x86_64 38/39 2026-02-25T08:09:45.654 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 39/39 2026-02-25T08:09:45.654 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:09:45.654 INFO:teuthology.orchestra.run.trial190.stdout:Upgraded: 2026-02-25T08:09:45.654 INFO:teuthology.orchestra.run.trial190.stdout: librados2-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:09:45.654 INFO:teuthology.orchestra.run.trial190.stdout: librbd1-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:09:45.654 INFO:teuthology.orchestra.run.trial190.stdout:Installed: 2026-02-25T08:09:45.654 INFO:teuthology.orchestra.run.trial190.stdout: boost-program-options-1.75.0-13.el9.x86_64 2026-02-25T08:09:45.655 INFO:teuthology.orchestra.run.trial190.stdout: ceph-base-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:09:45.655 INFO:teuthology.orchestra.run.trial190.stdout: ceph-common-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:09:45.655 INFO:teuthology.orchestra.run.trial190.stdout: ceph-radosgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:09:45.655 INFO:teuthology.orchestra.run.trial190.stdout: ceph-selinux-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:09:45.655 INFO:teuthology.orchestra.run.trial190.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2026-02-25T08:09:45.655 INFO:teuthology.orchestra.run.trial190.stdout: libarrow-9.0.0-15.el9.x86_64 2026-02-25T08:09:45.655 INFO:teuthology.orchestra.run.trial190.stdout: libarrow-doc-9.0.0-15.el9.noarch 2026-02-25T08:09:45.655 INFO:teuthology.orchestra.run.trial190.stdout: libcephfs-daemon-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:09:45.655 INFO:teuthology.orchestra.run.trial190.stdout: libcephfs-proxy2-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:09:45.655 INFO:teuthology.orchestra.run.trial190.stdout: libcephfs2-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:09:45.655 INFO:teuthology.orchestra.run.trial190.stdout: libnbd-1.20.3-4.el9.x86_64 2026-02-25T08:09:45.655 INFO:teuthology.orchestra.run.trial190.stdout: liboath-2.6.12-1.el9.x86_64 2026-02-25T08:09:45.656 INFO:teuthology.orchestra.run.trial190.stdout: libpmemobj-1.12.1-1.el9.x86_64 2026-02-25T08:09:45.656 INFO:teuthology.orchestra.run.trial190.stdout: librabbitmq-0.11.0-7.el9.x86_64 2026-02-25T08:09:45.656 INFO:teuthology.orchestra.run.trial190.stdout: libradosstriper1-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:09:45.656 INFO:teuthology.orchestra.run.trial190.stdout: librdkafka-1.6.1-102.el9.x86_64 2026-02-25T08:09:45.656 INFO:teuthology.orchestra.run.trial190.stdout: librgw2-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:09:45.656 INFO:teuthology.orchestra.run.trial190.stdout: libunwind-1.6.2-1.el9.x86_64 2026-02-25T08:09:45.656 INFO:teuthology.orchestra.run.trial190.stdout: lttng-ust-2.12.0-6.el9.x86_64 2026-02-25T08:09:45.656 INFO:teuthology.orchestra.run.trial190.stdout: mailcap-2.1.49-5.el9.noarch 2026-02-25T08:09:45.656 INFO:teuthology.orchestra.run.trial190.stdout: nvme-cli-2.16-1.el9.x86_64 2026-02-25T08:09:45.656 INFO:teuthology.orchestra.run.trial190.stdout: parquet-libs-9.0.0-15.el9.x86_64 2026-02-25T08:09:45.656 INFO:teuthology.orchestra.run.trial190.stdout: pciutils-3.7.0-7.el9.x86_64 2026-02-25T08:09:45.656 INFO:teuthology.orchestra.run.trial190.stdout: python3-ceph-argparse-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:09:45.657 INFO:teuthology.orchestra.run.trial190.stdout: python3-ceph-common-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:09:45.657 INFO:teuthology.orchestra.run.trial190.stdout: python3-cephfs-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:09:45.657 INFO:teuthology.orchestra.run.trial190.stdout: python3-rados-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:09:45.657 INFO:teuthology.orchestra.run.trial190.stdout: python3-rbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:09:45.657 INFO:teuthology.orchestra.run.trial190.stdout: python3-rgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:09:45.657 INFO:teuthology.orchestra.run.trial190.stdout: qatlib-25.08.0-2.el9.x86_64 2026-02-25T08:09:45.657 INFO:teuthology.orchestra.run.trial190.stdout: qatlib-service-25.08.0-2.el9.x86_64 2026-02-25T08:09:45.657 INFO:teuthology.orchestra.run.trial190.stdout: qatzip-libs-1.3.1-1.el9.x86_64 2026-02-25T08:09:45.657 INFO:teuthology.orchestra.run.trial190.stdout: re2-1:20211101-20.el9.x86_64 2026-02-25T08:09:45.657 INFO:teuthology.orchestra.run.trial190.stdout: thrift-0.15.0-4.el9.x86_64 2026-02-25T08:09:45.657 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:09:45.657 INFO:teuthology.orchestra.run.trial190.stdout:Complete! 2026-02-25T08:09:45.696 DEBUG:teuthology.orchestra.run.trial190:> sudo yum -y install ceph-test 2026-02-25T08:09:46.018 INFO:teuthology.orchestra.run.trial190.stdout:Last metadata expiration check: 0:00:54 ago on Wed 25 Feb 2026 08:08:52 AM UTC. 2026-02-25T08:09:46.100 INFO:teuthology.orchestra.run.trial190.stdout:Dependencies resolved. 2026-02-25T08:09:46.101 INFO:teuthology.orchestra.run.trial190.stdout:================================================================================ 2026-02-25T08:09:46.101 INFO:teuthology.orchestra.run.trial190.stdout: Package Arch Version Repository Size 2026-02-25T08:09:46.101 INFO:teuthology.orchestra.run.trial190.stdout:================================================================================ 2026-02-25T08:09:46.101 INFO:teuthology.orchestra.run.trial190.stdout:Installing: 2026-02-25T08:09:46.101 INFO:teuthology.orchestra.run.trial190.stdout: ceph-test x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 127 M 2026-02-25T08:09:46.102 INFO:teuthology.orchestra.run.trial190.stdout:Installing dependencies: 2026-02-25T08:09:46.102 INFO:teuthology.orchestra.run.trial190.stdout: hwloc-libs x86_64 2.4.1-6.el9 baseos 2.1 M 2026-02-25T08:09:46.102 INFO:teuthology.orchestra.run.trial190.stdout: libasan8 x86_64 15.2.1-7.el9 appstream 525 k 2026-02-25T08:09:46.102 INFO:teuthology.orchestra.run.trial190.stdout: libatomic x86_64 11.5.0-14.el9 baseos 23 k 2026-02-25T08:09:46.102 INFO:teuthology.orchestra.run.trial190.stdout: libcephsqlite x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 153 k 2026-02-25T08:09:46.102 INFO:teuthology.orchestra.run.trial190.stdout: libubsan x86_64 11.5.0-14.el9 appstream 176 k 2026-02-25T08:09:46.102 INFO:teuthology.orchestra.run.trial190.stdout: libxslt x86_64 1.1.34-12.el9 appstream 233 k 2026-02-25T08:09:46.102 INFO:teuthology.orchestra.run.trial190.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2026-02-25T08:09:46.102 INFO:teuthology.orchestra.run.trial190.stdout: protobuf x86_64 3.14.0-17.el9 appstream 1.0 M 2026-02-25T08:09:46.102 INFO:teuthology.orchestra.run.trial190.stdout: socat x86_64 1.7.4.1-8.el9 appstream 303 k 2026-02-25T08:09:46.102 INFO:teuthology.orchestra.run.trial190.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2026-02-25T08:09:46.102 INFO:teuthology.orchestra.run.trial190.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2026-02-25T08:09:46.103 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:09:46.103 INFO:teuthology.orchestra.run.trial190.stdout:Transaction Summary 2026-02-25T08:09:46.103 INFO:teuthology.orchestra.run.trial190.stdout:================================================================================ 2026-02-25T08:09:46.103 INFO:teuthology.orchestra.run.trial190.stdout:Install 12 Packages 2026-02-25T08:09:46.103 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:09:46.104 INFO:teuthology.orchestra.run.trial190.stdout:Total download size: 132 M 2026-02-25T08:09:46.104 INFO:teuthology.orchestra.run.trial190.stdout:Installed size: 988 M 2026-02-25T08:09:46.104 INFO:teuthology.orchestra.run.trial190.stdout:Downloading Packages: 2026-02-25T08:09:46.505 INFO:teuthology.orchestra.run.trial190.stdout:(1/12): hwloc-libs-2.4.1-6.el9.x86_64.rpm 12 MB/s | 2.1 MB 00:00 2026-02-25T08:09:46.527 INFO:teuthology.orchestra.run.trial190.stdout:(2/12): libatomic-11.5.0-14.el9.x86_64.rpm 1.0 MB/s | 23 kB 00:00 2026-02-25T08:09:46.550 INFO:teuthology.orchestra.run.trial190.stdout:(3/12): lksctp-tools-1.0.19-2.el9.x86_64.rpm 4.1 MB/s | 94 kB 00:00 2026-02-25T08:09:46.573 INFO:teuthology.orchestra.run.trial190.stdout:(4/12): libasan8-15.2.1-7.el9.x86_64.rpm 22 MB/s | 525 kB 00:00 2026-02-25T08:09:46.596 INFO:teuthology.orchestra.run.trial190.stdout:(5/12): libubsan-11.5.0-14.el9.x86_64.rpm 7.4 MB/s | 176 kB 00:00 2026-02-25T08:09:46.619 INFO:teuthology.orchestra.run.trial190.stdout:(6/12): libxslt-1.1.34-12.el9.x86_64.rpm 9.8 MB/s | 233 kB 00:00 2026-02-25T08:09:46.665 INFO:teuthology.orchestra.run.trial190.stdout:(7/12): protobuf-3.14.0-17.el9.x86_64.rpm 22 MB/s | 1.0 MB 00:00 2026-02-25T08:09:46.689 INFO:teuthology.orchestra.run.trial190.stdout:(8/12): socat-1.7.4.1-8.el9.x86_64.rpm 13 MB/s | 303 kB 00:00 2026-02-25T08:09:46.711 INFO:teuthology.orchestra.run.trial190.stdout:(9/12): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.8 MB/s | 64 kB 00:00 2026-02-25T08:09:46.851 INFO:teuthology.orchestra.run.trial190.stdout:(10/12): yaml-cpp-0.6.3-5.el9.x86_64.rpm 869 kB/s | 122 kB 00:00 2026-02-25T08:09:46.928 INFO:teuthology.orchestra.run.trial190.stdout:(11/12): libcephsqlite-20.3.0-5338.g6aeb6975.el 259 kB/s | 153 kB 00:00 2026-02-25T08:17:51.566 INFO:teuthology.orchestra.run.trial190.stdout:(12/12): ceph-test-20.3.0-5338.g6aeb6975.el9.x8 268 kB/s | 127 MB 08:05 2026-02-25T08:17:51.573 INFO:teuthology.orchestra.run.trial190.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:17:51.573 INFO:teuthology.orchestra.run.trial190.stdout:Total 278 kB/s | 132 MB 08:05 2026-02-25T08:17:51.641 INFO:teuthology.orchestra.run.trial190.stdout:Running transaction check 2026-02-25T08:17:51.652 INFO:teuthology.orchestra.run.trial190.stdout:Transaction check succeeded. 2026-02-25T08:17:51.652 INFO:teuthology.orchestra.run.trial190.stdout:Running transaction test 2026-02-25T08:17:52.083 INFO:teuthology.orchestra.run.trial190.stdout:Transaction test succeeded. 2026-02-25T08:17:52.083 INFO:teuthology.orchestra.run.trial190.stdout:Running transaction 2026-02-25T08:17:52.539 INFO:teuthology.orchestra.run.trial190.stdout: Preparing : 1/1 2026-02-25T08:17:52.548 INFO:teuthology.orchestra.run.trial190.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/12 2026-02-25T08:17:52.556 INFO:teuthology.orchestra.run.trial190.stdout: Installing : socat-1.7.4.1-8.el9.x86_64 2/12 2026-02-25T08:17:52.561 INFO:teuthology.orchestra.run.trial190.stdout: Installing : protobuf-3.14.0-17.el9.x86_64 3/12 2026-02-25T08:17:52.565 INFO:teuthology.orchestra.run.trial190.stdout: Installing : libxslt-1.1.34-12.el9.x86_64 4/12 2026-02-25T08:17:52.568 INFO:teuthology.orchestra.run.trial190.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 5/12 2026-02-25T08:17:52.572 INFO:teuthology.orchestra.run.trial190.stdout: Installing : libubsan-11.5.0-14.el9.x86_64 6/12 2026-02-25T08:17:52.573 INFO:teuthology.orchestra.run.trial190.stdout: Installing : libasan8-15.2.1-7.el9.x86_64 7/12 2026-02-25T08:17:52.589 INFO:teuthology.orchestra.run.trial190.stdout: Running scriptlet: libasan8-15.2.1-7.el9.x86_64 7/12 2026-02-25T08:17:52.592 INFO:teuthology.orchestra.run.trial190.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 8/12 2026-02-25T08:17:52.604 INFO:teuthology.orchestra.run.trial190.stdout: Installing : libatomic-11.5.0-14.el9.x86_64 9/12 2026-02-25T08:17:52.612 INFO:teuthology.orchestra.run.trial190.stdout: Installing : hwloc-libs-2.4.1-6.el9.x86_64 10/12 2026-02-25T08:17:52.613 INFO:teuthology.orchestra.run.trial190.stdout: Installing : libcephsqlite-2:20.3.0-5338.g6aeb6975.el9.x86_64 11/12 2026-02-25T08:17:59.563 INFO:teuthology.orchestra.run.trial190.stdout: Running scriptlet: libcephsqlite-2:20.3.0-5338.g6aeb6975.el9.x86_64 11/12 2026-02-25T08:17:59.581 INFO:teuthology.orchestra.run.trial190.stdout: Installing : ceph-test-2:20.3.0-5338.g6aeb6975.el9.x86_64 12/12 2026-02-25T08:18:00.368 INFO:teuthology.orchestra.run.trial190.stdout: Running scriptlet: ceph-test-2:20.3.0-5338.g6aeb6975.el9.x86_64 12/12 2026-02-25T08:18:00.368 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : ceph-test-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/12 2026-02-25T08:18:00.368 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : libcephsqlite-2:20.3.0-5338.g6aeb6975.el9.x86_64 2/12 2026-02-25T08:18:00.368 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : hwloc-libs-2.4.1-6.el9.x86_64 3/12 2026-02-25T08:18:00.368 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : libatomic-11.5.0-14.el9.x86_64 4/12 2026-02-25T08:18:00.368 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 5/12 2026-02-25T08:18:00.368 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : libasan8-15.2.1-7.el9.x86_64 6/12 2026-02-25T08:18:00.368 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : libubsan-11.5.0-14.el9.x86_64 7/12 2026-02-25T08:18:00.369 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : libxslt-1.1.34-12.el9.x86_64 8/12 2026-02-25T08:18:00.369 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : protobuf-3.14.0-17.el9.x86_64 9/12 2026-02-25T08:18:00.369 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : socat-1.7.4.1-8.el9.x86_64 10/12 2026-02-25T08:18:00.369 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 11/12 2026-02-25T08:18:00.425 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 12/12 2026-02-25T08:18:00.426 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:18:00.426 INFO:teuthology.orchestra.run.trial190.stdout:Installed: 2026-02-25T08:18:00.426 INFO:teuthology.orchestra.run.trial190.stdout: ceph-test-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:18:00.426 INFO:teuthology.orchestra.run.trial190.stdout: hwloc-libs-2.4.1-6.el9.x86_64 2026-02-25T08:18:00.426 INFO:teuthology.orchestra.run.trial190.stdout: libasan8-15.2.1-7.el9.x86_64 2026-02-25T08:18:00.426 INFO:teuthology.orchestra.run.trial190.stdout: libatomic-11.5.0-14.el9.x86_64 2026-02-25T08:18:00.426 INFO:teuthology.orchestra.run.trial190.stdout: libcephsqlite-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:18:00.426 INFO:teuthology.orchestra.run.trial190.stdout: libubsan-11.5.0-14.el9.x86_64 2026-02-25T08:18:00.426 INFO:teuthology.orchestra.run.trial190.stdout: libxslt-1.1.34-12.el9.x86_64 2026-02-25T08:18:00.426 INFO:teuthology.orchestra.run.trial190.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2026-02-25T08:18:00.426 INFO:teuthology.orchestra.run.trial190.stdout: protobuf-3.14.0-17.el9.x86_64 2026-02-25T08:18:00.427 INFO:teuthology.orchestra.run.trial190.stdout: socat-1.7.4.1-8.el9.x86_64 2026-02-25T08:18:00.427 INFO:teuthology.orchestra.run.trial190.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2026-02-25T08:18:00.427 INFO:teuthology.orchestra.run.trial190.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2026-02-25T08:18:00.427 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:18:00.427 INFO:teuthology.orchestra.run.trial190.stdout:Complete! 2026-02-25T08:18:00.467 DEBUG:teuthology.orchestra.run.trial190:> sudo yum -y install ceph 2026-02-25T08:18:00.780 INFO:teuthology.orchestra.run.trial190.stdout:Last metadata expiration check: 0:09:08 ago on Wed 25 Feb 2026 08:08:52 AM UTC. 2026-02-25T08:18:00.866 INFO:teuthology.orchestra.run.trial190.stdout:Dependencies resolved. 2026-02-25T08:18:00.869 INFO:teuthology.orchestra.run.trial190.stdout:======================================================================================= 2026-02-25T08:18:00.869 INFO:teuthology.orchestra.run.trial190.stdout: Package Arch Version Repository Size 2026-02-25T08:18:00.869 INFO:teuthology.orchestra.run.trial190.stdout:======================================================================================= 2026-02-25T08:18:00.869 INFO:teuthology.orchestra.run.trial190.stdout:Installing: 2026-02-25T08:18:00.869 INFO:teuthology.orchestra.run.trial190.stdout: ceph x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 6.5 k 2026-02-25T08:18:00.869 INFO:teuthology.orchestra.run.trial190.stdout:Installing dependencies: 2026-02-25T08:18:00.869 INFO:teuthology.orchestra.run.trial190.stdout: abseil-cpp x86_64 20211102.0-4.el9 epel 551 k 2026-02-25T08:18:00.869 INFO:teuthology.orchestra.run.trial190.stdout: ceph-grafana-dashboards noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 57 k 2026-02-25T08:18:00.869 INFO:teuthology.orchestra.run.trial190.stdout: ceph-mds x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 2.0 M 2026-02-25T08:18:00.869 INFO:teuthology.orchestra.run.trial190.stdout: ceph-mgr x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 937 k 2026-02-25T08:18:00.869 INFO:teuthology.orchestra.run.trial190.stdout: ceph-mgr-modules-core noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 297 k 2026-02-25T08:18:00.869 INFO:teuthology.orchestra.run.trial190.stdout: ceph-mon x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 4.9 M 2026-02-25T08:18:00.869 INFO:teuthology.orchestra.run.trial190.stdout: ceph-osd x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 163 k 2026-02-25T08:18:00.870 INFO:teuthology.orchestra.run.trial190.stdout: ceph-osd-classic x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 15 M 2026-02-25T08:18:00.870 INFO:teuthology.orchestra.run.trial190.stdout: ceph-osd-crimson x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 88 M 2026-02-25T08:18:00.870 INFO:teuthology.orchestra.run.trial190.stdout: ceph-prometheus-alerts noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 17 k 2026-02-25T08:18:00.870 INFO:teuthology.orchestra.run.trial190.stdout: cephadm noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 1.0 M 2026-02-25T08:18:00.870 INFO:teuthology.orchestra.run.trial190.stdout: cryptsetup x86_64 2.8.1-3.el9 baseos 351 k 2026-02-25T08:18:00.870 INFO:teuthology.orchestra.run.trial190.stdout: flexiblas x86_64 3.0.4-9.el9 appstream 30 k 2026-02-25T08:18:00.870 INFO:teuthology.orchestra.run.trial190.stdout: flexiblas-netlib x86_64 3.0.4-9.el9 appstream 3.0 M 2026-02-25T08:18:00.870 INFO:teuthology.orchestra.run.trial190.stdout: flexiblas-openblas-openmp x86_64 3.0.4-9.el9 appstream 15 k 2026-02-25T08:18:00.870 INFO:teuthology.orchestra.run.trial190.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2026-02-25T08:18:00.870 INFO:teuthology.orchestra.run.trial190.stdout: ledmon-libs x86_64 1.1.0-3.el9 baseos 40 k 2026-02-25T08:18:00.870 INFO:teuthology.orchestra.run.trial190.stdout: libconfig x86_64 1.7.2-9.el9 baseos 72 k 2026-02-25T08:18:00.870 INFO:teuthology.orchestra.run.trial190.stdout: libgfortran x86_64 11.5.0-14.el9 baseos 794 k 2026-02-25T08:18:00.870 INFO:teuthology.orchestra.run.trial190.stdout: libquadmath x86_64 11.5.0-14.el9 baseos 184 k 2026-02-25T08:18:00.870 INFO:teuthology.orchestra.run.trial190.stdout: libstoragemgmt x86_64 1.10.1-1.el9 appstream 246 k 2026-02-25T08:18:00.870 INFO:teuthology.orchestra.run.trial190.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2026-02-25T08:18:00.870 INFO:teuthology.orchestra.run.trial190.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2026-02-25T08:18:00.870 INFO:teuthology.orchestra.run.trial190.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2026-02-25T08:18:00.870 INFO:teuthology.orchestra.run.trial190.stdout: openblas x86_64 0.3.29-1.el9 appstream 42 k 2026-02-25T08:18:00.870 INFO:teuthology.orchestra.run.trial190.stdout: openblas-openmp x86_64 0.3.29-1.el9 appstream 5.3 M 2026-02-25T08:18:00.870 INFO:teuthology.orchestra.run.trial190.stdout: protobuf-compiler x86_64 3.14.0-17.el9 crb 862 k 2026-02-25T08:18:00.870 INFO:teuthology.orchestra.run.trial190.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2026-02-25T08:18:00.870 INFO:teuthology.orchestra.run.trial190.stdout: python3-autocommand noarch 2.2.2-8.el9 epel 29 k 2026-02-25T08:18:00.870 INFO:teuthology.orchestra.run.trial190.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2026-02-25T08:18:00.870 INFO:teuthology.orchestra.run.trial190.stdout: python3-backports-tarfile noarch 1.2.0-1.el9 epel 60 k 2026-02-25T08:18:00.870 INFO:teuthology.orchestra.run.trial190.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2026-02-25T08:18:00.870 INFO:teuthology.orchestra.run.trial190.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2026-02-25T08:18:00.870 INFO:teuthology.orchestra.run.trial190.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2026-02-25T08:18:00.870 INFO:teuthology.orchestra.run.trial190.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2026-02-25T08:18:00.870 INFO:teuthology.orchestra.run.trial190.stdout: python3-cheroot noarch 10.0.1-4.el9 epel 173 k 2026-02-25T08:18:00.870 INFO:teuthology.orchestra.run.trial190.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2026-02-25T08:18:00.870 INFO:teuthology.orchestra.run.trial190.stdout: python3-cryptography x86_64 36.0.1-5.el9 baseos 1.2 M 2026-02-25T08:18:00.870 INFO:teuthology.orchestra.run.trial190.stdout: python3-devel x86_64 3.9.25-3.el9 appstream 244 k 2026-02-25T08:18:00.870 INFO:teuthology.orchestra.run.trial190.stdout: python3-google-auth noarch 1:2.45.0-1.el9 epel 254 k 2026-02-25T08:18:00.870 INFO:teuthology.orchestra.run.trial190.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2026-02-25T08:18:00.870 INFO:teuthology.orchestra.run.trial190.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2026-02-25T08:18:00.870 INFO:teuthology.orchestra.run.trial190.stdout: python3-isodate noarch 0.6.1-3.el9 epel 56 k 2026-02-25T08:18:00.870 INFO:teuthology.orchestra.run.trial190.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2026-02-25T08:18:00.870 INFO:teuthology.orchestra.run.trial190.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2026-02-25T08:18:00.870 INFO:teuthology.orchestra.run.trial190.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2026-02-25T08:18:00.870 INFO:teuthology.orchestra.run.trial190.stdout: python3-jaraco-context noarch 6.0.1-3.el9 epel 20 k 2026-02-25T08:18:00.871 INFO:teuthology.orchestra.run.trial190.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2026-02-25T08:18:00.871 INFO:teuthology.orchestra.run.trial190.stdout: python3-jaraco-text noarch 4.0.0-2.el9 epel 26 k 2026-02-25T08:18:00.871 INFO:teuthology.orchestra.run.trial190.stdout: python3-jinja2 noarch 2.11.3-8.el9 appstream 249 k 2026-02-25T08:18:00.871 INFO:teuthology.orchestra.run.trial190.stdout: python3-jmespath noarch 1.0.1-1.el9 appstream 48 k 2026-02-25T08:18:00.871 INFO:teuthology.orchestra.run.trial190.stdout: python3-kubernetes noarch 1:26.1.0-3.el9 epel 1.0 M 2026-02-25T08:18:00.871 INFO:teuthology.orchestra.run.trial190.stdout: python3-libstoragemgmt x86_64 1.10.1-1.el9 appstream 177 k 2026-02-25T08:18:00.871 INFO:teuthology.orchestra.run.trial190.stdout: python3-lxml x86_64 4.6.5-3.el9 appstream 1.2 M 2026-02-25T08:18:00.871 INFO:teuthology.orchestra.run.trial190.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2026-02-25T08:18:00.871 INFO:teuthology.orchestra.run.trial190.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2026-02-25T08:18:00.871 INFO:teuthology.orchestra.run.trial190.stdout: python3-msgpack x86_64 1.0.3-2.el9 epel 86 k 2026-02-25T08:18:00.871 INFO:teuthology.orchestra.run.trial190.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2026-02-25T08:18:00.871 INFO:teuthology.orchestra.run.trial190.stdout: python3-numpy x86_64 1:1.23.5-2.el9 appstream 6.1 M 2026-02-25T08:18:00.871 INFO:teuthology.orchestra.run.trial190.stdout: python3-numpy-f2py x86_64 1:1.23.5-2.el9 appstream 442 k 2026-02-25T08:18:00.871 INFO:teuthology.orchestra.run.trial190.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2026-02-25T08:18:00.871 INFO:teuthology.orchestra.run.trial190.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2026-02-25T08:18:00.871 INFO:teuthology.orchestra.run.trial190.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2026-02-25T08:18:00.871 INFO:teuthology.orchestra.run.trial190.stdout: python3-protobuf noarch 3.14.0-17.el9 appstream 267 k 2026-02-25T08:18:00.871 INFO:teuthology.orchestra.run.trial190.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2026-02-25T08:18:00.871 INFO:teuthology.orchestra.run.trial190.stdout: python3-pyasn1 noarch 0.4.8-7.el9 appstream 157 k 2026-02-25T08:18:00.871 INFO:teuthology.orchestra.run.trial190.stdout: python3-pyasn1-modules noarch 0.4.8-7.el9 appstream 277 k 2026-02-25T08:18:00.871 INFO:teuthology.orchestra.run.trial190.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2026-02-25T08:18:00.871 INFO:teuthology.orchestra.run.trial190.stdout: python3-pyparsing noarch 2.4.7-9.el9 baseos 150 k 2026-02-25T08:18:00.871 INFO:teuthology.orchestra.run.trial190.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2026-02-25T08:18:00.871 INFO:teuthology.orchestra.run.trial190.stdout: python3-requests noarch 2.25.1-10.el9 baseos 126 k 2026-02-25T08:18:00.871 INFO:teuthology.orchestra.run.trial190.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2026-02-25T08:18:00.871 INFO:teuthology.orchestra.run.trial190.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2026-02-25T08:18:00.871 INFO:teuthology.orchestra.run.trial190.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2026-02-25T08:18:00.871 INFO:teuthology.orchestra.run.trial190.stdout: python3-scipy x86_64 1.9.3-2.el9 appstream 19 M 2026-02-25T08:18:00.871 INFO:teuthology.orchestra.run.trial190.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2026-02-25T08:18:00.871 INFO:teuthology.orchestra.run.trial190.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2026-02-25T08:18:00.871 INFO:teuthology.orchestra.run.trial190.stdout: python3-typing-extensions noarch 4.15.0-1.el9 epel 86 k 2026-02-25T08:18:00.871 INFO:teuthology.orchestra.run.trial190.stdout: python3-urllib3 noarch 1.26.5-7.el9 baseos 218 k 2026-02-25T08:18:00.871 INFO:teuthology.orchestra.run.trial190.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2026-02-25T08:18:00.871 INFO:teuthology.orchestra.run.trial190.stdout: python3-xmlsec x86_64 1.3.13-1.el9 epel 48 k 2026-02-25T08:18:00.871 INFO:teuthology.orchestra.run.trial190.stdout: python3-xmltodict noarch 0.12.0-15.el9 epel 22 k 2026-02-25T08:18:00.871 INFO:teuthology.orchestra.run.trial190.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2026-02-25T08:18:00.871 INFO:teuthology.orchestra.run.trial190.stdout: unzip x86_64 6.0-59.el9 baseos 182 k 2026-02-25T08:18:00.871 INFO:teuthology.orchestra.run.trial190.stdout: xmlsec1 x86_64 1.2.29-13.el9 appstream 189 k 2026-02-25T08:18:00.871 INFO:teuthology.orchestra.run.trial190.stdout: xmlsec1-openssl x86_64 1.2.29-13.el9 appstream 90 k 2026-02-25T08:18:00.871 INFO:teuthology.orchestra.run.trial190.stdout: zip x86_64 3.0-35.el9 baseos 266 k 2026-02-25T08:18:00.871 INFO:teuthology.orchestra.run.trial190.stdout:Installing weak dependencies: 2026-02-25T08:18:00.871 INFO:teuthology.orchestra.run.trial190.stdout: ceph-mgr-cephadm noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 184 k 2026-02-25T08:18:00.871 INFO:teuthology.orchestra.run.trial190.stdout: ceph-mgr-dashboard noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 51 M 2026-02-25T08:18:00.871 INFO:teuthology.orchestra.run.trial190.stdout: ceph-mgr-diskprediction-local noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 7.4 M 2026-02-25T08:18:00.872 INFO:teuthology.orchestra.run.trial190.stdout: ceph-mgr-k8sevents noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 22 k 2026-02-25T08:18:00.872 INFO:teuthology.orchestra.run.trial190.stdout: ceph-mgr-rook noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 50 k 2026-02-25T08:18:00.872 INFO:teuthology.orchestra.run.trial190.stdout: ceph-volume noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 301 k 2026-02-25T08:18:00.872 INFO:teuthology.orchestra.run.trial190.stdout: python3-influxdb noarch 5.3.1-1.el9 epel 139 k 2026-02-25T08:18:00.872 INFO:teuthology.orchestra.run.trial190.stdout: python3-saml noarch 1.16.0-1.el9 epel 125 k 2026-02-25T08:18:00.872 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:18:00.872 INFO:teuthology.orchestra.run.trial190.stdout:Transaction Summary 2026-02-25T08:18:00.872 INFO:teuthology.orchestra.run.trial190.stdout:======================================================================================= 2026-02-25T08:18:00.872 INFO:teuthology.orchestra.run.trial190.stdout:Install 96 Packages 2026-02-25T08:18:00.872 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:18:00.872 INFO:teuthology.orchestra.run.trial190.stdout:Total download size: 226 M 2026-02-25T08:18:00.872 INFO:teuthology.orchestra.run.trial190.stdout:Installed size: 1.5 G 2026-02-25T08:18:00.872 INFO:teuthology.orchestra.run.trial190.stdout:Downloading Packages: 2026-02-25T08:18:01.497 INFO:teuthology.orchestra.run.trial190.stdout:(1/96): ceph-20.3.0-5338.g6aeb6975.el9.x86_64.r 23 kB/s | 6.5 kB 00:00 2026-02-25T08:18:03.488 INFO:teuthology.orchestra.run.trial190.stdout:(2/96): ceph-mgr-20.3.0-5338.g6aeb6975.el9.x86_ 411 kB/s | 937 kB 00:02 2026-02-25T08:18:04.050 INFO:teuthology.orchestra.run.trial190.stdout:(3/96): ceph-osd-20.3.0-5338.g6aeb6975.el9.x86_ 291 kB/s | 163 kB 00:00 2026-02-25T08:18:10.782 INFO:teuthology.orchestra.run.trial190.stdout:(4/96): ceph-mds-20.3.0-5338.g6aeb6975.el9.x86_ 216 kB/s | 2.0 MB 00:09 2026-02-25T08:18:22.319 INFO:teuthology.orchestra.run.trial190.stdout:(5/96): ceph-mon-20.3.0-5338.g6aeb6975.el9.x86_ 239 kB/s | 4.9 MB 00:20 2026-02-25T08:18:22.765 INFO:teuthology.orchestra.run.trial190.stdout:(6/96): ceph-grafana-dashboards-20.3.0-5338.g6a 128 kB/s | 57 kB 00:00 2026-02-25T08:18:23.816 INFO:teuthology.orchestra.run.trial190.stdout:(7/96): ceph-mgr-cephadm-20.3.0-5338.g6aeb6975. 175 kB/s | 184 kB 00:01 2026-02-25T08:19:20.104 INFO:teuthology.orchestra.run.trial190.stdout:(8/96): ceph-osd-classic-20.3.0-5338.g6aeb6975. 199 kB/s | 15 MB 01:16 2026-02-25T08:19:59.344 INFO:teuthology.orchestra.run.trial190.stdout:(9/96): ceph-mgr-diskprediction-local-20.3.0-53 193 kB/s | 7.4 MB 00:39 2026-02-25T08:19:59.503 INFO:teuthology.orchestra.run.trial190.stdout:(10/96): ceph-mgr-k8sevents-20.3.0-5338.g6aeb69 141 kB/s | 22 kB 00:00 2026-02-25T08:20:01.036 INFO:teuthology.orchestra.run.trial190.stdout:(11/96): ceph-mgr-modules-core-20.3.0-5338.g6ae 193 kB/s | 297 kB 00:01 2026-02-25T08:20:01.457 INFO:teuthology.orchestra.run.trial190.stdout:(12/96): ceph-mgr-rook-20.3.0-5338.g6aeb6975.el 119 kB/s | 50 kB 00:00 2026-02-25T08:20:01.595 INFO:teuthology.orchestra.run.trial190.stdout:(13/96): ceph-prometheus-alerts-20.3.0-5338.g6a 126 kB/s | 17 kB 00:00 2026-02-25T08:20:03.140 INFO:teuthology.orchestra.run.trial190.stdout:(14/96): ceph-volume-20.3.0-5338.g6aeb6975.el9. 195 kB/s | 301 kB 00:01 2026-02-25T08:20:08.955 INFO:teuthology.orchestra.run.trial190.stdout:(15/96): cephadm-20.3.0-5338.g6aeb6975.el9.noar 179 kB/s | 1.0 MB 00:05 2026-02-25T08:20:09.493 INFO:teuthology.orchestra.run.trial190.stdout:(16/96): cryptsetup-2.8.1-3.el9.x86_64.rpm 652 kB/s | 351 kB 00:00 2026-02-25T08:20:09.577 INFO:teuthology.orchestra.run.trial190.stdout:(17/96): ledmon-libs-1.1.0-3.el9.x86_64.rpm 484 kB/s | 40 kB 00:00 2026-02-25T08:20:09.665 INFO:teuthology.orchestra.run.trial190.stdout:(18/96): libconfig-1.7.2-9.el9.x86_64.rpm 818 kB/s | 72 kB 00:00 2026-02-25T08:20:09.868 INFO:teuthology.orchestra.run.trial190.stdout:(19/96): libgfortran-11.5.0-14.el9.x86_64.rpm 3.8 MB/s | 794 kB 00:00 2026-02-25T08:20:09.956 INFO:teuthology.orchestra.run.trial190.stdout:(20/96): libquadmath-11.5.0-14.el9.x86_64.rpm 2.1 MB/s | 184 kB 00:00 2026-02-25T08:20:10.048 INFO:teuthology.orchestra.run.trial190.stdout:(21/96): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.7 MB/s | 253 kB 00:00 2026-02-25T08:20:10.198 INFO:teuthology.orchestra.run.trial190.stdout:(22/96): python3-cryptography-36.0.1-5.el9.x86_ 8.3 MB/s | 1.2 MB 00:00 2026-02-25T08:20:10.280 INFO:teuthology.orchestra.run.trial190.stdout:(23/96): python3-ply-3.11-14.el9.noarch.rpm 1.3 MB/s | 106 kB 00:00 2026-02-25T08:20:10.363 INFO:teuthology.orchestra.run.trial190.stdout:(24/96): python3-pycparser-2.20-6.el9.noarch.rp 1.6 MB/s | 135 kB 00:00 2026-02-25T08:20:10.447 INFO:teuthology.orchestra.run.trial190.stdout:(25/96): python3-pyparsing-2.4.7-9.el9.noarch.r 1.8 MB/s | 150 kB 00:00 2026-02-25T08:20:10.530 INFO:teuthology.orchestra.run.trial190.stdout:(26/96): python3-requests-2.25.1-10.el9.noarch. 1.5 MB/s | 126 kB 00:00 2026-02-25T08:20:10.617 INFO:teuthology.orchestra.run.trial190.stdout:(27/96): python3-urllib3-1.26.5-7.el9.noarch.rp 2.4 MB/s | 218 kB 00:00 2026-02-25T08:20:10.702 INFO:teuthology.orchestra.run.trial190.stdout:(28/96): unzip-6.0-59.el9.x86_64.rpm 2.1 MB/s | 182 kB 00:00 2026-02-25T08:20:10.790 INFO:teuthology.orchestra.run.trial190.stdout:(29/96): zip-3.0-35.el9.x86_64.rpm 2.9 MB/s | 266 kB 00:00 2026-02-25T08:20:10.833 INFO:teuthology.orchestra.run.trial190.stdout:(30/96): flexiblas-3.0.4-9.el9.x86_64.rpm 698 kB/s | 30 kB 00:00 2026-02-25T08:20:10.982 INFO:teuthology.orchestra.run.trial190.stdout:(31/96): flexiblas-netlib-3.0.4-9.el9.x86_64.rp 20 MB/s | 3.0 MB 00:00 2026-02-25T08:20:11.004 INFO:teuthology.orchestra.run.trial190.stdout:(32/96): flexiblas-openblas-openmp-3.0.4-9.el9. 686 kB/s | 15 kB 00:00 2026-02-25T08:20:11.028 INFO:teuthology.orchestra.run.trial190.stdout:(33/96): libstoragemgmt-1.10.1-1.el9.x86_64.rpm 10 MB/s | 246 kB 00:00 2026-02-25T08:20:11.051 INFO:teuthology.orchestra.run.trial190.stdout:(34/96): lua-5.4.4-4.el9.x86_64.rpm 7.9 MB/s | 188 kB 00:00 2026-02-25T08:20:11.073 INFO:teuthology.orchestra.run.trial190.stdout:(35/96): openblas-0.3.29-1.el9.x86_64.rpm 1.9 MB/s | 42 kB 00:00 2026-02-25T08:20:11.245 INFO:teuthology.orchestra.run.trial190.stdout:(36/96): openblas-openmp-0.3.29-1.el9.x86_64.rp 31 MB/s | 5.3 MB 00:00 2026-02-25T08:20:11.420 INFO:teuthology.orchestra.run.trial190.stdout:(37/96): python3-babel-2.9.1-2.el9.noarch.rpm 34 MB/s | 6.0 MB 00:00 2026-02-25T08:20:11.445 INFO:teuthology.orchestra.run.trial190.stdout:(38/96): python3-devel-3.9.25-3.el9.x86_64.rpm 9.6 MB/s | 244 kB 00:00 2026-02-25T08:20:11.470 INFO:teuthology.orchestra.run.trial190.stdout:(39/96): python3-jinja2-2.11.3-8.el9.noarch.rpm 10 MB/s | 249 kB 00:00 2026-02-25T08:20:11.492 INFO:teuthology.orchestra.run.trial190.stdout:(40/96): python3-jmespath-1.0.1-1.el9.noarch.rp 2.1 MB/s | 48 kB 00:00 2026-02-25T08:20:11.515 INFO:teuthology.orchestra.run.trial190.stdout:(41/96): python3-libstoragemgmt-1.10.1-1.el9.x8 7.5 MB/s | 177 kB 00:00 2026-02-25T08:20:11.562 INFO:teuthology.orchestra.run.trial190.stdout:(42/96): python3-lxml-4.6.5-3.el9.x86_64.rpm 26 MB/s | 1.2 MB 00:00 2026-02-25T08:20:11.585 INFO:teuthology.orchestra.run.trial190.stdout:(43/96): python3-markupsafe-1.1.1-12.el9.x86_64 1.5 MB/s | 35 kB 00:00 2026-02-25T08:20:11.776 INFO:teuthology.orchestra.run.trial190.stdout:(44/96): python3-numpy-1.23.5-2.el9.x86_64.rpm 32 MB/s | 6.1 MB 00:00 2026-02-25T08:20:11.802 INFO:teuthology.orchestra.run.trial190.stdout:(45/96): python3-numpy-f2py-1.23.5-2.el9.x86_64 17 MB/s | 442 kB 00:00 2026-02-25T08:20:11.825 INFO:teuthology.orchestra.run.trial190.stdout:(46/96): python3-packaging-20.9-5.el9.noarch.rp 3.3 MB/s | 77 kB 00:00 2026-02-25T08:20:11.848 INFO:teuthology.orchestra.run.trial190.stdout:(47/96): python3-protobuf-3.14.0-17.el9.noarch. 11 MB/s | 267 kB 00:00 2026-02-25T08:20:11.872 INFO:teuthology.orchestra.run.trial190.stdout:(48/96): python3-pyasn1-0.4.8-7.el9.noarch.rpm 6.5 MB/s | 157 kB 00:00 2026-02-25T08:20:11.897 INFO:teuthology.orchestra.run.trial190.stdout:(49/96): python3-pyasn1-modules-0.4.8-7.el9.noa 11 MB/s | 277 kB 00:00 2026-02-25T08:20:11.919 INFO:teuthology.orchestra.run.trial190.stdout:(50/96): python3-requests-oauthlib-1.3.0-12.el9 2.3 MB/s | 54 kB 00:00 2026-02-25T08:20:12.454 INFO:teuthology.orchestra.run.trial190.stdout:(51/96): python3-scipy-1.9.3-2.el9.x86_64.rpm 36 MB/s | 19 MB 00:00 2026-02-25T08:20:12.477 INFO:teuthology.orchestra.run.trial190.stdout:(52/96): python3-toml-0.10.2-6.el9.noarch.rpm 1.8 MB/s | 42 kB 00:00 2026-02-25T08:20:12.500 INFO:teuthology.orchestra.run.trial190.stdout:(53/96): xmlsec1-1.2.29-13.el9.x86_64.rpm 8.1 MB/s | 189 kB 00:00 2026-02-25T08:20:12.522 INFO:teuthology.orchestra.run.trial190.stdout:(54/96): xmlsec1-openssl-1.2.29-13.el9.x86_64.r 3.9 MB/s | 90 kB 00:00 2026-02-25T08:20:13.102 INFO:teuthology.orchestra.run.trial190.stdout:(55/96): lua-devel-5.4.4-4.el9.x86_64.rpm 38 kB/s | 22 kB 00:00 2026-02-25T08:20:14.327 INFO:teuthology.orchestra.run.trial190.stdout:(56/96): protobuf-compiler-3.14.0-17.el9.x86_64 704 kB/s | 862 kB 00:01 2026-02-25T08:20:14.526 INFO:teuthology.orchestra.run.trial190.stdout:(57/96): abseil-cpp-20211102.0-4.el9.x86_64.rpm 2.7 MB/s | 551 kB 00:00 2026-02-25T08:20:14.554 INFO:teuthology.orchestra.run.trial190.stdout:(58/96): grpc-data-1.46.7-10.el9.noarch.rpm 680 kB/s | 19 kB 00:00 2026-02-25T08:20:14.584 INFO:teuthology.orchestra.run.trial190.stdout:(59/96): luarocks-3.9.2-5.el9.noarch.rpm 4.9 MB/s | 151 kB 00:00 2026-02-25T08:20:14.617 INFO:teuthology.orchestra.run.trial190.stdout:(60/96): python3-asyncssh-2.13.2-5.el9.noarch.r 16 MB/s | 548 kB 00:00 2026-02-25T08:20:14.646 INFO:teuthology.orchestra.run.trial190.stdout:(61/96): python3-autocommand-2.2.2-8.el9.noarch 1.0 MB/s | 29 kB 00:00 2026-02-25T08:20:14.676 INFO:teuthology.orchestra.run.trial190.stdout:(62/96): python3-backports-tarfile-1.2.0-1.el9. 2.0 MB/s | 60 kB 00:00 2026-02-25T08:20:14.705 INFO:teuthology.orchestra.run.trial190.stdout:(63/96): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.4 MB/s | 43 kB 00:00 2026-02-25T08:20:14.734 INFO:teuthology.orchestra.run.trial190.stdout:(64/96): python3-cachetools-4.2.4-1.el9.noarch. 1.1 MB/s | 32 kB 00:00 2026-02-25T08:20:14.763 INFO:teuthology.orchestra.run.trial190.stdout:(65/96): python3-certifi-2023.05.07-4.el9.noarc 488 kB/s | 14 kB 00:00 2026-02-25T08:20:14.795 INFO:teuthology.orchestra.run.trial190.stdout:(66/96): python3-cheroot-10.0.1-4.el9.noarch.rp 5.4 MB/s | 173 kB 00:00 2026-02-25T08:20:14.831 INFO:teuthology.orchestra.run.trial190.stdout:(67/96): python3-cherrypy-18.6.1-2.el9.noarch.r 9.6 MB/s | 358 kB 00:00 2026-02-25T08:20:14.863 INFO:teuthology.orchestra.run.trial190.stdout:(68/96): python3-google-auth-2.45.0-1.el9.noarc 7.9 MB/s | 254 kB 00:00 2026-02-25T08:20:14.959 INFO:teuthology.orchestra.run.trial190.stdout:(69/96): python3-grpcio-1.46.7-10.el9.x86_64.rp 21 MB/s | 2.0 MB 00:00 2026-02-25T08:20:14.991 INFO:teuthology.orchestra.run.trial190.stdout:(70/96): python3-grpcio-tools-1.46.7-10.el9.x86 4.5 MB/s | 144 kB 00:00 2026-02-25T08:20:15.022 INFO:teuthology.orchestra.run.trial190.stdout:(71/96): python3-influxdb-5.3.1-1.el9.noarch.rp 4.3 MB/s | 139 kB 00:00 2026-02-25T08:20:15.052 INFO:teuthology.orchestra.run.trial190.stdout:(72/96): python3-isodate-0.6.1-3.el9.noarch.rpm 1.8 MB/s | 56 kB 00:00 2026-02-25T08:20:15.081 INFO:teuthology.orchestra.run.trial190.stdout:(73/96): python3-jaraco-8.2.1-3.el9.noarch.rpm 367 kB/s | 11 kB 00:00 2026-02-25T08:20:15.111 INFO:teuthology.orchestra.run.trial190.stdout:(74/96): python3-jaraco-classes-3.2.1-5.el9.noa 604 kB/s | 18 kB 00:00 2026-02-25T08:20:15.140 INFO:teuthology.orchestra.run.trial190.stdout:(75/96): python3-jaraco-collections-3.0.0-8.el9 789 kB/s | 23 kB 00:00 2026-02-25T08:20:15.169 INFO:teuthology.orchestra.run.trial190.stdout:(76/96): python3-jaraco-context-6.0.1-3.el9.noa 681 kB/s | 20 kB 00:00 2026-02-25T08:20:15.198 INFO:teuthology.orchestra.run.trial190.stdout:(77/96): python3-jaraco-functools-3.5.0-2.el9.n 673 kB/s | 19 kB 00:00 2026-02-25T08:20:15.227 INFO:teuthology.orchestra.run.trial190.stdout:(78/96): python3-jaraco-text-4.0.0-2.el9.noarch 906 kB/s | 26 kB 00:00 2026-02-25T08:20:15.261 INFO:teuthology.orchestra.run.trial190.stdout:(79/96): python3-kubernetes-26.1.0-3.el9.noarch 30 MB/s | 1.0 MB 00:00 2026-02-25T08:20:15.291 INFO:teuthology.orchestra.run.trial190.stdout:(80/96): python3-more-itertools-8.12.0-2.el9.no 2.6 MB/s | 79 kB 00:00 2026-02-25T08:20:15.321 INFO:teuthology.orchestra.run.trial190.stdout:(81/96): python3-msgpack-1.0.3-2.el9.x86_64.rpm 2.8 MB/s | 86 kB 00:00 2026-02-25T08:20:15.351 INFO:teuthology.orchestra.run.trial190.stdout:(82/96): python3-natsort-7.1.1-5.el9.noarch.rpm 1.9 MB/s | 58 kB 00:00 2026-02-25T08:20:15.380 INFO:teuthology.orchestra.run.trial190.stdout:(83/96): python3-portend-3.1.0-2.el9.noarch.rpm 572 kB/s | 16 kB 00:00 2026-02-25T08:20:15.410 INFO:teuthology.orchestra.run.trial190.stdout:(84/96): python3-pyOpenSSL-21.0.0-1.el9.noarch. 3.0 MB/s | 90 kB 00:00 2026-02-25T08:20:15.439 INFO:teuthology.orchestra.run.trial190.stdout:(85/96): python3-repoze-lru-0.7-16.el9.noarch.r 1.0 MB/s | 31 kB 00:00 2026-02-25T08:20:15.470 INFO:teuthology.orchestra.run.trial190.stdout:(86/96): python3-routes-2.5.1-5.el9.noarch.rpm 5.9 MB/s | 188 kB 00:00 2026-02-25T08:20:15.500 INFO:teuthology.orchestra.run.trial190.stdout:(87/96): python3-rsa-4.9-2.el9.noarch.rpm 1.9 MB/s | 59 kB 00:00 2026-02-25T08:20:15.531 INFO:teuthology.orchestra.run.trial190.stdout:(88/96): python3-saml-1.16.0-1.el9.noarch.rpm 4.0 MB/s | 125 kB 00:00 2026-02-25T08:20:15.560 INFO:teuthology.orchestra.run.trial190.stdout:(89/96): python3-tempora-5.0.0-2.el9.noarch.rpm 1.2 MB/s | 36 kB 00:00 2026-02-25T08:20:15.590 INFO:teuthology.orchestra.run.trial190.stdout:(90/96): python3-typing-extensions-4.15.0-1.el9 2.8 MB/s | 86 kB 00:00 2026-02-25T08:20:15.620 INFO:teuthology.orchestra.run.trial190.stdout:(91/96): python3-websocket-client-1.2.3-2.el9.n 3.0 MB/s | 90 kB 00:00 2026-02-25T08:20:15.650 INFO:teuthology.orchestra.run.trial190.stdout:(92/96): python3-xmlsec-1.3.13-1.el9.x86_64.rpm 1.6 MB/s | 48 kB 00:00 2026-02-25T08:20:15.679 INFO:teuthology.orchestra.run.trial190.stdout:(93/96): python3-xmltodict-0.12.0-15.el9.noarch 762 kB/s | 22 kB 00:00 2026-02-25T08:20:15.708 INFO:teuthology.orchestra.run.trial190.stdout:(94/96): python3-zc-lockfile-2.0-10.el9.noarch. 690 kB/s | 20 kB 00:00 2026-02-25T08:21:59.498 INFO:teuthology.orchestra.run.trial190.stdout:(95/96): ceph-mgr-dashboard-20.3.0-5338.g6aeb69 241 kB/s | 51 MB 03:35 2026-02-25T08:22:32.920 INFO:teuthology.orchestra.run.trial190.stdout:(96/96): ceph-osd-crimson-20.3.0-5338.g6aeb6975 342 kB/s | 88 MB 04:22 2026-02-25T08:22:32.923 INFO:teuthology.orchestra.run.trial190.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:22:32.923 INFO:teuthology.orchestra.run.trial190.stdout:Total 849 kB/s | 226 MB 04:32 2026-02-25T08:22:33.290 INFO:teuthology.orchestra.run.trial190.stdout:Running transaction check 2026-02-25T08:22:33.329 INFO:teuthology.orchestra.run.trial190.stdout:Transaction check succeeded. 2026-02-25T08:22:33.329 INFO:teuthology.orchestra.run.trial190.stdout:Running transaction test 2026-02-25T08:22:34.130 INFO:teuthology.orchestra.run.trial190.stdout:Transaction test succeeded. 2026-02-25T08:22:34.130 INFO:teuthology.orchestra.run.trial190.stdout:Running transaction 2026-02-25T08:22:34.956 INFO:teuthology.orchestra.run.trial190.stdout: Preparing : 1/1 2026-02-25T08:22:34.958 INFO:teuthology.orchestra.run.trial190.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/96 2026-02-25T08:22:34.961 INFO:teuthology.orchestra.run.trial190.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/96 2026-02-25T08:22:34.983 INFO:teuthology.orchestra.run.trial190.stdout: Installing : xmlsec1-1.2.29-13.el9.x86_64 3/96 2026-02-25T08:22:35.015 INFO:teuthology.orchestra.run.trial190.stdout: Installing : python3-pyasn1-0.4.8-7.el9.noarch 4/96 2026-02-25T08:22:35.021 INFO:teuthology.orchestra.run.trial190.stdout: Installing : python3-lxml-4.6.5-3.el9.x86_64 5/96 2026-02-25T08:22:35.024 INFO:teuthology.orchestra.run.trial190.stdout: Installing : lua-5.4.4-4.el9.x86_64 6/96 2026-02-25T08:22:35.028 INFO:teuthology.orchestra.run.trial190.stdout: Installing : flexiblas-3.0.4-9.el9.x86_64 7/96 2026-02-25T08:22:35.051 INFO:teuthology.orchestra.run.trial190.stdout: Installing : unzip-6.0-59.el9.x86_64 8/96 2026-02-25T08:22:35.063 INFO:teuthology.orchestra.run.trial190.stdout: Installing : python3-urllib3-1.26.5-7.el9.noarch 9/96 2026-02-25T08:22:35.067 INFO:teuthology.orchestra.run.trial190.stdout: Installing : python3-requests-2.25.1-10.el9.noarch 10/96 2026-02-25T08:22:35.073 INFO:teuthology.orchestra.run.trial190.stdout: Installing : libquadmath-11.5.0-14.el9.x86_64 11/96 2026-02-25T08:22:35.075 INFO:teuthology.orchestra.run.trial190.stdout: Installing : libgfortran-11.5.0-14.el9.x86_64 12/96 2026-02-25T08:22:35.086 INFO:teuthology.orchestra.run.trial190.stdout: Installing : ledmon-libs-1.1.0-3.el9.x86_64 13/96 2026-02-25T08:22:35.092 INFO:teuthology.orchestra.run.trial190.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 14/96 2026-02-25T08:22:35.116 INFO:teuthology.orchestra.run.trial190.stdout: Installing : zip-3.0-35.el9.x86_64 15/96 2026-02-25T08:22:35.119 INFO:teuthology.orchestra.run.trial190.stdout: Installing : luarocks-3.9.2-5.el9.noarch 16/96 2026-02-25T08:22:35.169 INFO:teuthology.orchestra.run.trial190.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 17/96 2026-02-25T08:22:35.181 INFO:teuthology.orchestra.run.trial190.stdout: Installing : python3-pyasn1-modules-0.4.8-7.el9.noarch 18/96 2026-02-25T08:22:35.184 INFO:teuthology.orchestra.run.trial190.stdout: Installing : python3-rsa-4.9-2.el9.noarch 19/96 2026-02-25T08:22:35.188 INFO:teuthology.orchestra.run.trial190.stdout: Installing : xmlsec1-openssl-1.2.29-13.el9.x86_64 20/96 2026-02-25T08:22:35.194 INFO:teuthology.orchestra.run.trial190.stdout: Installing : python3-xmlsec-1.3.13-1.el9.x86_64 21/96 2026-02-25T08:22:35.198 INFO:teuthology.orchestra.run.trial190.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 22/96 2026-02-25T08:22:35.201 INFO:teuthology.orchestra.run.trial190.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 23/96 2026-02-25T08:22:35.215 INFO:teuthology.orchestra.run.trial190.stdout: Installing : python3-xmltodict-0.12.0-15.el9.noarch 24/96 2026-02-25T08:22:35.220 INFO:teuthology.orchestra.run.trial190.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 25/96 2026-02-25T08:22:35.225 INFO:teuthology.orchestra.run.trial190.stdout: Installing : python3-typing-extensions-4.15.0-1.el9.noarch 26/96 2026-02-25T08:22:35.236 INFO:teuthology.orchestra.run.trial190.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 27/96 2026-02-25T08:22:35.245 INFO:teuthology.orchestra.run.trial190.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 28/96 2026-02-25T08:22:35.251 INFO:teuthology.orchestra.run.trial190.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 29/96 2026-02-25T08:22:35.270 INFO:teuthology.orchestra.run.trial190.stdout: Installing : python3-msgpack-1.0.3-2.el9.x86_64 30/96 2026-02-25T08:22:35.281 INFO:teuthology.orchestra.run.trial190.stdout: Installing : python3-influxdb-5.3.1-1.el9.noarch 31/96 2026-02-25T08:22:35.296 INFO:teuthology.orchestra.run.trial190.stdout: Installing : python3-isodate-0.6.1-3.el9.noarch 32/96 2026-02-25T08:22:35.301 INFO:teuthology.orchestra.run.trial190.stdout: Installing : python3-saml-1.16.0-1.el9.noarch 33/96 2026-02-25T08:22:35.308 INFO:teuthology.orchestra.run.trial190.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 34/96 2026-02-25T08:22:35.348 INFO:teuthology.orchestra.run.trial190.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 35/96 2026-02-25T08:22:35.667 INFO:teuthology.orchestra.run.trial190.stdout: Installing : python3-google-auth-1:2.45.0-1.el9.noarch 36/96 2026-02-25T08:22:35.677 INFO:teuthology.orchestra.run.trial190.stdout: Installing : python3-kubernetes-1:26.1.0-3.el9.noarch 37/96 2026-02-25T08:22:35.681 INFO:teuthology.orchestra.run.trial190.stdout: Installing : python3-backports-tarfile-1.2.0-1.el9.noarch 38/96 2026-02-25T08:22:35.686 INFO:teuthology.orchestra.run.trial190.stdout: Installing : python3-jaraco-context-6.0.1-3.el9.noarch 39/96 2026-02-25T08:22:35.688 INFO:teuthology.orchestra.run.trial190.stdout: Installing : python3-autocommand-2.2.2-8.el9.noarch 40/96 2026-02-25T08:22:35.716 INFO:teuthology.orchestra.run.trial190.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 41/96 2026-02-25T08:22:35.760 INFO:teuthology.orchestra.run.trial190.stdout: Installing : abseil-cpp-20211102.0-4.el9.x86_64 42/96 2026-02-25T08:22:35.768 INFO:teuthology.orchestra.run.trial190.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 43/96 2026-02-25T08:22:35.773 INFO:teuthology.orchestra.run.trial190.stdout: Installing : protobuf-compiler-3.14.0-17.el9.x86_64 44/96 2026-02-25T08:22:35.777 INFO:teuthology.orchestra.run.trial190.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 45/96 2026-02-25T08:22:35.782 INFO:teuthology.orchestra.run.trial190.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 46/96 2026-02-25T08:22:35.786 INFO:teuthology.orchestra.run.trial190.stdout: Installing : python3-jaraco-text-4.0.0-2.el9.noarch 47/96 2026-02-25T08:22:35.793 INFO:teuthology.orchestra.run.trial190.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 48/96 2026-02-25T08:22:35.796 INFO:teuthology.orchestra.run.trial190.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 49/96 2026-02-25T08:22:35.823 INFO:teuthology.orchestra.run.trial190.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 50/96 2026-02-25T08:22:35.833 INFO:teuthology.orchestra.run.trial190.stdout: Installing : python3-protobuf-3.14.0-17.el9.noarch 51/96 2026-02-25T08:22:35.839 INFO:teuthology.orchestra.run.trial190.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 52/96 2026-02-25T08:22:35.846 INFO:teuthology.orchestra.run.trial190.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 53/96 2026-02-25T08:22:35.879 INFO:teuthology.orchestra.run.trial190.stdout: Installing : python3-jmespath-1.0.1-1.el9.noarch 54/96 2026-02-25T08:22:36.100 INFO:teuthology.orchestra.run.trial190.stdout: Installing : python3-devel-3.9.25-3.el9.x86_64 55/96 2026-02-25T08:22:36.123 INFO:teuthology.orchestra.run.trial190.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 56/96 2026-02-25T08:22:36.127 INFO:teuthology.orchestra.run.trial190.stdout: Installing : python3-jinja2-2.11.3-8.el9.noarch 57/96 2026-02-25T08:22:36.185 INFO:teuthology.orchestra.run.trial190.stdout: Installing : openblas-0.3.29-1.el9.x86_64 58/96 2026-02-25T08:22:36.187 INFO:teuthology.orchestra.run.trial190.stdout: Installing : openblas-openmp-0.3.29-1.el9.x86_64 59/96 2026-02-25T08:22:36.208 INFO:teuthology.orchestra.run.trial190.stdout: Installing : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 60/96 2026-02-25T08:22:36.527 INFO:teuthology.orchestra.run.trial190.stdout: Installing : flexiblas-netlib-3.0.4-9.el9.x86_64 61/96 2026-02-25T08:22:36.599 INFO:teuthology.orchestra.run.trial190.stdout: Installing : python3-numpy-1:1.23.5-2.el9.x86_64 62/96 2026-02-25T08:22:37.258 INFO:teuthology.orchestra.run.trial190.stdout: Installing : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 63/96 2026-02-25T08:22:37.275 INFO:teuthology.orchestra.run.trial190.stdout: Installing : python3-scipy-1.9.3-2.el9.x86_64 64/96 2026-02-25T08:22:37.284 INFO:teuthology.orchestra.run.trial190.stdout: Installing : python3-pyparsing-2.4.7-9.el9.noarch 65/96 2026-02-25T08:22:37.291 INFO:teuthology.orchestra.run.trial190.stdout: Installing : python3-packaging-20.9-5.el9.noarch 66/96 2026-02-25T08:22:37.304 INFO:teuthology.orchestra.run.trial190.stdout: Installing : python3-ply-3.11-14.el9.noarch 67/96 2026-02-25T08:22:37.320 INFO:teuthology.orchestra.run.trial190.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 68/96 2026-02-25T08:22:37.394 INFO:teuthology.orchestra.run.trial190.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 69/96 2026-02-25T08:22:37.403 INFO:teuthology.orchestra.run.trial190.stdout: Installing : python3-cryptography-36.0.1-5.el9.x86_64 70/96 2026-02-25T08:22:37.427 INFO:teuthology.orchestra.run.trial190.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 71/96 2026-02-25T08:22:37.456 INFO:teuthology.orchestra.run.trial190.stdout: Installing : python3-cheroot-10.0.1-4.el9.noarch 72/96 2026-02-25T08:22:37.506 INFO:teuthology.orchestra.run.trial190.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 73/96 2026-02-25T08:22:37.512 INFO:teuthology.orchestra.run.trial190.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 74/96 2026-02-25T08:22:37.515 INFO:teuthology.orchestra.run.trial190.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 75/96 2026-02-25T08:22:37.517 INFO:teuthology.orchestra.run.trial190.stdout: Installing : libconfig-1.7.2-9.el9.x86_64 76/96 2026-02-25T08:22:37.535 INFO:teuthology.orchestra.run.trial190.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-25T08:22:37.535 INFO:teuthology.orchestra.run.trial190.stdout:Creating group 'libstoragemgmt' with GID 991. 2026-02-25T08:22:37.535 INFO:teuthology.orchestra.run.trial190.stdout:Creating user 'libstoragemgmt' (daemon account for libstoragemgmt) with UID 991 and GID 991. 2026-02-25T08:22:37.535 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:22:37.544 INFO:teuthology.orchestra.run.trial190.stdout: Installing : libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-25T08:22:37.569 INFO:teuthology.orchestra.run.trial190.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-25T08:22:37.570 INFO:teuthology.orchestra.run.trial190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/libstoragemgmt.service → /usr/lib/systemd/system/libstoragemgmt.service. 2026-02-25T08:22:37.570 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:22:37.603 INFO:teuthology.orchestra.run.trial190.stdout: Installing : python3-libstoragemgmt-1.10.1-1.el9.x86_64 78/96 2026-02-25T08:22:38.188 INFO:teuthology.orchestra.run.trial190.stdout: Installing : cryptsetup-2.8.1-3.el9.x86_64 79/96 2026-02-25T08:22:43.589 INFO:teuthology.orchestra.run.trial190.stdout: Installing : ceph-osd-classic-2:20.3.0-5338.g6aeb6975.el9.x86_6 80/96 2026-02-25T08:22:43.652 INFO:teuthology.orchestra.run.trial190.stdout: Installing : ceph-osd-crimson-2:20.3.0-5338.g6aeb6975.el9.x86_6 81/96 2026-02-25T08:22:43.654 INFO:teuthology.orchestra.run.trial190.stdout: Installing : ceph-volume-2:20.3.0-5338.g6aeb6975.el9.noarch 82/96 2026-02-25T08:22:43.668 INFO:teuthology.orchestra.run.trial190.stdout: Running scriptlet: ceph-volume-2:20.3.0-5338.g6aeb6975.el9.noarch 82/96 2026-02-25T08:22:43.668 INFO:teuthology.orchestra.run.trial190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T08:22:43.668 INFO:teuthology.orchestra.run.trial190.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2026-02-25T08:22:43.668 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:22:43.680 INFO:teuthology.orchestra.run.trial190.stdout: Installing : ceph-osd-2:20.3.0-5338.g6aeb6975.el9.x86_64 83/96 2026-02-25T08:22:43.704 INFO:teuthology.orchestra.run.trial190.stdout: Running scriptlet: ceph-osd-2:20.3.0-5338.g6aeb6975.el9.x86_64 83/96 2026-02-25T08:22:43.704 INFO:teuthology.orchestra.run.trial190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T08:22:43.704 INFO:teuthology.orchestra.run.trial190.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2026-02-25T08:22:43.704 INFO:teuthology.orchestra.run.trial190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-25T08:22:43.704 INFO:teuthology.orchestra.run.trial190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-25T08:22:43.704 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:22:43.760 INFO:teuthology.orchestra.run.trial190.stdout: Running scriptlet: cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch 84/96 2026-02-25T08:22:43.762 INFO:teuthology.orchestra.run.trial190.stdout: Installing : cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch 84/96 2026-02-25T08:22:43.771 INFO:teuthology.orchestra.run.trial190.stdout: Installing : ceph-prometheus-alerts-2:20.3.0-5338.g6aeb6975.el9 85/96 2026-02-25T08:22:43.795 INFO:teuthology.orchestra.run.trial190.stdout: Installing : ceph-grafana-dashboards-2:20.3.0-5338.g6aeb6975.el 86/96 2026-02-25T08:22:43.796 INFO:teuthology.orchestra.run.trial190.stdout: Installing : ceph-mgr-cephadm-2:20.3.0-5338.g6aeb6975.el9.noarc 87/96 2026-02-25T08:22:46.686 INFO:teuthology.orchestra.run.trial190.stdout: Running scriptlet: ceph-mgr-cephadm-2:20.3.0-5338.g6aeb6975.el9.noarc 87/96 2026-02-25T08:22:46.692 INFO:teuthology.orchestra.run.trial190.stdout: Installing : ceph-mgr-dashboard-2:20.3.0-5338.g6aeb6975.el9.noa 88/96 2026-02-25T08:22:47.146 INFO:teuthology.orchestra.run.trial190.stdout: Running scriptlet: ceph-mgr-dashboard-2:20.3.0-5338.g6aeb6975.el9.noa 88/96 2026-02-25T08:22:47.147 INFO:teuthology.orchestra.run.trial190.stdout: Installing : ceph-mgr-diskprediction-local-2:20.3.0-5338.g6aeb6 89/96 2026-02-25T08:22:47.158 INFO:teuthology.orchestra.run.trial190.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:20.3.0-5338.g6aeb6 89/96 2026-02-25T08:22:47.159 INFO:teuthology.orchestra.run.trial190.stdout: Installing : ceph-mgr-k8sevents-2:20.3.0-5338.g6aeb6975.el9.noa 90/96 2026-02-25T08:22:47.219 INFO:teuthology.orchestra.run.trial190.stdout: Running scriptlet: ceph-mgr-k8sevents-2:20.3.0-5338.g6aeb6975.el9.noa 90/96 2026-02-25T08:22:47.259 INFO:teuthology.orchestra.run.trial190.stdout: Installing : ceph-mgr-modules-core-2:20.3.0-5338.g6aeb6975.el9. 91/96 2026-02-25T08:22:47.260 INFO:teuthology.orchestra.run.trial190.stdout: Installing : ceph-mgr-2:20.3.0-5338.g6aeb6975.el9.x86_64 92/96 2026-02-25T08:22:47.280 INFO:teuthology.orchestra.run.trial190.stdout: Running scriptlet: ceph-mgr-2:20.3.0-5338.g6aeb6975.el9.x86_64 92/96 2026-02-25T08:22:47.280 INFO:teuthology.orchestra.run.trial190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T08:22:47.280 INFO:teuthology.orchestra.run.trial190.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2026-02-25T08:22:47.280 INFO:teuthology.orchestra.run.trial190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-25T08:22:47.281 INFO:teuthology.orchestra.run.trial190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-25T08:22:47.281 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:22:47.292 INFO:teuthology.orchestra.run.trial190.stdout: Installing : ceph-mgr-rook-2:20.3.0-5338.g6aeb6975.el9.noarch 93/96 2026-02-25T08:22:47.509 INFO:teuthology.orchestra.run.trial190.stdout: Running scriptlet: ceph-mgr-rook-2:20.3.0-5338.g6aeb6975.el9.noarch 93/96 2026-02-25T08:22:47.510 INFO:teuthology.orchestra.run.trial190.stdout: Installing : ceph-mon-2:20.3.0-5338.g6aeb6975.el9.x86_64 94/96 2026-02-25T08:22:47.531 INFO:teuthology.orchestra.run.trial190.stdout: Running scriptlet: ceph-mon-2:20.3.0-5338.g6aeb6975.el9.x86_64 94/96 2026-02-25T08:22:47.531 INFO:teuthology.orchestra.run.trial190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T08:22:47.531 INFO:teuthology.orchestra.run.trial190.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2026-02-25T08:22:47.531 INFO:teuthology.orchestra.run.trial190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-25T08:22:47.531 INFO:teuthology.orchestra.run.trial190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-25T08:22:47.531 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:22:47.614 INFO:teuthology.orchestra.run.trial190.stdout: Installing : ceph-mds-2:20.3.0-5338.g6aeb6975.el9.x86_64 95/96 2026-02-25T08:22:47.634 INFO:teuthology.orchestra.run.trial190.stdout: Running scriptlet: ceph-mds-2:20.3.0-5338.g6aeb6975.el9.x86_64 95/96 2026-02-25T08:22:47.634 INFO:teuthology.orchestra.run.trial190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T08:22:47.634 INFO:teuthology.orchestra.run.trial190.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2026-02-25T08:22:47.634 INFO:teuthology.orchestra.run.trial190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-25T08:22:47.634 INFO:teuthology.orchestra.run.trial190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-25T08:22:47.634 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:22:47.635 INFO:teuthology.orchestra.run.trial190.stdout: Installing : ceph-2:20.3.0-5338.g6aeb6975.el9.x86_64 96/96 2026-02-25T08:22:47.641 INFO:teuthology.orchestra.run.trial190.stdout: Running scriptlet: ceph-osd-classic-2:20.3.0-5338.g6aeb6975.el9.x86_6 96/96 2026-02-25T08:22:47.665 INFO:teuthology.orchestra.run.trial190.stdout: Running scriptlet: ceph-osd-crimson-2:20.3.0-5338.g6aeb6975.el9.x86_6 96/96 2026-02-25T08:22:49.081 INFO:teuthology.orchestra.run.trial190.stdout: Running scriptlet: ceph-2:20.3.0-5338.g6aeb6975.el9.x86_64 96/96 2026-02-25T08:22:49.081 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : ceph-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/96 2026-02-25T08:22:49.081 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : ceph-mds-2:20.3.0-5338.g6aeb6975.el9.x86_64 2/96 2026-02-25T08:22:49.081 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : ceph-mgr-2:20.3.0-5338.g6aeb6975.el9.x86_64 3/96 2026-02-25T08:22:49.081 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : ceph-mon-2:20.3.0-5338.g6aeb6975.el9.x86_64 4/96 2026-02-25T08:22:49.082 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : ceph-osd-2:20.3.0-5338.g6aeb6975.el9.x86_64 5/96 2026-02-25T08:22:49.082 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : ceph-osd-classic-2:20.3.0-5338.g6aeb6975.el9.x86_6 6/96 2026-02-25T08:22:49.082 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : ceph-osd-crimson-2:20.3.0-5338.g6aeb6975.el9.x86_6 7/96 2026-02-25T08:22:49.082 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : ceph-grafana-dashboards-2:20.3.0-5338.g6aeb6975.el 8/96 2026-02-25T08:22:49.082 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : ceph-mgr-cephadm-2:20.3.0-5338.g6aeb6975.el9.noarc 9/96 2026-02-25T08:22:49.082 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : ceph-mgr-dashboard-2:20.3.0-5338.g6aeb6975.el9.noa 10/96 2026-02-25T08:22:49.082 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : ceph-mgr-diskprediction-local-2:20.3.0-5338.g6aeb6 11/96 2026-02-25T08:22:49.082 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : ceph-mgr-k8sevents-2:20.3.0-5338.g6aeb6975.el9.noa 12/96 2026-02-25T08:22:49.082 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : ceph-mgr-modules-core-2:20.3.0-5338.g6aeb6975.el9. 13/96 2026-02-25T08:22:49.082 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : ceph-mgr-rook-2:20.3.0-5338.g6aeb6975.el9.noarch 14/96 2026-02-25T08:22:49.082 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : ceph-prometheus-alerts-2:20.3.0-5338.g6aeb6975.el9 15/96 2026-02-25T08:22:49.082 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : ceph-volume-2:20.3.0-5338.g6aeb6975.el9.noarch 16/96 2026-02-25T08:22:49.083 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch 17/96 2026-02-25T08:22:49.083 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : cryptsetup-2.8.1-3.el9.x86_64 18/96 2026-02-25T08:22:49.083 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : ledmon-libs-1.1.0-3.el9.x86_64 19/96 2026-02-25T08:22:49.083 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : libconfig-1.7.2-9.el9.x86_64 20/96 2026-02-25T08:22:49.083 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : libgfortran-11.5.0-14.el9.x86_64 21/96 2026-02-25T08:22:49.083 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : libquadmath-11.5.0-14.el9.x86_64 22/96 2026-02-25T08:22:49.083 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 23/96 2026-02-25T08:22:49.083 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : python3-cryptography-36.0.1-5.el9.x86_64 24/96 2026-02-25T08:22:49.083 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : python3-ply-3.11-14.el9.noarch 25/96 2026-02-25T08:22:49.083 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 26/96 2026-02-25T08:22:49.083 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : python3-pyparsing-2.4.7-9.el9.noarch 27/96 2026-02-25T08:22:49.083 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : python3-requests-2.25.1-10.el9.noarch 28/96 2026-02-25T08:22:49.083 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : python3-urllib3-1.26.5-7.el9.noarch 29/96 2026-02-25T08:22:49.084 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : unzip-6.0-59.el9.x86_64 30/96 2026-02-25T08:22:49.084 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : zip-3.0-35.el9.x86_64 31/96 2026-02-25T08:22:49.084 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : flexiblas-3.0.4-9.el9.x86_64 32/96 2026-02-25T08:22:49.084 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : flexiblas-netlib-3.0.4-9.el9.x86_64 33/96 2026-02-25T08:22:49.084 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 34/96 2026-02-25T08:22:49.084 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : libstoragemgmt-1.10.1-1.el9.x86_64 35/96 2026-02-25T08:22:49.084 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : lua-5.4.4-4.el9.x86_64 36/96 2026-02-25T08:22:49.084 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : openblas-0.3.29-1.el9.x86_64 37/96 2026-02-25T08:22:49.084 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : openblas-openmp-0.3.29-1.el9.x86_64 38/96 2026-02-25T08:22:49.084 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 39/96 2026-02-25T08:22:49.084 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : python3-devel-3.9.25-3.el9.x86_64 40/96 2026-02-25T08:22:49.084 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : python3-jinja2-2.11.3-8.el9.noarch 41/96 2026-02-25T08:22:49.084 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : python3-jmespath-1.0.1-1.el9.noarch 42/96 2026-02-25T08:22:49.085 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : python3-libstoragemgmt-1.10.1-1.el9.x86_64 43/96 2026-02-25T08:22:49.085 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : python3-lxml-4.6.5-3.el9.x86_64 44/96 2026-02-25T08:22:49.085 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 45/96 2026-02-25T08:22:49.085 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : python3-numpy-1:1.23.5-2.el9.x86_64 46/96 2026-02-25T08:22:49.085 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 47/96 2026-02-25T08:22:49.085 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 48/96 2026-02-25T08:22:49.085 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : python3-protobuf-3.14.0-17.el9.noarch 49/96 2026-02-25T08:22:49.085 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : python3-pyasn1-0.4.8-7.el9.noarch 50/96 2026-02-25T08:22:49.085 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : python3-pyasn1-modules-0.4.8-7.el9.noarch 51/96 2026-02-25T08:22:49.085 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 52/96 2026-02-25T08:22:49.085 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : python3-scipy-1.9.3-2.el9.x86_64 53/96 2026-02-25T08:22:49.085 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 54/96 2026-02-25T08:22:49.086 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : xmlsec1-1.2.29-13.el9.x86_64 55/96 2026-02-25T08:22:49.086 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : xmlsec1-openssl-1.2.29-13.el9.x86_64 56/96 2026-02-25T08:22:49.086 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 57/96 2026-02-25T08:22:49.086 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : protobuf-compiler-3.14.0-17.el9.x86_64 58/96 2026-02-25T08:22:49.086 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : abseil-cpp-20211102.0-4.el9.x86_64 59/96 2026-02-25T08:22:49.086 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 60/96 2026-02-25T08:22:49.086 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 61/96 2026-02-25T08:22:49.086 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 62/96 2026-02-25T08:22:49.086 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : python3-autocommand-2.2.2-8.el9.noarch 63/96 2026-02-25T08:22:49.086 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : python3-backports-tarfile-1.2.0-1.el9.noarch 64/96 2026-02-25T08:22:49.086 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 65/96 2026-02-25T08:22:49.086 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 66/96 2026-02-25T08:22:49.086 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 67/96 2026-02-25T08:22:49.087 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : python3-cheroot-10.0.1-4.el9.noarch 68/96 2026-02-25T08:22:49.087 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 69/96 2026-02-25T08:22:49.087 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : python3-google-auth-1:2.45.0-1.el9.noarch 70/96 2026-02-25T08:22:49.087 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 71/96 2026-02-25T08:22:49.087 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 72/96 2026-02-25T08:22:49.087 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : python3-influxdb-5.3.1-1.el9.noarch 73/96 2026-02-25T08:22:49.087 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : python3-isodate-0.6.1-3.el9.noarch 74/96 2026-02-25T08:22:49.087 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 75/96 2026-02-25T08:22:49.087 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 76/96 2026-02-25T08:22:49.087 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 77/96 2026-02-25T08:22:49.087 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : python3-jaraco-context-6.0.1-3.el9.noarch 78/96 2026-02-25T08:22:49.087 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 79/96 2026-02-25T08:22:49.087 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : python3-jaraco-text-4.0.0-2.el9.noarch 80/96 2026-02-25T08:22:49.088 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : python3-kubernetes-1:26.1.0-3.el9.noarch 81/96 2026-02-25T08:22:49.088 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 82/96 2026-02-25T08:22:49.088 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : python3-msgpack-1.0.3-2.el9.x86_64 83/96 2026-02-25T08:22:49.088 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 84/96 2026-02-25T08:22:49.088 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 85/96 2026-02-25T08:22:49.088 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/96 2026-02-25T08:22:49.088 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 87/96 2026-02-25T08:22:49.088 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 88/96 2026-02-25T08:22:49.088 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 89/96 2026-02-25T08:22:49.088 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : python3-saml-1.16.0-1.el9.noarch 90/96 2026-02-25T08:22:49.088 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 91/96 2026-02-25T08:22:49.088 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : python3-typing-extensions-4.15.0-1.el9.noarch 92/96 2026-02-25T08:22:49.088 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 93/96 2026-02-25T08:22:49.089 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : python3-xmlsec-1.3.13-1.el9.x86_64 94/96 2026-02-25T08:22:49.089 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : python3-xmltodict-0.12.0-15.el9.noarch 95/96 2026-02-25T08:22:49.149 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 96/96 2026-02-25T08:22:49.149 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:22:49.149 INFO:teuthology.orchestra.run.trial190.stdout:Installed: 2026-02-25T08:22:49.149 INFO:teuthology.orchestra.run.trial190.stdout: abseil-cpp-20211102.0-4.el9.x86_64 2026-02-25T08:22:49.149 INFO:teuthology.orchestra.run.trial190.stdout: ceph-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:22:49.149 INFO:teuthology.orchestra.run.trial190.stdout: ceph-grafana-dashboards-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:22:49.150 INFO:teuthology.orchestra.run.trial190.stdout: ceph-mds-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:22:49.150 INFO:teuthology.orchestra.run.trial190.stdout: ceph-mgr-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:22:49.150 INFO:teuthology.orchestra.run.trial190.stdout: ceph-mgr-cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:22:49.150 INFO:teuthology.orchestra.run.trial190.stdout: ceph-mgr-dashboard-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:22:49.150 INFO:teuthology.orchestra.run.trial190.stdout: ceph-mgr-diskprediction-local-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:22:49.150 INFO:teuthology.orchestra.run.trial190.stdout: ceph-mgr-k8sevents-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:22:49.150 INFO:teuthology.orchestra.run.trial190.stdout: ceph-mgr-modules-core-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:22:49.150 INFO:teuthology.orchestra.run.trial190.stdout: ceph-mgr-rook-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:22:49.150 INFO:teuthology.orchestra.run.trial190.stdout: ceph-mon-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:22:49.150 INFO:teuthology.orchestra.run.trial190.stdout: ceph-osd-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:22:49.150 INFO:teuthology.orchestra.run.trial190.stdout: ceph-osd-classic-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:22:49.150 INFO:teuthology.orchestra.run.trial190.stdout: ceph-osd-crimson-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:22:49.151 INFO:teuthology.orchestra.run.trial190.stdout: ceph-prometheus-alerts-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:22:49.151 INFO:teuthology.orchestra.run.trial190.stdout: ceph-volume-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:22:49.151 INFO:teuthology.orchestra.run.trial190.stdout: cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:22:49.151 INFO:teuthology.orchestra.run.trial190.stdout: cryptsetup-2.8.1-3.el9.x86_64 2026-02-25T08:22:49.151 INFO:teuthology.orchestra.run.trial190.stdout: flexiblas-3.0.4-9.el9.x86_64 2026-02-25T08:22:49.151 INFO:teuthology.orchestra.run.trial190.stdout: flexiblas-netlib-3.0.4-9.el9.x86_64 2026-02-25T08:22:49.151 INFO:teuthology.orchestra.run.trial190.stdout: flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 2026-02-25T08:22:49.151 INFO:teuthology.orchestra.run.trial190.stdout: grpc-data-1.46.7-10.el9.noarch 2026-02-25T08:22:49.151 INFO:teuthology.orchestra.run.trial190.stdout: ledmon-libs-1.1.0-3.el9.x86_64 2026-02-25T08:22:49.151 INFO:teuthology.orchestra.run.trial190.stdout: libconfig-1.7.2-9.el9.x86_64 2026-02-25T08:22:49.151 INFO:teuthology.orchestra.run.trial190.stdout: libgfortran-11.5.0-14.el9.x86_64 2026-02-25T08:22:49.151 INFO:teuthology.orchestra.run.trial190.stdout: libquadmath-11.5.0-14.el9.x86_64 2026-02-25T08:22:49.151 INFO:teuthology.orchestra.run.trial190.stdout: libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-25T08:22:49.152 INFO:teuthology.orchestra.run.trial190.stdout: lua-5.4.4-4.el9.x86_64 2026-02-25T08:22:49.152 INFO:teuthology.orchestra.run.trial190.stdout: lua-devel-5.4.4-4.el9.x86_64 2026-02-25T08:22:49.152 INFO:teuthology.orchestra.run.trial190.stdout: luarocks-3.9.2-5.el9.noarch 2026-02-25T08:22:49.152 INFO:teuthology.orchestra.run.trial190.stdout: openblas-0.3.29-1.el9.x86_64 2026-02-25T08:22:49.152 INFO:teuthology.orchestra.run.trial190.stdout: openblas-openmp-0.3.29-1.el9.x86_64 2026-02-25T08:22:49.152 INFO:teuthology.orchestra.run.trial190.stdout: protobuf-compiler-3.14.0-17.el9.x86_64 2026-02-25T08:22:49.152 INFO:teuthology.orchestra.run.trial190.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2026-02-25T08:22:49.152 INFO:teuthology.orchestra.run.trial190.stdout: python3-autocommand-2.2.2-8.el9.noarch 2026-02-25T08:22:49.152 INFO:teuthology.orchestra.run.trial190.stdout: python3-babel-2.9.1-2.el9.noarch 2026-02-25T08:22:49.152 INFO:teuthology.orchestra.run.trial190.stdout: python3-backports-tarfile-1.2.0-1.el9.noarch 2026-02-25T08:22:49.152 INFO:teuthology.orchestra.run.trial190.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2026-02-25T08:22:49.152 INFO:teuthology.orchestra.run.trial190.stdout: python3-cachetools-4.2.4-1.el9.noarch 2026-02-25T08:22:49.152 INFO:teuthology.orchestra.run.trial190.stdout: python3-certifi-2023.05.07-4.el9.noarch 2026-02-25T08:22:49.153 INFO:teuthology.orchestra.run.trial190.stdout: python3-cffi-1.14.5-5.el9.x86_64 2026-02-25T08:22:49.153 INFO:teuthology.orchestra.run.trial190.stdout: python3-cheroot-10.0.1-4.el9.noarch 2026-02-25T08:22:49.153 INFO:teuthology.orchestra.run.trial190.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2026-02-25T08:22:49.153 INFO:teuthology.orchestra.run.trial190.stdout: python3-cryptography-36.0.1-5.el9.x86_64 2026-02-25T08:22:49.153 INFO:teuthology.orchestra.run.trial190.stdout: python3-devel-3.9.25-3.el9.x86_64 2026-02-25T08:22:49.153 INFO:teuthology.orchestra.run.trial190.stdout: python3-google-auth-1:2.45.0-1.el9.noarch 2026-02-25T08:22:49.153 INFO:teuthology.orchestra.run.trial190.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2026-02-25T08:22:49.153 INFO:teuthology.orchestra.run.trial190.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2026-02-25T08:22:49.153 INFO:teuthology.orchestra.run.trial190.stdout: python3-influxdb-5.3.1-1.el9.noarch 2026-02-25T08:22:49.153 INFO:teuthology.orchestra.run.trial190.stdout: python3-isodate-0.6.1-3.el9.noarch 2026-02-25T08:22:49.153 INFO:teuthology.orchestra.run.trial190.stdout: python3-jaraco-8.2.1-3.el9.noarch 2026-02-25T08:22:49.153 INFO:teuthology.orchestra.run.trial190.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2026-02-25T08:22:49.153 INFO:teuthology.orchestra.run.trial190.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2026-02-25T08:22:49.154 INFO:teuthology.orchestra.run.trial190.stdout: python3-jaraco-context-6.0.1-3.el9.noarch 2026-02-25T08:22:49.154 INFO:teuthology.orchestra.run.trial190.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2026-02-25T08:22:49.154 INFO:teuthology.orchestra.run.trial190.stdout: python3-jaraco-text-4.0.0-2.el9.noarch 2026-02-25T08:22:49.154 INFO:teuthology.orchestra.run.trial190.stdout: python3-jinja2-2.11.3-8.el9.noarch 2026-02-25T08:22:49.154 INFO:teuthology.orchestra.run.trial190.stdout: python3-jmespath-1.0.1-1.el9.noarch 2026-02-25T08:22:49.154 INFO:teuthology.orchestra.run.trial190.stdout: python3-kubernetes-1:26.1.0-3.el9.noarch 2026-02-25T08:22:49.154 INFO:teuthology.orchestra.run.trial190.stdout: python3-libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-25T08:22:49.154 INFO:teuthology.orchestra.run.trial190.stdout: python3-lxml-4.6.5-3.el9.x86_64 2026-02-25T08:22:49.154 INFO:teuthology.orchestra.run.trial190.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2026-02-25T08:22:49.154 INFO:teuthology.orchestra.run.trial190.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2026-02-25T08:22:49.154 INFO:teuthology.orchestra.run.trial190.stdout: python3-msgpack-1.0.3-2.el9.x86_64 2026-02-25T08:22:49.154 INFO:teuthology.orchestra.run.trial190.stdout: python3-natsort-7.1.1-5.el9.noarch 2026-02-25T08:22:49.154 INFO:teuthology.orchestra.run.trial190.stdout: python3-numpy-1:1.23.5-2.el9.x86_64 2026-02-25T08:22:49.155 INFO:teuthology.orchestra.run.trial190.stdout: python3-numpy-f2py-1:1.23.5-2.el9.x86_64 2026-02-25T08:22:49.155 INFO:teuthology.orchestra.run.trial190.stdout: python3-packaging-20.9-5.el9.noarch 2026-02-25T08:22:49.155 INFO:teuthology.orchestra.run.trial190.stdout: python3-ply-3.11-14.el9.noarch 2026-02-25T08:22:49.155 INFO:teuthology.orchestra.run.trial190.stdout: python3-portend-3.1.0-2.el9.noarch 2026-02-25T08:22:49.155 INFO:teuthology.orchestra.run.trial190.stdout: python3-protobuf-3.14.0-17.el9.noarch 2026-02-25T08:22:49.155 INFO:teuthology.orchestra.run.trial190.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2026-02-25T08:22:49.155 INFO:teuthology.orchestra.run.trial190.stdout: python3-pyasn1-0.4.8-7.el9.noarch 2026-02-25T08:22:49.155 INFO:teuthology.orchestra.run.trial190.stdout: python3-pyasn1-modules-0.4.8-7.el9.noarch 2026-02-25T08:22:49.155 INFO:teuthology.orchestra.run.trial190.stdout: python3-pycparser-2.20-6.el9.noarch 2026-02-25T08:22:49.155 INFO:teuthology.orchestra.run.trial190.stdout: python3-pyparsing-2.4.7-9.el9.noarch 2026-02-25T08:22:49.155 INFO:teuthology.orchestra.run.trial190.stdout: python3-repoze-lru-0.7-16.el9.noarch 2026-02-25T08:22:49.155 INFO:teuthology.orchestra.run.trial190.stdout: python3-requests-2.25.1-10.el9.noarch 2026-02-25T08:22:49.155 INFO:teuthology.orchestra.run.trial190.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2026-02-25T08:22:49.155 INFO:teuthology.orchestra.run.trial190.stdout: python3-routes-2.5.1-5.el9.noarch 2026-02-25T08:22:49.156 INFO:teuthology.orchestra.run.trial190.stdout: python3-rsa-4.9-2.el9.noarch 2026-02-25T08:22:49.156 INFO:teuthology.orchestra.run.trial190.stdout: python3-saml-1.16.0-1.el9.noarch 2026-02-25T08:22:49.156 INFO:teuthology.orchestra.run.trial190.stdout: python3-scipy-1.9.3-2.el9.x86_64 2026-02-25T08:22:49.156 INFO:teuthology.orchestra.run.trial190.stdout: python3-tempora-5.0.0-2.el9.noarch 2026-02-25T08:22:49.156 INFO:teuthology.orchestra.run.trial190.stdout: python3-toml-0.10.2-6.el9.noarch 2026-02-25T08:22:49.156 INFO:teuthology.orchestra.run.trial190.stdout: python3-typing-extensions-4.15.0-1.el9.noarch 2026-02-25T08:22:49.156 INFO:teuthology.orchestra.run.trial190.stdout: python3-urllib3-1.26.5-7.el9.noarch 2026-02-25T08:22:49.156 INFO:teuthology.orchestra.run.trial190.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2026-02-25T08:22:49.156 INFO:teuthology.orchestra.run.trial190.stdout: python3-xmlsec-1.3.13-1.el9.x86_64 2026-02-25T08:22:49.156 INFO:teuthology.orchestra.run.trial190.stdout: python3-xmltodict-0.12.0-15.el9.noarch 2026-02-25T08:22:49.156 INFO:teuthology.orchestra.run.trial190.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2026-02-25T08:22:49.156 INFO:teuthology.orchestra.run.trial190.stdout: unzip-6.0-59.el9.x86_64 2026-02-25T08:22:49.156 INFO:teuthology.orchestra.run.trial190.stdout: xmlsec1-1.2.29-13.el9.x86_64 2026-02-25T08:22:49.157 INFO:teuthology.orchestra.run.trial190.stdout: xmlsec1-openssl-1.2.29-13.el9.x86_64 2026-02-25T08:22:49.157 INFO:teuthology.orchestra.run.trial190.stdout: zip-3.0-35.el9.x86_64 2026-02-25T08:22:49.157 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:22:49.157 INFO:teuthology.orchestra.run.trial190.stdout:Complete! 2026-02-25T08:22:49.218 DEBUG:teuthology.orchestra.run.trial190:> sudo yum -y install ceph-base 2026-02-25T08:22:49.543 INFO:teuthology.orchestra.run.trial190.stdout:Last metadata expiration check: 0:13:57 ago on Wed 25 Feb 2026 08:08:52 AM UTC. 2026-02-25T08:22:49.608 INFO:teuthology.orchestra.run.trial190.stdout:Package ceph-base-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:22:49.627 INFO:teuthology.orchestra.run.trial190.stdout:Dependencies resolved. 2026-02-25T08:22:49.627 INFO:teuthology.orchestra.run.trial190.stdout:Nothing to do. 2026-02-25T08:22:49.627 INFO:teuthology.orchestra.run.trial190.stdout:Complete! 2026-02-25T08:22:49.649 DEBUG:teuthology.orchestra.run.trial190:> sudo yum -y install cephadm 2026-02-25T08:22:49.969 INFO:teuthology.orchestra.run.trial190.stdout:Last metadata expiration check: 0:13:57 ago on Wed 25 Feb 2026 08:08:52 AM UTC. 2026-02-25T08:22:50.034 INFO:teuthology.orchestra.run.trial190.stdout:Package cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:22:50.053 INFO:teuthology.orchestra.run.trial190.stdout:Dependencies resolved. 2026-02-25T08:22:50.054 INFO:teuthology.orchestra.run.trial190.stdout:Nothing to do. 2026-02-25T08:22:50.054 INFO:teuthology.orchestra.run.trial190.stdout:Complete! 2026-02-25T08:22:50.076 DEBUG:teuthology.orchestra.run.trial190:> sudo yum -y install ceph-immutable-object-cache 2026-02-25T08:22:50.396 INFO:teuthology.orchestra.run.trial190.stdout:Last metadata expiration check: 0:13:58 ago on Wed 25 Feb 2026 08:08:52 AM UTC. 2026-02-25T08:22:50.480 INFO:teuthology.orchestra.run.trial190.stdout:Dependencies resolved. 2026-02-25T08:22:50.481 INFO:teuthology.orchestra.run.trial190.stdout:================================================================================ 2026-02-25T08:22:50.481 INFO:teuthology.orchestra.run.trial190.stdout: Package Arch Version Repo Size 2026-02-25T08:22:50.481 INFO:teuthology.orchestra.run.trial190.stdout:================================================================================ 2026-02-25T08:22:50.482 INFO:teuthology.orchestra.run.trial190.stdout:Installing: 2026-02-25T08:22:50.482 INFO:teuthology.orchestra.run.trial190.stdout: ceph-immutable-object-cache x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 149 k 2026-02-25T08:22:50.482 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:22:50.482 INFO:teuthology.orchestra.run.trial190.stdout:Transaction Summary 2026-02-25T08:22:50.482 INFO:teuthology.orchestra.run.trial190.stdout:================================================================================ 2026-02-25T08:22:50.482 INFO:teuthology.orchestra.run.trial190.stdout:Install 1 Package 2026-02-25T08:22:50.482 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:22:50.482 INFO:teuthology.orchestra.run.trial190.stdout:Total download size: 149 k 2026-02-25T08:22:50.482 INFO:teuthology.orchestra.run.trial190.stdout:Installed size: 455 k 2026-02-25T08:22:50.482 INFO:teuthology.orchestra.run.trial190.stdout:Downloading Packages: 2026-02-25T08:22:51.359 INFO:teuthology.orchestra.run.trial190.stdout:ceph-immutable-object-cache-20.3.0-5338.g6aeb69 170 kB/s | 149 kB 00:00 2026-02-25T08:22:51.360 INFO:teuthology.orchestra.run.trial190.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:22:51.360 INFO:teuthology.orchestra.run.trial190.stdout:Total 170 kB/s | 149 kB 00:00 2026-02-25T08:22:51.360 INFO:teuthology.orchestra.run.trial190.stdout:Running transaction check 2026-02-25T08:22:51.365 INFO:teuthology.orchestra.run.trial190.stdout:Transaction check succeeded. 2026-02-25T08:22:51.365 INFO:teuthology.orchestra.run.trial190.stdout:Running transaction test 2026-02-25T08:22:51.397 INFO:teuthology.orchestra.run.trial190.stdout:Transaction test succeeded. 2026-02-25T08:22:51.397 INFO:teuthology.orchestra.run.trial190.stdout:Running transaction 2026-02-25T08:22:51.453 INFO:teuthology.orchestra.run.trial190.stdout: Preparing : 1/1 2026-02-25T08:22:51.455 INFO:teuthology.orchestra.run.trial190.stdout: Installing : ceph-immutable-object-cache-2:20.3.0-5338.g6aeb6975. 1/1 2026-02-25T08:22:51.478 INFO:teuthology.orchestra.run.trial190.stdout: Running scriptlet: ceph-immutable-object-cache-2:20.3.0-5338.g6aeb6975. 1/1 2026-02-25T08:22:51.478 INFO:teuthology.orchestra.run.trial190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T08:22:51.478 INFO:teuthology.orchestra.run.trial190.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2026-02-25T08:22:51.478 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:22:51.798 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : ceph-immutable-object-cache-2:20.3.0-5338.g6aeb6975. 1/1 2026-02-25T08:22:51.799 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:22:51.799 INFO:teuthology.orchestra.run.trial190.stdout:Installed: 2026-02-25T08:22:51.799 INFO:teuthology.orchestra.run.trial190.stdout: ceph-immutable-object-cache-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:22:51.799 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:22:51.799 INFO:teuthology.orchestra.run.trial190.stdout:Complete! 2026-02-25T08:22:51.825 DEBUG:teuthology.orchestra.run.trial190:> sudo yum -y install ceph-mgr 2026-02-25T08:22:52.143 INFO:teuthology.orchestra.run.trial190.stdout:Last metadata expiration check: 0:14:00 ago on Wed 25 Feb 2026 08:08:52 AM UTC. 2026-02-25T08:22:52.209 INFO:teuthology.orchestra.run.trial190.stdout:Package ceph-mgr-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:22:52.227 INFO:teuthology.orchestra.run.trial190.stdout:Dependencies resolved. 2026-02-25T08:22:52.228 INFO:teuthology.orchestra.run.trial190.stdout:Nothing to do. 2026-02-25T08:22:52.228 INFO:teuthology.orchestra.run.trial190.stdout:Complete! 2026-02-25T08:22:52.250 DEBUG:teuthology.orchestra.run.trial190:> sudo yum -y install ceph-mgr-dashboard 2026-02-25T08:22:52.568 INFO:teuthology.orchestra.run.trial190.stdout:Last metadata expiration check: 0:14:00 ago on Wed 25 Feb 2026 08:08:52 AM UTC. 2026-02-25T08:22:52.633 INFO:teuthology.orchestra.run.trial190.stdout:Package ceph-mgr-dashboard-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:22:52.651 INFO:teuthology.orchestra.run.trial190.stdout:Dependencies resolved. 2026-02-25T08:22:52.652 INFO:teuthology.orchestra.run.trial190.stdout:Nothing to do. 2026-02-25T08:22:52.652 INFO:teuthology.orchestra.run.trial190.stdout:Complete! 2026-02-25T08:22:52.674 DEBUG:teuthology.orchestra.run.trial190:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-25T08:22:52.992 INFO:teuthology.orchestra.run.trial190.stdout:Last metadata expiration check: 0:14:00 ago on Wed 25 Feb 2026 08:08:52 AM UTC. 2026-02-25T08:22:53.058 INFO:teuthology.orchestra.run.trial190.stdout:Package ceph-mgr-diskprediction-local-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:22:53.076 INFO:teuthology.orchestra.run.trial190.stdout:Dependencies resolved. 2026-02-25T08:22:53.077 INFO:teuthology.orchestra.run.trial190.stdout:Nothing to do. 2026-02-25T08:22:53.077 INFO:teuthology.orchestra.run.trial190.stdout:Complete! 2026-02-25T08:22:53.098 DEBUG:teuthology.orchestra.run.trial190:> sudo yum -y install ceph-mgr-rook 2026-02-25T08:22:53.414 INFO:teuthology.orchestra.run.trial190.stdout:Last metadata expiration check: 0:14:01 ago on Wed 25 Feb 2026 08:08:52 AM UTC. 2026-02-25T08:22:53.479 INFO:teuthology.orchestra.run.trial190.stdout:Package ceph-mgr-rook-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:22:53.498 INFO:teuthology.orchestra.run.trial190.stdout:Dependencies resolved. 2026-02-25T08:22:53.499 INFO:teuthology.orchestra.run.trial190.stdout:Nothing to do. 2026-02-25T08:22:53.499 INFO:teuthology.orchestra.run.trial190.stdout:Complete! 2026-02-25T08:22:53.520 DEBUG:teuthology.orchestra.run.trial190:> sudo yum -y install ceph-mgr-cephadm 2026-02-25T08:22:53.837 INFO:teuthology.orchestra.run.trial190.stdout:Last metadata expiration check: 0:14:01 ago on Wed 25 Feb 2026 08:08:52 AM UTC. 2026-02-25T08:22:53.902 INFO:teuthology.orchestra.run.trial190.stdout:Package ceph-mgr-cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:22:53.921 INFO:teuthology.orchestra.run.trial190.stdout:Dependencies resolved. 2026-02-25T08:22:53.921 INFO:teuthology.orchestra.run.trial190.stdout:Nothing to do. 2026-02-25T08:22:53.921 INFO:teuthology.orchestra.run.trial190.stdout:Complete! 2026-02-25T08:22:53.943 DEBUG:teuthology.orchestra.run.trial190:> sudo yum -y install ceph-osd 2026-02-25T08:22:54.264 INFO:teuthology.orchestra.run.trial190.stdout:Last metadata expiration check: 0:14:02 ago on Wed 25 Feb 2026 08:08:52 AM UTC. 2026-02-25T08:22:54.330 INFO:teuthology.orchestra.run.trial190.stdout:Package ceph-osd-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:22:54.349 INFO:teuthology.orchestra.run.trial190.stdout:Dependencies resolved. 2026-02-25T08:22:54.349 INFO:teuthology.orchestra.run.trial190.stdout:Nothing to do. 2026-02-25T08:22:54.350 INFO:teuthology.orchestra.run.trial190.stdout:Complete! 2026-02-25T08:22:54.371 DEBUG:teuthology.orchestra.run.trial190:> sudo yum -y install ceph-osd-classic 2026-02-25T08:22:54.684 INFO:teuthology.orchestra.run.trial190.stdout:Last metadata expiration check: 0:14:02 ago on Wed 25 Feb 2026 08:08:52 AM UTC. 2026-02-25T08:22:54.750 INFO:teuthology.orchestra.run.trial190.stdout:Package ceph-osd-classic-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:22:54.768 INFO:teuthology.orchestra.run.trial190.stdout:Dependencies resolved. 2026-02-25T08:22:54.769 INFO:teuthology.orchestra.run.trial190.stdout:Nothing to do. 2026-02-25T08:22:54.769 INFO:teuthology.orchestra.run.trial190.stdout:Complete! 2026-02-25T08:22:54.790 DEBUG:teuthology.orchestra.run.trial190:> sudo yum -y install ceph-fuse 2026-02-25T08:22:55.107 INFO:teuthology.orchestra.run.trial190.stdout:Last metadata expiration check: 0:14:03 ago on Wed 25 Feb 2026 08:08:52 AM UTC. 2026-02-25T08:22:55.191 INFO:teuthology.orchestra.run.trial190.stdout:Dependencies resolved. 2026-02-25T08:22:55.192 INFO:teuthology.orchestra.run.trial190.stdout:================================================================================ 2026-02-25T08:22:55.192 INFO:teuthology.orchestra.run.trial190.stdout: Package Arch Version Repo Size 2026-02-25T08:22:55.192 INFO:teuthology.orchestra.run.trial190.stdout:================================================================================ 2026-02-25T08:22:55.192 INFO:teuthology.orchestra.run.trial190.stdout:Installing: 2026-02-25T08:22:55.192 INFO:teuthology.orchestra.run.trial190.stdout: ceph-fuse x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 914 k 2026-02-25T08:22:55.192 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:22:55.193 INFO:teuthology.orchestra.run.trial190.stdout:Transaction Summary 2026-02-25T08:22:55.193 INFO:teuthology.orchestra.run.trial190.stdout:================================================================================ 2026-02-25T08:22:55.193 INFO:teuthology.orchestra.run.trial190.stdout:Install 1 Package 2026-02-25T08:22:55.193 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:22:55.193 INFO:teuthology.orchestra.run.trial190.stdout:Total download size: 914 k 2026-02-25T08:22:55.193 INFO:teuthology.orchestra.run.trial190.stdout:Installed size: 2.9 M 2026-02-25T08:22:55.193 INFO:teuthology.orchestra.run.trial190.stdout:Downloading Packages: 2026-02-25T08:22:59.392 INFO:teuthology.orchestra.run.trial190.stdout:ceph-fuse-20.3.0-5338.g6aeb6975.el9.x86_64.rpm 218 kB/s | 914 kB 00:04 2026-02-25T08:22:59.393 INFO:teuthology.orchestra.run.trial190.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:22:59.393 INFO:teuthology.orchestra.run.trial190.stdout:Total 217 kB/s | 914 kB 00:04 2026-02-25T08:22:59.394 INFO:teuthology.orchestra.run.trial190.stdout:Running transaction check 2026-02-25T08:22:59.398 INFO:teuthology.orchestra.run.trial190.stdout:Transaction check succeeded. 2026-02-25T08:22:59.399 INFO:teuthology.orchestra.run.trial190.stdout:Running transaction test 2026-02-25T08:22:59.432 INFO:teuthology.orchestra.run.trial190.stdout:Transaction test succeeded. 2026-02-25T08:22:59.432 INFO:teuthology.orchestra.run.trial190.stdout:Running transaction 2026-02-25T08:22:59.522 INFO:teuthology.orchestra.run.trial190.stdout: Preparing : 1/1 2026-02-25T08:22:59.539 INFO:teuthology.orchestra.run.trial190.stdout: Installing : ceph-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:22:59.766 INFO:teuthology.orchestra.run.trial190.stdout: Running scriptlet: ceph-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:22:59.829 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : ceph-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:22:59.829 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:22:59.829 INFO:teuthology.orchestra.run.trial190.stdout:Installed: 2026-02-25T08:22:59.829 INFO:teuthology.orchestra.run.trial190.stdout: ceph-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:22:59.829 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:22:59.829 INFO:teuthology.orchestra.run.trial190.stdout:Complete! 2026-02-25T08:22:59.857 DEBUG:teuthology.orchestra.run.trial190:> sudo yum -y install ceph-volume 2026-02-25T08:23:00.177 INFO:teuthology.orchestra.run.trial190.stdout:Last metadata expiration check: 0:14:08 ago on Wed 25 Feb 2026 08:08:52 AM UTC. 2026-02-25T08:23:00.242 INFO:teuthology.orchestra.run.trial190.stdout:Package ceph-volume-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:23:00.260 INFO:teuthology.orchestra.run.trial190.stdout:Dependencies resolved. 2026-02-25T08:23:00.261 INFO:teuthology.orchestra.run.trial190.stdout:Nothing to do. 2026-02-25T08:23:00.261 INFO:teuthology.orchestra.run.trial190.stdout:Complete! 2026-02-25T08:23:00.283 DEBUG:teuthology.orchestra.run.trial190:> sudo yum -y install librados-devel 2026-02-25T08:23:00.601 INFO:teuthology.orchestra.run.trial190.stdout:Last metadata expiration check: 0:14:08 ago on Wed 25 Feb 2026 08:08:52 AM UTC. 2026-02-25T08:23:00.685 INFO:teuthology.orchestra.run.trial190.stdout:Dependencies resolved. 2026-02-25T08:23:00.685 INFO:teuthology.orchestra.run.trial190.stdout:================================================================================ 2026-02-25T08:23:00.685 INFO:teuthology.orchestra.run.trial190.stdout: Package Arch Version Repo Size 2026-02-25T08:23:00.685 INFO:teuthology.orchestra.run.trial190.stdout:================================================================================ 2026-02-25T08:23:00.685 INFO:teuthology.orchestra.run.trial190.stdout:Installing: 2026-02-25T08:23:00.685 INFO:teuthology.orchestra.run.trial190.stdout: librados-devel x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 124 k 2026-02-25T08:23:00.685 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:23:00.685 INFO:teuthology.orchestra.run.trial190.stdout:Transaction Summary 2026-02-25T08:23:00.685 INFO:teuthology.orchestra.run.trial190.stdout:================================================================================ 2026-02-25T08:23:00.685 INFO:teuthology.orchestra.run.trial190.stdout:Install 1 Package 2026-02-25T08:23:00.686 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:23:00.686 INFO:teuthology.orchestra.run.trial190.stdout:Total download size: 124 k 2026-02-25T08:23:00.686 INFO:teuthology.orchestra.run.trial190.stdout:Installed size: 434 k 2026-02-25T08:23:00.686 INFO:teuthology.orchestra.run.trial190.stdout:Downloading Packages: 2026-02-25T08:23:01.187 INFO:teuthology.orchestra.run.trial190.stdout:librados-devel-20.3.0-5338.g6aeb6975.el9.x86_64 247 kB/s | 124 kB 00:00 2026-02-25T08:23:01.187 INFO:teuthology.orchestra.run.trial190.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:23:01.187 INFO:teuthology.orchestra.run.trial190.stdout:Total 247 kB/s | 124 kB 00:00 2026-02-25T08:23:01.188 INFO:teuthology.orchestra.run.trial190.stdout:Running transaction check 2026-02-25T08:23:01.190 INFO:teuthology.orchestra.run.trial190.stdout:Transaction check succeeded. 2026-02-25T08:23:01.190 INFO:teuthology.orchestra.run.trial190.stdout:Running transaction test 2026-02-25T08:23:01.222 INFO:teuthology.orchestra.run.trial190.stdout:Transaction test succeeded. 2026-02-25T08:23:01.222 INFO:teuthology.orchestra.run.trial190.stdout:Running transaction 2026-02-25T08:23:01.278 INFO:teuthology.orchestra.run.trial190.stdout: Preparing : 1/1 2026-02-25T08:23:01.298 INFO:teuthology.orchestra.run.trial190.stdout: Installing : librados-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:23:01.383 INFO:teuthology.orchestra.run.trial190.stdout: Running scriptlet: librados-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:23:01.445 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : librados-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:23:01.445 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:23:01.445 INFO:teuthology.orchestra.run.trial190.stdout:Installed: 2026-02-25T08:23:01.445 INFO:teuthology.orchestra.run.trial190.stdout: librados-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:23:01.445 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:23:01.445 INFO:teuthology.orchestra.run.trial190.stdout:Complete! 2026-02-25T08:23:01.473 DEBUG:teuthology.orchestra.run.trial190:> sudo yum -y install libcephfs2 2026-02-25T08:23:01.792 INFO:teuthology.orchestra.run.trial190.stdout:Last metadata expiration check: 0:14:09 ago on Wed 25 Feb 2026 08:08:52 AM UTC. 2026-02-25T08:23:01.857 INFO:teuthology.orchestra.run.trial190.stdout:Package libcephfs2-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:23:01.877 INFO:teuthology.orchestra.run.trial190.stdout:Dependencies resolved. 2026-02-25T08:23:01.878 INFO:teuthology.orchestra.run.trial190.stdout:Nothing to do. 2026-02-25T08:23:01.878 INFO:teuthology.orchestra.run.trial190.stdout:Complete! 2026-02-25T08:23:01.898 DEBUG:teuthology.orchestra.run.trial190:> sudo yum -y install libcephfs-devel 2026-02-25T08:23:02.208 INFO:teuthology.orchestra.run.trial190.stdout:Last metadata expiration check: 0:14:10 ago on Wed 25 Feb 2026 08:08:52 AM UTC. 2026-02-25T08:23:02.292 INFO:teuthology.orchestra.run.trial190.stdout:Dependencies resolved. 2026-02-25T08:23:02.293 INFO:teuthology.orchestra.run.trial190.stdout:================================================================================ 2026-02-25T08:23:02.294 INFO:teuthology.orchestra.run.trial190.stdout: Package Arch Version Repo Size 2026-02-25T08:23:02.294 INFO:teuthology.orchestra.run.trial190.stdout:================================================================================ 2026-02-25T08:23:02.294 INFO:teuthology.orchestra.run.trial190.stdout:Installing: 2026-02-25T08:23:02.294 INFO:teuthology.orchestra.run.trial190.stdout: libcephfs-devel x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 37 k 2026-02-25T08:23:02.294 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:23:02.294 INFO:teuthology.orchestra.run.trial190.stdout:Transaction Summary 2026-02-25T08:23:02.294 INFO:teuthology.orchestra.run.trial190.stdout:================================================================================ 2026-02-25T08:23:02.294 INFO:teuthology.orchestra.run.trial190.stdout:Install 1 Package 2026-02-25T08:23:02.294 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:23:02.294 INFO:teuthology.orchestra.run.trial190.stdout:Total download size: 37 k 2026-02-25T08:23:02.294 INFO:teuthology.orchestra.run.trial190.stdout:Installed size: 171 k 2026-02-25T08:23:02.295 INFO:teuthology.orchestra.run.trial190.stdout:Downloading Packages: 2026-02-25T08:23:03.004 INFO:teuthology.orchestra.run.trial190.stdout:libcephfs-devel-20.3.0-5338.g6aeb6975.el9.x86_6 52 kB/s | 37 kB 00:00 2026-02-25T08:23:03.004 INFO:teuthology.orchestra.run.trial190.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:23:03.004 INFO:teuthology.orchestra.run.trial190.stdout:Total 52 kB/s | 37 kB 00:00 2026-02-25T08:23:03.005 INFO:teuthology.orchestra.run.trial190.stdout:Running transaction check 2026-02-25T08:23:03.008 INFO:teuthology.orchestra.run.trial190.stdout:Transaction check succeeded. 2026-02-25T08:23:03.008 INFO:teuthology.orchestra.run.trial190.stdout:Running transaction test 2026-02-25T08:23:03.010 INFO:teuthology.orchestra.run.trial190.stdout:Transaction test succeeded. 2026-02-25T08:23:03.010 INFO:teuthology.orchestra.run.trial190.stdout:Running transaction 2026-02-25T08:23:03.033 INFO:teuthology.orchestra.run.trial190.stdout: Preparing : 1/1 2026-02-25T08:23:03.050 INFO:teuthology.orchestra.run.trial190.stdout: Installing : libcephfs-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:23:03.111 INFO:teuthology.orchestra.run.trial190.stdout: Running scriptlet: libcephfs-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:23:03.172 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : libcephfs-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:23:03.172 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:23:03.172 INFO:teuthology.orchestra.run.trial190.stdout:Installed: 2026-02-25T08:23:03.172 INFO:teuthology.orchestra.run.trial190.stdout: libcephfs-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:23:03.172 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:23:03.172 INFO:teuthology.orchestra.run.trial190.stdout:Complete! 2026-02-25T08:23:03.197 DEBUG:teuthology.orchestra.run.trial190:> sudo yum -y install librados2 2026-02-25T08:23:03.512 INFO:teuthology.orchestra.run.trial190.stdout:Last metadata expiration check: 0:14:11 ago on Wed 25 Feb 2026 08:08:52 AM UTC. 2026-02-25T08:23:03.578 INFO:teuthology.orchestra.run.trial190.stdout:Package librados2-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:23:03.595 INFO:teuthology.orchestra.run.trial190.stdout:Dependencies resolved. 2026-02-25T08:23:03.596 INFO:teuthology.orchestra.run.trial190.stdout:Nothing to do. 2026-02-25T08:23:03.596 INFO:teuthology.orchestra.run.trial190.stdout:Complete! 2026-02-25T08:23:03.617 DEBUG:teuthology.orchestra.run.trial190:> sudo yum -y install librbd1 2026-02-25T08:23:03.930 INFO:teuthology.orchestra.run.trial190.stdout:Last metadata expiration check: 0:14:11 ago on Wed 25 Feb 2026 08:08:52 AM UTC. 2026-02-25T08:23:03.995 INFO:teuthology.orchestra.run.trial190.stdout:Package librbd1-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:23:04.013 INFO:teuthology.orchestra.run.trial190.stdout:Dependencies resolved. 2026-02-25T08:23:04.013 INFO:teuthology.orchestra.run.trial190.stdout:Nothing to do. 2026-02-25T08:23:04.014 INFO:teuthology.orchestra.run.trial190.stdout:Complete! 2026-02-25T08:23:04.034 DEBUG:teuthology.orchestra.run.trial190:> sudo yum -y install python3-rados 2026-02-25T08:23:04.347 INFO:teuthology.orchestra.run.trial190.stdout:Last metadata expiration check: 0:14:12 ago on Wed 25 Feb 2026 08:08:52 AM UTC. 2026-02-25T08:23:04.412 INFO:teuthology.orchestra.run.trial190.stdout:Package python3-rados-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:23:04.430 INFO:teuthology.orchestra.run.trial190.stdout:Dependencies resolved. 2026-02-25T08:23:04.431 INFO:teuthology.orchestra.run.trial190.stdout:Nothing to do. 2026-02-25T08:23:04.431 INFO:teuthology.orchestra.run.trial190.stdout:Complete! 2026-02-25T08:23:04.451 DEBUG:teuthology.orchestra.run.trial190:> sudo yum -y install python3-rgw 2026-02-25T08:23:04.764 INFO:teuthology.orchestra.run.trial190.stdout:Last metadata expiration check: 0:14:12 ago on Wed 25 Feb 2026 08:08:52 AM UTC. 2026-02-25T08:23:04.829 INFO:teuthology.orchestra.run.trial190.stdout:Package python3-rgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:23:04.847 INFO:teuthology.orchestra.run.trial190.stdout:Dependencies resolved. 2026-02-25T08:23:04.847 INFO:teuthology.orchestra.run.trial190.stdout:Nothing to do. 2026-02-25T08:23:04.848 INFO:teuthology.orchestra.run.trial190.stdout:Complete! 2026-02-25T08:23:04.869 DEBUG:teuthology.orchestra.run.trial190:> sudo yum -y install python3-cephfs 2026-02-25T08:23:05.182 INFO:teuthology.orchestra.run.trial190.stdout:Last metadata expiration check: 0:14:13 ago on Wed 25 Feb 2026 08:08:52 AM UTC. 2026-02-25T08:23:05.247 INFO:teuthology.orchestra.run.trial190.stdout:Package python3-cephfs-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:23:05.264 INFO:teuthology.orchestra.run.trial190.stdout:Dependencies resolved. 2026-02-25T08:23:05.265 INFO:teuthology.orchestra.run.trial190.stdout:Nothing to do. 2026-02-25T08:23:05.265 INFO:teuthology.orchestra.run.trial190.stdout:Complete! 2026-02-25T08:23:05.286 DEBUG:teuthology.orchestra.run.trial190:> sudo yum -y install python3-rbd 2026-02-25T08:23:05.599 INFO:teuthology.orchestra.run.trial190.stdout:Last metadata expiration check: 0:14:13 ago on Wed 25 Feb 2026 08:08:52 AM UTC. 2026-02-25T08:23:05.664 INFO:teuthology.orchestra.run.trial190.stdout:Package python3-rbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:23:05.681 INFO:teuthology.orchestra.run.trial190.stdout:Dependencies resolved. 2026-02-25T08:23:05.682 INFO:teuthology.orchestra.run.trial190.stdout:Nothing to do. 2026-02-25T08:23:05.682 INFO:teuthology.orchestra.run.trial190.stdout:Complete! 2026-02-25T08:23:05.703 DEBUG:teuthology.orchestra.run.trial190:> sudo yum -y install rbd-fuse 2026-02-25T08:23:06.019 INFO:teuthology.orchestra.run.trial190.stdout:Last metadata expiration check: 0:14:14 ago on Wed 25 Feb 2026 08:08:52 AM UTC. 2026-02-25T08:23:06.101 INFO:teuthology.orchestra.run.trial190.stdout:Dependencies resolved. 2026-02-25T08:23:06.102 INFO:teuthology.orchestra.run.trial190.stdout:================================================================================ 2026-02-25T08:23:06.102 INFO:teuthology.orchestra.run.trial190.stdout: Package Architecture Version Repository Size 2026-02-25T08:23:06.103 INFO:teuthology.orchestra.run.trial190.stdout:================================================================================ 2026-02-25T08:23:06.103 INFO:teuthology.orchestra.run.trial190.stdout:Installing: 2026-02-25T08:23:06.103 INFO:teuthology.orchestra.run.trial190.stdout: rbd-fuse x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 86 k 2026-02-25T08:23:06.103 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:23:06.103 INFO:teuthology.orchestra.run.trial190.stdout:Transaction Summary 2026-02-25T08:23:06.103 INFO:teuthology.orchestra.run.trial190.stdout:================================================================================ 2026-02-25T08:23:06.103 INFO:teuthology.orchestra.run.trial190.stdout:Install 1 Package 2026-02-25T08:23:06.103 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:23:06.103 INFO:teuthology.orchestra.run.trial190.stdout:Total download size: 86 k 2026-02-25T08:23:06.103 INFO:teuthology.orchestra.run.trial190.stdout:Installed size: 236 k 2026-02-25T08:23:06.103 INFO:teuthology.orchestra.run.trial190.stdout:Downloading Packages: 2026-02-25T08:23:06.827 INFO:teuthology.orchestra.run.trial190.stdout:rbd-fuse-20.3.0-5338.g6aeb6975.el9.x86_64.rpm 119 kB/s | 86 kB 00:00 2026-02-25T08:23:06.828 INFO:teuthology.orchestra.run.trial190.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:23:06.828 INFO:teuthology.orchestra.run.trial190.stdout:Total 119 kB/s | 86 kB 00:00 2026-02-25T08:23:06.828 INFO:teuthology.orchestra.run.trial190.stdout:Running transaction check 2026-02-25T08:23:06.832 INFO:teuthology.orchestra.run.trial190.stdout:Transaction check succeeded. 2026-02-25T08:23:06.832 INFO:teuthology.orchestra.run.trial190.stdout:Running transaction test 2026-02-25T08:23:06.863 INFO:teuthology.orchestra.run.trial190.stdout:Transaction test succeeded. 2026-02-25T08:23:06.863 INFO:teuthology.orchestra.run.trial190.stdout:Running transaction 2026-02-25T08:23:06.916 INFO:teuthology.orchestra.run.trial190.stdout: Preparing : 1/1 2026-02-25T08:23:06.933 INFO:teuthology.orchestra.run.trial190.stdout: Installing : rbd-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:23:07.023 INFO:teuthology.orchestra.run.trial190.stdout: Running scriptlet: rbd-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:23:07.086 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : rbd-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:23:07.086 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:23:07.086 INFO:teuthology.orchestra.run.trial190.stdout:Installed: 2026-02-25T08:23:07.086 INFO:teuthology.orchestra.run.trial190.stdout: rbd-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:23:07.086 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:23:07.086 INFO:teuthology.orchestra.run.trial190.stdout:Complete! 2026-02-25T08:23:07.113 DEBUG:teuthology.orchestra.run.trial190:> sudo yum -y install rbd-mirror 2026-02-25T08:23:07.427 INFO:teuthology.orchestra.run.trial190.stdout:Last metadata expiration check: 0:14:15 ago on Wed 25 Feb 2026 08:08:52 AM UTC. 2026-02-25T08:23:07.510 INFO:teuthology.orchestra.run.trial190.stdout:Dependencies resolved. 2026-02-25T08:23:07.510 INFO:teuthology.orchestra.run.trial190.stdout:================================================================================ 2026-02-25T08:23:07.511 INFO:teuthology.orchestra.run.trial190.stdout: Package Arch Version Repo Size 2026-02-25T08:23:07.511 INFO:teuthology.orchestra.run.trial190.stdout:================================================================================ 2026-02-25T08:23:07.511 INFO:teuthology.orchestra.run.trial190.stdout:Installing: 2026-02-25T08:23:07.511 INFO:teuthology.orchestra.run.trial190.stdout: rbd-mirror x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 2.6 M 2026-02-25T08:23:07.511 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:23:07.511 INFO:teuthology.orchestra.run.trial190.stdout:Transaction Summary 2026-02-25T08:23:07.511 INFO:teuthology.orchestra.run.trial190.stdout:================================================================================ 2026-02-25T08:23:07.511 INFO:teuthology.orchestra.run.trial190.stdout:Install 1 Package 2026-02-25T08:23:07.511 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:23:07.511 INFO:teuthology.orchestra.run.trial190.stdout:Total download size: 2.6 M 2026-02-25T08:23:07.512 INFO:teuthology.orchestra.run.trial190.stdout:Installed size: 11 M 2026-02-25T08:23:07.512 INFO:teuthology.orchestra.run.trial190.stdout:Downloading Packages: 2026-02-25T08:23:14.788 INFO:teuthology.orchestra.run.trial190.stdout:rbd-mirror-20.3.0-5338.g6aeb6975.el9.x86_64.rpm 365 kB/s | 2.6 MB 00:07 2026-02-25T08:23:14.788 INFO:teuthology.orchestra.run.trial190.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:23:14.789 INFO:teuthology.orchestra.run.trial190.stdout:Total 365 kB/s | 2.6 MB 00:07 2026-02-25T08:23:14.789 INFO:teuthology.orchestra.run.trial190.stdout:Running transaction check 2026-02-25T08:23:14.794 INFO:teuthology.orchestra.run.trial190.stdout:Transaction check succeeded. 2026-02-25T08:23:14.794 INFO:teuthology.orchestra.run.trial190.stdout:Running transaction test 2026-02-25T08:23:14.832 INFO:teuthology.orchestra.run.trial190.stdout:Transaction test succeeded. 2026-02-25T08:23:14.832 INFO:teuthology.orchestra.run.trial190.stdout:Running transaction 2026-02-25T08:23:14.999 INFO:teuthology.orchestra.run.trial190.stdout: Preparing : 1/1 2026-02-25T08:23:15.001 INFO:teuthology.orchestra.run.trial190.stdout: Installing : rbd-mirror-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:23:15.024 INFO:teuthology.orchestra.run.trial190.stdout: Running scriptlet: rbd-mirror-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:23:15.024 INFO:teuthology.orchestra.run.trial190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T08:23:15.024 INFO:teuthology.orchestra.run.trial190.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2026-02-25T08:23:15.024 INFO:teuthology.orchestra.run.trial190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-25T08:23:15.025 INFO:teuthology.orchestra.run.trial190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-25T08:23:15.025 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:23:15.360 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : rbd-mirror-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:23:15.360 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:23:15.360 INFO:teuthology.orchestra.run.trial190.stdout:Installed: 2026-02-25T08:23:15.360 INFO:teuthology.orchestra.run.trial190.stdout: rbd-mirror-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:23:15.360 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:23:15.360 INFO:teuthology.orchestra.run.trial190.stdout:Complete! 2026-02-25T08:23:15.388 DEBUG:teuthology.orchestra.run.trial190:> sudo yum -y install rbd-nbd 2026-02-25T08:23:15.712 INFO:teuthology.orchestra.run.trial190.stdout:Last metadata expiration check: 0:14:23 ago on Wed 25 Feb 2026 08:08:52 AM UTC. 2026-02-25T08:23:15.795 INFO:teuthology.orchestra.run.trial190.stdout:Dependencies resolved. 2026-02-25T08:23:15.796 INFO:teuthology.orchestra.run.trial190.stdout:================================================================================ 2026-02-25T08:23:15.796 INFO:teuthology.orchestra.run.trial190.stdout: Package Architecture Version Repository Size 2026-02-25T08:23:15.796 INFO:teuthology.orchestra.run.trial190.stdout:================================================================================ 2026-02-25T08:23:15.796 INFO:teuthology.orchestra.run.trial190.stdout:Installing: 2026-02-25T08:23:15.797 INFO:teuthology.orchestra.run.trial190.stdout: rbd-nbd x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 166 k 2026-02-25T08:23:15.797 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:23:15.797 INFO:teuthology.orchestra.run.trial190.stdout:Transaction Summary 2026-02-25T08:23:15.797 INFO:teuthology.orchestra.run.trial190.stdout:================================================================================ 2026-02-25T08:23:15.797 INFO:teuthology.orchestra.run.trial190.stdout:Install 1 Package 2026-02-25T08:23:15.797 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:23:15.797 INFO:teuthology.orchestra.run.trial190.stdout:Total download size: 166 k 2026-02-25T08:23:15.797 INFO:teuthology.orchestra.run.trial190.stdout:Installed size: 496 k 2026-02-25T08:23:15.797 INFO:teuthology.orchestra.run.trial190.stdout:Downloading Packages: 2026-02-25T08:23:16.433 INFO:teuthology.orchestra.run.trial190.stdout:rbd-nbd-20.3.0-5338.g6aeb6975.el9.x86_64.rpm 261 kB/s | 166 kB 00:00 2026-02-25T08:23:16.434 INFO:teuthology.orchestra.run.trial190.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:23:16.434 INFO:teuthology.orchestra.run.trial190.stdout:Total 261 kB/s | 166 kB 00:00 2026-02-25T08:23:16.434 INFO:teuthology.orchestra.run.trial190.stdout:Running transaction check 2026-02-25T08:23:16.439 INFO:teuthology.orchestra.run.trial190.stdout:Transaction check succeeded. 2026-02-25T08:23:16.439 INFO:teuthology.orchestra.run.trial190.stdout:Running transaction test 2026-02-25T08:23:16.470 INFO:teuthology.orchestra.run.trial190.stdout:Transaction test succeeded. 2026-02-25T08:23:16.470 INFO:teuthology.orchestra.run.trial190.stdout:Running transaction 2026-02-25T08:23:16.527 INFO:teuthology.orchestra.run.trial190.stdout: Preparing : 1/1 2026-02-25T08:23:16.545 INFO:teuthology.orchestra.run.trial190.stdout: Installing : rbd-nbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:23:16.633 INFO:teuthology.orchestra.run.trial190.stdout: Running scriptlet: rbd-nbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:23:16.696 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : rbd-nbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:23:16.696 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:23:16.696 INFO:teuthology.orchestra.run.trial190.stdout:Installed: 2026-02-25T08:23:16.696 INFO:teuthology.orchestra.run.trial190.stdout: rbd-nbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:23:16.696 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:23:16.696 INFO:teuthology.orchestra.run.trial190.stdout:Complete! 2026-02-25T08:23:16.725 DEBUG:teuthology.orchestra.run.trial190:> sudo yum -y install ceph-osd-crimson 2026-02-25T08:23:17.048 INFO:teuthology.orchestra.run.trial190.stdout:Last metadata expiration check: 0:14:25 ago on Wed 25 Feb 2026 08:08:52 AM UTC. 2026-02-25T08:23:17.113 INFO:teuthology.orchestra.run.trial190.stdout:Package ceph-osd-crimson-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:23:17.131 INFO:teuthology.orchestra.run.trial190.stdout:Dependencies resolved. 2026-02-25T08:23:17.131 INFO:teuthology.orchestra.run.trial190.stdout:Nothing to do. 2026-02-25T08:23:17.131 INFO:teuthology.orchestra.run.trial190.stdout:Complete! 2026-02-25T08:23:17.153 DEBUG:teuthology.orchestra.run.trial190:> sudo yum -y install ceph-osd-crimson-debuginfo 2026-02-25T08:23:17.468 INFO:teuthology.orchestra.run.trial190.stdout:Last metadata expiration check: 0:14:25 ago on Wed 25 Feb 2026 08:08:52 AM UTC. 2026-02-25T08:23:17.551 INFO:teuthology.orchestra.run.trial190.stdout:Dependencies resolved. 2026-02-25T08:23:17.552 INFO:teuthology.orchestra.run.trial190.stdout:================================================================================ 2026-02-25T08:23:17.552 INFO:teuthology.orchestra.run.trial190.stdout: Package Arch Version Repo Size 2026-02-25T08:23:17.552 INFO:teuthology.orchestra.run.trial190.stdout:================================================================================ 2026-02-25T08:23:17.553 INFO:teuthology.orchestra.run.trial190.stdout:Installing: 2026-02-25T08:23:17.553 INFO:teuthology.orchestra.run.trial190.stdout: ceph-osd-crimson-debuginfo x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 433 M 2026-02-25T08:23:17.553 INFO:teuthology.orchestra.run.trial190.stdout:Installing weak dependencies: 2026-02-25T08:23:17.553 INFO:teuthology.orchestra.run.trial190.stdout: ceph-debugsource x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 12 M 2026-02-25T08:23:17.553 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:23:17.553 INFO:teuthology.orchestra.run.trial190.stdout:Transaction Summary 2026-02-25T08:23:17.553 INFO:teuthology.orchestra.run.trial190.stdout:================================================================================ 2026-02-25T08:23:17.553 INFO:teuthology.orchestra.run.trial190.stdout:Install 2 Packages 2026-02-25T08:23:17.553 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:23:17.553 INFO:teuthology.orchestra.run.trial190.stdout:Total download size: 445 M 2026-02-25T08:23:17.553 INFO:teuthology.orchestra.run.trial190.stdout:Installed size: 2.7 G 2026-02-25T08:23:17.554 INFO:teuthology.orchestra.run.trial190.stdout:Downloading Packages: 2026-02-25T08:23:46.237 INFO:teuthology.orchestra.run.trial190.stdout:(1/2): ceph-debugsource-20.3.0-5338.g6aeb6975.e 430 kB/s | 12 MB 00:28 2026-02-25T08:39:47.993 INFO:teuthology.orchestra.run.trial190.stdout:(2/2): ceph-osd-crimson-debuginfo-20.3.0-5338.g 447 kB/s | 433 MB 16:30 2026-02-25T08:39:47.994 INFO:teuthology.orchestra.run.trial190.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:39:47.995 INFO:teuthology.orchestra.run.trial190.stdout:Total 460 kB/s | 445 MB 16:30 2026-02-25T08:39:47.995 INFO:teuthology.orchestra.run.trial190.stdout:Running transaction check 2026-02-25T08:39:48.001 INFO:teuthology.orchestra.run.trial190.stdout:Transaction check succeeded. 2026-02-25T08:39:48.001 INFO:teuthology.orchestra.run.trial190.stdout:Running transaction test 2026-02-25T08:39:49.373 INFO:teuthology.orchestra.run.trial190.stdout:Transaction test succeeded. 2026-02-25T08:39:49.373 INFO:teuthology.orchestra.run.trial190.stdout:Running transaction 2026-02-25T08:39:53.107 INFO:teuthology.orchestra.run.trial190.stdout: Preparing : 1/1 2026-02-25T08:40:15.422 INFO:teuthology.orchestra.run.trial190.stdout: Installing : ceph-debugsource-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/2 2026-02-25T08:40:15.440 INFO:teuthology.orchestra.run.trial190.stdout: Installing : ceph-osd-crimson-debuginfo-2:20.3.0-5338.g6aeb6975.e 2/2 2026-02-25T08:40:15.594 INFO:teuthology.orchestra.run.trial190.stdout: Running scriptlet: ceph-osd-crimson-debuginfo-2:20.3.0-5338.g6aeb6975.e 2/2 2026-02-25T08:40:15.595 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : ceph-debugsource-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/2 2026-02-25T08:40:15.657 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : ceph-osd-crimson-debuginfo-2:20.3.0-5338.g6aeb6975.e 2/2 2026-02-25T08:40:15.657 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:40:15.657 INFO:teuthology.orchestra.run.trial190.stdout:Installed: 2026-02-25T08:40:15.658 INFO:teuthology.orchestra.run.trial190.stdout: ceph-debugsource-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:40:15.658 INFO:teuthology.orchestra.run.trial190.stdout: ceph-osd-crimson-debuginfo-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:40:15.658 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:40:15.658 INFO:teuthology.orchestra.run.trial190.stdout:Complete! 2026-02-25T08:40:15.735 DEBUG:teuthology.parallel:result is None 2026-02-25T08:40:15.736 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=6aeb6975bc9a4790a399c9cc70dba9635d2a7819 2026-02-25T08:40:15.909 DEBUG:teuthology.orchestra.run.trial190:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-25T08:40:15.930 INFO:teuthology.orchestra.run.trial190.stdout:20.3.0-5338.g6aeb6975.el9 2026-02-25T08:40:15.931 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5338.g6aeb6975.el9 2026-02-25T08:40:15.932 INFO:teuthology.task.install:The correct ceph version 20.3.0-5338.g6aeb6975 is installed. 2026-02-25T08:40:15.934 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2026-02-25T08:40:15.934 DEBUG:teuthology.orchestra.run.trial190:> set -ex 2026-02-25T08:40:15.934 DEBUG:teuthology.orchestra.run.trial190:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-25T08:40:15.997 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2026-02-25T08:40:15.997 DEBUG:teuthology.orchestra.run.trial190:> set -ex 2026-02-25T08:40:15.997 DEBUG:teuthology.orchestra.run.trial190:> sudo dd of=/usr/bin/daemon-helper 2026-02-25T08:40:16.060 DEBUG:teuthology.orchestra.run.trial190:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-25T08:40:16.124 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2026-02-25T08:40:16.125 DEBUG:teuthology.orchestra.run.trial190:> set -ex 2026-02-25T08:40:16.125 DEBUG:teuthology.orchestra.run.trial190:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-25T08:40:16.188 DEBUG:teuthology.orchestra.run.trial190:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-25T08:40:16.252 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2026-02-25T08:40:16.253 DEBUG:teuthology.orchestra.run.trial190:> set -ex 2026-02-25T08:40:16.253 DEBUG:teuthology.orchestra.run.trial190:> sudo dd of=/usr/bin/stdin-killer 2026-02-25T08:40:16.316 DEBUG:teuthology.orchestra.run.trial190:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-25T08:40:16.380 INFO:teuthology.run_tasks:Running task pexec... 2026-02-25T08:40:16.398 INFO:teuthology.task.pexec:Executing custom commands... 2026-02-25T08:40:16.398 DEBUG:teuthology.orchestra.run.trial190:> TESTDIR=/home/ubuntu/cephtest bash -s 2026-02-25T08:40:16.422 INFO:teuthology.task.pexec:Running commands on host ubuntu@trial190.front.sepia.ceph.com 2026-02-25T08:40:16.422 INFO:teuthology.task.pexec:sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson 2026-02-25T08:40:16.473 DEBUG:teuthology.parallel:result is None 2026-02-25T08:40:16.473 INFO:teuthology.run_tasks:Running task install... 2026-02-25T08:40:16.487 DEBUG:teuthology.task.install:project ceph 2026-02-25T08:40:16.487 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'debug', 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819'}} 2026-02-25T08:40:16.487 DEBUG:teuthology.task.install:config {'flavor': 'debug', 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819'} 2026-02-25T08:40:16.487 INFO:teuthology.task.install:Using flavor: debug 2026-02-25T08:40:16.496 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-osd', 'ceph-osd-classic', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-osd', 'ceph-osd-classic', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-25T08:40:16.496 INFO:teuthology.task.install:extra packages: [] 2026-02-25T08:40:16.497 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'debug', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819', 'tag': None, 'wait_for_package': False} 2026-02-25T08:40:16.497 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=6aeb6975bc9a4790a399c9cc70dba9635d2a7819 2026-02-25T08:40:16.654 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/ 2026-02-25T08:40:16.655 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5338.g6aeb6975 2026-02-25T08:40:16.785 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-25T08:40:16.785 DEBUG:teuthology.orchestra.run.trial190:> set -ex 2026-02-25T08:40:16.785 DEBUG:teuthology.orchestra.run.trial190:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-25T08:40:16.810 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2026-02-25T08:40:16.810 DEBUG:teuthology.orchestra.run.trial190:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-25T08:40:16.874 DEBUG:teuthology.orchestra.run.trial190:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-25T08:40:16.938 DEBUG:teuthology.orchestra.run.trial190:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-25T08:40:16.994 INFO:teuthology.orchestra.run.trial190.stdout:check_obsoletes = 1 2026-02-25T08:40:17.003 DEBUG:teuthology.orchestra.run.trial190:> sudo yum clean all 2026-02-25T08:40:17.163 INFO:teuthology.orchestra.run.trial190.stdout:56 files removed 2026-02-25T08:40:17.178 DEBUG:teuthology.orchestra.run.trial190:> sudo yum -y install ceph-radosgw 2026-02-25T08:40:17.496 INFO:teuthology.orchestra.run.trial190.stdout:ceph packages for x86_64 574 kB/s | 94 kB 00:00 2026-02-25T08:40:17.640 INFO:teuthology.orchestra.run.trial190.stdout:ceph noarch packages 211 kB/s | 24 kB 00:00 2026-02-25T08:40:17.788 INFO:teuthology.orchestra.run.trial190.stdout:ceph source packages 17 kB/s | 2.1 kB 00:00 2026-02-25T08:40:18.490 INFO:teuthology.orchestra.run.trial190.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.9 MB 00:00 2026-02-25T08:40:21.734 INFO:teuthology.orchestra.run.trial190.stdout:CentOS Stream 9 - AppStream 9.9 MB/s | 27 MB 00:02 2026-02-25T08:40:25.722 INFO:teuthology.orchestra.run.trial190.stdout:CentOS Stream 9 - CRB 4.9 MB/s | 8.0 MB 00:01 2026-02-25T08:40:28.806 INFO:teuthology.orchestra.run.trial190.stdout:CentOS Stream 9 - Extras packages 8.6 kB/s | 20 kB 00:02 2026-02-25T08:40:34.763 INFO:teuthology.orchestra.run.trial190.stdout:Extra Packages for Enterprise Linux 3.4 MB/s | 20 MB 00:05 2026-02-25T08:40:37.998 INFO:teuthology.orchestra.run.trial190.stdout:lab-extras 2.6 MB/s | 50 kB 00:00 2026-02-25T08:40:39.171 INFO:teuthology.orchestra.run.trial190.stdout:Package ceph-radosgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:39.189 INFO:teuthology.orchestra.run.trial190.stdout:Dependencies resolved. 2026-02-25T08:40:39.190 INFO:teuthology.orchestra.run.trial190.stdout:Nothing to do. 2026-02-25T08:40:39.190 INFO:teuthology.orchestra.run.trial190.stdout:Complete! 2026-02-25T08:40:39.213 DEBUG:teuthology.orchestra.run.trial190:> sudo yum -y install ceph-test 2026-02-25T08:40:39.533 INFO:teuthology.orchestra.run.trial190.stdout:Last metadata expiration check: 0:00:02 ago on Wed 25 Feb 2026 08:40:37 AM UTC. 2026-02-25T08:40:39.598 INFO:teuthology.orchestra.run.trial190.stdout:Package ceph-test-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:39.616 INFO:teuthology.orchestra.run.trial190.stdout:Dependencies resolved. 2026-02-25T08:40:39.616 INFO:teuthology.orchestra.run.trial190.stdout:Nothing to do. 2026-02-25T08:40:39.616 INFO:teuthology.orchestra.run.trial190.stdout:Complete! 2026-02-25T08:40:39.637 DEBUG:teuthology.orchestra.run.trial190:> sudo yum -y install ceph 2026-02-25T08:40:39.956 INFO:teuthology.orchestra.run.trial190.stdout:Last metadata expiration check: 0:00:02 ago on Wed 25 Feb 2026 08:40:37 AM UTC. 2026-02-25T08:40:40.021 INFO:teuthology.orchestra.run.trial190.stdout:Package ceph-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:40.039 INFO:teuthology.orchestra.run.trial190.stdout:Dependencies resolved. 2026-02-25T08:40:40.039 INFO:teuthology.orchestra.run.trial190.stdout:Nothing to do. 2026-02-25T08:40:40.039 INFO:teuthology.orchestra.run.trial190.stdout:Complete! 2026-02-25T08:40:40.061 DEBUG:teuthology.orchestra.run.trial190:> sudo yum -y install ceph-base 2026-02-25T08:40:40.381 INFO:teuthology.orchestra.run.trial190.stdout:Last metadata expiration check: 0:00:03 ago on Wed 25 Feb 2026 08:40:37 AM UTC. 2026-02-25T08:40:40.447 INFO:teuthology.orchestra.run.trial190.stdout:Package ceph-base-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:40.464 INFO:teuthology.orchestra.run.trial190.stdout:Dependencies resolved. 2026-02-25T08:40:40.465 INFO:teuthology.orchestra.run.trial190.stdout:Nothing to do. 2026-02-25T08:40:40.465 INFO:teuthology.orchestra.run.trial190.stdout:Complete! 2026-02-25T08:40:40.487 DEBUG:teuthology.orchestra.run.trial190:> sudo yum -y install cephadm 2026-02-25T08:40:40.803 INFO:teuthology.orchestra.run.trial190.stdout:Last metadata expiration check: 0:00:03 ago on Wed 25 Feb 2026 08:40:37 AM UTC. 2026-02-25T08:40:40.868 INFO:teuthology.orchestra.run.trial190.stdout:Package cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:40:40.886 INFO:teuthology.orchestra.run.trial190.stdout:Dependencies resolved. 2026-02-25T08:40:40.887 INFO:teuthology.orchestra.run.trial190.stdout:Nothing to do. 2026-02-25T08:40:40.887 INFO:teuthology.orchestra.run.trial190.stdout:Complete! 2026-02-25T08:40:40.908 DEBUG:teuthology.orchestra.run.trial190:> sudo yum -y install ceph-immutable-object-cache 2026-02-25T08:40:41.226 INFO:teuthology.orchestra.run.trial190.stdout:Last metadata expiration check: 0:00:04 ago on Wed 25 Feb 2026 08:40:37 AM UTC. 2026-02-25T08:40:41.291 INFO:teuthology.orchestra.run.trial190.stdout:Package ceph-immutable-object-cache-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:41.308 INFO:teuthology.orchestra.run.trial190.stdout:Dependencies resolved. 2026-02-25T08:40:41.309 INFO:teuthology.orchestra.run.trial190.stdout:Nothing to do. 2026-02-25T08:40:41.309 INFO:teuthology.orchestra.run.trial190.stdout:Complete! 2026-02-25T08:40:41.330 DEBUG:teuthology.orchestra.run.trial190:> sudo yum -y install ceph-mgr 2026-02-25T08:40:41.650 INFO:teuthology.orchestra.run.trial190.stdout:Last metadata expiration check: 0:00:04 ago on Wed 25 Feb 2026 08:40:37 AM UTC. 2026-02-25T08:40:41.715 INFO:teuthology.orchestra.run.trial190.stdout:Package ceph-mgr-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:41.733 INFO:teuthology.orchestra.run.trial190.stdout:Dependencies resolved. 2026-02-25T08:40:41.733 INFO:teuthology.orchestra.run.trial190.stdout:Nothing to do. 2026-02-25T08:40:41.733 INFO:teuthology.orchestra.run.trial190.stdout:Complete! 2026-02-25T08:40:41.755 DEBUG:teuthology.orchestra.run.trial190:> sudo yum -y install ceph-mgr-dashboard 2026-02-25T08:40:42.074 INFO:teuthology.orchestra.run.trial190.stdout:Last metadata expiration check: 0:00:05 ago on Wed 25 Feb 2026 08:40:37 AM UTC. 2026-02-25T08:40:42.139 INFO:teuthology.orchestra.run.trial190.stdout:Package ceph-mgr-dashboard-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:40:42.157 INFO:teuthology.orchestra.run.trial190.stdout:Dependencies resolved. 2026-02-25T08:40:42.158 INFO:teuthology.orchestra.run.trial190.stdout:Nothing to do. 2026-02-25T08:40:42.158 INFO:teuthology.orchestra.run.trial190.stdout:Complete! 2026-02-25T08:40:42.179 DEBUG:teuthology.orchestra.run.trial190:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-25T08:40:42.498 INFO:teuthology.orchestra.run.trial190.stdout:Last metadata expiration check: 0:00:05 ago on Wed 25 Feb 2026 08:40:37 AM UTC. 2026-02-25T08:40:42.563 INFO:teuthology.orchestra.run.trial190.stdout:Package ceph-mgr-diskprediction-local-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:40:42.581 INFO:teuthology.orchestra.run.trial190.stdout:Dependencies resolved. 2026-02-25T08:40:42.581 INFO:teuthology.orchestra.run.trial190.stdout:Nothing to do. 2026-02-25T08:40:42.581 INFO:teuthology.orchestra.run.trial190.stdout:Complete! 2026-02-25T08:40:42.602 DEBUG:teuthology.orchestra.run.trial190:> sudo yum -y install ceph-mgr-rook 2026-02-25T08:40:42.918 INFO:teuthology.orchestra.run.trial190.stdout:Last metadata expiration check: 0:00:05 ago on Wed 25 Feb 2026 08:40:37 AM UTC. 2026-02-25T08:40:42.984 INFO:teuthology.orchestra.run.trial190.stdout:Package ceph-mgr-rook-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:40:43.001 INFO:teuthology.orchestra.run.trial190.stdout:Dependencies resolved. 2026-02-25T08:40:43.002 INFO:teuthology.orchestra.run.trial190.stdout:Nothing to do. 2026-02-25T08:40:43.002 INFO:teuthology.orchestra.run.trial190.stdout:Complete! 2026-02-25T08:40:43.023 DEBUG:teuthology.orchestra.run.trial190:> sudo yum -y install ceph-mgr-cephadm 2026-02-25T08:40:43.341 INFO:teuthology.orchestra.run.trial190.stdout:Last metadata expiration check: 0:00:06 ago on Wed 25 Feb 2026 08:40:37 AM UTC. 2026-02-25T08:40:43.406 INFO:teuthology.orchestra.run.trial190.stdout:Package ceph-mgr-cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:40:43.424 INFO:teuthology.orchestra.run.trial190.stdout:Dependencies resolved. 2026-02-25T08:40:43.425 INFO:teuthology.orchestra.run.trial190.stdout:Nothing to do. 2026-02-25T08:40:43.425 INFO:teuthology.orchestra.run.trial190.stdout:Complete! 2026-02-25T08:40:43.446 DEBUG:teuthology.orchestra.run.trial190:> sudo yum -y install ceph-osd 2026-02-25T08:40:43.766 INFO:teuthology.orchestra.run.trial190.stdout:Last metadata expiration check: 0:00:06 ago on Wed 25 Feb 2026 08:40:37 AM UTC. 2026-02-25T08:40:43.831 INFO:teuthology.orchestra.run.trial190.stdout:Package ceph-osd-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:43.848 INFO:teuthology.orchestra.run.trial190.stdout:Dependencies resolved. 2026-02-25T08:40:43.849 INFO:teuthology.orchestra.run.trial190.stdout:Nothing to do. 2026-02-25T08:40:43.849 INFO:teuthology.orchestra.run.trial190.stdout:Complete! 2026-02-25T08:40:43.871 DEBUG:teuthology.orchestra.run.trial190:> sudo yum -y install ceph-osd-classic 2026-02-25T08:40:44.190 INFO:teuthology.orchestra.run.trial190.stdout:Last metadata expiration check: 0:00:07 ago on Wed 25 Feb 2026 08:40:37 AM UTC. 2026-02-25T08:40:44.255 INFO:teuthology.orchestra.run.trial190.stdout:Package ceph-osd-classic-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:44.273 INFO:teuthology.orchestra.run.trial190.stdout:Dependencies resolved. 2026-02-25T08:40:44.273 INFO:teuthology.orchestra.run.trial190.stdout:Nothing to do. 2026-02-25T08:40:44.274 INFO:teuthology.orchestra.run.trial190.stdout:Complete! 2026-02-25T08:40:44.296 DEBUG:teuthology.orchestra.run.trial190:> sudo yum -y install ceph-fuse 2026-02-25T08:40:44.618 INFO:teuthology.orchestra.run.trial190.stdout:Last metadata expiration check: 0:00:07 ago on Wed 25 Feb 2026 08:40:37 AM UTC. 2026-02-25T08:40:44.683 INFO:teuthology.orchestra.run.trial190.stdout:Package ceph-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:44.700 INFO:teuthology.orchestra.run.trial190.stdout:Dependencies resolved. 2026-02-25T08:40:44.701 INFO:teuthology.orchestra.run.trial190.stdout:Nothing to do. 2026-02-25T08:40:44.701 INFO:teuthology.orchestra.run.trial190.stdout:Complete! 2026-02-25T08:40:44.723 DEBUG:teuthology.orchestra.run.trial190:> sudo yum -y install ceph-volume 2026-02-25T08:40:45.043 INFO:teuthology.orchestra.run.trial190.stdout:Last metadata expiration check: 0:00:08 ago on Wed 25 Feb 2026 08:40:37 AM UTC. 2026-02-25T08:40:45.109 INFO:teuthology.orchestra.run.trial190.stdout:Package ceph-volume-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:40:45.126 INFO:teuthology.orchestra.run.trial190.stdout:Dependencies resolved. 2026-02-25T08:40:45.127 INFO:teuthology.orchestra.run.trial190.stdout:Nothing to do. 2026-02-25T08:40:45.127 INFO:teuthology.orchestra.run.trial190.stdout:Complete! 2026-02-25T08:40:45.149 DEBUG:teuthology.orchestra.run.trial190:> sudo yum -y install librados-devel 2026-02-25T08:40:45.465 INFO:teuthology.orchestra.run.trial190.stdout:Last metadata expiration check: 0:00:08 ago on Wed 25 Feb 2026 08:40:37 AM UTC. 2026-02-25T08:40:45.530 INFO:teuthology.orchestra.run.trial190.stdout:Package librados-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:45.548 INFO:teuthology.orchestra.run.trial190.stdout:Dependencies resolved. 2026-02-25T08:40:45.548 INFO:teuthology.orchestra.run.trial190.stdout:Nothing to do. 2026-02-25T08:40:45.548 INFO:teuthology.orchestra.run.trial190.stdout:Complete! 2026-02-25T08:40:45.569 DEBUG:teuthology.orchestra.run.trial190:> sudo yum -y install libcephfs2 2026-02-25T08:40:45.886 INFO:teuthology.orchestra.run.trial190.stdout:Last metadata expiration check: 0:00:08 ago on Wed 25 Feb 2026 08:40:37 AM UTC. 2026-02-25T08:40:45.952 INFO:teuthology.orchestra.run.trial190.stdout:Package libcephfs2-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:45.970 INFO:teuthology.orchestra.run.trial190.stdout:Dependencies resolved. 2026-02-25T08:40:45.970 INFO:teuthology.orchestra.run.trial190.stdout:Nothing to do. 2026-02-25T08:40:45.970 INFO:teuthology.orchestra.run.trial190.stdout:Complete! 2026-02-25T08:40:45.991 DEBUG:teuthology.orchestra.run.trial190:> sudo yum -y install libcephfs-devel 2026-02-25T08:40:46.305 INFO:teuthology.orchestra.run.trial190.stdout:Last metadata expiration check: 0:00:09 ago on Wed 25 Feb 2026 08:40:37 AM UTC. 2026-02-25T08:40:46.370 INFO:teuthology.orchestra.run.trial190.stdout:Package libcephfs-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:46.388 INFO:teuthology.orchestra.run.trial190.stdout:Dependencies resolved. 2026-02-25T08:40:46.389 INFO:teuthology.orchestra.run.trial190.stdout:Nothing to do. 2026-02-25T08:40:46.389 INFO:teuthology.orchestra.run.trial190.stdout:Complete! 2026-02-25T08:40:46.410 DEBUG:teuthology.orchestra.run.trial190:> sudo yum -y install librados2 2026-02-25T08:40:46.727 INFO:teuthology.orchestra.run.trial190.stdout:Last metadata expiration check: 0:00:09 ago on Wed 25 Feb 2026 08:40:37 AM UTC. 2026-02-25T08:40:46.792 INFO:teuthology.orchestra.run.trial190.stdout:Package librados2-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:46.809 INFO:teuthology.orchestra.run.trial190.stdout:Dependencies resolved. 2026-02-25T08:40:46.810 INFO:teuthology.orchestra.run.trial190.stdout:Nothing to do. 2026-02-25T08:40:46.810 INFO:teuthology.orchestra.run.trial190.stdout:Complete! 2026-02-25T08:40:46.831 DEBUG:teuthology.orchestra.run.trial190:> sudo yum -y install librbd1 2026-02-25T08:40:47.151 INFO:teuthology.orchestra.run.trial190.stdout:Last metadata expiration check: 0:00:10 ago on Wed 25 Feb 2026 08:40:37 AM UTC. 2026-02-25T08:40:47.217 INFO:teuthology.orchestra.run.trial190.stdout:Package librbd1-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:47.234 INFO:teuthology.orchestra.run.trial190.stdout:Dependencies resolved. 2026-02-25T08:40:47.235 INFO:teuthology.orchestra.run.trial190.stdout:Nothing to do. 2026-02-25T08:40:47.235 INFO:teuthology.orchestra.run.trial190.stdout:Complete! 2026-02-25T08:40:47.257 DEBUG:teuthology.orchestra.run.trial190:> sudo yum -y install python3-rados 2026-02-25T08:40:47.580 INFO:teuthology.orchestra.run.trial190.stdout:Last metadata expiration check: 0:00:10 ago on Wed 25 Feb 2026 08:40:37 AM UTC. 2026-02-25T08:40:47.646 INFO:teuthology.orchestra.run.trial190.stdout:Package python3-rados-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:47.664 INFO:teuthology.orchestra.run.trial190.stdout:Dependencies resolved. 2026-02-25T08:40:47.664 INFO:teuthology.orchestra.run.trial190.stdout:Nothing to do. 2026-02-25T08:40:47.664 INFO:teuthology.orchestra.run.trial190.stdout:Complete! 2026-02-25T08:40:47.686 DEBUG:teuthology.orchestra.run.trial190:> sudo yum -y install python3-rgw 2026-02-25T08:40:48.002 INFO:teuthology.orchestra.run.trial190.stdout:Last metadata expiration check: 0:00:11 ago on Wed 25 Feb 2026 08:40:37 AM UTC. 2026-02-25T08:40:48.068 INFO:teuthology.orchestra.run.trial190.stdout:Package python3-rgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:48.085 INFO:teuthology.orchestra.run.trial190.stdout:Dependencies resolved. 2026-02-25T08:40:48.086 INFO:teuthology.orchestra.run.trial190.stdout:Nothing to do. 2026-02-25T08:40:48.086 INFO:teuthology.orchestra.run.trial190.stdout:Complete! 2026-02-25T08:40:48.108 DEBUG:teuthology.orchestra.run.trial190:> sudo yum -y install python3-cephfs 2026-02-25T08:40:48.427 INFO:teuthology.orchestra.run.trial190.stdout:Last metadata expiration check: 0:00:11 ago on Wed 25 Feb 2026 08:40:37 AM UTC. 2026-02-25T08:40:48.492 INFO:teuthology.orchestra.run.trial190.stdout:Package python3-cephfs-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:48.510 INFO:teuthology.orchestra.run.trial190.stdout:Dependencies resolved. 2026-02-25T08:40:48.510 INFO:teuthology.orchestra.run.trial190.stdout:Nothing to do. 2026-02-25T08:40:48.510 INFO:teuthology.orchestra.run.trial190.stdout:Complete! 2026-02-25T08:40:48.533 DEBUG:teuthology.orchestra.run.trial190:> sudo yum -y install python3-rbd 2026-02-25T08:40:48.850 INFO:teuthology.orchestra.run.trial190.stdout:Last metadata expiration check: 0:00:11 ago on Wed 25 Feb 2026 08:40:37 AM UTC. 2026-02-25T08:40:48.915 INFO:teuthology.orchestra.run.trial190.stdout:Package python3-rbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:48.933 INFO:teuthology.orchestra.run.trial190.stdout:Dependencies resolved. 2026-02-25T08:40:48.933 INFO:teuthology.orchestra.run.trial190.stdout:Nothing to do. 2026-02-25T08:40:48.933 INFO:teuthology.orchestra.run.trial190.stdout:Complete! 2026-02-25T08:40:48.954 DEBUG:teuthology.orchestra.run.trial190:> sudo yum -y install rbd-fuse 2026-02-25T08:40:49.273 INFO:teuthology.orchestra.run.trial190.stdout:Last metadata expiration check: 0:00:12 ago on Wed 25 Feb 2026 08:40:37 AM UTC. 2026-02-25T08:40:49.338 INFO:teuthology.orchestra.run.trial190.stdout:Package rbd-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:49.356 INFO:teuthology.orchestra.run.trial190.stdout:Dependencies resolved. 2026-02-25T08:40:49.356 INFO:teuthology.orchestra.run.trial190.stdout:Nothing to do. 2026-02-25T08:40:49.356 INFO:teuthology.orchestra.run.trial190.stdout:Complete! 2026-02-25T08:40:49.378 DEBUG:teuthology.orchestra.run.trial190:> sudo yum -y install rbd-mirror 2026-02-25T08:40:49.693 INFO:teuthology.orchestra.run.trial190.stdout:Last metadata expiration check: 0:00:12 ago on Wed 25 Feb 2026 08:40:37 AM UTC. 2026-02-25T08:40:49.758 INFO:teuthology.orchestra.run.trial190.stdout:Package rbd-mirror-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:49.776 INFO:teuthology.orchestra.run.trial190.stdout:Dependencies resolved. 2026-02-25T08:40:49.776 INFO:teuthology.orchestra.run.trial190.stdout:Nothing to do. 2026-02-25T08:40:49.776 INFO:teuthology.orchestra.run.trial190.stdout:Complete! 2026-02-25T08:40:49.798 DEBUG:teuthology.orchestra.run.trial190:> sudo yum -y install rbd-nbd 2026-02-25T08:40:50.116 INFO:teuthology.orchestra.run.trial190.stdout:Last metadata expiration check: 0:00:13 ago on Wed 25 Feb 2026 08:40:37 AM UTC. 2026-02-25T08:40:50.182 INFO:teuthology.orchestra.run.trial190.stdout:Package rbd-nbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:50.200 INFO:teuthology.orchestra.run.trial190.stdout:Dependencies resolved. 2026-02-25T08:40:50.200 INFO:teuthology.orchestra.run.trial190.stdout:Nothing to do. 2026-02-25T08:40:50.200 INFO:teuthology.orchestra.run.trial190.stdout:Complete! 2026-02-25T08:40:50.226 DEBUG:teuthology.parallel:result is None 2026-02-25T08:40:50.226 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=6aeb6975bc9a4790a399c9cc70dba9635d2a7819 2026-02-25T08:40:50.400 DEBUG:teuthology.orchestra.run.trial190:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-25T08:40:50.422 INFO:teuthology.orchestra.run.trial190.stdout:20.3.0-5338.g6aeb6975.el9 2026-02-25T08:40:50.422 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5338.g6aeb6975.el9 2026-02-25T08:40:50.423 INFO:teuthology.task.install:The correct ceph version 20.3.0-5338.g6aeb6975 is installed. 2026-02-25T08:40:50.426 INFO:teuthology.task.install.util:Utilities already shipped, skip it... 2026-02-25T08:40:50.426 INFO:teuthology.run_tasks:Running task ceph... 2026-02-25T08:40:50.526 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2026-02-25T08:40:50.526 DEBUG:teuthology.orchestra.run.trial190:> sudo chmod 777 /var/log/ceph 2026-02-25T08:40:50.551 INFO:tasks.ceph:Disabling ceph logrotate... 2026-02-25T08:40:50.551 DEBUG:teuthology.orchestra.run.trial190:> sudo rm -f -- /etc/logrotate.d/ceph 2026-02-25T08:40:50.614 INFO:tasks.ceph:Creating extra log directories... 2026-02-25T08:40:50.615 DEBUG:teuthology.orchestra.run.trial190:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2026-02-25T08:40:50.684 INFO:tasks.ceph:Creating ceph cluster ceph... 2026-02-25T08:40:50.684 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore main device type': 'RANDOM_BLOCK_SSD', 'seastore max concurrent transactions': 8}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2026-02-25T08:40:50.684 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/mnt/teuthology/matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial/69096', 'branch': 'wip-logstoremanager', 'description': 'crimson-rados/perf/{0-crimson_install clusters/{crimson-fixed-1} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/rbm$/{crimson_seastore_rbm} perf_ignorelist settings/optimized workloads/radosbench_4K_rand_read}', 'email': None, 'first_in_suite': False, 'flavor': 'debug', 'job_id': '69096', 'kernel': {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'trial', 'name': 'matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial', 'no_nested_subset': False, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-logstoremanager'}, 'ceph': {'conf': {'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd': {'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore main device type': 'RANDOM_BLOCK_SSD', 'seastore max concurrent transactions': 8}}, 'crimson_compat': True, 'flavor': 'debug', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH'], 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'global': {'osd crush chooseleaf type': 0, 'osd pool default pg num': 128, 'osd pool default pgp num': 128, 'osd pool default size': 2}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'debug', 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819'}}, 'workunit': {'branch': 'main', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': 'c32469671c6b0222f786dd36c34eb702f780a11d'}}, 'owner': 'scheduled_matan@soko04.front.sepia.ceph.com', 'priority': 70, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 869, 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph.git', 'suite_sha1': 'c32469671c6b0222f786dd36c34eb702f780a11d', 'targets': {'trial190.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE1WLmp6Pwq3/iMPqif60sc3ZuonJypHy0xLtT+Qq0rcxyi9hnFxqABKTBjZnITZcBPYfOZq90iGBR+2c7Zbems='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'flavor': 'debug', 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819'}}, {'pexec': {'all': ['sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson']}}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore main device type': 'RANDOM_BLOCK_SSD', 'seastore max concurrent transactions': 8}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}}, 'wait-for-scrub': False, 'crimson_compat': True, 'flavor': 'debug', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH'], 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819', 'cluster': 'ceph'}}, {'ssh_keys': None}, {'cbt': {'benchmarks': {'radosbench': {'concurrent_ops': 4, 'concurrent_procs': 2, 'op_size': [4096], 'pool_monitoring_list': ['collectl'], 'pool_profile': 'replicated', 'readmode': 'rand', 'run_monitoring_list': ['collectl'], 'time': 60, 'write_only': False}}, 'cluster': {'iterations': 1, 'osds_per_node': 3, 'pool_profiles': {'replicated': {'pg_size': 256, 'pgp_size': 256, 'replication': 'replicated'}}, 'user': 'ubuntu'}, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'fa17720d0088c3ac28e473468bfc79eeaff5cd38', 'timestamp': '2026-02-25_07:55:35', 'tube': 'trial', 'user': 'matan', 'verbose': False, 'worker_log': '/home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.904699'} 2026-02-25T08:40:50.685 DEBUG:teuthology.orchestra.run.trial190:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2026-02-25T08:40:50.735 DEBUG:teuthology.orchestra.run.trial190:> sudo install -d -m0777 -- /var/run/ceph 2026-02-25T08:40:50.798 DEBUG:teuthology.orchestra.run.trial190:> set -ex 2026-02-25T08:40:50.798 DEBUG:teuthology.orchestra.run.trial190:> dd if=/scratch_devs of=/dev/stdout 2026-02-25T08:40:50.854 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2026-02-25T08:40:50.854 DEBUG:teuthology.orchestra.run.trial190:> stat /dev/vg_nvme/lv_1 2026-02-25T08:40:50.910 INFO:teuthology.orchestra.run.trial190.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2026-02-25T08:40:50.910 INFO:teuthology.orchestra.run.trial190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-25T08:40:50.910 INFO:teuthology.orchestra.run.trial190.stdout:Device: 6h/6d Inode: 1309 Links: 1 2026-02-25T08:40:50.910 INFO:teuthology.orchestra.run.trial190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-25T08:40:50.910 INFO:teuthology.orchestra.run.trial190.stdout:Context: system_u:object_r:device_t:s0 2026-02-25T08:40:50.910 INFO:teuthology.orchestra.run.trial190.stdout:Access: 2026-02-25 08:23:15.163143174 +0000 2026-02-25T08:40:50.910 INFO:teuthology.orchestra.run.trial190.stdout:Modify: 2026-02-25 08:09:39.005667705 +0000 2026-02-25T08:40:50.910 INFO:teuthology.orchestra.run.trial190.stdout:Change: 2026-02-25 08:09:39.005667705 +0000 2026-02-25T08:40:50.910 INFO:teuthology.orchestra.run.trial190.stdout: Birth: 2026-02-25 08:09:39.005667705 +0000 2026-02-25T08:40:50.910 DEBUG:teuthology.orchestra.run.trial190:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2026-02-25T08:40:50.971 INFO:teuthology.orchestra.run.trial190.stderr:1+0 records in 2026-02-25T08:40:50.971 INFO:teuthology.orchestra.run.trial190.stderr:1+0 records out 2026-02-25T08:40:50.971 INFO:teuthology.orchestra.run.trial190.stderr:512 bytes copied, 0.00023505 s, 2.2 MB/s 2026-02-25T08:40:50.973 DEBUG:teuthology.orchestra.run.trial190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2026-02-25T08:40:51.029 DEBUG:teuthology.orchestra.run.trial190:> stat /dev/vg_nvme/lv_2 2026-02-25T08:40:51.085 INFO:teuthology.orchestra.run.trial190.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2026-02-25T08:40:51.085 INFO:teuthology.orchestra.run.trial190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-25T08:40:51.085 INFO:teuthology.orchestra.run.trial190.stdout:Device: 6h/6d Inode: 1314 Links: 1 2026-02-25T08:40:51.086 INFO:teuthology.orchestra.run.trial190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-25T08:40:51.086 INFO:teuthology.orchestra.run.trial190.stdout:Context: system_u:object_r:device_t:s0 2026-02-25T08:40:51.086 INFO:teuthology.orchestra.run.trial190.stdout:Access: 2026-02-25 08:23:15.163143174 +0000 2026-02-25T08:40:51.086 INFO:teuthology.orchestra.run.trial190.stdout:Modify: 2026-02-25 08:09:39.006668101 +0000 2026-02-25T08:40:51.086 INFO:teuthology.orchestra.run.trial190.stdout:Change: 2026-02-25 08:09:39.006668101 +0000 2026-02-25T08:40:51.086 INFO:teuthology.orchestra.run.trial190.stdout: Birth: 2026-02-25 08:09:39.006668101 +0000 2026-02-25T08:40:51.086 DEBUG:teuthology.orchestra.run.trial190:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2026-02-25T08:40:51.149 INFO:teuthology.orchestra.run.trial190.stderr:1+0 records in 2026-02-25T08:40:51.149 INFO:teuthology.orchestra.run.trial190.stderr:1+0 records out 2026-02-25T08:40:51.149 INFO:teuthology.orchestra.run.trial190.stderr:512 bytes copied, 0.000211576 s, 2.4 MB/s 2026-02-25T08:40:51.150 DEBUG:teuthology.orchestra.run.trial190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2026-02-25T08:40:51.206 DEBUG:teuthology.orchestra.run.trial190:> stat /dev/vg_nvme/lv_3 2026-02-25T08:40:51.262 INFO:teuthology.orchestra.run.trial190.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2026-02-25T08:40:51.262 INFO:teuthology.orchestra.run.trial190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-25T08:40:51.262 INFO:teuthology.orchestra.run.trial190.stdout:Device: 6h/6d Inode: 1296 Links: 1 2026-02-25T08:40:51.262 INFO:teuthology.orchestra.run.trial190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-25T08:40:51.262 INFO:teuthology.orchestra.run.trial190.stdout:Context: system_u:object_r:device_t:s0 2026-02-25T08:40:51.262 INFO:teuthology.orchestra.run.trial190.stdout:Access: 2026-02-25 08:23:15.164143169 +0000 2026-02-25T08:40:51.262 INFO:teuthology.orchestra.run.trial190.stdout:Modify: 2026-02-25 08:09:39.005667705 +0000 2026-02-25T08:40:51.262 INFO:teuthology.orchestra.run.trial190.stdout:Change: 2026-02-25 08:09:39.005667705 +0000 2026-02-25T08:40:51.262 INFO:teuthology.orchestra.run.trial190.stdout: Birth: 2026-02-25 08:09:39.005667705 +0000 2026-02-25T08:40:51.262 DEBUG:teuthology.orchestra.run.trial190:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2026-02-25T08:40:51.323 INFO:teuthology.orchestra.run.trial190.stderr:1+0 records in 2026-02-25T08:40:51.324 INFO:teuthology.orchestra.run.trial190.stderr:1+0 records out 2026-02-25T08:40:51.324 INFO:teuthology.orchestra.run.trial190.stderr:512 bytes copied, 0.000152877 s, 3.3 MB/s 2026-02-25T08:40:51.325 DEBUG:teuthology.orchestra.run.trial190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2026-02-25T08:40:51.382 DEBUG:teuthology.orchestra.run.trial190:> stat /dev/vg_nvme/lv_4 2026-02-25T08:40:51.437 INFO:teuthology.orchestra.run.trial190.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2026-02-25T08:40:51.437 INFO:teuthology.orchestra.run.trial190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-25T08:40:51.437 INFO:teuthology.orchestra.run.trial190.stdout:Device: 6h/6d Inode: 1302 Links: 1 2026-02-25T08:40:51.437 INFO:teuthology.orchestra.run.trial190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-25T08:40:51.437 INFO:teuthology.orchestra.run.trial190.stdout:Context: system_u:object_r:device_t:s0 2026-02-25T08:40:51.437 INFO:teuthology.orchestra.run.trial190.stdout:Access: 2026-02-25 08:23:15.164143169 +0000 2026-02-25T08:40:51.437 INFO:teuthology.orchestra.run.trial190.stdout:Modify: 2026-02-25 08:09:39.005667705 +0000 2026-02-25T08:40:51.437 INFO:teuthology.orchestra.run.trial190.stdout:Change: 2026-02-25 08:09:39.005667705 +0000 2026-02-25T08:40:51.437 INFO:teuthology.orchestra.run.trial190.stdout: Birth: 2026-02-25 08:09:39.005667705 +0000 2026-02-25T08:40:51.437 DEBUG:teuthology.orchestra.run.trial190:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2026-02-25T08:40:51.500 INFO:teuthology.orchestra.run.trial190.stderr:1+0 records in 2026-02-25T08:40:51.500 INFO:teuthology.orchestra.run.trial190.stderr:1+0 records out 2026-02-25T08:40:51.500 INFO:teuthology.orchestra.run.trial190.stderr:512 bytes copied, 0.000134081 s, 3.8 MB/s 2026-02-25T08:40:51.502 DEBUG:teuthology.orchestra.run.trial190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2026-02-25T08:40:51.558 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-25T08:40:51.559 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@trial190.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2026-02-25T08:40:51.559 INFO:tasks.ceph:Generating config... 2026-02-25T08:40:51.561 INFO:tasks.ceph:[osd] debug monc = 20 2026-02-25T08:40:51.561 INFO:tasks.ceph:[osd] crimson cpu num = 4 2026-02-25T08:40:51.562 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2026-02-25T08:40:51.562 INFO:tasks.ceph:[osd] debug filestore = 0/0 2026-02-25T08:40:51.562 INFO:tasks.ceph:[osd] debug journal = 0/0 2026-02-25T08:40:51.562 INFO:tasks.ceph:[osd] debug ms = 0/0 2026-02-25T08:40:51.562 INFO:tasks.ceph:[osd] debug osd = 0/0 2026-02-25T08:40:51.562 INFO:tasks.ceph:[osd] debug seastore = 20 2026-02-25T08:40:51.562 INFO:tasks.ceph:[osd] debug seastore backref = 20 2026-02-25T08:40:51.562 INFO:tasks.ceph:[osd] debug seastore cache = 20 2026-02-25T08:40:51.562 INFO:tasks.ceph:[osd] debug seastore cleaner = 20 2026-02-25T08:40:51.562 INFO:tasks.ceph:[osd] debug seastore device = 20 2026-02-25T08:40:51.562 INFO:tasks.ceph:[osd] debug seastore epm = 20 2026-02-25T08:40:51.563 INFO:tasks.ceph:[osd] debug seastore fixedkv tree = 20 2026-02-25T08:40:51.563 INFO:tasks.ceph:[osd] debug seastore journal = 20 2026-02-25T08:40:51.563 INFO:tasks.ceph:[osd] debug seastore lba = 20 2026-02-25T08:40:51.563 INFO:tasks.ceph:[osd] debug seastore odata = 20 2026-02-25T08:40:51.563 INFO:tasks.ceph:[osd] debug seastore omap = 20 2026-02-25T08:40:51.563 INFO:tasks.ceph:[osd] debug seastore onode = 20 2026-02-25T08:40:51.563 INFO:tasks.ceph:[osd] debug seastore t = 20 2026-02-25T08:40:51.563 INFO:tasks.ceph:[osd] debug seastore tm = 20 2026-02-25T08:40:51.563 INFO:tasks.ceph:[osd] osd objectstore = seastore 2026-02-25T08:40:51.563 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2026-02-25T08:40:51.563 INFO:tasks.ceph:[osd] seastore cachepin size pershard = 64M 2026-02-25T08:40:51.563 INFO:tasks.ceph:[osd] seastore main device type = RANDOM_BLOCK_SSD 2026-02-25T08:40:51.563 INFO:tasks.ceph:[osd] seastore max concurrent transactions = 8 2026-02-25T08:40:51.564 INFO:tasks.ceph:[global] auth client required = none 2026-02-25T08:40:51.564 INFO:tasks.ceph:[global] auth cluster required = none 2026-02-25T08:40:51.564 INFO:tasks.ceph:[global] auth service required = none 2026-02-25T08:40:51.564 INFO:tasks.ceph:[global] auth supported = none 2026-02-25T08:40:51.564 INFO:tasks.ceph:[global] debug asok = 0/0 2026-02-25T08:40:51.564 INFO:tasks.ceph:[global] debug auth = 0/0 2026-02-25T08:40:51.564 INFO:tasks.ceph:[global] debug bdev = 0/0 2026-02-25T08:40:51.564 INFO:tasks.ceph:[global] debug bluefs = 0/0 2026-02-25T08:40:51.564 INFO:tasks.ceph:[global] debug bluestore = 0/0 2026-02-25T08:40:51.564 INFO:tasks.ceph:[global] debug buffer = 0/0 2026-02-25T08:40:51.564 INFO:tasks.ceph:[global] debug civetweb = 0/0 2026-02-25T08:40:51.564 INFO:tasks.ceph:[global] debug client = 0/0 2026-02-25T08:40:51.564 INFO:tasks.ceph:[global] debug compressor = 0/0 2026-02-25T08:40:51.565 INFO:tasks.ceph:[global] debug context = 0/0 2026-02-25T08:40:51.565 INFO:tasks.ceph:[global] debug crush = 0/0 2026-02-25T08:40:51.565 INFO:tasks.ceph:[global] debug crypto = 0/0 2026-02-25T08:40:51.565 INFO:tasks.ceph:[global] debug dpdk = 0/0 2026-02-25T08:40:51.565 INFO:tasks.ceph:[global] debug eventtrace = 0/0 2026-02-25T08:40:51.565 INFO:tasks.ceph:[global] debug filer = 0/0 2026-02-25T08:40:51.565 INFO:tasks.ceph:[global] debug filestore = 0/0 2026-02-25T08:40:51.565 INFO:tasks.ceph:[global] debug finisher = 0/0 2026-02-25T08:40:51.565 INFO:tasks.ceph:[global] debug fuse = 0/0 2026-02-25T08:40:51.565 INFO:tasks.ceph:[global] debug heartbeatmap = 0/0 2026-02-25T08:40:51.565 INFO:tasks.ceph:[global] debug javaclient = 0/0 2026-02-25T08:40:51.565 INFO:tasks.ceph:[global] debug journal = 0/0 2026-02-25T08:40:51.565 INFO:tasks.ceph:[global] debug journaler = 0/0 2026-02-25T08:40:51.566 INFO:tasks.ceph:[global] debug kstore = 0/0 2026-02-25T08:40:51.566 INFO:tasks.ceph:[global] debug leveldb = 0/0 2026-02-25T08:40:51.566 INFO:tasks.ceph:[global] debug lockdep = 0/0 2026-02-25T08:40:51.566 INFO:tasks.ceph:[global] debug mds = 0/0 2026-02-25T08:40:51.566 INFO:tasks.ceph:[global] debug mds balancer = 0/0 2026-02-25T08:40:51.566 INFO:tasks.ceph:[global] debug mds locker = 0/0 2026-02-25T08:40:51.566 INFO:tasks.ceph:[global] debug mds log = 0/0 2026-02-25T08:40:51.566 INFO:tasks.ceph:[global] debug mds log expire = 0/0 2026-02-25T08:40:51.566 INFO:tasks.ceph:[global] debug mds migrator = 0/0 2026-02-25T08:40:51.566 INFO:tasks.ceph:[global] debug memdb = 0/0 2026-02-25T08:40:51.566 INFO:tasks.ceph:[global] debug mgr = 0/0 2026-02-25T08:40:51.566 INFO:tasks.ceph:[global] debug mgrc = 0/0 2026-02-25T08:40:51.566 INFO:tasks.ceph:[global] debug mon = 0/0 2026-02-25T08:40:51.567 INFO:tasks.ceph:[global] debug monc = 0/0 2026-02-25T08:40:51.567 INFO:tasks.ceph:[global] debug ms = 0/0 2026-02-25T08:40:51.567 INFO:tasks.ceph:[global] debug objclass = 0/0 2026-02-25T08:40:51.567 INFO:tasks.ceph:[global] debug objectcacher = 0/0 2026-02-25T08:40:51.567 INFO:tasks.ceph:[global] debug objecter = 0/0 2026-02-25T08:40:51.567 INFO:tasks.ceph:[global] debug optracker = 0/0 2026-02-25T08:40:51.567 INFO:tasks.ceph:[global] debug osd = 0/0 2026-02-25T08:40:51.567 INFO:tasks.ceph:[global] debug paxos = 0/0 2026-02-25T08:40:51.567 INFO:tasks.ceph:[global] debug perfcounter = 0/0 2026-02-25T08:40:51.567 INFO:tasks.ceph:[global] debug rados = 0/0 2026-02-25T08:40:51.567 INFO:tasks.ceph:[global] debug rbd = 0/0 2026-02-25T08:40:51.567 INFO:tasks.ceph:[global] debug rbd mirror = 0/0 2026-02-25T08:40:51.567 INFO:tasks.ceph:[global] debug rbd replay = 0/0 2026-02-25T08:40:51.567 INFO:tasks.ceph:[global] debug refs = 0/0 2026-02-25T08:40:51.568 INFO:tasks.ceph:[global] debug rgw = 0/0 2026-02-25T08:40:51.568 INFO:tasks.ceph:[global] debug rgw lifecycle = 0/0 2026-02-25T08:40:51.568 INFO:tasks.ceph:[global] debug rgw notification = 0/0 2026-02-25T08:40:51.568 INFO:tasks.ceph:[global] debug rgw sync = 0/0 2026-02-25T08:40:51.568 INFO:tasks.ceph:[global] debug rocksdb = 0/0 2026-02-25T08:40:51.568 INFO:tasks.ceph:[global] debug striper = 0/0 2026-02-25T08:40:51.568 INFO:tasks.ceph:[global] debug throttle = 0/0 2026-02-25T08:40:51.568 INFO:tasks.ceph:[global] debug timer = 0/0 2026-02-25T08:40:51.568 INFO:tasks.ceph:[global] debug tp = 0/0 2026-02-25T08:40:51.568 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2026-02-25T08:40:51.568 INFO:tasks.ceph:[global] ms client mode = crc 2026-02-25T08:40:51.568 INFO:tasks.ceph:[global] ms cluster mode = crc 2026-02-25T08:40:51.568 INFO:tasks.ceph:[global] ms mon client mode = crc 2026-02-25T08:40:51.568 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2026-02-25T08:40:51.569 INFO:tasks.ceph:[global] ms mon service mode = crc 2026-02-25T08:40:51.569 INFO:tasks.ceph:[global] ms service mode = crc 2026-02-25T08:40:51.569 INFO:tasks.ceph:[mgr] debug mgr = 20 2026-02-25T08:40:51.569 INFO:tasks.ceph:[mgr] debug ms = 1 2026-02-25T08:40:51.569 INFO:tasks.ceph:[mon] debug mon = 0/0 2026-02-25T08:40:51.569 INFO:tasks.ceph:[mon] debug ms = 0/0 2026-02-25T08:40:51.569 INFO:tasks.ceph:[mon] debug paxos = 0/0 2026-02-25T08:40:51.569 INFO:tasks.ceph:[mon] osd pool default crimson = True 2026-02-25T08:40:51.569 INFO:tasks.ceph:Setting up mon.a... 2026-02-25T08:40:51.570 DEBUG:teuthology.orchestra.run.trial190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2026-02-25T08:40:51.639 INFO:teuthology.orchestra.run.trial190.stdout:creating /etc/ceph/ceph.keyring 2026-02-25T08:40:51.642 DEBUG:teuthology.orchestra.run.trial190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2026-02-25T08:40:51.726 DEBUG:teuthology.orchestra.run.trial190:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-25T08:40:51.790 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '10.20.193.190')] 2026-02-25T08:40:51.791 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'none', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '10.20.193.190', 'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': 'true', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True, 'seastore cachepin size pershard': '64M', 'seastore main device type': 'RANDOM_BLOCK_SSD', 'seastore max concurrent transactions': 8}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': '0/0', 'debug mon': '0/0', 'debug paxos': '0/0', 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}} 2026-02-25T08:40:51.792 DEBUG:teuthology.orchestra.run.trial190:> set -ex 2026-02-25T08:40:51.792 DEBUG:teuthology.orchestra.run.trial190:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2026-02-25T08:40:51.845 DEBUG:teuthology.orchestra.run.trial190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 10.20.193.190 --print /home/ubuntu/cephtest/ceph.monmap 2026-02-25T08:40:51.924 INFO:teuthology.orchestra.run.trial190.stderr:2026-02-25T08:40:51.922+0000 7f939b041f40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:51.924 INFO:teuthology.orchestra.run.trial190.stderr:2026-02-25T08:40:51.922+0000 7f939b041f40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:51.924 INFO:teuthology.orchestra.run.trial190.stderr:2026-02-25T08:40:51.922+0000 7f939b041f40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:51.925 INFO:teuthology.orchestra.run.trial190.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2026-02-25T08:40:51.925 INFO:teuthology.orchestra.run.trial190.stdout:monmaptool: generated fsid 2a6cc9ca-2439-4b64-860c-0d3000bf169b 2026-02-25T08:40:51.926 INFO:teuthology.orchestra.run.trial190.stdout:setting min_mon_release = tentacle 2026-02-25T08:40:51.926 INFO:teuthology.orchestra.run.trial190.stdout:epoch 0 2026-02-25T08:40:51.926 INFO:teuthology.orchestra.run.trial190.stdout:fsid 2a6cc9ca-2439-4b64-860c-0d3000bf169b 2026-02-25T08:40:51.926 INFO:teuthology.orchestra.run.trial190.stdout:last_changed 2026-02-25T08:40:51.923348+0000 2026-02-25T08:40:51.926 INFO:teuthology.orchestra.run.trial190.stdout:created 2026-02-25T08:40:51.923348+0000 2026-02-25T08:40:51.926 INFO:teuthology.orchestra.run.trial190.stdout:min_mon_release 20 (tentacle) 2026-02-25T08:40:51.926 INFO:teuthology.orchestra.run.trial190.stdout:election_strategy: 1 2026-02-25T08:40:51.926 INFO:teuthology.orchestra.run.trial190.stdout:0: [v2:10.20.193.190:3300/0,v1:10.20.193.190:6789/0] mon.a 2026-02-25T08:40:51.926 INFO:teuthology.orchestra.run.trial190.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2026-02-25T08:40:51.927 DEBUG:teuthology.orchestra.run.trial190:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2026-02-25T08:40:51.982 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 2a6cc9ca-2439-4b64-860c-0d3000bf169b... 2026-02-25T08:40:51.983 DEBUG:teuthology.orchestra.run.trial190:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2026-02-25T08:40:52.064 INFO:teuthology.orchestra.run.trial190.stdout:[global] 2026-02-25T08:40:52.064 INFO:teuthology.orchestra.run.trial190.stdout: chdir = "" 2026-02-25T08:40:52.064 INFO:teuthology.orchestra.run.trial190.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2026-02-25T08:40:52.065 INFO:teuthology.orchestra.run.trial190.stdout: auth supported = none 2026-02-25T08:40:52.065 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:40:52.065 INFO:teuthology.orchestra.run.trial190.stdout: filestore xattr use omap = true 2026-02-25T08:40:52.065 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:40:52.065 INFO:teuthology.orchestra.run.trial190.stdout: mon clock drift allowed = 1.000 2026-02-25T08:40:52.065 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:40:52.065 INFO:teuthology.orchestra.run.trial190.stdout: osd crush chooseleaf type = 0 2026-02-25T08:40:52.065 INFO:teuthology.orchestra.run.trial190.stdout: auth debug = true 2026-02-25T08:40:52.065 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:40:52.065 INFO:teuthology.orchestra.run.trial190.stdout: ms die on old message = true 2026-02-25T08:40:52.065 INFO:teuthology.orchestra.run.trial190.stdout: ms die on bug = true 2026-02-25T08:40:52.066 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:40:52.066 INFO:teuthology.orchestra.run.trial190.stdout: mon max pg per osd = 10000 # >= luminous 2026-02-25T08:40:52.066 INFO:teuthology.orchestra.run.trial190.stdout: mon pg warn max object skew = 0 2026-02-25T08:40:52.066 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:40:52.066 INFO:teuthology.orchestra.run.trial190.stdout: # disable pg_autoscaler by default for new pools 2026-02-25T08:40:52.066 INFO:teuthology.orchestra.run.trial190.stdout: osd_pool_default_pg_autoscale_mode = off 2026-02-25T08:40:52.066 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:40:52.066 INFO:teuthology.orchestra.run.trial190.stdout: osd pool default size = 2 2026-02-25T08:40:52.066 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:40:52.066 INFO:teuthology.orchestra.run.trial190.stdout: mon osd allow primary affinity = true 2026-02-25T08:40:52.066 INFO:teuthology.orchestra.run.trial190.stdout: mon osd allow pg remap = true 2026-02-25T08:40:52.066 INFO:teuthology.orchestra.run.trial190.stdout: mon warn on legacy crush tunables = false 2026-02-25T08:40:52.067 INFO:teuthology.orchestra.run.trial190.stdout: mon warn on crush straw calc version zero = false 2026-02-25T08:40:52.067 INFO:teuthology.orchestra.run.trial190.stdout: mon warn on no sortbitwise = false 2026-02-25T08:40:52.067 INFO:teuthology.orchestra.run.trial190.stdout: mon warn on osd down out interval zero = false 2026-02-25T08:40:52.067 INFO:teuthology.orchestra.run.trial190.stdout: mon warn on too few osds = false 2026-02-25T08:40:52.067 INFO:teuthology.orchestra.run.trial190.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2026-02-25T08:40:52.067 INFO:teuthology.orchestra.run.trial190.stdout: mon_warn_on_pool_no_redundancy = false 2026-02-25T08:40:52.067 INFO:teuthology.orchestra.run.trial190.stdout: mon_allow_pool_size_one = true 2026-02-25T08:40:52.067 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:40:52.067 INFO:teuthology.orchestra.run.trial190.stdout: osd pool default erasure code profile = plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2026-02-25T08:40:52.067 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:40:52.067 INFO:teuthology.orchestra.run.trial190.stdout: osd default data pool replay window = 5 2026-02-25T08:40:52.067 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:40:52.067 INFO:teuthology.orchestra.run.trial190.stdout: mon allow pool delete = true 2026-02-25T08:40:52.068 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:40:52.068 INFO:teuthology.orchestra.run.trial190.stdout: mon cluster log file level = debug 2026-02-25T08:40:52.068 INFO:teuthology.orchestra.run.trial190.stdout: debug asserts on shutdown = true 2026-02-25T08:40:52.068 INFO:teuthology.orchestra.run.trial190.stdout: mon health detail to clog = false 2026-02-25T08:40:52.068 INFO:teuthology.orchestra.run.trial190.stdout: mon host = 10.20.193.190 2026-02-25T08:40:52.068 INFO:teuthology.orchestra.run.trial190.stdout: auth client required = none 2026-02-25T08:40:52.068 INFO:teuthology.orchestra.run.trial190.stdout: auth cluster required = none 2026-02-25T08:40:52.068 INFO:teuthology.orchestra.run.trial190.stdout: auth service required = none 2026-02-25T08:40:52.068 INFO:teuthology.orchestra.run.trial190.stdout: debug asok = 0/0 2026-02-25T08:40:52.068 INFO:teuthology.orchestra.run.trial190.stdout: debug auth = 0/0 2026-02-25T08:40:52.068 INFO:teuthology.orchestra.run.trial190.stdout: debug bdev = 0/0 2026-02-25T08:40:52.068 INFO:teuthology.orchestra.run.trial190.stdout: debug bluefs = 0/0 2026-02-25T08:40:52.069 INFO:teuthology.orchestra.run.trial190.stdout: debug bluestore = 0/0 2026-02-25T08:40:52.069 INFO:teuthology.orchestra.run.trial190.stdout: debug buffer = 0/0 2026-02-25T08:40:52.069 INFO:teuthology.orchestra.run.trial190.stdout: debug civetweb = 0/0 2026-02-25T08:40:52.069 INFO:teuthology.orchestra.run.trial190.stdout: debug client = 0/0 2026-02-25T08:40:52.069 INFO:teuthology.orchestra.run.trial190.stdout: debug compressor = 0/0 2026-02-25T08:40:52.069 INFO:teuthology.orchestra.run.trial190.stdout: debug context = 0/0 2026-02-25T08:40:52.069 INFO:teuthology.orchestra.run.trial190.stdout: debug crush = 0/0 2026-02-25T08:40:52.069 INFO:teuthology.orchestra.run.trial190.stdout: debug crypto = 0/0 2026-02-25T08:40:52.069 INFO:teuthology.orchestra.run.trial190.stdout: debug dpdk = 0/0 2026-02-25T08:40:52.069 INFO:teuthology.orchestra.run.trial190.stdout: debug eventtrace = 0/0 2026-02-25T08:40:52.069 INFO:teuthology.orchestra.run.trial190.stdout: debug filer = 0/0 2026-02-25T08:40:52.069 INFO:teuthology.orchestra.run.trial190.stdout: debug filestore = 0/0 2026-02-25T08:40:52.069 INFO:teuthology.orchestra.run.trial190.stdout: debug finisher = 0/0 2026-02-25T08:40:52.070 INFO:teuthology.orchestra.run.trial190.stdout: debug fuse = 0/0 2026-02-25T08:40:52.070 INFO:teuthology.orchestra.run.trial190.stdout: debug heartbeatmap = 0/0 2026-02-25T08:40:52.070 INFO:teuthology.orchestra.run.trial190.stdout: debug javaclient = 0/0 2026-02-25T08:40:52.070 INFO:teuthology.orchestra.run.trial190.stdout: debug journal = 0/0 2026-02-25T08:40:52.070 INFO:teuthology.orchestra.run.trial190.stdout: debug journaler = 0/0 2026-02-25T08:40:52.070 INFO:teuthology.orchestra.run.trial190.stdout: debug kstore = 0/0 2026-02-25T08:40:52.070 INFO:teuthology.orchestra.run.trial190.stdout: debug leveldb = 0/0 2026-02-25T08:40:52.070 INFO:teuthology.orchestra.run.trial190.stdout: debug lockdep = 0/0 2026-02-25T08:40:52.070 INFO:teuthology.orchestra.run.trial190.stdout: debug mds = 0/0 2026-02-25T08:40:52.070 INFO:teuthology.orchestra.run.trial190.stdout: debug mds balancer = 0/0 2026-02-25T08:40:52.070 INFO:teuthology.orchestra.run.trial190.stdout: debug mds locker = 0/0 2026-02-25T08:40:52.070 INFO:teuthology.orchestra.run.trial190.stdout: debug mds log = 0/0 2026-02-25T08:40:52.070 INFO:teuthology.orchestra.run.trial190.stdout: debug mds log expire = 0/0 2026-02-25T08:40:52.071 INFO:teuthology.orchestra.run.trial190.stdout: debug mds migrator = 0/0 2026-02-25T08:40:52.071 INFO:teuthology.orchestra.run.trial190.stdout: debug memdb = 0/0 2026-02-25T08:40:52.071 INFO:teuthology.orchestra.run.trial190.stdout: debug mgr = 0/0 2026-02-25T08:40:52.071 INFO:teuthology.orchestra.run.trial190.stdout: debug mgrc = 0/0 2026-02-25T08:40:52.071 INFO:teuthology.orchestra.run.trial190.stdout: debug mon = 0/0 2026-02-25T08:40:52.071 INFO:teuthology.orchestra.run.trial190.stdout: debug monc = 0/0 2026-02-25T08:40:52.071 INFO:teuthology.orchestra.run.trial190.stdout: debug ms = 0/0 2026-02-25T08:40:52.071 INFO:teuthology.orchestra.run.trial190.stdout: debug objclass = 0/0 2026-02-25T08:40:52.071 INFO:teuthology.orchestra.run.trial190.stdout: debug objectcacher = 0/0 2026-02-25T08:40:52.071 INFO:teuthology.orchestra.run.trial190.stdout: debug objecter = 0/0 2026-02-25T08:40:52.071 INFO:teuthology.orchestra.run.trial190.stdout: debug optracker = 0/0 2026-02-25T08:40:52.071 INFO:teuthology.orchestra.run.trial190.stdout: debug osd = 0/0 2026-02-25T08:40:52.071 INFO:teuthology.orchestra.run.trial190.stdout: debug paxos = 0/0 2026-02-25T08:40:52.072 INFO:teuthology.orchestra.run.trial190.stdout: debug perfcounter = 0/0 2026-02-25T08:40:52.072 INFO:teuthology.orchestra.run.trial190.stdout: debug rados = 0/0 2026-02-25T08:40:52.072 INFO:teuthology.orchestra.run.trial190.stdout: debug rbd = 0/0 2026-02-25T08:40:52.072 INFO:teuthology.orchestra.run.trial190.stdout: debug rbd mirror = 0/0 2026-02-25T08:40:52.072 INFO:teuthology.orchestra.run.trial190.stdout: debug rbd replay = 0/0 2026-02-25T08:40:52.072 INFO:teuthology.orchestra.run.trial190.stdout: debug refs = 0/0 2026-02-25T08:40:52.072 INFO:teuthology.orchestra.run.trial190.stdout: debug rgw = 0/0 2026-02-25T08:40:52.072 INFO:teuthology.orchestra.run.trial190.stdout: debug rgw lifecycle = 0/0 2026-02-25T08:40:52.072 INFO:teuthology.orchestra.run.trial190.stdout: debug rgw notification = 0/0 2026-02-25T08:40:52.072 INFO:teuthology.orchestra.run.trial190.stdout: debug rgw sync = 0/0 2026-02-25T08:40:52.072 INFO:teuthology.orchestra.run.trial190.stdout: debug rocksdb = 0/0 2026-02-25T08:40:52.072 INFO:teuthology.orchestra.run.trial190.stdout: debug striper = 0/0 2026-02-25T08:40:52.072 INFO:teuthology.orchestra.run.trial190.stdout: debug throttle = 0/0 2026-02-25T08:40:52.072 INFO:teuthology.orchestra.run.trial190.stdout: debug timer = 0/0 2026-02-25T08:40:52.073 INFO:teuthology.orchestra.run.trial190.stdout: debug tp = 0/0 2026-02-25T08:40:52.073 INFO:teuthology.orchestra.run.trial190.stdout: enable experimental unrecoverable data corrupting features = crimson 2026-02-25T08:40:52.073 INFO:teuthology.orchestra.run.trial190.stdout: ms client mode = crc 2026-02-25T08:40:52.073 INFO:teuthology.orchestra.run.trial190.stdout: ms cluster mode = crc 2026-02-25T08:40:52.073 INFO:teuthology.orchestra.run.trial190.stdout: ms mon client mode = crc 2026-02-25T08:40:52.073 INFO:teuthology.orchestra.run.trial190.stdout: ms mon cluster mode = crc 2026-02-25T08:40:52.073 INFO:teuthology.orchestra.run.trial190.stdout: ms mon service mode = crc 2026-02-25T08:40:52.073 INFO:teuthology.orchestra.run.trial190.stdout: ms service mode = crc 2026-02-25T08:40:52.073 INFO:teuthology.orchestra.run.trial190.stdout: fsid = 2a6cc9ca-2439-4b64-860c-0d3000bf169b 2026-02-25T08:40:52.073 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:40:52.073 INFO:teuthology.orchestra.run.trial190.stdout:[osd] 2026-02-25T08:40:52.073 INFO:teuthology.orchestra.run.trial190.stdout: osd journal size = 100 2026-02-25T08:40:52.073 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:40:52.074 INFO:teuthology.orchestra.run.trial190.stdout: osd scrub load threshold = 5.0 2026-02-25T08:40:52.074 INFO:teuthology.orchestra.run.trial190.stdout: osd scrub max interval = 600 2026-02-25T08:40:52.074 INFO:teuthology.orchestra.run.trial190.stdout: osd mclock profile = high_recovery_ops 2026-02-25T08:40:52.074 INFO:teuthology.orchestra.run.trial190.stdout: osd mclock skip benchmark = true 2026-02-25T08:40:52.074 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:40:52.074 INFO:teuthology.orchestra.run.trial190.stdout: osd recover clone overlap = true 2026-02-25T08:40:52.074 INFO:teuthology.orchestra.run.trial190.stdout: osd recovery max chunk = 1048576 2026-02-25T08:40:52.074 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:40:52.074 INFO:teuthology.orchestra.run.trial190.stdout: osd debug shutdown = true 2026-02-25T08:40:52.074 INFO:teuthology.orchestra.run.trial190.stdout: osd debug op order = true 2026-02-25T08:40:52.074 INFO:teuthology.orchestra.run.trial190.stdout: osd debug verify stray on activate = true 2026-02-25T08:40:52.074 INFO:teuthology.orchestra.run.trial190.stdout: osd debug trim objects = true 2026-02-25T08:40:52.075 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:40:52.075 INFO:teuthology.orchestra.run.trial190.stdout: osd open classes on start = true 2026-02-25T08:40:52.075 INFO:teuthology.orchestra.run.trial190.stdout: osd debug pg log writeout = true 2026-02-25T08:40:52.075 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:40:52.075 INFO:teuthology.orchestra.run.trial190.stdout: osd deep scrub update digest min age = 30 2026-02-25T08:40:52.075 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:40:52.075 INFO:teuthology.orchestra.run.trial190.stdout: osd map max advance = 10 2026-02-25T08:40:52.075 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:40:52.075 INFO:teuthology.orchestra.run.trial190.stdout: journal zero on create = true 2026-02-25T08:40:52.075 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:40:52.075 INFO:teuthology.orchestra.run.trial190.stdout: filestore ondisk finisher threads = 3 2026-02-25T08:40:52.075 INFO:teuthology.orchestra.run.trial190.stdout: filestore apply finisher threads = 3 2026-02-25T08:40:52.075 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:40:52.076 INFO:teuthology.orchestra.run.trial190.stdout: bdev debug aio = true 2026-02-25T08:40:52.076 INFO:teuthology.orchestra.run.trial190.stdout: osd debug misdirected ops = true 2026-02-25T08:40:52.076 INFO:teuthology.orchestra.run.trial190.stdout: debug monc = 20 2026-02-25T08:40:52.076 INFO:teuthology.orchestra.run.trial190.stdout: crimson cpu num = 4 2026-02-25T08:40:52.076 INFO:teuthology.orchestra.run.trial190.stdout: crimson osd obc lru size = 10 2026-02-25T08:40:52.076 INFO:teuthology.orchestra.run.trial190.stdout: debug filestore = 0/0 2026-02-25T08:40:52.076 INFO:teuthology.orchestra.run.trial190.stdout: debug journal = 0/0 2026-02-25T08:40:52.076 INFO:teuthology.orchestra.run.trial190.stdout: debug ms = 0/0 2026-02-25T08:40:52.076 INFO:teuthology.orchestra.run.trial190.stdout: debug osd = 0/0 2026-02-25T08:40:52.076 INFO:teuthology.orchestra.run.trial190.stdout: debug seastore = 20 2026-02-25T08:40:52.076 INFO:teuthology.orchestra.run.trial190.stdout: debug seastore backref = 20 2026-02-25T08:40:52.076 INFO:teuthology.orchestra.run.trial190.stdout: debug seastore cache = 20 2026-02-25T08:40:52.076 INFO:teuthology.orchestra.run.trial190.stdout: debug seastore cleaner = 20 2026-02-25T08:40:52.076 INFO:teuthology.orchestra.run.trial190.stdout: debug seastore device = 20 2026-02-25T08:40:52.077 INFO:teuthology.orchestra.run.trial190.stdout: debug seastore epm = 20 2026-02-25T08:40:52.077 INFO:teuthology.orchestra.run.trial190.stdout: debug seastore fixedkv tree = 20 2026-02-25T08:40:52.077 INFO:teuthology.orchestra.run.trial190.stdout: debug seastore journal = 20 2026-02-25T08:40:52.077 INFO:teuthology.orchestra.run.trial190.stdout: debug seastore lba = 20 2026-02-25T08:40:52.077 INFO:teuthology.orchestra.run.trial190.stdout: debug seastore odata = 20 2026-02-25T08:40:52.077 INFO:teuthology.orchestra.run.trial190.stdout: debug seastore omap = 20 2026-02-25T08:40:52.077 INFO:teuthology.orchestra.run.trial190.stdout: debug seastore onode = 20 2026-02-25T08:40:52.077 INFO:teuthology.orchestra.run.trial190.stdout: debug seastore t = 20 2026-02-25T08:40:52.077 INFO:teuthology.orchestra.run.trial190.stdout: debug seastore tm = 20 2026-02-25T08:40:52.077 INFO:teuthology.orchestra.run.trial190.stdout: osd objectstore = seastore 2026-02-25T08:40:52.077 INFO:teuthology.orchestra.run.trial190.stdout: osd shutdown pgref assert = True 2026-02-25T08:40:52.077 INFO:teuthology.orchestra.run.trial190.stdout: seastore cachepin size pershard = 64M 2026-02-25T08:40:52.077 INFO:teuthology.orchestra.run.trial190.stdout: seastore main device type = RANDOM_BLOCK_SSD 2026-02-25T08:40:52.078 INFO:teuthology.orchestra.run.trial190.stdout: seastore max concurrent transactions = 8 2026-02-25T08:40:52.078 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:40:52.078 INFO:teuthology.orchestra.run.trial190.stdout:[mgr] 2026-02-25T08:40:52.078 INFO:teuthology.orchestra.run.trial190.stdout: debug ms = 1 2026-02-25T08:40:52.078 INFO:teuthology.orchestra.run.trial190.stdout: debug mgr = 20 2026-02-25T08:40:52.078 INFO:teuthology.orchestra.run.trial190.stdout: debug mon = 20 2026-02-25T08:40:52.078 INFO:teuthology.orchestra.run.trial190.stdout: debug auth = 20 2026-02-25T08:40:52.078 INFO:teuthology.orchestra.run.trial190.stdout: mon reweight min pgs per osd = 4 2026-02-25T08:40:52.078 INFO:teuthology.orchestra.run.trial190.stdout: mon reweight min bytes per osd = 10 2026-02-25T08:40:52.078 INFO:teuthology.orchestra.run.trial190.stdout: mgr/telemetry/nag = false 2026-02-25T08:40:52.078 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:40:52.078 INFO:teuthology.orchestra.run.trial190.stdout:[mon] 2026-02-25T08:40:52.078 INFO:teuthology.orchestra.run.trial190.stdout: debug ms = 0/0 2026-02-25T08:40:52.079 INFO:teuthology.orchestra.run.trial190.stdout: debug mon = 0/0 2026-02-25T08:40:52.079 INFO:teuthology.orchestra.run.trial190.stdout: debug paxos = 0/0 2026-02-25T08:40:52.079 INFO:teuthology.orchestra.run.trial190.stdout: debug auth = 20 2026-02-25T08:40:52.079 INFO:teuthology.orchestra.run.trial190.stdout: mon data avail warn = 5 2026-02-25T08:40:52.079 INFO:teuthology.orchestra.run.trial190.stdout: mon mgr mkfs grace = 240 2026-02-25T08:40:52.079 INFO:teuthology.orchestra.run.trial190.stdout: mon reweight min pgs per osd = 4 2026-02-25T08:40:52.079 INFO:teuthology.orchestra.run.trial190.stdout: mon osd reporter subtree level = osd 2026-02-25T08:40:52.079 INFO:teuthology.orchestra.run.trial190.stdout: mon osd prime pg temp = true 2026-02-25T08:40:52.079 INFO:teuthology.orchestra.run.trial190.stdout: mon reweight min bytes per osd = 10 2026-02-25T08:40:52.079 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:40:52.079 INFO:teuthology.orchestra.run.trial190.stdout: # rotate auth tickets quickly to exercise renewal paths 2026-02-25T08:40:52.079 INFO:teuthology.orchestra.run.trial190.stdout: auth mon ticket ttl = 660 # 11m 2026-02-25T08:40:52.079 INFO:teuthology.orchestra.run.trial190.stdout: auth service ticket ttl = 240 # 4m 2026-02-25T08:40:52.080 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:40:52.080 INFO:teuthology.orchestra.run.trial190.stdout: # don't complain about insecure global_id in the test suite 2026-02-25T08:40:52.080 INFO:teuthology.orchestra.run.trial190.stdout: mon_warn_on_insecure_global_id_reclaim = false 2026-02-25T08:40:52.080 INFO:teuthology.orchestra.run.trial190.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2026-02-25T08:40:52.080 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:40:52.080 INFO:teuthology.orchestra.run.trial190.stdout: # 1m isn't quite enough 2026-02-25T08:40:52.080 INFO:teuthology.orchestra.run.trial190.stdout: mon_down_mkfs_grace = 2m 2026-02-25T08:40:52.080 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:40:52.080 INFO:teuthology.orchestra.run.trial190.stdout: mon_warn_on_filestore_osds = false 2026-02-25T08:40:52.080 INFO:teuthology.orchestra.run.trial190.stdout: osd pool default crimson = True 2026-02-25T08:40:52.080 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:40:52.080 INFO:teuthology.orchestra.run.trial190.stdout:[client] 2026-02-25T08:40:52.080 INFO:teuthology.orchestra.run.trial190.stdout: rgw cache enabled = true 2026-02-25T08:40:52.081 INFO:teuthology.orchestra.run.trial190.stdout: rgw enable ops log = true 2026-02-25T08:40:52.081 INFO:teuthology.orchestra.run.trial190.stdout: rgw enable usage log = true 2026-02-25T08:40:52.081 INFO:teuthology.orchestra.run.trial190.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2026-02-25T08:40:52.081 INFO:teuthology.orchestra.run.trial190.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2026-02-25T08:40:52.081 INFO:teuthology.orchestra.run.trial190.stdout:[mon.a] 2026-02-25T08:40:52.082 INFO:tasks.ceph:Creating admin key on mon.a... 2026-02-25T08:40:52.082 DEBUG:teuthology.orchestra.run.trial190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2026-02-25T08:40:52.172 INFO:tasks.ceph:Copying monmap to all nodes... 2026-02-25T08:40:52.172 DEBUG:teuthology.orchestra.run.trial190:> set -ex 2026-02-25T08:40:52.172 DEBUG:teuthology.orchestra.run.trial190:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2026-02-25T08:40:52.189 DEBUG:teuthology.orchestra.run.trial190:> set -ex 2026-02-25T08:40:52.189 DEBUG:teuthology.orchestra.run.trial190:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2026-02-25T08:40:52.245 INFO:tasks.ceph:Sending monmap to node ubuntu@trial190.front.sepia.ceph.com 2026-02-25T08:40:52.245 DEBUG:teuthology.orchestra.run.trial190:> set -ex 2026-02-25T08:40:52.245 DEBUG:teuthology.orchestra.run.trial190:> sudo dd of=/etc/ceph/ceph.keyring 2026-02-25T08:40:52.245 DEBUG:teuthology.orchestra.run.trial190:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-25T08:40:52.318 DEBUG:teuthology.orchestra.run.trial190:> set -ex 2026-02-25T08:40:52.318 DEBUG:teuthology.orchestra.run.trial190:> dd of=/home/ubuntu/cephtest/ceph.monmap 2026-02-25T08:40:52.375 INFO:tasks.ceph:Setting up mon nodes... 2026-02-25T08:40:52.375 INFO:tasks.ceph:Setting up mgr nodes... 2026-02-25T08:40:52.376 DEBUG:teuthology.orchestra.run.trial190:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2026-02-25T08:40:52.465 INFO:teuthology.orchestra.run.trial190.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2026-02-25T08:40:52.468 INFO:tasks.ceph:Setting up mds nodes... 2026-02-25T08:40:52.468 INFO:tasks.ceph_client:Setting up client nodes... 2026-02-25T08:40:52.469 DEBUG:teuthology.orchestra.run.trial190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2026-02-25T08:40:52.510 INFO:teuthology.orchestra.run.trial190.stdout:creating /etc/ceph/ceph.client.0.keyring 2026-02-25T08:40:52.521 INFO:tasks.ceph:Running mkfs on osd nodes... 2026-02-25T08:40:52.521 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@trial190.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2026-02-25T08:40:52.521 DEBUG:teuthology.orchestra.run.trial190:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2026-02-25T08:40:52.585 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-25T08:40:52.585 INFO:tasks.ceph:role: osd.0 2026-02-25T08:40:52.585 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@trial190.front.sepia.ceph.com 2026-02-25T08:40:52.585 DEBUG:teuthology.orchestra.run.trial190:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2026-02-25T08:40:52.651 INFO:teuthology.orchestra.run.trial190.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=45875200 blks 2026-02-25T08:40:52.651 INFO:teuthology.orchestra.run.trial190.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-25T08:40:52.651 INFO:teuthology.orchestra.run.trial190.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-25T08:40:52.652 INFO:teuthology.orchestra.run.trial190.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-25T08:40:52.652 INFO:teuthology.orchestra.run.trial190.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-25T08:40:52.652 INFO:teuthology.orchestra.run.trial190.stdout: = sunit=0 swidth=0 blks 2026-02-25T08:40:52.652 INFO:teuthology.orchestra.run.trial190.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-25T08:40:52.652 INFO:teuthology.orchestra.run.trial190.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-25T08:40:52.652 INFO:teuthology.orchestra.run.trial190.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-25T08:40:52.652 INFO:teuthology.orchestra.run.trial190.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-25T08:40:53.230 INFO:teuthology.orchestra.run.trial190.stdout:Discarding blocks...Done. 2026-02-25T08:40:53.232 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@trial190.front.sepia.ceph.com -o noatime 2026-02-25T08:40:53.233 DEBUG:teuthology.orchestra.run.trial190:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2026-02-25T08:40:53.301 DEBUG:teuthology.orchestra.run.trial190:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2026-02-25T08:40:53.368 DEBUG:teuthology.orchestra.run.trial190:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2026-02-25T08:40:53.431 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-25T08:40:53.432 INFO:tasks.ceph:role: osd.1 2026-02-25T08:40:53.432 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@trial190.front.sepia.ceph.com 2026-02-25T08:40:53.432 DEBUG:teuthology.orchestra.run.trial190:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2026-02-25T08:40:53.496 INFO:teuthology.orchestra.run.trial190.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=45875200 blks 2026-02-25T08:40:53.496 INFO:teuthology.orchestra.run.trial190.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-25T08:40:53.496 INFO:teuthology.orchestra.run.trial190.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-25T08:40:53.496 INFO:teuthology.orchestra.run.trial190.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-25T08:40:53.496 INFO:teuthology.orchestra.run.trial190.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-25T08:40:53.496 INFO:teuthology.orchestra.run.trial190.stdout: = sunit=0 swidth=0 blks 2026-02-25T08:40:53.496 INFO:teuthology.orchestra.run.trial190.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-25T08:40:53.496 INFO:teuthology.orchestra.run.trial190.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-25T08:40:53.496 INFO:teuthology.orchestra.run.trial190.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-25T08:40:53.496 INFO:teuthology.orchestra.run.trial190.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-25T08:40:54.055 INFO:teuthology.orchestra.run.trial190.stdout:Discarding blocks...Done. 2026-02-25T08:40:54.057 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@trial190.front.sepia.ceph.com -o noatime 2026-02-25T08:40:54.057 DEBUG:teuthology.orchestra.run.trial190:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2026-02-25T08:40:54.126 DEBUG:teuthology.orchestra.run.trial190:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2026-02-25T08:40:54.192 DEBUG:teuthology.orchestra.run.trial190:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2026-02-25T08:40:54.254 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2026-02-25T08:40:54.254 INFO:tasks.ceph:role: osd.2 2026-02-25T08:40:54.254 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@trial190.front.sepia.ceph.com 2026-02-25T08:40:54.254 DEBUG:teuthology.orchestra.run.trial190:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2026-02-25T08:40:54.319 INFO:teuthology.orchestra.run.trial190.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=45875200 blks 2026-02-25T08:40:54.319 INFO:teuthology.orchestra.run.trial190.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-25T08:40:54.319 INFO:teuthology.orchestra.run.trial190.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-25T08:40:54.319 INFO:teuthology.orchestra.run.trial190.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-25T08:40:54.319 INFO:teuthology.orchestra.run.trial190.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-25T08:40:54.319 INFO:teuthology.orchestra.run.trial190.stdout: = sunit=0 swidth=0 blks 2026-02-25T08:40:54.319 INFO:teuthology.orchestra.run.trial190.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-25T08:40:54.319 INFO:teuthology.orchestra.run.trial190.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-25T08:40:54.319 INFO:teuthology.orchestra.run.trial190.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-25T08:40:54.319 INFO:teuthology.orchestra.run.trial190.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-25T08:40:54.888 INFO:teuthology.orchestra.run.trial190.stdout:Discarding blocks...Done. 2026-02-25T08:40:54.889 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@trial190.front.sepia.ceph.com -o noatime 2026-02-25T08:40:54.890 DEBUG:teuthology.orchestra.run.trial190:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2026-02-25T08:40:54.957 DEBUG:teuthology.orchestra.run.trial190:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2026-02-25T08:40:55.024 DEBUG:teuthology.orchestra.run.trial190:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-25T08:40:55.174 INFO:teuthology.orchestra.run.trial190.stderr:INFO  osd - OSD::main: parsing early config 2026-02-25T08:40:55.226 INFO:teuthology.orchestra.run.trial190.stderr:INFO 2026-02-25 08:40:55,226 seastar - Reactor backend: linux-aio 2026-02-25T08:40:55.236 INFO:teuthology.orchestra.run.trial190.stderr:INFO 2026-02-25 08:40:55,235 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:40:55.236 INFO:teuthology.orchestra.run.trial190.stderr:INFO 2026-02-25 08:40:55,235 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:40:55.238 INFO:teuthology.orchestra.run.trial190.stderr:==39676==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-25T08:40:55.297 INFO:teuthology.orchestra.run.trial190.stderr:DEBUG 2026-02-25 08:40:55,296 [shard 0:main] seastar - reactor::drain 2026-02-25T08:40:57.383 INFO:teuthology.orchestra.run.trial190.stderr:----------------------------------------------------- 2026-02-25T08:40:57.383 INFO:teuthology.orchestra.run.trial190.stderr:Suppressions used: 2026-02-25T08:40:57.383 INFO:teuthology.orchestra.run.trial190.stderr: count bytes template 2026-02-25T08:40:57.383 INFO:teuthology.orchestra.run.trial190.stderr: 1 8 MallocExtension::Register 2026-02-25T08:40:57.383 INFO:teuthology.orchestra.run.trial190.stderr:----------------------------------------------------- 2026-02-25T08:40:57.383 INFO:teuthology.orchestra.run.trial190.stderr: 2026-02-25T08:40:57.402 INFO:teuthology.orchestra.run.trial190.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-25T08:40:57.402 INFO:teuthology.orchestra.run.trial190.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-25T08:40:57.444 INFO:teuthology.orchestra.run.trial190.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-25T08:40:57.444 INFO:teuthology.orchestra.run.trial190.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-25T08:40:57.451 INFO:teuthology.orchestra.run.trial190.stderr:INFO 2026-02-25 08:40:57,450 seastar - Reactor backend: linux-aio 2026-02-25T08:40:57.462 INFO:teuthology.orchestra.run.trial190.stderr:INFO 2026-02-25 08:40:57,461 [shard 2:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:40:57.462 INFO:teuthology.orchestra.run.trial190.stderr:INFO 2026-02-25 08:40:57,461 [shard 2:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:40:57.466 INFO:teuthology.orchestra.run.trial190.stderr:==39674==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-25T08:40:57.466 INFO:teuthology.orchestra.run.trial190.stderr:INFO 2026-02-25 08:40:57,465 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-25T08:40:57.605 INFO:teuthology.orchestra.run.trial190.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0xcca009 0x10a279e 0xf47938 0x9e216df 0x9e21ba9 0x9e21e1c 0x9e23b43 0x9e24090 0x9e2428f 0x9f095b1 0x9f09d0c 0x9f13f9a 0x9f18413 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7044244 0x7048045 0x704845b 0x73b7c8f 0x73b7d14 0x73b7e16 0x73b7f0e 0x74780b4 0x72214da 0x7266e18 0x7398e9a 0x73efcd2 0x7221561 0x726701f 0x755dcc5 0x73b7f93 0x73b8071 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:57.605 INFO:teuthology.orchestra.run.trial190.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xb04f2 /lib64/libasan.so.8+0xb1437 0xb7ac87 0x10a27ac 0xf47938 0x9e216df 0x9e21ba9 0x9e21e1c 0x9e23b43 0x9f0751a 0x9f0916c 0x9f097c6 0x9f09d0c 0x9f13f9a 0x9f18413 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7044244 0x7048045 0x704845b 0x73b7c8f 0x73b7d14 0x73b7e16 0x73b7f0e 0x74780b4 0x72214da 0x7266e18 0x7398e9a 0x73efcd2 0x7221561 0x726701f 0x755dcc5 0x73b7f93 0x73b8071 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:57.605 INFO:teuthology.orchestra.run.trial190.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x1137fca 0xf4792d 0x9e216df 0x9e21ba9 0x9e21e1c 0x9e220d2 0x9e22b81 0x9e2307f 0x9e236af 0x9e23819 0x9f03b89 0x9f1698c 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7044244 0x7048045 0x704845b 0x73b7c8f 0x73b7d14 0x73b7e16 0x73b7f0e 0x74780b4 0x72214da 0x7266e18 0x7398e9a 0x73efcd2 0x7221561 0x726701f 0x755dcc5 0x73b7f93 0x73b8071 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:57.756 INFO:teuthology.orchestra.run.trial190.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xb04e6 /lib64/libasan.so.8+0xb1437 0xb7ac87 0x10a27ac 0xf47938 0x9e218ac 0x9e21ba9 0x9e21e1c 0x9e220d2 0x9e22b81 0x9e2307f 0x9e236af 0x9e23819 0x9f03b89 0x9f1698c 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x6fec95e 0x6ed2592 0x73b5492 0x73b5517 0x73b5619 0x73b5711 0x7477f38 0x72201a5 0x7260160 0x73973b5 0x73ea713 0x722022c 0x7260367 0x755b9c5 0x73b5796 0x73b5874 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:57.756 INFO:teuthology.orchestra.run.trial190.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1297db /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x15e37 0x9e2ac86 0x9e2b21b 0x9f13d68 0x9f18413 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x6fec95e 0x6ed2592 0x73b5492 0x73b5517 0x73b5619 0x73b5711 0x7477f38 0x72201a5 0x7260160 0x73973b5 0x73ea713 0x722022c 0x7260367 0x755b9c5 0x73b5796 0x73b5874 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:57.756 INFO:teuthology.orchestra.run.trial190.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x9de1889 0x9e1ae74 0x9eec204 0x9f18680 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x6fec95e 0x6ed2592 0x73b5492 0x73b5517 0x73b5619 0x73b5711 0x7477f38 0x72201a5 0x7260160 0x73973b5 0x73ea713 0x722022c 0x7260367 0x755b9c5 0x73b5796 0x73b5874 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:59.900 INFO:teuthology.orchestra.run.trial190.stderr:----------------------------------------------------- 2026-02-25T08:40:59.900 INFO:teuthology.orchestra.run.trial190.stderr:Suppressions used: 2026-02-25T08:40:59.900 INFO:teuthology.orchestra.run.trial190.stderr: count bytes template 2026-02-25T08:40:59.900 INFO:teuthology.orchestra.run.trial190.stderr: 1 8 MallocExtension::Register 2026-02-25T08:40:59.900 INFO:teuthology.orchestra.run.trial190.stderr:----------------------------------------------------- 2026-02-25T08:40:59.900 INFO:teuthology.orchestra.run.trial190.stderr: 2026-02-25T08:40:59.929 DEBUG:teuthology.orchestra.run.trial190:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2026-02-25T08:40:59.955 DEBUG:teuthology.orchestra.run.trial190:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-25T08:41:00.104 INFO:teuthology.orchestra.run.trial190.stderr:INFO  osd - OSD::main: parsing early config 2026-02-25T08:41:00.161 INFO:teuthology.orchestra.run.trial190.stderr:INFO 2026-02-25 08:41:00,160 seastar - Reactor backend: linux-aio 2026-02-25T08:41:00.171 INFO:teuthology.orchestra.run.trial190.stderr:INFO 2026-02-25 08:41:00,170 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:41:00.171 INFO:teuthology.orchestra.run.trial190.stderr:INFO 2026-02-25 08:41:00,170 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:41:00.173 INFO:teuthology.orchestra.run.trial190.stderr:==39744==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-25T08:41:00.232 INFO:teuthology.orchestra.run.trial190.stderr:DEBUG 2026-02-25 08:41:00,231 [shard 0:main] seastar - reactor::drain 2026-02-25T08:41:02.303 INFO:teuthology.orchestra.run.trial190.stderr:----------------------------------------------------- 2026-02-25T08:41:02.303 INFO:teuthology.orchestra.run.trial190.stderr:Suppressions used: 2026-02-25T08:41:02.303 INFO:teuthology.orchestra.run.trial190.stderr: count bytes template 2026-02-25T08:41:02.303 INFO:teuthology.orchestra.run.trial190.stderr: 1 8 MallocExtension::Register 2026-02-25T08:41:02.304 INFO:teuthology.orchestra.run.trial190.stderr:----------------------------------------------------- 2026-02-25T08:41:02.304 INFO:teuthology.orchestra.run.trial190.stderr: 2026-02-25T08:41:02.323 INFO:teuthology.orchestra.run.trial190.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-25T08:41:02.323 INFO:teuthology.orchestra.run.trial190.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-25T08:41:02.364 INFO:teuthology.orchestra.run.trial190.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-25T08:41:02.364 INFO:teuthology.orchestra.run.trial190.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-25T08:41:02.371 INFO:teuthology.orchestra.run.trial190.stderr:INFO 2026-02-25 08:41:02,370 seastar - Reactor backend: linux-aio 2026-02-25T08:41:02.383 INFO:teuthology.orchestra.run.trial190.stderr:INFO 2026-02-25 08:41:02,382 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:41:02.383 INFO:teuthology.orchestra.run.trial190.stderr:INFO 2026-02-25 08:41:02,382 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:41:02.387 INFO:teuthology.orchestra.run.trial190.stderr:==39742==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-25T08:41:02.388 INFO:teuthology.orchestra.run.trial190.stderr:INFO 2026-02-25 08:41:02,387 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-25T08:41:02.533 INFO:teuthology.orchestra.run.trial190.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0xb7ac65 0x10a27ac 0xf47938 0x9e216df 0x9e21ba9 0x9e21e1c 0x9e220d2 0x9e22b81 0x9e2307f 0x9e236af 0x9e23819 0x9f03b89 0x9f1698c 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7044244 0x7048045 0x704845b 0x73b7c8f 0x73b7d14 0x73b7e16 0x73b7f0e 0x74780b4 0x72214da 0x7266e18 0x7398e9a 0x73efcd2 0x7221561 0x726701f 0x755dcc5 0x73b7f93 0x73b8071 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:41:02.533 INFO:teuthology.orchestra.run.trial190.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x318129f 0x9e21b22 0x9e21e1c 0x9e23b43 0x9e24090 0x9e2428f 0x9f095b1 0x9f09d0c 0x9f13f9a 0x9f18413 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7044244 0x7048045 0x704845b 0x73b7c8f 0x73b7d14 0x73b7e16 0x73b7f0e 0x74780b4 0x72214da 0x7266e18 0x7398e9a 0x73efcd2 0x7221561 0x726701f 0x755dcc5 0x73b7f93 0x73b8071 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:41:02.533 INFO:teuthology.orchestra.run.trial190.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x9edcdc5 0x9edcf47 0x9edd1b7 0x9edd2bc 0x9f0f6c7 0x9f0fe6a 0x9f1a760 0x78b69db 0x7ba17e8 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7044244 0x7048045 0x704845b 0x73b7c8f 0x73b7d14 0x73b7e16 0x73b7f0e 0x74780b4 0x72214da 0x7266e18 0x7398e9a 0x73efcd2 0x7221561 0x726701f 0x755dcc5 0x73b7f93 0x73b8071 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:41:02.683 INFO:teuthology.orchestra.run.trial190.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x9e30306 0x9ef6683 0x9ef684b 0x9ef69ac 0x9ef6ea0 0x9f1372b 0x9f18413 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x6fec95e 0x6ed2592 0x73b5492 0x73b5517 0x73b5619 0x73b5711 0x7477f38 0x72201a5 0x7260160 0x73973b5 0x73ea713 0x722022c 0x7260367 0x755b9c5 0x73b5796 0x73b5874 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:41:02.683 INFO:teuthology.orchestra.run.trial190.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x8cc0c /lib64/libasan.so.8+0x10069c /lib64/libasan.so.8+0xc65b9 /lib64/libasan.so.8+0x15197 0x35fbdb7 0x35fbfe6 0x361a32a 0x361a79f 0x361a859 0x9ef6a64 0x9ef6ea0 0x9f1372b 0x9f18413 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x6fec95e 0x6ed2592 0x73b5492 0x73b5517 0x73b5619 0x73b5711 0x7477f38 0x72201a5 0x7260160 0x73973b5 0x73ea713 0x722022c 0x7260367 0x755b9c5 0x73b5796 0x73b5874 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:41:02.684 INFO:teuthology.orchestra.run.trial190.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc597 /lib64/libasan.so.8+0x8eea /lib64/libasan.so.8+0xedc45 0xe40f5d 0x9ef6526 0x9ef66e2 0x9ef684b 0x9ef6976 0x9ef6ea0 0x9f1372b 0x9f18413 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x6fec95e 0x6ed2592 0x73b5492 0x73b5517 0x73b5619 0x73b5711 0x7477f38 0x72201a5 0x7260160 0x73973b5 0x73ea713 0x722022c 0x7260367 0x755b9c5 0x73b5796 0x73b5874 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:41:02.684 INFO:teuthology.orchestra.run.trial190.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffab29431b 0xffffffffab420057 0xffffffffab4203b0 0xffffffffab29525e 0xffffffffab5e7269 0xffffffffab5ca7d5 0xffffffffab5d4bab 0xffffffffab5d500b 0xffffffffab2910d2 0xffffffffabf50451 0xffffffffac000ba2 2026-02-25T08:41:04.848 INFO:teuthology.orchestra.run.trial190.stderr:----------------------------------------------------- 2026-02-25T08:41:04.848 INFO:teuthology.orchestra.run.trial190.stderr:Suppressions used: 2026-02-25T08:41:04.848 INFO:teuthology.orchestra.run.trial190.stderr: count bytes template 2026-02-25T08:41:04.848 INFO:teuthology.orchestra.run.trial190.stderr: 1 8 MallocExtension::Register 2026-02-25T08:41:04.848 INFO:teuthology.orchestra.run.trial190.stderr:----------------------------------------------------- 2026-02-25T08:41:04.848 INFO:teuthology.orchestra.run.trial190.stderr: 2026-02-25T08:41:04.878 DEBUG:teuthology.orchestra.run.trial190:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2026-02-25T08:41:04.903 DEBUG:teuthology.orchestra.run.trial190:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-25T08:41:05.054 INFO:teuthology.orchestra.run.trial190.stderr:INFO  osd - OSD::main: parsing early config 2026-02-25T08:41:05.113 INFO:teuthology.orchestra.run.trial190.stderr:INFO 2026-02-25 08:41:05,112 seastar - Reactor backend: linux-aio 2026-02-25T08:41:05.127 INFO:teuthology.orchestra.run.trial190.stderr:INFO 2026-02-25 08:41:05,126 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:41:05.127 INFO:teuthology.orchestra.run.trial190.stderr:INFO 2026-02-25 08:41:05,126 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:41:05.129 INFO:teuthology.orchestra.run.trial190.stderr:==39811==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-25T08:41:05.188 INFO:teuthology.orchestra.run.trial190.stderr:DEBUG 2026-02-25 08:41:05,187 [shard 0:main] seastar - reactor::drain 2026-02-25T08:41:07.261 INFO:teuthology.orchestra.run.trial190.stderr:----------------------------------------------------- 2026-02-25T08:41:07.262 INFO:teuthology.orchestra.run.trial190.stderr:Suppressions used: 2026-02-25T08:41:07.262 INFO:teuthology.orchestra.run.trial190.stderr: count bytes template 2026-02-25T08:41:07.262 INFO:teuthology.orchestra.run.trial190.stderr: 1 8 MallocExtension::Register 2026-02-25T08:41:07.262 INFO:teuthology.orchestra.run.trial190.stderr:----------------------------------------------------- 2026-02-25T08:41:07.262 INFO:teuthology.orchestra.run.trial190.stderr: 2026-02-25T08:41:07.281 INFO:teuthology.orchestra.run.trial190.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-25T08:41:07.282 INFO:teuthology.orchestra.run.trial190.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-25T08:41:07.324 INFO:teuthology.orchestra.run.trial190.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-25T08:41:07.324 INFO:teuthology.orchestra.run.trial190.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-25T08:41:07.331 INFO:teuthology.orchestra.run.trial190.stderr:INFO 2026-02-25 08:41:07,330 seastar - Reactor backend: linux-aio 2026-02-25T08:41:07.343 INFO:teuthology.orchestra.run.trial190.stderr:INFO 2026-02-25 08:41:07,342 [shard 3:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:41:07.343 INFO:teuthology.orchestra.run.trial190.stderr:INFO 2026-02-25 08:41:07,342 [shard 3:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:41:07.347 INFO:teuthology.orchestra.run.trial190.stderr:==39809==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-25T08:41:07.347 INFO:teuthology.orchestra.run.trial190.stderr:INFO 2026-02-25 08:41:07,347 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-25T08:41:07.491 INFO:teuthology.orchestra.run.trial190.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0xb7fb41 0x10a2729 0xf47938 0x9e216df 0x9e21ba9 0x9e21e1c 0x9e220d2 0x9e22b81 0x9e2307f 0x9e236af 0x9e23819 0x9f03b89 0x9f1698c 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7044244 0x7048045 0x704845b 0x73b7c8f 0x73b7d14 0x73b7e16 0x73b7f0e 0x74780b4 0x72214da 0x7266e18 0x7398e9a 0x73efcd2 0x7221561 0x726701f 0x755dcc5 0x73b7f93 0x73b8071 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:41:07.491 INFO:teuthology.orchestra.run.trial190.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x31826c8 0x9e21e0a 0x9e220d2 0x9e22b81 0x9e2307f 0x9e236af 0x9e23819 0x9f03b89 0x9f1698c 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7044244 0x7048045 0x704845b 0x73b7c8f 0x73b7d14 0x73b7e16 0x73b7f0e 0x74780b4 0x72214da 0x7266e18 0x7398e9a 0x73efcd2 0x7221561 0x726701f 0x755dcc5 0x73b7f93 0x73b8071 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:41:07.491 INFO:teuthology.orchestra.run.trial190.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x15865 0x9e21e70 0x9e220d2 0x9e22b81 0x9e2307f 0x9e236af 0x9e23819 0x9f03b89 0x9f1698c 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7044244 0x7048045 0x704845b 0x73b7c8f 0x73b7d14 0x73b7e16 0x73b7f0e 0x74780b4 0x72214da 0x7266e18 0x7398e9a 0x73efcd2 0x7221561 0x726701f 0x755dcc5 0x73b7f93 0x73b8071 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:41:07.645 INFO:teuthology.orchestra.run.trial190.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xf4e8f /lib64/libasan.so.8+0x10069c /lib64/libasan.so.8+0xc65b9 /lib64/libasan.so.8+0x15197 0x318273f 0x9e21d5d 0x9e220d2 0x9e22b81 0x9e2307f 0x9e236af 0x9e23819 0x9f03b89 0x9f1698c 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x6fec95e 0x6ed2592 0x73b5492 0x73b5517 0x73b5619 0x73b5711 0x7477f38 0x72201a5 0x7260160 0x73973b5 0x73ea713 0x722022c 0x7260367 0x755b9c5 0x73b5796 0x73b5874 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:41:07.646 INFO:teuthology.orchestra.run.trial190.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x151d9 0x9e21748 0x9e21ba9 0x9e21e1c 0x9e220d2 0x9e22694 0x9e2ab3b 0x9e2b21b 0x9f13d68 0x9f18413 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x6fec95e 0x6ed2592 0x73b5492 0x73b5517 0x73b5619 0x73b5711 0x7477f38 0x72201a5 0x7260160 0x73973b5 0x73ea713 0x722022c 0x7260367 0x755b9c5 0x73b5796 0x73b5874 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:41:07.646 INFO:teuthology.orchestra.run.trial190.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x15225 0x318273f 0x9e21e0a 0x9e220d2 0x9e22b81 0x9e2307f 0x9e236af 0x9e23819 0x9f03b89 0x9f1698c 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x6fec95e 0x6ed2592 0x73b5492 0x73b5517 0x73b5619 0x73b5711 0x7477f38 0x72201a5 0x7260160 0x73973b5 0x73ea713 0x722022c 0x7260367 0x755b9c5 0x73b5796 0x73b5874 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:41:09.741 INFO:teuthology.orchestra.run.trial190.stderr:----------------------------------------------------- 2026-02-25T08:41:09.741 INFO:teuthology.orchestra.run.trial190.stderr:Suppressions used: 2026-02-25T08:41:09.741 INFO:teuthology.orchestra.run.trial190.stderr: count bytes template 2026-02-25T08:41:09.742 INFO:teuthology.orchestra.run.trial190.stderr: 1 8 MallocExtension::Register 2026-02-25T08:41:09.742 INFO:teuthology.orchestra.run.trial190.stderr:----------------------------------------------------- 2026-02-25T08:41:09.742 INFO:teuthology.orchestra.run.trial190.stderr: 2026-02-25T08:41:09.772 DEBUG:teuthology.orchestra.run.trial190:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2026-02-25T08:41:09.796 INFO:tasks.ceph:Reading keys from all nodes... 2026-02-25T08:41:09.796 DEBUG:teuthology.orchestra.run.trial190:> set -ex 2026-02-25T08:41:09.796 DEBUG:teuthology.orchestra.run.trial190:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2026-02-25T08:41:09.860 DEBUG:teuthology.orchestra.run.trial190:> set -ex 2026-02-25T08:41:09.860 DEBUG:teuthology.orchestra.run.trial190:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2026-02-25T08:41:09.925 DEBUG:teuthology.orchestra.run.trial190:> set -ex 2026-02-25T08:41:09.925 DEBUG:teuthology.orchestra.run.trial190:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2026-02-25T08:41:09.988 DEBUG:teuthology.orchestra.run.trial190:> set -ex 2026-02-25T08:41:09.988 DEBUG:teuthology.orchestra.run.trial190:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2026-02-25T08:41:10.051 DEBUG:teuthology.orchestra.run.trial190:> set -ex 2026-02-25T08:41:10.051 DEBUG:teuthology.orchestra.run.trial190:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2026-02-25T08:41:10.107 INFO:tasks.ceph:Adding keys to all mons... 2026-02-25T08:41:10.108 DEBUG:teuthology.orchestra.run.trial190:> sudo tee -a /etc/ceph/ceph.keyring 2026-02-25T08:41:10.169 INFO:teuthology.orchestra.run.trial190.stdout:[mgr.x] 2026-02-25T08:41:10.170 INFO:teuthology.orchestra.run.trial190.stdout: key = AQCUtZ5p3bq4GxAA5UNt7iXs3TMf5rvTcqsMPQ== 2026-02-25T08:41:10.170 INFO:teuthology.orchestra.run.trial190.stdout:[osd.0] 2026-02-25T08:41:10.170 INFO:teuthology.orchestra.run.trial190.stdout: key = AQCZtZ5pJqWQHxAAAanKmd35NQCyFUOKs062NA== 2026-02-25T08:41:10.170 INFO:teuthology.orchestra.run.trial190.stdout:[osd.1] 2026-02-25T08:41:10.170 INFO:teuthology.orchestra.run.trial190.stdout: key = AQCetZ5pPNkhGxAANgeLEclw/gLV/PDQWWYs8w== 2026-02-25T08:41:10.170 INFO:teuthology.orchestra.run.trial190.stdout:[osd.2] 2026-02-25T08:41:10.170 INFO:teuthology.orchestra.run.trial190.stdout: key = AQCjtZ5pY9a0GBAAlzTHAvk4//KSTO3e9hiNLQ== 2026-02-25T08:41:10.170 INFO:teuthology.orchestra.run.trial190.stdout:[client.0] 2026-02-25T08:41:10.170 INFO:teuthology.orchestra.run.trial190.stdout: key = AQCUtZ5ptBFcHhAAxKvRoqgKvs+lwtZTYxh3Jg== 2026-02-25T08:41:10.172 DEBUG:teuthology.orchestra.run.trial190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2026-02-25T08:41:10.257 DEBUG:teuthology.orchestra.run.trial190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-25T08:41:10.305 DEBUG:teuthology.orchestra.run.trial190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-25T08:41:10.350 DEBUG:teuthology.orchestra.run.trial190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-25T08:41:10.398 DEBUG:teuthology.orchestra.run.trial190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-25T08:41:10.446 INFO:tasks.ceph:Running mkfs on mon nodes... 2026-02-25T08:41:10.446 DEBUG:teuthology.orchestra.run.trial190:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2026-02-25T08:41:10.469 DEBUG:teuthology.orchestra.run.trial190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2026-02-25T08:41:10.554 INFO:teuthology.orchestra.run.trial190.stderr:2026-02-25T08:41:10.553+0000 7fd37ff23c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:10.554 INFO:teuthology.orchestra.run.trial190.stderr:2026-02-25T08:41:10.553+0000 7fd37ff23c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:10.561 DEBUG:teuthology.orchestra.run.trial190:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2026-02-25T08:41:10.585 DEBUG:teuthology.orchestra.run.trial190:> rm -- /home/ubuntu/cephtest/ceph.monmap 2026-02-25T08:41:10.640 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2026-02-25T08:41:10.640 INFO:tasks.ceph.mon.a:Restarting daemon 2026-02-25T08:41:10.640 DEBUG:teuthology.orchestra.run.trial190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2026-02-25T08:41:10.682 INFO:tasks.ceph.mon.a:Started 2026-02-25T08:41:10.682 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2026-02-25T08:41:10.682 INFO:tasks.ceph.mgr.x:Restarting daemon 2026-02-25T08:41:10.682 DEBUG:teuthology.orchestra.run.trial190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2026-02-25T08:41:10.684 INFO:tasks.ceph.mgr.x:Started 2026-02-25T08:41:10.684 DEBUG:tasks.ceph:set 0 configs 2026-02-25T08:41:10.684 DEBUG:teuthology.orchestra.run.trial190:> sudo ceph --cluster ceph config dump 2026-02-25T08:41:10.743 INFO:tasks.ceph.mon.a.trial190.stderr:2026-02-25T08:41:10.741+0000 7f160b03ac00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:10.743 INFO:tasks.ceph.mon.a.trial190.stderr:2026-02-25T08:41:10.741+0000 7f160b03ac00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:10.743 INFO:tasks.ceph.mon.a.trial190.stderr:2026-02-25T08:41:10.741+0000 7f160b03ac00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:10.745 INFO:tasks.ceph.mgr.x.trial190.stderr:2026-02-25T08:41:10.743+0000 7f449aff3fc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:10.760 INFO:teuthology.orchestra.run.trial190.stderr:2026-02-25T08:41:10.757+0000 7f35525dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:10.761 INFO:teuthology.orchestra.run.trial190.stderr:2026-02-25T08:41:10.760+0000 7f35525dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:10.792 INFO:teuthology.orchestra.run.trial190.stdout:WHO MASK LEVEL OPTION VALUE RO 2026-02-25T08:41:10.801 INFO:tasks.ceph:Setting crush tunables to default 2026-02-25T08:41:10.801 DEBUG:teuthology.orchestra.run.trial190:> sudo ceph --cluster ceph osd crush tunables default 2026-02-25T08:41:10.875 INFO:teuthology.orchestra.run.trial190.stderr:2026-02-25T08:41:10.870+0000 7f251ca5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:10.876 INFO:teuthology.orchestra.run.trial190.stderr:2026-02-25T08:41:10.874+0000 7f251ca5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:10.908 INFO:teuthology.orchestra.run.trial190.stderr:adjusted tunables profile to default 2026-02-25T08:41:10.917 INFO:tasks.ceph:check_enable_crimson: True 2026-02-25T08:41:10.917 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2026-02-25T08:41:10.917 DEBUG:teuthology.orchestra.run.trial190:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2026-02-25T08:41:10.948 INFO:tasks.ceph.mgr.x.trial190.stderr:2026-02-25T08:41:10.946+0000 7f449aff3fc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:10.949 INFO:tasks.ceph.mgr.x.trial190.stderr:2026-02-25T08:41:10.947+0000 7f449aff3fc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:10.990 INFO:teuthology.orchestra.run.trial190.stderr:2026-02-25T08:41:10.985+0000 7fb6306cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:10.991 INFO:teuthology.orchestra.run.trial190.stderr:2026-02-25T08:41:10.989+0000 7fb6306cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:11.029 INFO:tasks.ceph.mon.a.trial190.stderr:2026-02-25T08:41:11.027+0000 7f16023f1640 -1 WARNING: experimental feature 'crimson' is enabled 2026-02-25T08:41:11.029 INFO:tasks.ceph.mon.a.trial190.stderr:Please be aware that this feature is experimental, untested, 2026-02-25T08:41:11.029 INFO:tasks.ceph.mon.a.trial190.stderr:unsupported, and may result in data corruption, data loss, 2026-02-25T08:41:11.029 INFO:tasks.ceph.mon.a.trial190.stderr:and/or irreparable damage to your cluster. Do not use 2026-02-25T08:41:11.029 INFO:tasks.ceph.mon.a.trial190.stderr:feature with important data. 2026-02-25T08:41:11.029 INFO:tasks.ceph.mon.a.trial190.stderr: 2026-02-25T08:41:11.795 INFO:tasks.ceph.mgr.x.trial190.stderr:/usr/lib64/python3.9/site-packages/scipy/__init__.py:73: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2026-02-25T08:41:11.795 INFO:tasks.ceph.mgr.x.trial190.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2026-02-25T08:41:11.795 INFO:tasks.ceph.mgr.x.trial190.stderr: from numpy import show_config as show_numpy_config 2026-02-25T08:41:11.919 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2026-02-25T08:41:11.920 DEBUG:teuthology.orchestra.run.trial190:> set -ex 2026-02-25T08:41:11.920 DEBUG:teuthology.orchestra.run.trial190:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2026-02-25T08:41:11.984 DEBUG:teuthology.orchestra.run.trial190:> set -ex 2026-02-25T08:41:11.985 DEBUG:teuthology.orchestra.run.trial190:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2026-02-25T08:41:12.047 DEBUG:teuthology.orchestra.run.trial190:> set -ex 2026-02-25T08:41:12.048 DEBUG:teuthology.orchestra.run.trial190:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2026-02-25T08:41:12.112 DEBUG:teuthology.orchestra.run.trial190:> sudo ceph --cluster ceph osd new 0911493a-668e-4736-b7cc-46e0fced3475 0 2026-02-25T08:41:12.226 INFO:teuthology.orchestra.run.trial190.stderr:2026-02-25T08:41:12.221+0000 7f1c6990b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:12.228 INFO:teuthology.orchestra.run.trial190.stderr:2026-02-25T08:41:12.226+0000 7f1c6990b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:12.264 INFO:teuthology.orchestra.run.trial190.stdout:0 2026-02-25T08:41:12.272 DEBUG:teuthology.orchestra.run.trial190:> sudo ceph --cluster ceph osd new 5c5fdbaf-a331-4e28-918d-94481631225d 1 2026-02-25T08:41:12.346 INFO:teuthology.orchestra.run.trial190.stderr:2026-02-25T08:41:12.341+0000 7f0d11d5c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:12.347 INFO:teuthology.orchestra.run.trial190.stderr:2026-02-25T08:41:12.346+0000 7f0d11d5c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:12.383 INFO:teuthology.orchestra.run.trial190.stdout:1 2026-02-25T08:41:12.392 DEBUG:teuthology.orchestra.run.trial190:> sudo ceph --cluster ceph osd new e824df79-8af8-42e3-bd56-9ed43ac6524e 2 2026-02-25T08:41:12.465 INFO:teuthology.orchestra.run.trial190.stderr:2026-02-25T08:41:12.461+0000 7f3ce6c46640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:12.466 INFO:teuthology.orchestra.run.trial190.stderr:2026-02-25T08:41:12.465+0000 7f3ce6c46640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:12.503 INFO:teuthology.orchestra.run.trial190.stdout:2 2026-02-25T08:41:12.511 DEBUG:teuthology.orchestra.run.trial190:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2026-02-25T08:41:12.540 INFO:tasks.ceph.osd.0:Restarting daemon 2026-02-25T08:41:12.540 DEBUG:teuthology.orchestra.run.trial190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2026-02-25T08:41:12.582 INFO:tasks.ceph.osd.0:Started 2026-02-25T08:41:12.582 DEBUG:teuthology.orchestra.run.trial190:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2026-02-25T08:41:12.612 INFO:tasks.ceph.osd.1:Restarting daemon 2026-02-25T08:41:12.612 DEBUG:teuthology.orchestra.run.trial190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2026-02-25T08:41:12.654 INFO:tasks.ceph.osd.1:Started 2026-02-25T08:41:12.654 DEBUG:teuthology.orchestra.run.trial190:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2026-02-25T08:41:12.681 INFO:tasks.ceph.osd.2:Restarting daemon 2026-02-25T08:41:12.681 DEBUG:teuthology.orchestra.run.trial190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2026-02-25T08:41:12.724 INFO:tasks.ceph.osd.2:Started 2026-02-25T08:41:12.724 DEBUG:teuthology.orchestra.run.trial190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-25T08:41:12.799 INFO:teuthology.orchestra.run.trial190.stderr:2026-02-25T08:41:12.795+0000 7fa48d9b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:12.800 INFO:teuthology.orchestra.run.trial190.stderr:2026-02-25T08:41:12.799+0000 7fa48d9b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:12.833 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:41:12.833 INFO:teuthology.orchestra.run.trial190.stdout:{"epoch":6,"fsid":"2a6cc9ca-2439-4b64-860c-0d3000bf169b","created":"2026-02-25T08:41:10.753011+0000","modified":"2026-02-25T08:41:12.501670+0000","last_up_change":"0.000000","last_in_change":"2026-02-25T08:41:12.501670+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"0911493a-668e-4736-b7cc-46e0fced3475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c5fdbaf-a331-4e28-918d-94481631225d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e824df79-8af8-42e3-bd56-9ed43ac6524e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-25T08:41:12.840 INFO:tasks.ceph.ceph_manager.ceph:[] 2026-02-25T08:41:12.840 INFO:tasks.ceph:Waiting for OSDs to come up 2026-02-25T08:41:13.142 DEBUG:teuthology.orchestra.run.trial190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-25T08:41:13.210 INFO:teuthology.misc.health.trial190.stderr:2026-02-25T08:41:13.206+0000 7fd79ef0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:13.211 INFO:teuthology.misc.health.trial190.stderr:2026-02-25T08:41:13.209+0000 7fd79ef0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:13.242 INFO:teuthology.misc.health.trial190.stdout: 2026-02-25T08:41:13.242 INFO:teuthology.misc.health.trial190.stdout:{"epoch":6,"fsid":"2a6cc9ca-2439-4b64-860c-0d3000bf169b","created":"2026-02-25T08:41:10.753011+0000","modified":"2026-02-25T08:41:12.501670+0000","last_up_change":"0.000000","last_in_change":"2026-02-25T08:41:12.501670+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"0911493a-668e-4736-b7cc-46e0fced3475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c5fdbaf-a331-4e28-918d-94481631225d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e824df79-8af8-42e3-bd56-9ed43ac6524e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-25T08:41:13.248 DEBUG:teuthology.misc:0 of 3 OSDs are up 2026-02-25T08:41:14.197 INFO:tasks.ceph.mgr.x.trial190.stderr:2026-02-25T08:41:14.195+0000 7f43b5f9a640 -1 mgr.server handle_report got status from non-daemon mon.a 2026-02-25T08:41:19.551 DEBUG:teuthology.orchestra.run.trial190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-25T08:41:19.622 INFO:teuthology.misc.health.trial190.stderr:2026-02-25T08:41:19.619+0000 7fe0ff983640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:19.623 INFO:teuthology.misc.health.trial190.stderr:2026-02-25T08:41:19.622+0000 7fe0ff983640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:19.733 INFO:teuthology.misc.health.trial190.stdout: 2026-02-25T08:41:19.734 INFO:teuthology.misc.health.trial190.stdout:{"epoch":8,"fsid":"2a6cc9ca-2439-4b64-860c-0d3000bf169b","created":"2026-02-25T08:41:10.753011+0000","modified":"2026-02-25T08:41:19.129447+0000","last_up_change":"0.000000","last_in_change":"2026-02-25T08:41:12.501670+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"0911493a-668e-4736-b7cc-46e0fced3475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c5fdbaf-a331-4e28-918d-94481631225d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e824df79-8af8-42e3-bd56-9ed43ac6524e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-25T08:41:19.741 DEBUG:teuthology.misc:0 of 3 OSDs are up 2026-02-25T08:41:26.044 DEBUG:teuthology.orchestra.run.trial190:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-25T08:41:26.114 INFO:teuthology.misc.health.trial190.stderr:2026-02-25T08:41:26.111+0000 7ffb3ce5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:26.116 INFO:teuthology.misc.health.trial190.stderr:2026-02-25T08:41:26.114+0000 7ffb3ce5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:26.227 INFO:teuthology.misc.health.trial190.stdout: 2026-02-25T08:41:26.227 INFO:teuthology.misc.health.trial190.stdout:{"epoch":13,"fsid":"2a6cc9ca-2439-4b64-860c-0d3000bf169b","created":"2026-02-25T08:41:10.753011+0000","modified":"2026-02-25T08:41:24.149773+0000","last_up_change":"2026-02-25T08:41:22.141251+0000","last_in_change":"2026-02-25T08:41:12.501670+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-25T08:41:21.204867+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0911493a-668e-4736-b7cc-46e0fced3475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.190:6809","nonce":1401965930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.190:6810","nonce":1401965930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.190:6814","nonce":1401965930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.190:6813","nonce":1401965930}]},"public_addr":"10.20.193.190:6809/1401965930","cluster_addr":"10.20.193.190:6810/1401965930","heartbeat_back_addr":"10.20.193.190:6814/1401965930","heartbeat_front_addr":"10.20.193.190:6813/1401965930","state":["exists","up"]},{"osd":1,"uuid":"5c5fdbaf-a331-4e28-918d-94481631225d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.190:6803","nonce":2036318314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.190:6804","nonce":2036318314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.190:6807","nonce":2036318314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.190:6806","nonce":2036318314}]},"public_addr":"10.20.193.190:6803/2036318314","cluster_addr":"10.20.193.190:6804/2036318314","heartbeat_back_addr":"10.20.193.190:6807/2036318314","heartbeat_front_addr":"10.20.193.190:6806/2036318314","state":["exists","up"]},{"osd":2,"uuid":"e824df79-8af8-42e3-bd56-9ed43ac6524e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.190:6805","nonce":1083467540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.190:6802","nonce":1083467540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.190:6811","nonce":1083467540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.190:6812","nonce":1083467540}]},"public_addr":"10.20.193.190:6805/1083467540","cluster_addr":"10.20.193.190:6802/1083467540","heartbeat_back_addr":"10.20.193.190:6811/1083467540","heartbeat_front_addr":"10.20.193.190:6812/1083467540","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-25T08:41:26.235 DEBUG:teuthology.misc:3 of 3 OSDs are up 2026-02-25T08:41:26.235 INFO:tasks.ceph:Creating RBD pool 2026-02-25T08:41:26.235 DEBUG:teuthology.orchestra.run.trial190:> sudo ceph --cluster ceph osd pool create rbd 8 2026-02-25T08:41:26.308 INFO:teuthology.orchestra.run.trial190.stderr:2026-02-25T08:41:26.303+0000 7fe11fbf5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:26.309 INFO:teuthology.orchestra.run.trial190.stderr:2026-02-25T08:41:26.308+0000 7fe11fbf5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:27.162 INFO:teuthology.orchestra.run.trial190.stderr:pool 'rbd' created 2026-02-25T08:41:27.176 DEBUG:teuthology.orchestra.run.trial190:> rbd --cluster ceph pool init rbd 2026-02-25T08:41:27.218 INFO:teuthology.orchestra.run.trial190.stderr:2026-02-25T08:41:27.217+0000 7fe5f9a76240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:27.219 INFO:teuthology.orchestra.run.trial190.stderr:2026-02-25T08:41:27.217+0000 7fe5f9a76240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:27.220 INFO:teuthology.orchestra.run.trial190.stderr:2026-02-25T08:41:27.218+0000 7fe5f9a76240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:30.193 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2026-02-25T08:41:30.193 DEBUG:teuthology.orchestra.run.trial190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2026-02-25T08:41:30.194 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2026-02-25T08:41:30.309 INFO:teuthology.orchestra.run.trial190.stderr:2026-02-25T08:41:30.304+0000 7fd32f9b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:30.310 INFO:teuthology.orchestra.run.trial190.stderr:2026-02-25T08:41:30.308+0000 7fd32f9b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:30.417 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:41:30.426 INFO:teuthology.orchestra.run.trial190.stdout:[{"version":1,"timestamp":"0.000000","name":"","changes":[]}] 2026-02-25T08:41:30.426 INFO:tasks.ceph_manager:config epoch is 1 2026-02-25T08:41:30.427 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2026-02-25T08:41:30.427 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2026-02-25T08:41:30.427 DEBUG:teuthology.orchestra.run.trial190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2026-02-25T08:41:30.505 INFO:teuthology.orchestra.run.trial190.stderr:2026-02-25T08:41:30.501+0000 7f464330b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:30.506 INFO:teuthology.orchestra.run.trial190.stderr:2026-02-25T08:41:30.504+0000 7f464330b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:30.646 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:41:30.654 INFO:teuthology.orchestra.run.trial190.stdout:{"epoch":5,"flags":0,"active_gid":4104,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.190:6800","nonce":4211196260},{"type":"v1","addr":"10.20.193.190:6801","nonce":4211196260}]},"active_addr":"10.20.193.190:6801/4211196260","active_change":"2026-02-25T08:41:13.190648+0000","active_mgr_features":4544132024016699391,"available":true,"standbys":[],"modules":["iostat","nfs"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_pg_upmap_activity":{"name":"update_pg_upmap_activity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Updates pg_upmap activity stats to be used in `balancer status detail`","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often the agent on each host will gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"The first TCP port the agent will try to bind to. Up to 1000 subsequent ports will be attempted if this port cannot be bound","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this option can facilitate debugging daemons that encounter runtime problems.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"How frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"Fraction of total system memory to divide among autotuned daemons. Converged systems hosting both Ceph and compute should set a lower value; systems with more than 10 GiB of memory per OSD may benefit from a slightly larger value so that the abundance of RAM can be fully utilized","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command persistent logging","long_desc":"","tags":[],"see_also":[]},"certificate_automated_rotation_enabled":{"name":"certificate_automated_rotation_enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"This flag controls whether cephadm automatically rotates certificates upon expiration.","long_desc":"","tags":[],"see_also":[]},"certificate_check_debug_mode":{"name":"certificate_check_debug_mode","type":"bool","level":"dev","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"FOR TESTING ONLY: This flag forces the certificate check instead of waiting for certificate_check_period.","long_desc":"","tags":[],"see_also":[]},"certificate_check_period":{"name":"certificate_check_period","type":"int","level":"advanced","flags":0,"default_value":"1","min":"0","max":"30","enum_allowed":[],"desc":"Specifies how often (in days) the certificate should be checked for validity.","long_desc":"","tags":[],"see_also":[]},"certificate_duration_days":{"name":"certificate_duration_days","type":"int","level":"advanced","flags":0,"default_value":"1095","min":"90","max":"3650","enum_allowed":[],"desc":"Specifies the duration of self-signed certificates generated and signed by the cephadm root CA","long_desc":"","tags":[],"see_also":[]},"certificate_renewal_threshold_days":{"name":"certificate_renewal_threshold_days","type":"int","level":"advanced","flags":0,"default_value":"30","min":"10","max":"90","enum_allowed":[],"desc":"Specifies the lead time in days to initiate certificate renewal before expiration.","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently Podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.28.1","min":"","max":"","enum_allowed":[],"desc":"Alertmanager container image","long_desc":"","tags":[],"see_also":[]},"container_image_alloy":{"name":"container_image_alloy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/alloy:latest","min":"","max":"","enum_allowed":[],"desc":"Alloy container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"Elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:12.3.1","min":"","max":"","enum_allowed":[],"desc":"Grafana container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"Haproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.9.1","min":"","max":"","enum_allowed":[],"desc":"Node exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.6","min":"","max":"","enum_allowed":[],"desc":"Nvmeof container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"Oauth2 proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v3.6.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba metrics container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"Snmp gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"quay.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"Location of dashboards to include in Grafana","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"How frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether the hardware monitoring daemon be deployed on every host?","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at the debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"Max number of daemons per service per host: guard against mistakes and runaways","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"Max number of OSDs that will be drained simultaneously when OSDs are removed","long_desc":"","tags":[],"see_also":[]},"max_parallel_osd_upgrades":{"name":"max_parallel_osd_upgrades","type":"int","level":"advanced","flags":0,"default_value":"16","min":"","max":"","enum_allowed":[],"desc":"Maximum number of OSD daemons upgraded in parallel.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"Mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default IP address for RedFish API (OOB management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"Location of alerts to include in Prometheus Alertmanager","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry URL for login purposes. This is not the default registry.","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"Cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Customized SSH config file to distribute to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many sequential SSH connection liveness check failures before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often SSH connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"stray_daemon_check_interval":{"name":"stray_daemon_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"How frequently cephadm should check for the presence of stray daemons","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use the cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest to ensure that all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROM_ALERT_CREDENTIAL_CACHE_TTL":{"name":"PROM_ALERT_CREDENTIAL_CACHE_TTL","type":"int","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_HOSTNAME_PER_DAEMON":{"name":"RGW_HOSTNAME_PER_DAEMON","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crypto_caller":{"name":"crypto_caller","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":true,"error_string":"","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"healthcheck_history_max_entries":{"name":"healthcheck_history_max_entries","type":"int","level":"advanced","flags":1,"default_value":"1000","min":"","max":"","enum_allowed":[],"desc":"Maximum number of health check history entries to keep","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"prometheus_tls_secret_name":{"name":"prometheus_tls_secret_name","type":"str","level":"advanced","flags":0,"default_value":"rook-ceph-prometheus-server-tls","min":"","max":"","enum_allowed":[],"desc":"name of tls secret in k8s for prometheus","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_cloning":{"name":"pause_cloning","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_purging":{"name":"pause_purging","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous subvolume purge threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"tentacle":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"force_disabled_modules":{},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"10.20.193.190:0","nonce":515578833}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"10.20.193.190:0","nonce":116286728}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"10.20.193.190:0","nonce":1842533799}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"10.20.193.190:0","nonce":4033684833}]}]} 2026-02-25T08:41:30.656 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2026-02-25T08:41:30.656 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2026-02-25T08:41:30.656 DEBUG:teuthology.orchestra.run.trial190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-25T08:41:30.730 INFO:teuthology.orchestra.run.trial190.stderr:2026-02-25T08:41:30.727+0000 7fe67c50b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:30.731 INFO:teuthology.orchestra.run.trial190.stderr:2026-02-25T08:41:30.730+0000 7fe67c50b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:30.840 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:41:30.841 INFO:teuthology.orchestra.run.trial190.stdout:{"epoch":17,"fsid":"2a6cc9ca-2439-4b64-860c-0d3000bf169b","created":"2026-02-25T08:41:10.753011+0000","modified":"2026-02-25T08:41:30.169801+0000","last_up_change":"2026-02-25T08:41:22.141251+0000","last_in_change":"2026-02-25T08:41:12.501670+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-25T08:41:21.204867+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-25T08:41:26.425506+0000","flags":270337,"flags_names":"hashpspool,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0911493a-668e-4736-b7cc-46e0fced3475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.190:6809","nonce":1401965930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.190:6810","nonce":1401965930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.190:6814","nonce":1401965930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.190:6813","nonce":1401965930}]},"public_addr":"10.20.193.190:6809/1401965930","cluster_addr":"10.20.193.190:6810/1401965930","heartbeat_back_addr":"10.20.193.190:6814/1401965930","heartbeat_front_addr":"10.20.193.190:6813/1401965930","state":["exists","up"]},{"osd":1,"uuid":"5c5fdbaf-a331-4e28-918d-94481631225d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.190:6803","nonce":2036318314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.190:6804","nonce":2036318314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.190:6807","nonce":2036318314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.190:6806","nonce":2036318314}]},"public_addr":"10.20.193.190:6803/2036318314","cluster_addr":"10.20.193.190:6804/2036318314","heartbeat_back_addr":"10.20.193.190:6807/2036318314","heartbeat_front_addr":"10.20.193.190:6806/2036318314","state":["exists","up"]},{"osd":2,"uuid":"e824df79-8af8-42e3-bd56-9ed43ac6524e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.190:6805","nonce":1083467540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.190:6802","nonce":1083467540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.190:6811","nonce":1083467540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.190:6812","nonce":1083467540}]},"public_addr":"10.20.193.190:6805/1083467540","cluster_addr":"10.20.193.190:6802/1083467540","heartbeat_back_addr":"10.20.193.190:6811/1083467540","heartbeat_front_addr":"10.20.193.190:6812/1083467540","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-25T08:41:30.850 INFO:tasks.ceph.ceph_manager.ceph:all up! 2026-02-25T08:41:30.850 DEBUG:teuthology.orchestra.run.trial190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-25T08:41:30.927 INFO:teuthology.orchestra.run.trial190.stderr:2026-02-25T08:41:30.923+0000 7fe87670b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:30.928 INFO:teuthology.orchestra.run.trial190.stderr:2026-02-25T08:41:30.926+0000 7fe87670b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:31.038 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:41:31.038 INFO:teuthology.orchestra.run.trial190.stdout:{"epoch":17,"fsid":"2a6cc9ca-2439-4b64-860c-0d3000bf169b","created":"2026-02-25T08:41:10.753011+0000","modified":"2026-02-25T08:41:30.169801+0000","last_up_change":"2026-02-25T08:41:22.141251+0000","last_in_change":"2026-02-25T08:41:12.501670+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-25T08:41:21.204867+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-25T08:41:26.425506+0000","flags":270337,"flags_names":"hashpspool,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0911493a-668e-4736-b7cc-46e0fced3475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.190:6809","nonce":1401965930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.190:6810","nonce":1401965930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.190:6814","nonce":1401965930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.190:6813","nonce":1401965930}]},"public_addr":"10.20.193.190:6809/1401965930","cluster_addr":"10.20.193.190:6810/1401965930","heartbeat_back_addr":"10.20.193.190:6814/1401965930","heartbeat_front_addr":"10.20.193.190:6813/1401965930","state":["exists","up"]},{"osd":1,"uuid":"5c5fdbaf-a331-4e28-918d-94481631225d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.190:6803","nonce":2036318314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.190:6804","nonce":2036318314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.190:6807","nonce":2036318314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.190:6806","nonce":2036318314}]},"public_addr":"10.20.193.190:6803/2036318314","cluster_addr":"10.20.193.190:6804/2036318314","heartbeat_back_addr":"10.20.193.190:6807/2036318314","heartbeat_front_addr":"10.20.193.190:6806/2036318314","state":["exists","up"]},{"osd":2,"uuid":"e824df79-8af8-42e3-bd56-9ed43ac6524e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.190:6805","nonce":1083467540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.190:6802","nonce":1083467540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.190:6811","nonce":1083467540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.190:6812","nonce":1083467540}]},"public_addr":"10.20.193.190:6805/1083467540","cluster_addr":"10.20.193.190:6802/1083467540","heartbeat_back_addr":"10.20.193.190:6811/1083467540","heartbeat_front_addr":"10.20.193.190:6812/1083467540","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-25T08:41:31.048 DEBUG:teuthology.orchestra.run.trial190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-25T08:41:31.049 DEBUG:teuthology.orchestra.run.trial190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-25T08:41:31.049 DEBUG:teuthology.orchestra.run.trial190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-25T08:41:31.129 INFO:teuthology.orchestra.run.trial190.stderr:2026-02-25T08:41:31.124+0000 7f3f8d61a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:31.130 INFO:teuthology.orchestra.run.trial190.stderr:2026-02-25T08:41:31.128+0000 7f3f8d61a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:31.130 INFO:teuthology.orchestra.run.trial190.stderr:2026-02-25T08:41:31.126+0000 7f3f4441a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:31.131 INFO:teuthology.orchestra.run.trial190.stderr:2026-02-25T08:41:31.126+0000 7f17e2fdc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:31.131 INFO:teuthology.orchestra.run.trial190.stderr:2026-02-25T08:41:31.130+0000 7f3f4441a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:31.132 INFO:teuthology.orchestra.run.trial190.stderr:2026-02-25T08:41:31.130+0000 7f17e2fdc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:31.145 INFO:teuthology.orchestra.run.trial190.stdout:38654705674 2026-02-25T08:41:31.145 DEBUG:teuthology.orchestra.run.trial190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-25T08:41:31.146 INFO:teuthology.orchestra.run.trial190.stdout:42949672969 2026-02-25T08:41:31.147 DEBUG:teuthology.orchestra.run.trial190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-25T08:41:31.147 INFO:teuthology.orchestra.run.trial190.stdout:47244640264 2026-02-25T08:41:31.147 DEBUG:teuthology.orchestra.run.trial190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-25T08:41:31.267 INFO:teuthology.orchestra.run.trial190.stderr:2026-02-25T08:41:31.263+0000 7fcc19011640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:31.267 INFO:teuthology.orchestra.run.trial190.stderr:2026-02-25T08:41:31.263+0000 7f279dc5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:31.268 INFO:teuthology.orchestra.run.trial190.stderr:2026-02-25T08:41:31.263+0000 7f0e402cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:31.268 INFO:teuthology.orchestra.run.trial190.stderr:2026-02-25T08:41:31.267+0000 7fcc19011640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:31.269 INFO:teuthology.orchestra.run.trial190.stderr:2026-02-25T08:41:31.267+0000 7f279dc5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:31.269 INFO:teuthology.orchestra.run.trial190.stderr:2026-02-25T08:41:31.268+0000 7f0e402cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:31.381 INFO:teuthology.orchestra.run.trial190.stdout:47244640264 2026-02-25T08:41:31.382 INFO:teuthology.orchestra.run.trial190.stdout:38654705673 2026-02-25T08:41:31.388 INFO:teuthology.orchestra.run.trial190.stdout:42949672964 2026-02-25T08:41:31.392 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640264 got 47244640264 for osd.0 2026-02-25T08:41:31.392 DEBUG:teuthology.parallel:result is None 2026-02-25T08:41:31.393 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705674 got 38654705673 for osd.1 2026-02-25T08:41:31.397 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672969 got 42949672964 for osd.2 2026-02-25T08:41:32.393 DEBUG:teuthology.orchestra.run.trial190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-25T08:41:32.399 DEBUG:teuthology.orchestra.run.trial190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-25T08:41:32.473 INFO:teuthology.orchestra.run.trial190.stderr:2026-02-25T08:41:32.469+0000 7fd00afa4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:32.475 INFO:teuthology.orchestra.run.trial190.stderr:2026-02-25T08:41:32.473+0000 7fd00afa4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:32.513 INFO:teuthology.orchestra.run.trial190.stderr:2026-02-25T08:41:32.509+0000 7feeee374640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:32.514 INFO:teuthology.orchestra.run.trial190.stderr:2026-02-25T08:41:32.513+0000 7feeee374640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:32.587 INFO:teuthology.orchestra.run.trial190.stdout:38654705674 2026-02-25T08:41:32.597 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705674 got 38654705674 for osd.1 2026-02-25T08:41:32.597 DEBUG:teuthology.parallel:result is None 2026-02-25T08:41:32.625 INFO:teuthology.orchestra.run.trial190.stdout:42949672969 2026-02-25T08:41:32.634 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672969 got 42949672969 for osd.2 2026-02-25T08:41:32.635 DEBUG:teuthology.parallel:result is None 2026-02-25T08:41:32.635 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-25T08:41:32.635 DEBUG:teuthology.orchestra.run.trial190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-25T08:41:32.749 INFO:teuthology.orchestra.run.trial190.stderr:2026-02-25T08:41:32.745+0000 7f54d6ccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:32.750 INFO:teuthology.orchestra.run.trial190.stderr:2026-02-25T08:41:32.749+0000 7f54d6ccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:32.858 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:41:32.858 INFO:teuthology.orchestra.run.trial190.stderr:dumped all 2026-02-25T08:41:32.866 INFO:teuthology.orchestra.run.trial190.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2026-02-25T08:41:31.194368+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":157286352,"kb_used":62917324,"kb_used_data":62917324,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":94369028,"statfs":{"total":161061224448,"available":96633884672,"internally_reserved":0,"allocated":64427339776,"data_stored":64427339776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.032338"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:30.178708+0000","last_change":"2026-02-25T08:41:30.180859+0000","last_active":"2026-02-25T08:41:30.178708+0000","last_peered":"2026-02-25T08:41:30.178708+0000","last_clean":"2026-02-25T08:41:30.178708+0000","last_became_active":"2026-02-25T08:41:28.197609+0000","last_became_peered":"2026-02-25T08:41:28.197609+0000","last_unstale":"2026-02-25T08:41:30.178708+0000","last_undegraded":"2026-02-25T08:41:30.178708+0000","last_fullsized":"2026-02-25T08:41:30.178708+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:27.157892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:27.157892+0000","last_clean_scrub_stamp":"2026-02-25T08:41:27.157892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:30.180032+0000","last_change":"2026-02-25T08:41:30.181787+0000","last_active":"2026-02-25T08:41:30.180032+0000","last_peered":"2026-02-25T08:41:30.180032+0000","last_clean":"2026-02-25T08:41:30.180032+0000","last_became_active":"2026-02-25T08:41:28.197115+0000","last_became_peered":"2026-02-25T08:41:28.197115+0000","last_unstale":"2026-02-25T08:41:30.180032+0000","last_undegraded":"2026-02-25T08:41:30.180032+0000","last_fullsized":"2026-02-25T08:41:30.180032+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:27.157892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:27.157892+0000","last_clean_scrub_stamp":"2026-02-25T08:41:27.157892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:30.180796+0000","last_change":"2026-02-25T08:41:30.183489+0000","last_active":"2026-02-25T08:41:30.180796+0000","last_peered":"2026-02-25T08:41:30.180796+0000","last_clean":"2026-02-25T08:41:30.180796+0000","last_became_active":"2026-02-25T08:41:28.200225+0000","last_became_peered":"2026-02-25T08:41:28.200225+0000","last_unstale":"2026-02-25T08:41:30.180796+0000","last_undegraded":"2026-02-25T08:41:30.180796+0000","last_fullsized":"2026-02-25T08:41:30.180796+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:27.157892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:27.157892+0000","last_clean_scrub_stamp":"2026-02-25T08:41:27.157892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:30.180207+0000","last_change":"2026-02-25T08:41:30.181981+0000","last_active":"2026-02-25T08:41:30.180207+0000","last_peered":"2026-02-25T08:41:30.180207+0000","last_clean":"2026-02-25T08:41:30.180207+0000","last_became_active":"2026-02-25T08:41:28.201556+0000","last_became_peered":"2026-02-25T08:41:28.201556+0000","last_unstale":"2026-02-25T08:41:30.180207+0000","last_undegraded":"2026-02-25T08:41:30.180207+0000","last_fullsized":"2026-02-25T08:41:30.180207+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:27.157892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:27.157892+0000","last_clean_scrub_stamp":"2026-02-25T08:41:27.157892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'1","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:30.181554+0000","last_change":"2026-02-25T08:41:30.181554+0000","last_active":"2026-02-25T08:41:30.181554+0000","last_peered":"2026-02-25T08:41:30.181554+0000","last_clean":"2026-02-25T08:41:30.181554+0000","last_became_active":"2026-02-25T08:41:28.211423+0000","last_became_peered":"2026-02-25T08:41:28.211423+0000","last_unstale":"2026-02-25T08:41:30.181554+0000","last_undegraded":"2026-02-25T08:41:30.181554+0000","last_fullsized":"2026-02-25T08:41:30.181554+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:27.157892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:27.157892+0000","last_clean_scrub_stamp":"2026-02-25T08:41:27.157892+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:30.180545+0000","last_change":"2026-02-25T08:41:30.182651+0000","last_active":"2026-02-25T08:41:30.180545+0000","last_peered":"2026-02-25T08:41:30.180545+0000","last_clean":"2026-02-25T08:41:30.180545+0000","last_became_active":"2026-02-25T08:41:28.200920+0000","last_became_peered":"2026-02-25T08:41:28.200920+0000","last_unstale":"2026-02-25T08:41:30.180545+0000","last_undegraded":"2026-02-25T08:41:30.180545+0000","last_fullsized":"2026-02-25T08:41:30.180545+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:27.157892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:27.157892+0000","last_clean_scrub_stamp":"2026-02-25T08:41:27.157892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:30.178398+0000","last_change":"2026-02-25T08:41:30.180887+0000","last_active":"2026-02-25T08:41:30.178398+0000","last_peered":"2026-02-25T08:41:30.178398+0000","last_clean":"2026-02-25T08:41:30.178398+0000","last_became_active":"2026-02-25T08:41:28.210490+0000","last_became_peered":"2026-02-25T08:41:28.210490+0000","last_unstale":"2026-02-25T08:41:30.178398+0000","last_undegraded":"2026-02-25T08:41:30.178398+0000","last_fullsized":"2026-02-25T08:41:30.178398+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:27.157892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:27.157892+0000","last_clean_scrub_stamp":"2026-02-25T08:41:27.157892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":16,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:30.180164+0000","last_change":"2026-02-25T08:41:30.183473+0000","last_active":"2026-02-25T08:41:30.180164+0000","last_peered":"2026-02-25T08:41:30.180164+0000","last_clean":"2026-02-25T08:41:30.180164+0000","last_became_active":"2026-02-25T08:41:28.197113+0000","last_became_peered":"2026-02-25T08:41:28.197113+0000","last_unstale":"2026-02-25T08:41:30.180164+0000","last_undegraded":"2026-02-25T08:41:30.180164+0000","last_fullsized":"2026-02-25T08:41:30.180164+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:27.157892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:27.157892+0000","last_clean_scrub_stamp":"2026-02-25T08:41:27.157892+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":90,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:30.178369+0000","last_change":"2026-02-25T08:41:23.178708+0000","last_active":"2026-02-25T08:41:30.178369+0000","last_peered":"2026-02-25T08:41:30.178369+0000","last_clean":"2026-02-25T08:41:30.178369+0000","last_became_active":"2026-02-25T08:41:23.172000+0000","last_became_peered":"2026-02-25T08:41:23.172000+0000","last_unstale":"2026-02-25T08:41:30.178369+0000","last_undegraded":"2026-02-25T08:41:30.178369+0000","last_fullsized":"2026-02-25T08:41:30.178369+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:22.141251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:22.141251+0000","last_clean_scrub_stamp":"2026-02-25T08:41:22.141251+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":161061224448,"available":96633860096,"internally_reserved":0,"allocated":64427364352,"data_stored":64427364352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53687074816,"available":32210939904,"internally_reserved":0,"allocated":21476134912,"data_stored":21476134912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":0,"up_from":11,"seq":47244640264,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972684,"kb_used_data":20972684,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456100,"statfs":{"total":53687074816,"available":32211046400,"internally_reserved":0,"allocated":21476028416,"data_stored":21476028416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20971860,"kb_used_data":20971860,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456924,"statfs":{"total":53687074816,"available":32211890176,"internally_reserved":0,"allocated":21475184640,"data_stored":21475184640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972780,"kb_used_data":20972780,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456004,"statfs":{"total":53687074816,"available":32210948096,"internally_reserved":0,"allocated":21476126720,"data_stored":21476126720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53687074816,"available":32210939904,"internally_reserved":0,"allocated":21476134912,"data_stored":21476134912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53687074816,"available":32211038208,"internally_reserved":0,"allocated":21476036608,"data_stored":21476036608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53687074816,"available":32210939904,"internally_reserved":0,"allocated":21476134912,"data_stored":21476134912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53687074816,"available":32211881984,"internally_reserved":0,"allocated":21475192832,"data_stored":21475192832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-25T08:41:32.867 DEBUG:teuthology.orchestra.run.trial190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-25T08:41:32.942 INFO:teuthology.orchestra.run.trial190.stderr:2026-02-25T08:41:32.938+0000 7f118d4c2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:32.943 INFO:teuthology.orchestra.run.trial190.stderr:2026-02-25T08:41:32.941+0000 7f118d4c2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:33.050 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:41:33.051 INFO:teuthology.orchestra.run.trial190.stderr:dumped all 2026-02-25T08:41:33.060 INFO:teuthology.orchestra.run.trial190.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2026-02-25T08:41:31.194368+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":157286352,"kb_used":62917324,"kb_used_data":62917324,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":94369028,"statfs":{"total":161061224448,"available":96633884672,"internally_reserved":0,"allocated":64427339776,"data_stored":64427339776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.032338"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:30.178708+0000","last_change":"2026-02-25T08:41:30.180859+0000","last_active":"2026-02-25T08:41:30.178708+0000","last_peered":"2026-02-25T08:41:30.178708+0000","last_clean":"2026-02-25T08:41:30.178708+0000","last_became_active":"2026-02-25T08:41:28.197609+0000","last_became_peered":"2026-02-25T08:41:28.197609+0000","last_unstale":"2026-02-25T08:41:30.178708+0000","last_undegraded":"2026-02-25T08:41:30.178708+0000","last_fullsized":"2026-02-25T08:41:30.178708+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:27.157892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:27.157892+0000","last_clean_scrub_stamp":"2026-02-25T08:41:27.157892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:30.180032+0000","last_change":"2026-02-25T08:41:30.181787+0000","last_active":"2026-02-25T08:41:30.180032+0000","last_peered":"2026-02-25T08:41:30.180032+0000","last_clean":"2026-02-25T08:41:30.180032+0000","last_became_active":"2026-02-25T08:41:28.197115+0000","last_became_peered":"2026-02-25T08:41:28.197115+0000","last_unstale":"2026-02-25T08:41:30.180032+0000","last_undegraded":"2026-02-25T08:41:30.180032+0000","last_fullsized":"2026-02-25T08:41:30.180032+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:27.157892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:27.157892+0000","last_clean_scrub_stamp":"2026-02-25T08:41:27.157892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:30.180796+0000","last_change":"2026-02-25T08:41:30.183489+0000","last_active":"2026-02-25T08:41:30.180796+0000","last_peered":"2026-02-25T08:41:30.180796+0000","last_clean":"2026-02-25T08:41:30.180796+0000","last_became_active":"2026-02-25T08:41:28.200225+0000","last_became_peered":"2026-02-25T08:41:28.200225+0000","last_unstale":"2026-02-25T08:41:30.180796+0000","last_undegraded":"2026-02-25T08:41:30.180796+0000","last_fullsized":"2026-02-25T08:41:30.180796+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:27.157892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:27.157892+0000","last_clean_scrub_stamp":"2026-02-25T08:41:27.157892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:30.180207+0000","last_change":"2026-02-25T08:41:30.181981+0000","last_active":"2026-02-25T08:41:30.180207+0000","last_peered":"2026-02-25T08:41:30.180207+0000","last_clean":"2026-02-25T08:41:30.180207+0000","last_became_active":"2026-02-25T08:41:28.201556+0000","last_became_peered":"2026-02-25T08:41:28.201556+0000","last_unstale":"2026-02-25T08:41:30.180207+0000","last_undegraded":"2026-02-25T08:41:30.180207+0000","last_fullsized":"2026-02-25T08:41:30.180207+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:27.157892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:27.157892+0000","last_clean_scrub_stamp":"2026-02-25T08:41:27.157892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'1","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:30.181554+0000","last_change":"2026-02-25T08:41:30.181554+0000","last_active":"2026-02-25T08:41:30.181554+0000","last_peered":"2026-02-25T08:41:30.181554+0000","last_clean":"2026-02-25T08:41:30.181554+0000","last_became_active":"2026-02-25T08:41:28.211423+0000","last_became_peered":"2026-02-25T08:41:28.211423+0000","last_unstale":"2026-02-25T08:41:30.181554+0000","last_undegraded":"2026-02-25T08:41:30.181554+0000","last_fullsized":"2026-02-25T08:41:30.181554+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:27.157892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:27.157892+0000","last_clean_scrub_stamp":"2026-02-25T08:41:27.157892+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:30.180545+0000","last_change":"2026-02-25T08:41:30.182651+0000","last_active":"2026-02-25T08:41:30.180545+0000","last_peered":"2026-02-25T08:41:30.180545+0000","last_clean":"2026-02-25T08:41:30.180545+0000","last_became_active":"2026-02-25T08:41:28.200920+0000","last_became_peered":"2026-02-25T08:41:28.200920+0000","last_unstale":"2026-02-25T08:41:30.180545+0000","last_undegraded":"2026-02-25T08:41:30.180545+0000","last_fullsized":"2026-02-25T08:41:30.180545+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:27.157892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:27.157892+0000","last_clean_scrub_stamp":"2026-02-25T08:41:27.157892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:30.178398+0000","last_change":"2026-02-25T08:41:30.180887+0000","last_active":"2026-02-25T08:41:30.178398+0000","last_peered":"2026-02-25T08:41:30.178398+0000","last_clean":"2026-02-25T08:41:30.178398+0000","last_became_active":"2026-02-25T08:41:28.210490+0000","last_became_peered":"2026-02-25T08:41:28.210490+0000","last_unstale":"2026-02-25T08:41:30.178398+0000","last_undegraded":"2026-02-25T08:41:30.178398+0000","last_fullsized":"2026-02-25T08:41:30.178398+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:27.157892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:27.157892+0000","last_clean_scrub_stamp":"2026-02-25T08:41:27.157892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":16,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:30.180164+0000","last_change":"2026-02-25T08:41:30.183473+0000","last_active":"2026-02-25T08:41:30.180164+0000","last_peered":"2026-02-25T08:41:30.180164+0000","last_clean":"2026-02-25T08:41:30.180164+0000","last_became_active":"2026-02-25T08:41:28.197113+0000","last_became_peered":"2026-02-25T08:41:28.197113+0000","last_unstale":"2026-02-25T08:41:30.180164+0000","last_undegraded":"2026-02-25T08:41:30.180164+0000","last_fullsized":"2026-02-25T08:41:30.180164+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:27.157892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:27.157892+0000","last_clean_scrub_stamp":"2026-02-25T08:41:27.157892+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":90,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:30.178369+0000","last_change":"2026-02-25T08:41:23.178708+0000","last_active":"2026-02-25T08:41:30.178369+0000","last_peered":"2026-02-25T08:41:30.178369+0000","last_clean":"2026-02-25T08:41:30.178369+0000","last_became_active":"2026-02-25T08:41:23.172000+0000","last_became_peered":"2026-02-25T08:41:23.172000+0000","last_unstale":"2026-02-25T08:41:30.178369+0000","last_undegraded":"2026-02-25T08:41:30.178369+0000","last_fullsized":"2026-02-25T08:41:30.178369+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:22.141251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:22.141251+0000","last_clean_scrub_stamp":"2026-02-25T08:41:22.141251+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":161061224448,"available":96633860096,"internally_reserved":0,"allocated":64427364352,"data_stored":64427364352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53687074816,"available":32210939904,"internally_reserved":0,"allocated":21476134912,"data_stored":21476134912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":0,"up_from":11,"seq":47244640264,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972684,"kb_used_data":20972684,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456100,"statfs":{"total":53687074816,"available":32211046400,"internally_reserved":0,"allocated":21476028416,"data_stored":21476028416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20971860,"kb_used_data":20971860,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456924,"statfs":{"total":53687074816,"available":32211890176,"internally_reserved":0,"allocated":21475184640,"data_stored":21475184640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972780,"kb_used_data":20972780,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456004,"statfs":{"total":53687074816,"available":32210948096,"internally_reserved":0,"allocated":21476126720,"data_stored":21476126720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53687074816,"available":32210939904,"internally_reserved":0,"allocated":21476134912,"data_stored":21476134912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53687074816,"available":32211038208,"internally_reserved":0,"allocated":21476036608,"data_stored":21476036608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53687074816,"available":32210939904,"internally_reserved":0,"allocated":21476134912,"data_stored":21476134912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53687074816,"available":32211881984,"internally_reserved":0,"allocated":21475192832,"data_stored":21475192832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-25T08:41:33.061 INFO:tasks.ceph.ceph_manager.ceph:clean! 2026-02-25T08:41:33.061 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2026-02-25T08:41:33.061 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2026-02-25T08:41:33.061 DEBUG:teuthology.orchestra.run.trial190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-25T08:41:33.137 INFO:teuthology.orchestra.run.trial190.stderr:2026-02-25T08:41:33.134+0000 7f11df68b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:33.138 INFO:teuthology.orchestra.run.trial190.stderr:2026-02-25T08:41:33.136+0000 7f11df68b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:33.276 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:41:33.276 INFO:teuthology.orchestra.run.trial190.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2026-02-25T08:41:33.286 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2026-02-25T08:41:33.286 INFO:teuthology.run_tasks:Running task ssh_keys... 2026-02-25T08:41:33.364 DEBUG:teuthology.orchestra.run.trial190:> echo 'StrictHostKeyChecking no 2026-02-25T08:41:33.364 DEBUG:teuthology.orchestra.run.trial190:> ' > /home/ubuntu/.ssh/config && echo 'UserKnownHostsFile ' /dev/null >> /home/ubuntu/.ssh/config && chmod 600 /home/ubuntu/.ssh/config 2026-02-25T08:41:33.380 INFO:teuthology.task.ssh_keys:generated public key AAAAB3NzaC1yc2EAAAADAQABAAABAQCjHvixrqDIPfzFFP9L7ZijE4IpeNENbX2WQTsjz8GHWryGVLtBrl9gjjsxMuDfBoVEfb6JSUijCenpyZp/T9f4TnGv9FA7WKjlMMKHHY5jC5gj+UIUwY5KghBJHUsXmoukPQtAIUGYB4QcCSEdnFUkfLixXfv5zmttsYXh7IGo0L7TAqxR7Mjqkjc23OCCDMoJOOcBqpdG9AmfXFzeIcKZFwUdO6WmvUB1K5kx1F6V8W1v9VDkccCUk6cLdZcPm7KZCtt59TVp4aGgnJGl0njrV3QQUbAEJLjFh1UZMOB4tHgkVI0cdd0nLv925W5vWFXrxBHxNX/oU/Cq13xr5RKX 2026-02-25T08:41:33.380 INFO:teuthology.task.ssh_keys:pushing keys to trial190.front.sepia.ceph.com for ubuntu 2026-02-25T08:41:33.380 DEBUG:teuthology.orchestra.run.trial190:> rm -f -- /home/ubuntu/.ssh/id_rsa 2026-02-25T08:41:33.436 DEBUG:teuthology.orchestra.run.trial190:> set -ex 2026-02-25T08:41:33.436 DEBUG:teuthology.orchestra.run.trial190:> dd of=/home/ubuntu/.ssh/id_rsa 2026-02-25T08:41:33.436 DEBUG:teuthology.orchestra.run.trial190:> chmod 0500 /home/ubuntu/.ssh/id_rsa 2026-02-25T08:41:33.492 DEBUG:teuthology.orchestra.run.trial190:> rm -f -- /home/ubuntu/.ssh/id_rsa.pub 2026-02-25T08:41:33.547 DEBUG:teuthology.orchestra.run.trial190:> set -ex 2026-02-25T08:41:33.548 DEBUG:teuthology.orchestra.run.trial190:> dd of=/home/ubuntu/.ssh/id_rsa.pub 2026-02-25T08:41:33.604 DEBUG:teuthology.orchestra.run.trial190:> cp -v -a /home/ubuntu/.ssh/authorized_keys /home/ubuntu/.ssh/authorized_keys_2026-02-25_08:41:33:604511 2026-02-25T08:41:33.660 INFO:teuthology.orchestra.run.trial190.stdout:'/home/ubuntu/.ssh/authorized_keys' -> '/home/ubuntu/.ssh/authorized_keys_2026-02-25_08:41:33:604511' 2026-02-25T08:41:33.660 DEBUG:teuthology.orchestra.run.trial190:> set -ex 2026-02-25T08:41:33.660 DEBUG:teuthology.orchestra.run.trial190:> dd of=/home/ubuntu/.ssh/authorized_keys conv=notrunc oflag=append 2026-02-25T08:41:33.716 INFO:teuthology.run_tasks:Running task cbt... 2026-02-25T08:41:33.731 INFO:tasks.cbt:cbt configuration is {'cluster': {'user': 'ubuntu', 'head': 'ubuntu@trial190.front.sepia.ceph.com', 'osds': ['ubuntu@trial190.front.sepia.ceph.com'], 'mons': ['ubuntu@trial190.front.sepia.ceph.com'], 'clients': ['ubuntu@trial190.front.sepia.ceph.com'], 'rgws': {'ubuntu@trial190.front.sepia.ceph.com': None}, 'osds_per_node': 3, 'rebuild_every_test': False, 'use_existing': True, 'is_teuthology': True, 'iterations': 1, 'tmp_dir': '/tmp/cbt', 'pool_profiles': {'replicated': {'pg_size': 256, 'pgp_size': 256, 'replication': 'replicated'}}, 'pid_dir': '/var/run/ceph'}, 'benchmarks': {'radosbench': {'concurrent_ops': 4, 'concurrent_procs': 2, 'op_size': [4096], 'pool_monitoring_list': ['collectl'], 'pool_profile': 'replicated', 'readmode': 'rand', 'run_monitoring_list': ['collectl'], 'time': 60, 'write_only': False}}, 'client_endpoints': None, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}} 2026-02-25T08:41:33.731 DEBUG:teuthology.orchestra.run.trial190:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/cbt 2026-02-25T08:41:33.773 DEBUG:teuthology.orchestra.run.trial190:> set -ex 2026-02-25T08:41:33.773 DEBUG:teuthology.orchestra.run.trial190:> dd of=/home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2026-02-25T08:41:33.828 DEBUG:teuthology.orchestra.run.trial190:> git clone --depth 1 -b master https://github.com/ceph/cbt.git /home/ubuntu/cephtest/cbt 2026-02-25T08:41:33.889 INFO:teuthology.orchestra.run.trial190.stderr:Cloning into '/home/ubuntu/cephtest/cbt'... 2026-02-25T08:41:34.190 INFO:tasks.cbt:Installing collectl 2026-02-25T08:41:34.191 DEBUG:teuthology.orchestra.run.trial190:> sudo mkdir collectl && cd collectl && sudo wget https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download -O collectl.tar.gz && sudo tar -xvf collectl.tar.gz && cd collectl-4.3.1 && sudo ./INSTALL 2026-02-25T08:41:34.265 INFO:teuthology.orchestra.run.trial190.stderr:--2026-02-25 08:41:34-- https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download 2026-02-25T08:41:34.318 INFO:teuthology.orchestra.run.trial190.stderr:Resolving sourceforge.net (sourceforge.net)... 104.18.13.149, 104.18.12.149, 2606:4700::6812:c95, ... 2026-02-25T08:41:34.321 INFO:teuthology.orchestra.run.trial190.stderr:Connecting to sourceforge.net (sourceforge.net)|104.18.13.149|:443... connected. 2026-02-25T08:41:34.696 INFO:teuthology.orchestra.run.trial190.stderr:HTTP request sent, awaiting response... 302 Found 2026-02-25T08:41:34.697 INFO:teuthology.orchestra.run.trial190.stderr:Location: https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABpnrW-gfm6UKt7igsYmhVMjM3GT6ko3Grq5JvGhEv6iDR2Gui7rvsZQeTYmgltSn_ANtzepSTacWWxijSv_ZQ70VtYcA%3D%3D&use_mirror=psychz&r= [following] 2026-02-25T08:41:34.697 INFO:teuthology.orchestra.run.trial190.stderr:--2026-02-25 08:41:34-- https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABpnrW-gfm6UKt7igsYmhVMjM3GT6ko3Grq5JvGhEv6iDR2Gui7rvsZQeTYmgltSn_ANtzepSTacWWxijSv_ZQ70VtYcA%3D%3D&use_mirror=psychz&r= 2026-02-25T08:41:34.739 INFO:teuthology.orchestra.run.trial190.stderr:Resolving downloads.sourceforge.net (downloads.sourceforge.net)... 104.18.13.149, 104.18.12.149, 2606:4700::6812:c95, ... 2026-02-25T08:41:34.742 INFO:teuthology.orchestra.run.trial190.stderr:Connecting to downloads.sourceforge.net (downloads.sourceforge.net)|104.18.13.149|:443... connected. 2026-02-25T08:41:35.374 INFO:teuthology.orchestra.run.trial190.stderr:HTTP request sent, awaiting response... 302 Found 2026-02-25T08:41:35.374 INFO:teuthology.orchestra.run.trial190.stderr:Location: https://psychz.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?viasf=1 [following] 2026-02-25T08:41:35.374 INFO:teuthology.orchestra.run.trial190.stderr:--2026-02-25 08:41:35-- https://psychz.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?viasf=1 2026-02-25T08:41:35.379 INFO:teuthology.orchestra.run.trial190.stderr:Resolving psychz.dl.sourceforge.net (psychz.dl.sourceforge.net)... 208.87.241.191 2026-02-25T08:41:35.443 INFO:teuthology.orchestra.run.trial190.stderr:Connecting to psychz.dl.sourceforge.net (psychz.dl.sourceforge.net)|208.87.241.191|:443... connected. 2026-02-25T08:41:35.751 INFO:teuthology.orchestra.run.trial190.stderr:HTTP request sent, awaiting response... 200 OK 2026-02-25T08:41:35.751 INFO:teuthology.orchestra.run.trial190.stderr:Length: 620693 (606K) [application/x-gzip] 2026-02-25T08:41:35.751 INFO:teuthology.orchestra.run.trial190.stderr:Saving to: ‘collectl.tar.gz’ 2026-02-25T08:41:35.751 INFO:teuthology.orchestra.run.trial190.stderr: 2026-02-25T08:41:35.814 INFO:teuthology.orchestra.run.trial190.stderr: 0K .......... .......... .......... .......... .......... 8% 793K 1s 2026-02-25T08:41:35.877 INFO:teuthology.orchestra.run.trial190.stderr: 50K .......... .......... .......... .......... .......... 16% 797K 1s 2026-02-25T08:41:35.877 INFO:teuthology.orchestra.run.trial190.stderr: 100K .......... .......... .......... .......... .......... 24% 205M 0s 2026-02-25T08:41:35.879 INFO:teuthology.orchestra.run.trial190.stderr: 150K .......... .......... .......... .......... .......... 32% 23.3M 0s 2026-02-25T08:41:35.944 INFO:teuthology.orchestra.run.trial190.stderr: 200K .......... .......... .......... .......... .......... 41% 829K 0s 2026-02-25T08:41:35.944 INFO:teuthology.orchestra.run.trial190.stderr: 250K .......... .......... .......... .......... .......... 49% 192M 0s 2026-02-25T08:41:35.944 INFO:teuthology.orchestra.run.trial190.stderr: 300K .......... .......... .......... .......... .......... 57% 237M 0s 2026-02-25T08:41:35.944 INFO:teuthology.orchestra.run.trial190.stderr: 350K .......... .......... .......... .......... .......... 65% 27.9M 0s 2026-02-25T08:41:36.006 INFO:teuthology.orchestra.run.trial190.stderr: 400K .......... .......... .......... .......... .......... 74% 829K 0s 2026-02-25T08:41:36.006 INFO:teuthology.orchestra.run.trial190.stderr: 450K .......... .......... .......... .......... .......... 82% 62.4M 0s 2026-02-25T08:41:36.006 INFO:teuthology.orchestra.run.trial190.stderr: 500K .......... .......... .......... .......... .......... 90% 288M 0s 2026-02-25T08:41:36.006 INFO:teuthology.orchestra.run.trial190.stderr: 550K .......... .......... .......... .......... .......... 98% 395M 0s 2026-02-25T08:41:36.007 INFO:teuthology.orchestra.run.trial190.stderr: 600K ...... 100% 11.4T=0.3s 2026-02-25T08:41:36.007 INFO:teuthology.orchestra.run.trial190.stderr: 2026-02-25T08:41:36.007 INFO:teuthology.orchestra.run.trial190.stderr:2026-02-25 08:41:36 (2.35 MB/s) - ‘collectl.tar.gz’ saved [620693/620693] 2026-02-25T08:41:36.007 INFO:teuthology.orchestra.run.trial190.stderr: 2026-02-25T08:41:36.018 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/ 2026-02-25T08:41:36.019 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/collectl.conf 2026-02-25T08:41:36.019 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/docs/ 2026-02-25T08:41:36.019 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/docs/SlowProc.html 2026-02-25T08:41:36.019 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/docs/BuddyInfo.html 2026-02-25T08:41:36.019 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/docs/WhySummary.html 2026-02-25T08:41:36.019 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/docs/Socket.html 2026-02-25T08:41:36.019 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/docs/colmux.html 2026-02-25T08:41:36.019 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/docs/Tutorial.html 2026-02-25T08:41:36.019 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/docs/Gexpr.html 2026-02-25T08:41:36.019 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/docs/Import.html 2026-02-25T08:41:36.019 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/docs/style.css 2026-02-25T08:41:36.019 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/docs/Ganglia3.jpg 2026-02-25T08:41:36.019 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/docs/Ganglia2.jpg 2026-02-25T08:41:36.019 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/docs/FileNaming.html 2026-02-25T08:41:36.019 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/docs/TheMath.html 2026-02-25T08:41:36.019 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/docs/Environmental.html 2026-02-25T08:41:36.019 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/docs/Process.html 2026-02-25T08:41:36.020 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/docs/OutputFiles.html 2026-02-25T08:41:36.020 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/docs/HiResTime.html 2026-02-25T08:41:36.020 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/docs/OutputFormats.html 2026-02-25T08:41:36.020 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/docs/slub.jpg 2026-02-25T08:41:36.020 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/docs/Lustre.html 2026-02-25T08:41:36.020 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/docs/Documentation.html 2026-02-25T08:41:36.020 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/docs/Inodes.html 2026-02-25T08:41:36.020 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/docs/Graphite.html 2026-02-25T08:41:36.020 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/docs/InputFiles.html 2026-02-25T08:41:36.020 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/docs/Export.html 2026-02-25T08:41:36.020 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/docs/Interrupts.html 2026-02-25T08:41:36.020 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/docs/Startup.html 2026-02-25T08:41:36.020 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/docs/Performance.html 2026-02-25T08:41:36.020 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/docs/Messages.html 2026-02-25T08:41:36.020 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/docs/Logging.html 2026-02-25T08:41:36.020 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/docs/Data-detail.html 2026-02-25T08:41:36.020 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/docs/CPUs.html 2026-02-25T08:41:36.020 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/docs/Matrix.html 2026-02-25T08:41:36.020 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/docs/Data-brief.html 2026-02-25T08:41:36.020 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/docs/SlabInfo.html 2026-02-25T08:41:36.020 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/docs/Misc.html 2026-02-25T08:41:36.020 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/docs/Examples.html 2026-02-25T08:41:36.020 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/docs/OpenStack.html 2026-02-25T08:41:36.020 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/docs/Plotfiles.html 2026-02-25T08:41:36.020 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/docs/ColmuxCPU.jpg 2026-02-25T08:41:36.020 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/docs/Tutorial-Lustre.html 2026-02-25T08:41:36.020 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/docs/Data-verbose.html 2026-02-25T08:41:36.021 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/docs/Data.html 2026-02-25T08:41:36.021 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/docs/Exceptions.html 2026-02-25T08:41:36.021 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/docs/RunningAsAService.html 2026-02-25T08:41:36.021 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/docs/FAQ-collectl.html 2026-02-25T08:41:36.021 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/docs/Ganglia.jpg 2026-02-25T08:41:36.021 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/docs/OperationalModes.html 2026-02-25T08:41:36.021 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/docs/Playback.html 2026-02-25T08:41:36.021 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/docs/Hello.html 2026-02-25T08:41:36.021 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/docs/Sockets.html 2026-02-25T08:41:36.021 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/docs/Network.html 2026-02-25T08:41:36.021 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/docs/Infiniband.html 2026-02-25T08:41:36.021 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/docs/Memory.html 2026-02-25T08:41:36.021 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/docs/ColmuxLustre.jpg 2026-02-25T08:41:36.022 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/docs/Disks.html 2026-02-25T08:41:36.022 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/docs/NfsInfo.html 2026-02-25T08:41:36.022 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/service/ 2026-02-25T08:41:36.022 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/service/collectl.service 2026-02-25T08:41:36.022 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/initd/ 2026-02-25T08:41:36.022 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/initd/collectl-suse 2026-02-25T08:41:36.022 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/initd/collectl-debian 2026-02-25T08:41:36.022 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/initd/collectl-generic 2026-02-25T08:41:36.022 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/initd/collectl 2026-02-25T08:41:36.022 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/misc.ph 2026-02-25T08:41:36.022 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/gexpr.ph 2026-02-25T08:41:36.022 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/statsd.ph 2026-02-25T08:41:36.022 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/envrules.std 2026-02-25T08:41:36.022 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/hello.ph 2026-02-25T08:41:36.022 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/vmsum.ph 2026-02-25T08:41:36.022 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/proctree.ph 2026-02-25T08:41:36.022 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/README-WINDOWS 2026-02-25T08:41:36.022 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/formatit.ph 2026-02-25T08:41:36.022 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/colmux 2026-02-25T08:41:36.023 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/INSTALL 2026-02-25T08:41:36.023 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/graphite.ph 2026-02-25T08:41:36.023 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/RELEASE-collectl 2026-02-25T08:41:36.023 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/vmstat.ph 2026-02-25T08:41:36.023 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/README 2026-02-25T08:41:36.023 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/man1/ 2026-02-25T08:41:36.023 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/man1/collectl.1 2026-02-25T08:41:36.023 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/man1/colmux.1 2026-02-25T08:41:36.024 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/lexpr.ph 2026-02-25T08:41:36.024 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/client.pl 2026-02-25T08:41:36.024 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/collectl 2026-02-25T08:41:36.025 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/COPYING 2026-02-25T08:41:36.025 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/vnet.ph 2026-02-25T08:41:36.025 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/GPL 2026-02-25T08:41:36.025 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/UNINSTALL 2026-02-25T08:41:36.025 INFO:teuthology.orchestra.run.trial190.stdout:collectl-4.3.1/ARTISTIC 2026-02-25T08:41:36.069 INFO:teuthology.orchestra.run.trial190.stderr:./INSTALL: line 90: chkconfig: command not found 2026-02-25T08:41:36.071 DEBUG:teuthology.orchestra.run.trial190:> sudo yum -y install librbd-devel pdsh pdsh-rcmd-ssh perf 2026-02-25T08:41:36.436 INFO:teuthology.orchestra.run.trial190.stdout:Last metadata expiration check: 0:00:59 ago on Wed 25 Feb 2026 08:40:37 AM UTC. 2026-02-25T08:41:36.521 INFO:teuthology.orchestra.run.trial190.stdout:Dependencies resolved. 2026-02-25T08:41:36.522 INFO:teuthology.orchestra.run.trial190.stdout:================================================================================ 2026-02-25T08:41:36.522 INFO:teuthology.orchestra.run.trial190.stdout: Package Arch Version Repository Size 2026-02-25T08:41:36.523 INFO:teuthology.orchestra.run.trial190.stdout:================================================================================ 2026-02-25T08:41:36.523 INFO:teuthology.orchestra.run.trial190.stdout:Installing: 2026-02-25T08:41:36.523 INFO:teuthology.orchestra.run.trial190.stdout: librbd-devel x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 24 k 2026-02-25T08:41:36.523 INFO:teuthology.orchestra.run.trial190.stdout: pdsh x86_64 2.34-7.el9 epel 118 k 2026-02-25T08:41:36.523 INFO:teuthology.orchestra.run.trial190.stdout: pdsh-rcmd-ssh x86_64 2.34-7.el9 epel 13 k 2026-02-25T08:41:36.523 INFO:teuthology.orchestra.run.trial190.stdout: perf x86_64 5.14.0-686.el9 appstream 3.3 M 2026-02-25T08:41:36.523 INFO:teuthology.orchestra.run.trial190.stdout:Installing dependencies: 2026-02-25T08:41:36.523 INFO:teuthology.orchestra.run.trial190.stdout: libradospp-devel x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 31 k 2026-02-25T08:41:36.523 INFO:teuthology.orchestra.run.trial190.stdout: libtraceevent x86_64 1.8.4-2.el9 baseos 290 k 2026-02-25T08:41:36.523 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:41:36.523 INFO:teuthology.orchestra.run.trial190.stdout:Transaction Summary 2026-02-25T08:41:36.523 INFO:teuthology.orchestra.run.trial190.stdout:================================================================================ 2026-02-25T08:41:36.524 INFO:teuthology.orchestra.run.trial190.stdout:Install 6 Packages 2026-02-25T08:41:36.524 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:41:36.524 INFO:teuthology.orchestra.run.trial190.stdout:Total download size: 3.8 M 2026-02-25T08:41:36.524 INFO:teuthology.orchestra.run.trial190.stdout:Installed size: 14 M 2026-02-25T08:41:36.524 INFO:teuthology.orchestra.run.trial190.stdout:Downloading Packages: 2026-02-25T08:41:37.393 INFO:teuthology.orchestra.run.trial190.stdout:(1/6): libradospp-devel-20.3.0-5338.g6aeb6975.e 452 kB/s | 31 kB 00:00 2026-02-25T08:41:37.393 INFO:teuthology.orchestra.run.trial190.stdout:(2/6): librbd-devel-20.3.0-5338.g6aeb6975.el9.x 339 kB/s | 24 kB 00:00 2026-02-25T08:41:37.437 INFO:teuthology.orchestra.run.trial190.stdout:(3/6): pdsh-2.34-7.el9.x86_64.rpm 2.7 MB/s | 118 kB 00:00 2026-02-25T08:41:37.445 INFO:teuthology.orchestra.run.trial190.stdout:(4/6): pdsh-rcmd-ssh-2.34-7.el9.x86_64.rpm 1.7 MB/s | 13 kB 00:00 2026-02-25T08:41:38.177 INFO:teuthology.orchestra.run.trial190.stdout:(5/6): libtraceevent-1.8.4-2.el9.x86_64.rpm 340 kB/s | 290 kB 00:00 2026-02-25T08:41:41.538 INFO:teuthology.orchestra.run.trial190.stdout:(6/6): perf-5.14.0-686.el9.x86_64.rpm 815 kB/s | 3.3 MB 00:04 2026-02-25T08:41:41.541 INFO:teuthology.orchestra.run.trial190.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:41:41.541 INFO:teuthology.orchestra.run.trial190.stdout:Total 768 kB/s | 3.8 MB 00:05 2026-02-25T08:41:41.565 INFO:teuthology.orchestra.run.trial190.stdout:Running transaction check 2026-02-25T08:41:41.575 INFO:teuthology.orchestra.run.trial190.stdout:Transaction check succeeded. 2026-02-25T08:41:41.576 INFO:teuthology.orchestra.run.trial190.stdout:Running transaction test 2026-02-25T08:41:41.634 INFO:teuthology.orchestra.run.trial190.stdout:Transaction test succeeded. 2026-02-25T08:41:41.635 INFO:teuthology.orchestra.run.trial190.stdout:Running transaction 2026-02-25T08:41:41.713 INFO:teuthology.orchestra.run.trial190.stdout: Preparing : 1/1 2026-02-25T08:41:41.718 INFO:teuthology.orchestra.run.trial190.stdout: Installing : pdsh-rcmd-ssh-2.34-7.el9.x86_64 1/6 2026-02-25T08:41:41.759 INFO:teuthology.orchestra.run.trial190.stdout: Installing : pdsh-2.34-7.el9.x86_64 2/6 2026-02-25T08:41:41.763 INFO:teuthology.orchestra.run.trial190.stdout: Installing : libtraceevent-1.8.4-2.el9.x86_64 3/6 2026-02-25T08:41:41.765 INFO:teuthology.orchestra.run.trial190.stdout: Installing : libradospp-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 4/6 2026-02-25T08:41:41.928 INFO:teuthology.orchestra.run.trial190.stdout: Installing : librbd-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 5/6 2026-02-25T08:41:41.950 INFO:teuthology.orchestra.run.trial190.stdout: Installing : perf-5.14.0-686.el9.x86_64 6/6 2026-02-25T08:41:42.063 INFO:teuthology.orchestra.run.trial190.stdout: Running scriptlet: perf-5.14.0-686.el9.x86_64 6/6 2026-02-25T08:41:42.063 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : libradospp-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/6 2026-02-25T08:41:42.063 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : librbd-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 2/6 2026-02-25T08:41:42.063 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : libtraceevent-1.8.4-2.el9.x86_64 3/6 2026-02-25T08:41:42.063 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : perf-5.14.0-686.el9.x86_64 4/6 2026-02-25T08:41:42.063 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : pdsh-2.34-7.el9.x86_64 5/6 2026-02-25T08:41:42.127 INFO:teuthology.orchestra.run.trial190.stdout: Verifying : pdsh-rcmd-ssh-2.34-7.el9.x86_64 6/6 2026-02-25T08:41:42.127 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:41:42.127 INFO:teuthology.orchestra.run.trial190.stdout:Installed: 2026-02-25T08:41:42.128 INFO:teuthology.orchestra.run.trial190.stdout: libradospp-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:41:42.128 INFO:teuthology.orchestra.run.trial190.stdout: librbd-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:41:42.128 INFO:teuthology.orchestra.run.trial190.stdout: libtraceevent-1.8.4-2.el9.x86_64 2026-02-25T08:41:42.128 INFO:teuthology.orchestra.run.trial190.stdout: pdsh-2.34-7.el9.x86_64 2026-02-25T08:41:42.128 INFO:teuthology.orchestra.run.trial190.stdout: pdsh-rcmd-ssh-2.34-7.el9.x86_64 2026-02-25T08:41:42.128 INFO:teuthology.orchestra.run.trial190.stdout: perf-5.14.0-686.el9.x86_64 2026-02-25T08:41:42.128 INFO:teuthology.orchestra.run.trial190.stdout: 2026-02-25T08:41:42.128 INFO:teuthology.orchestra.run.trial190.stdout:Complete! 2026-02-25T08:41:42.159 DEBUG:teuthology.orchestra.run.trial190:> python3 -m pip install -r /home/ubuntu/cephtest/cbt/requirements.txt 2026-02-25T08:41:42.299 INFO:teuthology.orchestra.run.trial190.stdout:Defaulting to user installation because normal site-packages is not writeable 2026-02-25T08:41:42.310 INFO:teuthology.orchestra.run.trial190.stdout:Requirement already satisfied: pyyaml in /usr/lib64/python3.9/site-packages (from -r /home/ubuntu/cephtest/cbt/requirements.txt (line 1)) (5.4.1) 2026-02-25T08:41:42.310 INFO:teuthology.orchestra.run.trial190.stdout:Requirement already satisfied: lxml in /usr/lib64/python3.9/site-packages (from -r /home/ubuntu/cephtest/cbt/requirements.txt (line 2)) (4.6.5) 2026-02-25T08:41:42.639 INFO:teuthology.orchestra.run.trial190.stdout:Collecting matplotlib 2026-02-25T08:41:42.658 INFO:teuthology.orchestra.run.trial190.stdout: Downloading matplotlib-3.9.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (8.3 MB) 2026-02-25T08:41:42.866 INFO:teuthology.orchestra.run.trial190.stdout:Collecting contourpy>=1.0.1 2026-02-25T08:41:42.870 INFO:teuthology.orchestra.run.trial190.stdout: Downloading contourpy-1.3.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (321 kB) 2026-02-25T08:41:42.906 INFO:teuthology.orchestra.run.trial190.stdout:Collecting importlib-resources>=3.2.0 2026-02-25T08:41:42.910 INFO:teuthology.orchestra.run.trial190.stdout: Downloading importlib_resources-6.5.2-py3-none-any.whl (37 kB) 2026-02-25T08:41:42.911 INFO:teuthology.orchestra.run.trial190.stdout:Requirement already satisfied: packaging>=20.0 in /usr/lib/python3.9/site-packages (from matplotlib->-r /home/ubuntu/cephtest/cbt/requirements.txt (line 3)) (20.9) 2026-02-25T08:41:42.921 INFO:teuthology.orchestra.run.trial190.stdout:Collecting cycler>=0.10 2026-02-25T08:41:42.925 INFO:teuthology.orchestra.run.trial190.stdout: Downloading cycler-0.12.1-py3-none-any.whl (8.3 kB) 2026-02-25T08:41:42.926 INFO:teuthology.orchestra.run.trial190.stdout:Requirement already satisfied: python-dateutil>=2.7 in /usr/lib/python3.9/site-packages (from matplotlib->-r /home/ubuntu/cephtest/cbt/requirements.txt (line 3)) (2.9.0.post0) 2026-02-25T08:41:43.056 INFO:teuthology.orchestra.run.trial190.stdout:Collecting kiwisolver>=1.3.1 2026-02-25T08:41:43.063 INFO:teuthology.orchestra.run.trial190.stdout: Downloading kiwisolver-1.4.7-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.6 MB) 2026-02-25T08:41:43.072 INFO:teuthology.orchestra.run.trial190.stdout:Requirement already satisfied: pyparsing>=2.3.1 in /usr/lib/python3.9/site-packages (from matplotlib->-r /home/ubuntu/cephtest/cbt/requirements.txt (line 3)) (2.4.7) 2026-02-25T08:41:43.464 INFO:teuthology.orchestra.run.trial190.stdout:Collecting pillow>=8 2026-02-25T08:41:43.470 INFO:teuthology.orchestra.run.trial190.stdout: Downloading pillow-11.3.0-cp39-cp39-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (6.6 MB) 2026-02-25T08:41:43.848 INFO:teuthology.orchestra.run.trial190.stdout:Collecting fonttools>=4.22.0 2026-02-25T08:41:43.855 INFO:teuthology.orchestra.run.trial190.stdout: Downloading fonttools-4.60.2-cp39-cp39-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (4.8 MB) 2026-02-25T08:41:43.882 INFO:teuthology.orchestra.run.trial190.stdout:Requirement already satisfied: numpy>=1.23 in /usr/lib64/python3.9/site-packages (from matplotlib->-r /home/ubuntu/cephtest/cbt/requirements.txt (line 3)) (1.23.5) 2026-02-25T08:41:43.932 INFO:teuthology.orchestra.run.trial190.stdout:Collecting zipp>=3.1.0 2026-02-25T08:41:43.936 INFO:teuthology.orchestra.run.trial190.stdout: Downloading zipp-3.23.0-py3-none-any.whl (10 kB) 2026-02-25T08:41:43.947 INFO:teuthology.orchestra.run.trial190.stdout:Requirement already satisfied: six>=1.5 in /usr/lib/python3.9/site-packages (from python-dateutil>=2.7->matplotlib->-r /home/ubuntu/cephtest/cbt/requirements.txt (line 3)) (1.15.0) 2026-02-25T08:41:44.076 INFO:teuthology.orchestra.run.trial190.stdout:Installing collected packages: zipp, pillow, kiwisolver, importlib-resources, fonttools, cycler, contourpy, matplotlib 2026-02-25T08:41:45.120 INFO:teuthology.orchestra.run.trial190.stdout:Successfully installed contourpy-1.3.0 cycler-0.12.1 fonttools-4.60.2 importlib-resources-6.5.2 kiwisolver-1.4.7 matplotlib-3.9.4 pillow-11.3.0 zipp-3.23.0 2026-02-25T08:41:45.235 INFO:tasks.cbt:benchmark: radosbench 2026-02-25T08:41:45.235 DEBUG:teuthology.orchestra.run.trial190:> sudo /sbin/sysctl -q -w kernel.perf_event_paranoid=0 2026-02-25T08:41:45.260 DEBUG:teuthology.orchestra.run.trial190:> /home/ubuntu/cephtest/cbt/cbt.py -a /home/ubuntu/cephtest/archive/cbt /home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2026-02-25T08:41:45.372 INFO:teuthology.orchestra.run.trial190.stderr:08:41:45 - DEBUG - cbt - Settings.cluster: 2026-02-25T08:41:45.372 INFO:teuthology.orchestra.run.trial190.stderr: {'archive_dir': '/home/ubuntu/cephtest/archive/cbt', 2026-02-25T08:41:45.372 INFO:teuthology.orchestra.run.trial190.stderr: 'clients': ['ubuntu@trial190.front.sepia.ceph.com'], 2026-02-25T08:41:45.372 INFO:teuthology.orchestra.run.trial190.stderr: 'conf_file': '/etc/ceph/ceph.conf', 2026-02-25T08:41:45.372 INFO:teuthology.orchestra.run.trial190.stderr: 'head': 'ubuntu@trial190.front.sepia.ceph.com', 2026-02-25T08:41:45.372 INFO:teuthology.orchestra.run.trial190.stderr: 'is_teuthology': True, 2026-02-25T08:41:45.372 INFO:teuthology.orchestra.run.trial190.stderr: 'iterations': 1, 2026-02-25T08:41:45.372 INFO:teuthology.orchestra.run.trial190.stderr: 'mons': ['ubuntu@trial190.front.sepia.ceph.com'], 2026-02-25T08:41:45.372 INFO:teuthology.orchestra.run.trial190.stderr: 'osds': ['ubuntu@trial190.front.sepia.ceph.com'], 2026-02-25T08:41:45.372 INFO:teuthology.orchestra.run.trial190.stderr: 'osds_per_node': 3, 2026-02-25T08:41:45.372 INFO:teuthology.orchestra.run.trial190.stderr: 'pid_dir': '/var/run/ceph', 2026-02-25T08:41:45.372 INFO:teuthology.orchestra.run.trial190.stderr: 'pool_profiles': {'replicated': {'pg_size': 256, 2026-02-25T08:41:45.372 INFO:teuthology.orchestra.run.trial190.stderr: 'pgp_size': 256, 2026-02-25T08:41:45.372 INFO:teuthology.orchestra.run.trial190.stderr: 'replication': 'replicated'}}, 2026-02-25T08:41:45.372 INFO:teuthology.orchestra.run.trial190.stderr: 'rebuild_every_test': False, 2026-02-25T08:41:45.372 INFO:teuthology.orchestra.run.trial190.stderr: 'rgws': {'ubuntu@trial190.front.sepia.ceph.com': None}, 2026-02-25T08:41:45.372 INFO:teuthology.orchestra.run.trial190.stderr: 'tmp_dir': '/tmp/cbt', 2026-02-25T08:41:45.372 INFO:teuthology.orchestra.run.trial190.stderr: 'use_existing': True, 2026-02-25T08:41:45.372 INFO:teuthology.orchestra.run.trial190.stderr: 'user': 'ubuntu'} 2026-02-25T08:41:45.372 INFO:teuthology.orchestra.run.trial190.stderr:08:41:45 - INFO - cbt - Results dir: /home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3 2026-02-25T08:41:45.372 INFO:teuthology.orchestra.run.trial190.stderr:08:41:45 - INFO - cbt - Pausing for 60s for idle monitoring. 2026-02-25T08:41:45.373 INFO:teuthology.orchestra.run.trial190.stderr:08:41:45 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial190.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/idle_monitoring/collectl 2026-02-25T08:41:45.549 INFO:teuthology.orchestra.run.trial190.stderr:08:41:45 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial190.front.sepia.ceph.com collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/idle_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2026-02-25T08:41:45.551 INFO:teuthology.orchestra.run.trial190.stderr:08:41:45 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial190.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/idle_monitoring/perf 2026-02-25T08:41:45.716 INFO:teuthology.orchestra.run.trial190.stderr:08:41:45 - DEBUG - cbt - PerfMonitoring: remote_node 2026-02-25T08:41:45.716 INFO:teuthology.orchestra.run.trial190.stderr:08:41:45 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial190.front.sepia.ceph.com for pid in `cat /var/run/ceph/crimson-osd.*.pid`; do perf stat -p ${pid} -o /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/idle_monitoring/perf/perf_stat.${pid} & done 2026-02-25T08:42:45.778 INFO:teuthology.orchestra.run.trial190.stderr:08:42:45 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial190.front.sepia.ceph.com pkill -SIGINT -f collectl 2026-02-25T08:42:45.951 INFO:teuthology.orchestra.run.trial190.stderr:08:42:45 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial190.front.sepia.ceph.com sudo pkill -SIGINT -f perf\ 2026-02-25T08:42:46.131 INFO:teuthology.orchestra.run.trial190.stderr:08:42:46 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial190.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/idle_monitoring/perf/perf.data 2026-02-25T08:42:46.133 INFO:teuthology.orchestra.run.trial190.stderr:08:42:46 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial190.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/idle_monitoring/perf/perf_stat.* 2026-02-25T08:42:46.135 INFO:teuthology.orchestra.run.trial190.stderr:08:42:46 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial190.front.sepia.ceph.com sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/* 2026-02-25T08:42:46.316 INFO:teuthology.orchestra.run.trial190.stderr:08:42:46 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=rpdcp -f 10 -R ssh -w ubuntu@trial190.front.sepia.ceph.com -r /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/* /home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3 2026-02-25T08:42:46.478 INFO:teuthology.orchestra.run.trial190.stderr:08:42:46 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial190.front.sepia.ceph.com sudo killall -9 rados 2026-02-25T08:42:46.651 INFO:teuthology.orchestra.run.trial190.stderr:08:42:46 - INFO - cbt - Results dir: /home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3 2026-02-25T08:42:46.651 INFO:teuthology.orchestra.run.trial190.stderr:08:42:46 - INFO - cbt - Skipping existing test in /home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3. 2026-02-25T08:42:46.651 INFO:teuthology.orchestra.run.trial190.stderr:08:42:46 - INFO - cbt - Running benchmark /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004 2026-02-25T08:42:46.651 INFO:teuthology.orchestra.run.trial190.stderr:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3 2026-02-25T08:42:46.651 INFO:teuthology.orchestra.run.trial190.stderr:{'concurrent_ops': 4, 'concurrent_procs': 2, 'pool_profile': 'replicated', 'readmode': 'rand', 'time': 60, 'write_only': False, 'op_size': 4096, 'pool_monitoring_list': 'collectl', 'run_monitoring_list': 'collectl', 'benchmark': 'radosbench', 'iteration': 0} == iteration 0 == 2026-02-25T08:42:46.651 INFO:teuthology.orchestra.run.trial190.stderr:08:42:46 - INFO - cbt - Setting OSD Read Ahead to: 0 2026-02-25T08:42:46.652 INFO:teuthology.orchestra.run.trial190.stderr:08:42:46 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial190.front.sepia.ceph.com find /dev/disk/by-partlabel/osd-device-*data -exec readlink {} \; | cut -d"/" -f 3 | sed "s/[0-9]$//" | xargs -I{} sudo sh -c "echo 0 > /sys/block/'{}'/queue/read_ahead_kb" 2026-02-25T08:42:46.652 INFO:teuthology.orchestra.run.trial190.stderr:08:42:46 - DEBUG - cbt - Cleaning existing temporary run directory: /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004 2026-02-25T08:42:46.653 INFO:teuthology.orchestra.run.trial190.stderr:08:42:46 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial190.front.sepia.ceph.com sudo rm -rf /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004 2026-02-25T08:42:46.838 INFO:teuthology.orchestra.run.trial190.stderr:08:42:46 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial190.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/pool_monitoring/collectl 2026-02-25T08:42:46.999 INFO:teuthology.orchestra.run.trial190.stderr:08:42:46 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial190.front.sepia.ceph.com collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/pool_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2026-02-25T08:42:47.001 INFO:teuthology.orchestra.run.trial190.stderr:08:42:47 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial190.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/pool_monitoring/perf 2026-02-25T08:42:47.165 INFO:teuthology.orchestra.run.trial190.stderr:08:42:47 - DEBUG - cbt - PerfMonitoring: remote_node 2026-02-25T08:42:47.165 INFO:teuthology.orchestra.run.trial190.stderr:08:42:47 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial190.front.sepia.ceph.com for pid in `cat /var/run/ceph/crimson-osd.*.pid`; do perf stat -p ${pid} -o /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/pool_monitoring/perf/perf_stat.${pid} & done 2026-02-25T08:42:47.167 INFO:teuthology.orchestra.run.trial190.stderr:08:42:47 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial190.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool delete rados-bench-cbt rados-bench-cbt --yes-i-really-really-mean-it 2026-02-25T08:42:47.505 INFO:teuthology.orchestra.run.trial190.stderr:08:42:47 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial190.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool create rados-bench-cbt 256 256 2026-02-25T08:42:48.303 INFO:teuthology.orchestra.run.trial190.stderr:08:42:48 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial190.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool application enable rados-bench-cbt radosbench 2026-02-25T08:42:49.306 INFO:teuthology.orchestra.run.trial190.stderr:08:42:49 - INFO - cbt - Checking Health after pool creation. 2026-02-25T08:42:54.311 INFO:teuthology.orchestra.run.trial190.stderr:08:42:54 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial190.front.sepia.ceph.com /usr/bin/ceph -c /etc/ceph/ceph.conf health 2026-02-25T08:42:54.673 INFO:teuthology.orchestra.run.trial190.stderr:08:42:54 - INFO - cbt - Final Pool Health Check. 2026-02-25T08:42:59.679 INFO:teuthology.orchestra.run.trial190.stderr:08:42:59 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial190.front.sepia.ceph.com /usr/bin/ceph -c /etc/ceph/ceph.conf health 2026-02-25T08:43:00.041 INFO:teuthology.orchestra.run.trial190.stderr:08:43:00 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial190.front.sepia.ceph.com pkill -SIGINT -f collectl 2026-02-25T08:43:00.207 INFO:teuthology.orchestra.run.trial190.stderr:08:43:00 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial190.front.sepia.ceph.com sudo pkill -SIGINT -f perf\ 2026-02-25T08:43:00.382 INFO:teuthology.orchestra.run.trial190.stderr:08:43:00 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial190.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/pool_monitoring/perf/perf.data 2026-02-25T08:43:00.383 INFO:teuthology.orchestra.run.trial190.stderr:08:43:00 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial190.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/pool_monitoring/perf/perf_stat.* 2026-02-25T08:43:00.385 INFO:teuthology.orchestra.run.trial190.stderr:08:43:00 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial190.front.sepia.ceph.com sync 2026-02-25T08:43:00.559 INFO:teuthology.orchestra.run.trial190.stderr:08:43:00 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial190.front.sepia.ceph.com echo 3 | sudo tee /proc/sys/vm/drop_caches 2026-02-25T08:43:01.582 INFO:teuthology.orchestra.run.trial190.stderr:08:43:01 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial190.front.sepia.ceph.com /usr/bin/rados -c /etc/ceph/ceph.conf -v 2026-02-25T08:43:01.792 INFO:teuthology.orchestra.run.trial190.stderr:08:43:01 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial190.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/write 2026-02-25T08:43:01.958 INFO:teuthology.orchestra.run.trial190.stderr:08:43:01 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial190.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf daemon osd.0 config show > /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/write/ceph_settings.out 2026-02-25T08:43:02.241 INFO:teuthology.orchestra.run.trial190.stderr:08:43:02 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial190.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/write/collectl 2026-02-25T08:43:02.403 INFO:teuthology.orchestra.run.trial190.stderr:08:43:02 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial190.front.sepia.ceph.com collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/write/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2026-02-25T08:43:02.405 INFO:teuthology.orchestra.run.trial190.stderr:08:43:02 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial190.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/write/perf 2026-02-25T08:43:02.570 INFO:teuthology.orchestra.run.trial190.stderr:08:43:02 - DEBUG - cbt - PerfMonitoring: remote_node 2026-02-25T08:43:02.570 INFO:teuthology.orchestra.run.trial190.stderr:08:43:02 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial190.front.sepia.ceph.com for pid in `cat /var/run/ceph/crimson-osd.*.pid`; do perf stat -p ${pid} -o /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/write/perf/perf_stat.${pid} & done 2026-02-25T08:43:02.572 INFO:teuthology.orchestra.run.trial190.stderr:08:43:02 - INFO - cbt - Running radosbench write test. 2026-02-25T08:43:02.572 INFO:teuthology.orchestra.run.trial190.stderr:08:43:02 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial190.front.sepia.ceph.com /usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench -b 4096 60 write --concurrent-ios 4 --run-name `hostname -f`-0 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/write/objecter.0.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/write/output.0 2026-02-25T08:43:02.574 INFO:teuthology.orchestra.run.trial190.stderr:08:43:02 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial190.front.sepia.ceph.com /usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench -b 4096 60 write --concurrent-ios 4 --run-name `hostname -f`-1 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/write/objecter.1.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/write/output.1 2026-02-25T08:44:03.786 INFO:teuthology.orchestra.run.trial190.stderr:08:44:03 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial190.front.sepia.ceph.com pkill -SIGINT -f collectl 2026-02-25T08:44:03.959 INFO:teuthology.orchestra.run.trial190.stderr:08:44:03 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial190.front.sepia.ceph.com sudo pkill -SIGINT -f perf\ 2026-02-25T08:44:04.135 INFO:teuthology.orchestra.run.trial190.stderr:08:44:04 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial190.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/write/perf/perf.data 2026-02-25T08:44:04.136 INFO:teuthology.orchestra.run.trial190.stderr:08:44:04 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial190.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/write/perf/perf_stat.* 2026-02-25T08:44:04.138 INFO:teuthology.orchestra.run.trial190.stderr:08:44:04 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial190.front.sepia.ceph.com find /var/run/ceph/ceph-osd*.asok -maxdepth 1 -exec sudo /usr/bin/ceph --admin-daemon {} dump_historic_ops \; > /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/write/historic_ops.out 2026-02-25T08:44:04.499 INFO:teuthology.orchestra.run.trial190.stderr:08:44:04 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial190.front.sepia.ceph.com sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/write/* 2026-02-25T08:44:04.666 INFO:teuthology.orchestra.run.trial190.stderr:08:44:04 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=rpdcp -f 10 -R ssh -w ubuntu@trial190.front.sepia.ceph.com -r /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/write/* /home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/write 2026-02-25T08:44:04.832 INFO:teuthology.orchestra.run.trial190.stderr:08:44:04 - INFO - cbt - Convert results to json format. 2026-02-25T08:44:04.833 INFO:teuthology.orchestra.run.trial190.stderr:08:44:04 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial190.front.sepia.ceph.com sync 2026-02-25T08:44:05.110 INFO:teuthology.orchestra.run.trial190.stderr:08:44:05 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial190.front.sepia.ceph.com echo 3 | sudo tee /proc/sys/vm/drop_caches 2026-02-25T08:44:05.578 INFO:teuthology.orchestra.run.trial190.stderr:08:44:05 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial190.front.sepia.ceph.com /usr/bin/rados -c /etc/ceph/ceph.conf -v 2026-02-25T08:44:05.797 INFO:teuthology.orchestra.run.trial190.stderr:08:44:05 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial190.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/rand 2026-02-25T08:44:05.962 INFO:teuthology.orchestra.run.trial190.stderr:08:44:05 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial190.front.sepia.ceph.com sudo /usr/bin/ceph -c /etc/ceph/ceph.conf daemon osd.0 config show > /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/rand/ceph_settings.out 2026-02-25T08:44:06.243 INFO:teuthology.orchestra.run.trial190.stderr:08:44:06 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial190.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/rand/collectl 2026-02-25T08:44:06.401 INFO:teuthology.orchestra.run.trial190.stderr:08:44:06 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial190.front.sepia.ceph.com collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/rand/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2026-02-25T08:44:06.403 INFO:teuthology.orchestra.run.trial190.stderr:08:44:06 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial190.front.sepia.ceph.com mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/rand/perf 2026-02-25T08:44:06.568 INFO:teuthology.orchestra.run.trial190.stderr:08:44:06 - DEBUG - cbt - PerfMonitoring: remote_node 2026-02-25T08:44:06.568 INFO:teuthology.orchestra.run.trial190.stderr:08:44:06 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial190.front.sepia.ceph.com for pid in `cat /var/run/ceph/crimson-osd.*.pid`; do perf stat -p ${pid} -o /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/rand/perf/perf_stat.${pid} & done 2026-02-25T08:44:06.570 INFO:teuthology.orchestra.run.trial190.stderr:08:44:06 - INFO - cbt - Running radosbench rand test. 2026-02-25T08:44:06.570 INFO:teuthology.orchestra.run.trial190.stderr:08:44:06 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial190.front.sepia.ceph.com /usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench 60 rand --concurrent-ios 4 --run-name `hostname -f`-0 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/rand/objecter.0.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/rand/output.0 2026-02-25T08:44:06.571 INFO:teuthology.orchestra.run.trial190.stderr:08:44:06 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial190.front.sepia.ceph.com /usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench 60 rand --concurrent-ios 4 --run-name `hostname -f`-1 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/rand/objecter.1.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/rand/output.1 2026-02-25T08:45:07.781 INFO:teuthology.orchestra.run.trial190.stderr:08:45:07 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial190.front.sepia.ceph.com pkill -SIGINT -f collectl 2026-02-25T08:45:07.955 INFO:teuthology.orchestra.run.trial190.stderr:08:45:07 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial190.front.sepia.ceph.com sudo pkill -SIGINT -f perf\ 2026-02-25T08:45:08.135 INFO:teuthology.orchestra.run.trial190.stderr:08:45:08 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial190.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/rand/perf/perf.data 2026-02-25T08:45:08.136 INFO:teuthology.orchestra.run.trial190.stderr:08:45:08 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial190.front.sepia.ceph.com sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/rand/perf/perf_stat.* 2026-02-25T08:45:08.138 INFO:teuthology.orchestra.run.trial190.stderr:08:45:08 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=False args=pdsh -f 1 -R ssh -w ubuntu@trial190.front.sepia.ceph.com find /var/run/ceph/ceph-osd*.asok -maxdepth 1 -exec sudo /usr/bin/ceph --admin-daemon {} dump_historic_ops \; > /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/rand/historic_ops.out 2026-02-25T08:45:08.492 INFO:teuthology.orchestra.run.trial190.stderr:08:45:08 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@trial190.front.sepia.ceph.com sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/rand/* 2026-02-25T08:45:08.665 INFO:teuthology.orchestra.run.trial190.stderr:08:45:08 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=rpdcp -f 10 -R ssh -w ubuntu@trial190.front.sepia.ceph.com -r /tmp/cbt/00000000/Radosbench/osd_ra-00000000/op_size-00004096/concurrent_ops-00000004/rand/* /home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/rand 2026-02-25T08:45:08.828 INFO:teuthology.orchestra.run.trial190.stderr:08:45:08 - INFO - cbt - Convert results to json format. 2026-02-25T08:45:08.839 DEBUG:teuthology.run_tasks:Unwinding manager cbt 2026-02-25T08:45:08.860 DEBUG:teuthology.orchestra.run.trial190:> rm --one-file-system -rf -- /home/ubuntu/cephtest/cbt 2026-02-25T08:45:08.886 INFO:tasks.cbt_performance:Collecting CBT performance data config 2026-02-25T08:45:08.886 INFO:tasks.cbt_performance:reading cbt preferences 2026-02-25T08:45:08.887 DEBUG:teuthology.orchestra.run.trial190:> find /home/ubuntu/cephtest -name 'json_output.*' 2026-02-25T08:45:08.944 INFO:teuthology.orchestra.run.trial190.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/write/json_output.0.trial190.front.sepia.ceph.com 2026-02-25T08:45:08.944 INFO:teuthology.orchestra.run.trial190.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/write/json_output.1.trial190.front.sepia.ceph.com 2026-02-25T08:45:08.944 INFO:teuthology.orchestra.run.trial190.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/rand/json_output.0.trial190.front.sepia.ceph.com 2026-02-25T08:45:08.945 INFO:teuthology.orchestra.run.trial190.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/rand/json_output.1.trial190.front.sepia.ceph.com 2026-02-25T08:45:08.945 DEBUG:teuthology.orchestra.run.trial190:> set -ex 2026-02-25T08:45:08.946 DEBUG:teuthology.orchestra.run.trial190:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/write/json_output.0.trial190.front.sepia.ceph.com of=/dev/stdout 2026-02-25T08:45:09.002 INFO:tasks.cbt_performance:reading total cpu cycles 2026-02-25T08:45:09.002 DEBUG:teuthology.orchestra.run.trial190:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/write -name 'perf_stat.*' 2026-02-25T08:45:09.057 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2026-02-25T08:45:09.058 INFO:tasks.cbt_performance:total cpu cycles: 0 2026-02-25T08:45:09.058 DEBUG:teuthology.orchestra.run.trial190:> set -ex 2026-02-25T08:45:09.058 DEBUG:teuthology.orchestra.run.trial190:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/write/json_output.1.trial190.front.sepia.ceph.com of=/dev/stdout 2026-02-25T08:45:09.115 INFO:tasks.cbt_performance:reading total cpu cycles 2026-02-25T08:45:09.115 DEBUG:teuthology.orchestra.run.trial190:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/write -name 'perf_stat.*' 2026-02-25T08:45:09.171 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2026-02-25T08:45:09.171 INFO:tasks.cbt_performance:total cpu cycles: 0 2026-02-25T08:45:09.172 DEBUG:teuthology.orchestra.run.trial190:> set -ex 2026-02-25T08:45:09.172 DEBUG:teuthology.orchestra.run.trial190:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/rand/json_output.0.trial190.front.sepia.ceph.com of=/dev/stdout 2026-02-25T08:45:09.227 INFO:tasks.cbt_performance:reading total cpu cycles 2026-02-25T08:45:09.228 DEBUG:teuthology.orchestra.run.trial190:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/rand -name 'perf_stat.*' 2026-02-25T08:45:09.282 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2026-02-25T08:45:09.283 INFO:tasks.cbt_performance:total cpu cycles: 0 2026-02-25T08:45:09.283 DEBUG:teuthology.orchestra.run.trial190:> set -ex 2026-02-25T08:45:09.283 DEBUG:teuthology.orchestra.run.trial190:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/rand/json_output.1.trial190.front.sepia.ceph.com of=/dev/stdout 2026-02-25T08:45:09.339 INFO:tasks.cbt_performance:reading total cpu cycles 2026-02-25T08:45:09.339 DEBUG:teuthology.orchestra.run.trial190:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/rand -name 'perf_stat.*' 2026-02-25T08:45:09.395 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2026-02-25T08:45:09.395 INFO:tasks.cbt_performance:total cpu cycles: 0 2026-02-25T08:45:09.400 WARNING:tasks.cbt_performance:Could not send performance data to http://mira118.front.sepia.ceph.com:4000/cbt_performance: HTTPConnectionPool(host='mira118.front.sepia.ceph.com', port=4000): Max retries exceeded with url: /cbt_performance (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -2] Name or service not known')) 2026-02-25T08:45:09.400 WARNING:tasks.cbt_performance:Continuing without performance data collection 2026-02-25T08:45:09.400 DEBUG:teuthology.run_tasks:Unwinding manager ssh_keys 2026-02-25T08:45:09.415 INFO:teuthology.task.ssh_keys:Cleaning up SSH keys 2026-02-25T08:45:09.415 INFO:teuthology.task.ssh_keys:cleaning up keys added for testing 2026-02-25T08:45:09.415 INFO:teuthology.task.ssh_keys: cleaning up keys for user ubuntu on trial190.front.sepia.ceph.com 2026-02-25T08:45:09.416 DEBUG:teuthology.orchestra.run.trial190:> rm -- /home/ubuntu/.ssh/id_rsa 2026-02-25T08:45:09.451 DEBUG:teuthology.orchestra.run.trial190:> rm -- /home/ubuntu/.ssh/id_rsa.pub 2026-02-25T08:45:09.507 DEBUG:teuthology.orchestra.run.trial190:> stat -c '"%a"' /home/ubuntu/.ssh/authorized_keys 2026-02-25T08:45:09.562 INFO:teuthology.orchestra.run.trial190.stdout:"600" 2026-02-25T08:45:09.563 DEBUG:teuthology.orchestra.run.trial190:> mv -- /home/ubuntu/.ssh/authorized_keys_2026-02-25_08:41:33:604511 /home/ubuntu/.ssh/authorized_keys 2026-02-25T08:45:09.620 DEBUG:teuthology.orchestra.run.trial190:> chmod 600 /home/ubuntu/.ssh/authorized_keys 2026-02-25T08:45:09.675 DEBUG:teuthology.orchestra.run.trial190:> rm /home/ubuntu/.ssh/config 2026-02-25T08:45:09.731 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2026-02-25T08:45:09.742 DEBUG:teuthology.orchestra.run.trial190:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2026-02-25T08:45:09.845 INFO:teuthology.orchestra.run.trial190.stderr:2026-02-25T08:45:09.840+0000 7f51d625a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:09.846 INFO:teuthology.orchestra.run.trial190.stderr:2026-02-25T08:45:09.844+0000 7f51d625a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:09.973 INFO:teuthology.misc:Shutting down mds daemons... 2026-02-25T08:45:09.973 INFO:teuthology.misc:Shutting down osd daemons... 2026-02-25T08:45:09.973 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2026-02-25T08:45:09.974 INFO:teuthology.orchestra.run:waiting for 300 2026-02-25T08:45:10.063 INFO:tasks.ceph.osd.0:Stopped 2026-02-25T08:45:10.064 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2026-02-25T08:45:10.064 INFO:teuthology.orchestra.run:waiting for 300 2026-02-25T08:45:10.156 INFO:tasks.ceph.osd.1:Stopped 2026-02-25T08:45:10.156 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2026-02-25T08:45:10.157 INFO:teuthology.orchestra.run:waiting for 300 2026-02-25T08:45:10.506 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2026-02-25T08:45:15.707 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~5s 2026-02-25T08:45:16.257 INFO:tasks.ceph.osd.2:Stopped 2026-02-25T08:45:16.258 INFO:teuthology.misc:Shutting down mgr daemons... 2026-02-25T08:45:16.258 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2026-02-25T08:45:16.258 INFO:teuthology.orchestra.run:waiting for 300 2026-02-25T08:45:16.298 INFO:tasks.ceph.mgr.x:Stopped 2026-02-25T08:45:16.298 INFO:teuthology.misc:Shutting down mon daemons... 2026-02-25T08:45:16.298 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2026-02-25T08:45:16.298 INFO:teuthology.orchestra.run:waiting for 300 2026-02-25T08:45:16.347 INFO:tasks.ceph.mon.a:Stopped 2026-02-25T08:45:16.347 INFO:tasks.ceph:Checking cluster log for badness... 2026-02-25T08:45:16.347 DEBUG:teuthology.orchestra.run.trial190:> sudo grep -E '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | grep -E -v '\(MDS_ALL_DOWN\)' | grep -E -v '\(MDS_UP_LESS_THAN_MAX\)' | grep -E -v '\(POOL_APP_NOT_ENABLED\)' | grep -E -v '\(PG_' | grep -E -v '\(OSD_' | grep -E -v '\(OBJECT_' | grep -E -v 'overall HEALTH' | head -n 1 2026-02-25T08:45:16.413 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@trial190.front.sepia.ceph.com 2026-02-25T08:45:16.413 DEBUG:teuthology.orchestra.run.trial190:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2026-02-25T08:45:16.489 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@trial190.front.sepia.ceph.com 2026-02-25T08:45:16.489 DEBUG:teuthology.orchestra.run.trial190:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2026-02-25T08:45:16.558 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@trial190.front.sepia.ceph.com 2026-02-25T08:45:16.559 DEBUG:teuthology.orchestra.run.trial190:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2026-02-25T08:45:16.633 INFO:tasks.ceph:Archiving mon data... 2026-02-25T08:45:16.634 DEBUG:teuthology.misc:Transferring archived files from trial190:/var/lib/ceph/mon/ceph-a to /home/teuthworker/mnt/teuthology/matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial/69096/data/mon.a.tgz 2026-02-25T08:45:16.635 DEBUG:teuthology.orchestra.run.trial190:> mktemp 2026-02-25T08:45:16.689 INFO:teuthology.orchestra.run.trial190.stdout:/tmp/tmp.c24ysABBAy 2026-02-25T08:45:16.689 DEBUG:teuthology.orchestra.run.trial190:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.c24ysABBAy 2026-02-25T08:45:16.780 DEBUG:teuthology.orchestra.run.trial190:> sudo chmod 0666 /tmp/tmp.c24ysABBAy 2026-02-25T08:45:16.859 DEBUG:teuthology.orchestra.remote:trial190:/tmp/tmp.c24ysABBAy is 208KB 2026-02-25T08:45:16.919 DEBUG:teuthology.orchestra.run.trial190:> rm -fr /tmp/tmp.c24ysABBAy 2026-02-25T08:45:16.934 INFO:tasks.ceph:Cleaning ceph cluster... 2026-02-25T08:45:16.934 DEBUG:teuthology.orchestra.run.trial190:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2026-02-25T08:45:17.054 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2026-02-25T08:45:17.054 INFO:tasks.ceph:Archiving crash dumps... 2026-02-25T08:45:17.056 DEBUG:teuthology.misc:Transferring archived files from trial190:/var/lib/ceph/crash to /home/teuthworker/mnt/teuthology/matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial/69096/remote/trial190/crash 2026-02-25T08:45:17.058 DEBUG:teuthology.orchestra.run.trial190:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2026-02-25T08:45:17.117 INFO:tasks.ceph:Compressing logs... 2026-02-25T08:45:17.117 DEBUG:teuthology.orchestra.run.trial190:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2026-02-25T08:45:17.181 INFO:teuthology.orchestra.run.trial190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2026-02-25T08:45:17.194 INFO:teuthology.orchestra.run.trial190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41728.log 2026-02-25T08:45:17.194 INFO:teuthology.orchestra.run.trial190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2026-02-25T08:45:17.194 INFO:teuthology.orchestra.run.trial190.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2026-02-25T08:45:17.194 INFO:teuthology.orchestra.run.trial190.stderr:/var/log/ceph/ceph-client.admin.41728.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41728.log.gz 2026-02-25T08:45:17.194 INFO:teuthology.orchestra.run.trial190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40706.log 2026-02-25T08:45:17.194 INFO:teuthology.orchestra.run.trial190.stderr:/var/log/ceph/ceph-mon.a.log: /var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41581.log 2026-02-25T08:45:17.194 INFO:teuthology.orchestra.run.trial190.stderr:/var/log/ceph/ceph-client.admin.40706.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42017.log 2026-02-25T08:45:17.194 INFO:teuthology.orchestra.run.trial190.stderr:/var/log/ceph/ceph-client.admin.41581.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40706.log.gzgzip 2026-02-25T08:45:17.194 INFO:teuthology.orchestra.run.trial190.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.45418.log 2026-02-25T08:45:17.195 INFO:teuthology.orchestra.run.trial190.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41581.log.gz/var/log/ceph/ceph-client.admin.42017.log: 2026-02-25T08:45:17.195 INFO:teuthology.orchestra.run.trial190.stderr: 91.2% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2026-02-25T08:45:17.195 INFO:teuthology.orchestra.run.trial190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41102.log 2026-02-25T08:45:17.195 INFO:teuthology.orchestra.run.trial190.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42017.log.gz 2026-02-25T08:45:17.195 INFO:teuthology.orchestra.run.trial190.stderr:/var/log/ceph/ceph-client.admin.45418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41968.log 2026-02-25T08:45:17.195 INFO:teuthology.orchestra.run.trial190.stderr: 66.7% -- replaced with /var/log/ceph/ceph-client.admin.45418.log.gz 2026-02-25T08:45:17.195 INFO:teuthology.orchestra.run.trial190.stderr:/var/log/ceph/ceph-client.admin.41102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46106.log 2026-02-25T08:45:17.195 INFO:teuthology.orchestra.run.trial190.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41102.log.gz 2026-02-25T08:45:17.195 INFO:teuthology.orchestra.run.trial190.stderr:/var/log/ceph/ceph-client.admin.41968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44703.log 2026-02-25T08:45:17.195 INFO:teuthology.orchestra.run.trial190.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41968.log.gz 2026-02-25T08:45:17.195 INFO:teuthology.orchestra.run.trial190.stderr:/var/log/ceph/ceph-client.admin.46106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44647.log 2026-02-25T08:45:17.195 INFO:teuthology.orchestra.run.trial190.stderr: 66.4% -- replaced with /var/log/ceph/ceph-client.admin.46106.log.gz 2026-02-25T08:45:17.196 INFO:teuthology.orchestra.run.trial190.stderr:/var/log/ceph/ceph-client.admin.44703.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41580.log 2026-02-25T08:45:17.196 INFO:teuthology.orchestra.run.trial190.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44703.log.gz 2026-02-25T08:45:17.196 INFO:teuthology.orchestra.run.trial190.stderr:/var/log/ceph/ceph-client.admin.44647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40955.log 2026-02-25T08:45:17.196 INFO:teuthology.orchestra.run.trial190.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.44647.log.gz 2026-02-25T08:45:17.196 INFO:teuthology.orchestra.run.trial190.stderr:/var/log/ceph/ceph-client.admin.41580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44807.log 2026-02-25T08:45:17.196 INFO:teuthology.orchestra.run.trial190.stderr:/var/log/ceph/ceph-client.admin.40955.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41580.log.gzgzip 2026-02-25T08:45:17.196 INFO:teuthology.orchestra.run.trial190.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40659.log 2026-02-25T08:45:17.196 INFO:teuthology.orchestra.run.trial190.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40955.log.gz 2026-02-25T08:45:17.196 INFO:teuthology.orchestra.run.trial190.stderr:/var/log/ceph/ceph-client.admin.44807.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46105.log 2026-02-25T08:45:17.196 INFO:teuthology.orchestra.run.trial190.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.44807.log.gz 2026-02-25T08:45:17.196 INFO:teuthology.orchestra.run.trial190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40475.log 2026-02-25T08:45:17.196 INFO:teuthology.orchestra.run.trial190.stderr:/var/log/ceph/ceph-client.admin.40659.log: /var/log/ceph/ceph-client.admin.46105.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44861.log 2026-02-25T08:45:17.196 INFO:teuthology.orchestra.run.trial190.stderr: 50.0% 66.4% -- replaced with /var/log/ceph/ceph-client.admin.46105.log.gz 2026-02-25T08:45:17.196 INFO:teuthology.orchestra.run.trial190.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40659.log.gz 2026-02-25T08:45:17.197 INFO:teuthology.orchestra.run.trial190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40925.log 2026-02-25T08:45:17.197 INFO:teuthology.orchestra.run.trial190.stderr:/var/log/ceph/ceph-client.admin.40475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41727.log 2026-02-25T08:45:17.197 INFO:teuthology.orchestra.run.trial190.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40475.log.gz 2026-02-25T08:45:17.197 INFO:teuthology.orchestra.run.trial190.stderr:/var/log/ceph/ceph-client.admin.44861.log: /var/log/ceph/ceph-client.admin.40925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41848.log 2026-02-25T08:45:17.197 INFO:teuthology.orchestra.run.trial190.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.44861.log.gz 2026-02-25T08:45:17.197 INFO:teuthology.orchestra.run.trial190.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40925.log.gz 2026-02-25T08:45:17.197 INFO:teuthology.orchestra.run.trial190.stderr:/var/log/ceph/ceph-client.admin.41727.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41579.log 2026-02-25T08:45:17.197 INFO:teuthology.orchestra.run.trial190.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41727.log.gz 2026-02-25T08:45:17.197 INFO:teuthology.orchestra.run.trial190.stderr:/var/log/ceph/ceph-client.admin.41848.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.39041.log 2026-02-25T08:45:17.197 INFO:teuthology.orchestra.run.trial190.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41848.log.gz 2026-02-25T08:45:17.197 INFO:teuthology.orchestra.run.trial190.stderr:/var/log/ceph/ceph-client.admin.41579.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40333.log 2026-02-25T08:45:17.197 INFO:teuthology.orchestra.run.trial190.stderr:/var/log/ceph/ceph.tmp-client.admin.39041.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41579.log.gz 2026-02-25T08:45:17.197 INFO:teuthology.orchestra.run.trial190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2026-02-25T08:45:17.198 INFO:teuthology.orchestra.run.trial190.stderr: 67.7% -- replaced with /var/log/ceph/ceph.tmp-client.admin.39041.log.gz 2026-02-25T08:45:17.198 INFO:teuthology.orchestra.run.trial190.stderr:/var/log/ceph/ceph-client.admin.40333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41331.log 2026-02-25T08:45:17.198 INFO:teuthology.orchestra.run.trial190.stderr:/var/log/ceph/ceph.audit.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40333.log.gz 2026-02-25T08:45:17.198 INFO:teuthology.orchestra.run.trial190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41478.log 2026-02-25T08:45:17.198 INFO:teuthology.orchestra.run.trial190.stderr:/var/log/ceph/ceph-client.admin.41331.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41919.log 2026-02-25T08:45:17.198 INFO:teuthology.orchestra.run.trial190.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41331.log.gz 2026-02-25T08:45:17.198 INFO:teuthology.orchestra.run.trial190.stderr: 87.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2026-02-25T08:45:17.198 INFO:teuthology.orchestra.run.trial190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2026-02-25T08:45:17.198 INFO:teuthology.orchestra.run.trial190.stderr:/var/log/ceph/ceph-client.admin.41478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46783.log 2026-02-25T08:45:17.198 INFO:teuthology.orchestra.run.trial190.stderr:/var/log/ceph/ceph-client.admin.41919.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41478.log.gz 2026-02-25T08:45:17.198 INFO:teuthology.orchestra.run.trial190.stderr:/var/log/ceph/ceph-osd.1.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41919.log.gz 2026-02-25T08:45:17.198 INFO:teuthology.orchestra.run.trial190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41171.log 2026-02-25T08:45:17.198 INFO:teuthology.orchestra.run.trial190.stderr:/var/log/ceph/ceph-client.admin.46783.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41266.log 2026-02-25T08:45:17.199 INFO:teuthology.orchestra.run.trial190.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46783.log.gz 2026-02-25T08:45:17.199 INFO:teuthology.orchestra.run.trial190.stderr:/var/log/ceph/ceph-client.admin.41171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41380.log 2026-02-25T08:45:17.199 INFO:teuthology.orchestra.run.trial190.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41171.log.gz 2026-02-25T08:45:17.199 INFO:teuthology.orchestra.run.trial190.stderr:/var/log/ceph/ceph-client.admin.41266.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45416.log 2026-02-25T08:45:17.199 INFO:teuthology.orchestra.run.trial190.stderr: 66.7% -- replaced with /var/log/ceph/ceph-client.admin.41266.log.gz 2026-02-25T08:45:17.199 INFO:teuthology.orchestra.run.trial190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2026-02-25T08:45:17.199 INFO:teuthology.orchestra.run.trial190.stderr:/var/log/ceph/ceph-client.admin.41380.log: /var/log/ceph/ceph-client.admin.45416.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41380.log.gz 2026-02-25T08:45:17.199 INFO:teuthology.orchestra.run.trial190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41429.log 2026-02-25T08:45:17.199 INFO:teuthology.orchestra.run.trial190.stderr: 66.4% -- replaced with /var/log/ceph/ceph-client.admin.45416.log.gz 2026-02-25T08:45:17.199 INFO:teuthology.orchestra.run.trial190.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41821.log 2026-02-25T08:45:17.199 INFO:teuthology.orchestra.run.trial190.stderr:/var/log/ceph/ceph-client.admin.41429.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2026-02-25T08:45:17.199 INFO:teuthology.orchestra.run.trial190.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41429.log.gz 2026-02-25T08:45:17.199 INFO:teuthology.orchestra.run.trial190.stderr:/var/log/ceph/ceph-client.admin.41821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41726.log 2026-02-25T08:45:17.200 INFO:teuthology.orchestra.run.trial190.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41821.log.gz 2026-02-25T08:45:17.200 INFO:teuthology.orchestra.run.trial190.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40612.log 2026-02-25T08:45:17.200 INFO:teuthology.orchestra.run.trial190.stderr:/var/log/ceph/ceph-client.admin.41726.log: 86.1% -- replaced with /var/log/ceph/ceph.log.gz 2026-02-25T08:45:17.200 INFO:teuthology.orchestra.run.trial190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41241.log 2026-02-25T08:45:17.200 INFO:teuthology.orchestra.run.trial190.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41726.log.gz 2026-02-25T08:45:17.200 INFO:teuthology.orchestra.run.trial190.stderr:/var/log/ceph/ceph-client.admin.40612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44762.log 2026-02-25T08:45:17.200 INFO:teuthology.orchestra.run.trial190.stderr:/var/log/ceph/ceph-client.admin.41241.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40428.log 2026-02-25T08:45:17.200 INFO:teuthology.orchestra.run.trial190.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40612.log.gz 2026-02-25T08:45:17.200 INFO:teuthology.orchestra.run.trial190.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41241.log.gz 2026-02-25T08:45:17.200 INFO:teuthology.orchestra.run.trial190.stderr:/var/log/ceph/ceph-client.admin.44762.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.44762.log.gz 2026-02-25T08:45:17.200 INFO:teuthology.orchestra.run.trial190.stderr:/var/log/ceph/ceph-client.admin.40428.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40428.log.gz 2026-02-25T08:45:17.201 INFO:teuthology.orchestra.run.trial190.stderr: 92.1% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2026-02-25T08:45:20.709 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2026-02-25T08:45:22.471 INFO:teuthology.orchestra.run.trial190.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2026-02-25T08:45:22.716 INFO:teuthology.orchestra.run.trial190.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2026-02-25T08:45:23.162 INFO:teuthology.orchestra.run.trial190.stderr: 92.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2026-02-25T08:45:23.164 INFO:teuthology.orchestra.run.trial190.stderr: 2026-02-25T08:45:23.164 INFO:teuthology.orchestra.run.trial190.stderr:real 0m5.991s 2026-02-25T08:45:23.164 INFO:teuthology.orchestra.run.trial190.stderr:user 0m15.715s 2026-02-25T08:45:23.165 INFO:teuthology.orchestra.run.trial190.stderr:sys 0m1.076s 2026-02-25T08:45:23.165 INFO:tasks.ceph:Archiving logs... 2026-02-25T08:45:23.165 DEBUG:teuthology.misc:Transferring archived files from trial190:/var/log/ceph to /home/teuthworker/mnt/teuthology/matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial/69096/remote/trial190/log 2026-02-25T08:45:23.167 DEBUG:teuthology.orchestra.run.trial190:> sudo tar c -f - -C /var/log/ceph -- . 2026-02-25T08:45:24.570 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-25T08:45:24.639 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-25T08:45:24.650 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2026-02-25T08:45:24.650 DEBUG:teuthology.orchestra.run.trial190:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-25T08:45:24.676 DEBUG:teuthology.run_tasks:Unwinding manager clock 2026-02-25T08:45:24.688 INFO:teuthology.task.clock:Checking final clock skew... 2026-02-25T08:45:24.688 DEBUG:teuthology.orchestra.run.trial190:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-25T08:45:24.731 INFO:teuthology.orchestra.run.trial190.stderr:bash: line 1: ntpq: command not found 2026-02-25T08:45:24.755 INFO:teuthology.orchestra.run.trial190.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-25T08:45:24.755 INFO:teuthology.orchestra.run.trial190.stdout:=============================================================================== 2026-02-25T08:45:24.756 INFO:teuthology.orchestra.run.trial190.stdout:^* nyc2.us.ntp.li 2 8 377 73 -207us[ -251us] +/- 2466us 2026-02-25T08:45:24.756 INFO:teuthology.orchestra.run.trial190.stdout:^- dns-e.ns4v.icu 2 6 377 6 -19us[ -19us] +/- 55ms 2026-02-25T08:45:24.756 INFO:teuthology.orchestra.run.trial190.stdout:^- sushi.ruselabs.com 2 7 377 74 -1078us[-1122us] +/- 35ms 2026-02-25T08:45:24.756 INFO:teuthology.orchestra.run.trial190.stdout:^- 50.218.103.254 2 8 377 78 +1554us[+1510us] +/- 40ms 2026-02-25T08:45:24.757 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2026-02-25T08:45:24.771 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2026-02-25T08:45:24.772 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2026-02-25T08:45:24.787 DEBUG:teuthology.orchestra.run.trial190:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2026-02-25T08:45:24.858 DEBUG:teuthology.orchestra.run.trial190:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-25T08:45:24.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-25T08:45:24.885 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2026-02-25T08:45:24.897 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2026-02-25T08:45:24.910 INFO:teuthology.task.internal:Duration was 2364.159934 seconds 2026-02-25T08:45:24.910 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2026-02-25T08:45:24.925 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2026-02-25T08:45:24.925 DEBUG:teuthology.orchestra.run.trial190:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-25T08:45:24.964 INFO:teuthology.orchestra.run.trial190.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-25T08:45:25.109 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2026-02-25T08:45:25.109 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial190.front.sepia.ceph.com 2026-02-25T08:45:25.109 DEBUG:teuthology.orchestra.run.trial190:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-25T08:45:25.171 INFO:teuthology.task.internal.syslog:Gathering journactl... 2026-02-25T08:45:25.171 DEBUG:teuthology.orchestra.run.trial190:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-25T08:45:25.278 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2026-02-25T08:45:25.278 DEBUG:teuthology.orchestra.run.trial190:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-25T08:45:25.302 INFO:teuthology.orchestra.run.trial190.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-25T08:45:25.302 INFO:teuthology.orchestra.run.trial190.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-25T08:45:25.302 INFO:teuthology.orchestra.run.trial190.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-02-25T08:45:25.303 INFO:teuthology.orchestra.run.trial190.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-25T08:45:25.303 INFO:teuthology.orchestra.run.trial190.stderr:/home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-25T08:45:25.313 INFO:teuthology.orchestra.run.trial190.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: 83.1% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-25T08:45:25.315 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2026-02-25T08:45:25.336 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2026-02-25T08:45:25.336 DEBUG:teuthology.orchestra.run.trial190:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-25T08:45:25.378 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2026-02-25T08:45:25.390 DEBUG:teuthology.orchestra.run.trial190:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-25T08:45:25.441 INFO:teuthology.orchestra.run.trial190.stdout:kernel.core_pattern = core 2026-02-25T08:45:25.456 DEBUG:teuthology.orchestra.run.trial190:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-25T08:45:25.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-25T08:45:25.512 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2026-02-25T08:45:25.529 INFO:teuthology.task.internal:Transferring archived files... 2026-02-25T08:45:25.530 DEBUG:teuthology.misc:Transferring archived files from trial190:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial/69096/remote/trial190 2026-02-25T08:45:25.530 DEBUG:teuthology.orchestra.run.trial190:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-25T08:45:25.634 INFO:teuthology.task.internal:Removing archive directory... 2026-02-25T08:45:25.634 DEBUG:teuthology.orchestra.run.trial190:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-25T08:45:25.651 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2026-02-25T08:45:25.673 INFO:teuthology.task.internal:Not uploading archives. 2026-02-25T08:45:25.673 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2026-02-25T08:45:25.690 INFO:teuthology.task.internal:Tidying up after the test... 2026-02-25T08:45:25.691 DEBUG:teuthology.orchestra.run.trial190:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-25T08:45:25.707 INFO:teuthology.orchestra.run.trial190.stdout: 920834 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 25 08:45 /home/ubuntu/cephtest 2026-02-25T08:45:25.708 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2026-02-25T08:45:25.727 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2026-02-25T08:45:25.767 INFO:teuthology.run:Summary data: description: crimson-rados/perf/{0-crimson_install clusters/{crimson-fixed-1} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/rbm$/{crimson_seastore_rbm} perf_ignorelist settings/optimized workloads/radosbench_4K_rand_read} duration: 2364.1599338054657 flavor: debug owner: scheduled_matan@soko04.front.sepia.ceph.com success: true 2026-02-25T08:45:25.767 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-25T08:45:25.877 INFO:teuthology.run:pass